The build workflow is at https://github.com/E3SM-Project/zstash/blob/main/.github/workflows/build_workflow.yml. See comments in the file for in-depth explanations of each step.

. See comments in the file for in-depth explanations of each step.

The release workflow is at https://github.com/E3SM-Project/zstash/blob/main/.github/workflows/release_workflow.yml. See comments in the file for in-depth explanations of each step.

Both of these workflows are run by GitHub Actions. See https://github.com/E3SM-Project/zstash/actions.

When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch.

When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the 

This documentation is created using
`Sphinx <http://www.sphinx-doc.org/en/stable>`_. Sphinx is an open-source tool
that makes it easy to create intelligent and beautiful documentation, written
by Georg Brandl and licensed under the BSD license.

. Sphinx is an open-source tool

The documentation is maintained in the ``main`` branch of the GitHub repository.
You can include code and its corresponding documentation updates in a single pull request (PR).

 branch of the GitHub repository.

After merging a PR, GitHub Actions automates the documentation building process.
It pushes the HTML build to the ``gh-pages`` branch, which is hosted on GitHub Pages.

It pushes the HTML build to the 

Sphinx uses `reStructuredText <http://docutils.sourceforge.net/rst.html>`_
as its markup language. For more information on how to write documentation
using Sphinx, you can refer to

`First Steps with Sphinx <http://www.sphinx-doc.org/en/stable/tutorial.html>`_

`reStructuredText Primer <http://www.sphinx-doc.org/en/stable/rest.html#external-links>`_

Make sure that you have conda in your path. On NERSC machines, you can load it with: ::

$ module load python/3.7-anaconda-2019.10

Make sure that you have conda in your path. On NERSC machines, you can load it with:

Clone the repository and checkout a branch from `main`: ::

$ cd <myDir>
$ git clone https://github.com/<your-github-username>/zstash.git
$ cd zstash
$ git checkout -b <branch-name> main

$ cd zstash

Create and activate the conda development environment ::

$ cd <myDir>/zstash
$ conda env create -f conda/dev.yml
$ conda activate zstash_dev

To modify the documentation, simply edit the files under ``/docs/source``.

To see the changes you made to the documentation, rebuild the web pages ::

$ cd <myDir>/zstash/docs
$ make html

To see the changes you made to the documentation, rebuild the web pages

View them locally in a web browser at ``file:///<myDir>/zstash/docs/_build/html/index.html``.

- Sometimes the browser caches Sphinx docs, so you might need to delete your cache to view changes.

Sometimes the browser caches Sphinx docs, so you might need to delete your cache to view changes.

Once you are satisfied with your modifications, commit and push them back to the repository: ::

 $ cd <myDir>/zstash
 $ # `/docs/_build` is ignored by git since it does not need to be pushed
 $ git add .
 $ git commit
 $ git push <fork-origin> <branch-name>

Once you are satisfied with your modifications, commit and push them back to the repository:

$ # `/docs/_build` is ignored by git since it does not need to be pushed
$ git add .
$ git commit
$ git push <fork-origin> <branch-name>

<`OPTIONAL`> If you want to generate and view versioned docs: ::

 $ # After commiting to your branch
 $ cd <myDir>/zstash/docs
 $ sphinx-multiversion source _build/html
 $ # Check the `_build/html` folder for all generated versioned docs
 $ # Open `_build/html/<your-branch>/index.html` to view in browser

.. figure:: _static/figures/docs-version-selector.png
   :alt: Docs version selector

   Docs version selector dropdown in the bottom left-hand corner

$ # After commiting to your branch
$ sphinx-multiversion source _build/html
$ # Check the `_build/html` folder for all generated versioned docs
$ # Open `_build/html/<your-branch>/index.html` to view in browser

Create a pull request from ``<your-fork>/zstash/branch-name`` to ``E3SM-Project/zstash/main``.

Once this pull request is merged and GitHub Actions finishes building the docs, changes will be available on the
`zstash documentation page <https://e3sm-project.github.io/zstash/>`_.

`zstash documentation page <https://e3sm-project.github.io/zstash/>`_

The `sphinx-multiversion <https://github.com/Holzhaus/sphinx-multiversion>`_ package manages documentation versioning.

``sphinx-multiversion`` is configured to generate versioned docs for available tags and
branches on local, ``origin`` (the likely name for your ``<fork-origin>``) and
``upstream`` (the likely name for your ``<upstream-origin>``)..

 is configured to generate versioned docs for available tags and
branches on local, 

Branches or tags that don’t contain both the sphinx ``source`` directory and the ``conf.py`` file will be skipped automatically.

- Skipped versions include releases ``<= v1.0.1`` since the documention source was not included in those tagged releases.
- Run ``sphinx-multiversion source _build/html --dump-metadata`` to see which tags/branches matched.

Skipped versions include releases ``<= v1.0.1`` since the documention source was not included in those tagged releases.

 since the documention source was not included in those tagged releases.

Run ``sphinx-multiversion source _build/html --dump-metadata`` to see which tags/branches matched.

The instructions below only apply for the initial configuration of the
Sphinx documentation on the Github repository. They are documented here
for reference only. Do not follow them unless you are setting up documentation
for a new repository. (Adapted from `Sphinx documentation on GitHub
<http://datadesk.latimes.com/posts/2012/01/sphinx-on-github>`_.)

for a new repository. (Adapted from 

`Sphinx documentation on GitHub
<http://datadesk.latimes.com/posts/2012/01/sphinx-on-github>`_

.. code-block:: bash

  git branch gh-pages
  git checkout gh-pages

  git symbolic-ref HEAD refs/heads/gh-pages
  rm .git/index
  git clean -fdx

git symbolic-ref HEAD refs/heads/gh-pages
rm .git/index
git clean -fdx

Change the Sphinx theme to 'ReadTheDocs'. Edit 'source/conf.py and change

Change the Sphinx theme to ‘ReadTheDocs’. Edit ‘source/conf.py and change

.. code-block:: python

  import sphinx_rtd_theme
  html_theme = "sphinx_rtd_theme"
  html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

import sphinx_rtd_theme
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

Create an empty .nojekyll file to indicate to Github.com that this
is not a Jekyll static website:

<meta http-equiv="refresh" content="0; url=./docs/html/index.html" />

  git add .
  git commit
  git push origin gh-pages

Use this guide for repository conventions, testing, release work, and
implementation details that are mainly useful to contributors.

If you haven't already, please visit :ref:`"(b) Development Environment" <dev-env>`
to set up your local development environment.

The repository uses a fork-based Git workflow with tag releases.

.. figure:: /_static/figures/git-flow.svg
   :alt: Git Flow Diagram

Make sure ``pre-commit`` checks pass when committing (enforced in CI/CD build)

Once the CI/CD build passes and PR is approved, **squash and rebase** your
commits

The repository uses the ``pre-commit`` package to manage pre-commit hooks.
These hooks help enforce quality assurance standards and identify simple issues
at the commit level before submitting code reviews.

 package to manage pre-commit hooks.

.. figure:: /_static/figures/pre-commit-flow.svg
   :alt: Pre-commit Flow Diagram

   ``pre-commit`` Flow

    conda activate zstash_dev
    pre-commit install

  # Tip: If there is an issue with pre-commit, you can bypass with the `--no-verify` flag. Please do NOT use this on a regular basis.
  git commit -m '...'

# Tip: If there is an issue with pre-commit, you can bypass with the `--no-verify` flag. Please do NOT use this on a regular basis.
git commit -m '...'

.. figure:: ../_static/figures/pre-commit-passing.png
   :alt: pre-commit Output

   ``pre-commit`` Output

    # Available hook ids: trailing-whitespace, end-of-file-fixer, check-yaml, black, isort, flake8, mypy
    pre-commit run <hook_id>

# Available hook ids: trailing-whitespace, end-of-file-fixer, check-yaml, black, isort, flake8, mypy
pre-commit run <hook_id>

Before you merge a support branch back into ``main``, the branch is typically
squashed down to a single* buildable commit, and then rebased on top of the main repo's ``main`` branch.

, the branch is typically
squashed down to a single* buildable commit, and then rebased on top of the main repo’s 

\* *In some cases, it might be logical to have multiple squashed commits, as long as each commit passes the CI/CD build*

*In some cases, it might be logical to have multiple squashed commits, as long as each commit passes the CI/CD build*

In some cases, it might be logical to have multiple squashed commits, as long as each commit passes the CI/CD build

Makes handling conflicts from rebasing simple since there are fewer commits to fix conflicts on

Makes ``git bisect`` easier and more effective to use. For example, it will show the exact commit that introduced a bug since the commit contains a relatively small changeset. On the otherhand, a merge commit bringing in many commits makes it much harder since it includes a large changeset. A changeset is a collection of commits that are logically grouped together.

 easier and more effective to use. For example, it will show the exact commit that introduced a bug since the commit contains a relatively small changeset. On the otherhand, a merge commit bringing in many commits makes it much harder since it includes a large changeset. A changeset is a collection of commits that are logically grouped together.

Assuming that you followed :ref:`"(b) Development Environment" <dev-env>`:

Sync ``main`` with the main repo's ``main``

 .. code-block:: bash

     git checkout main
     git rebase <upstream-origin>/main
     git push -f <fork-origin> main

    git checkout main
    git rebase <upstream-origin>/main
    git push -f <fork-origin> main

git checkout main
git rebase <upstream-origin>/main
git push -f <fork-origin> main

Get the SHA of the commit OR number of commits to rebase to

     git log --graph --decorate --pretty=oneline --abbrev-commit

    git log --graph --decorate --pretty=oneline --abbrev-commit

Squash commits

     git rebase -i [SHA]
     # OR:
     git rebase -i HEAD~[NUMBER OF COMMITS]

    git rebase -i [SHA]
    # OR:
    git rebase -i HEAD~[NUMBER OF COMMITS]

git rebase -i [SHA]
# OR:
git rebase -i HEAD~[NUMBER OF COMMITS]

Rebase branch onto ``main``

     git checkout <branch-name>
     git rebase main
     git push -f <fork-origin> <branch-name>

    git checkout <branch-name>
    git rebase main
    git push -f <fork-origin> <branch-name>

git checkout <branch-name>
git rebase main
git push -f <fork-origin> <branch-name>

Force push to remote branch

     # You have to force push because the rebase rewrites the commit SHAs

    # You have to force push because the rebase rewrites the commit SHAs

# You have to force push because the rebase rewrites the commit SHAs

This project uses several tools for code formatting, linting, and type checking listed below.

Code Formatting: `black <https://black.readthedocs.io/en/stable/>`__

Linting: `flake8 <https://github.com/PyCQA/flake8#flake8>`__, `isort <https://pycqa.github.io/isort/>`__

You can run them as hooks manually/automatically when committing using ``pre-commit``, or manually through the terminal or IDE/text editor.

You can run them as hooks manually/automatically when committing using 

    # Available tool names: black, flake8, isort, mypy
    <tool_name> .

# Available tool names: black, flake8, isort, mypy
<tool_name> .

This project uses `GitHub Actions <https://github.com/E3SM-Project/zstash/actions>`_ to run two CI/CD workflows.

`GitHub Actions <https://github.com/E3SM-Project/zstash/actions>`_

This workflow is triggered by Git ``pull_request`` and ``push`` (merging PRs) events to the the main repo's ``main``.

Jobs:

  1. Run ``pre-commit`` for formatting, linting, and type checking
  2. Run test suite in a conda environment
  3. Publish latest `main` docs (depends on jobs 1 and 2)

1. Run ``pre-commit`` for formatting, linting, and type checking
2. Run test suite in a conda environment
3. Publish latest `main` docs (depends on jobs 1 and 2)

This workflow is triggered by the Git ``publish`` event, which occurs when a new release is tagged.

Jobs:

  1. Publish new release docs
  2. Publish Anaconda package

This project follows the Black code style. Please read about it more `here <https://black.readthedocs.io/en/stable/the_black_code_style.html>`__.

This project follows the Black code style. Please read about it more 

If necessary, documentation should explain **why** something is done, its purpose, and its goal.
The code shows **how** it is done, so commenting on this can be redundant.

Embrace documentation as an integral part of the overall
development process

Treat documentation as code and follow principles such as *Don't
Repeat Yourself* and *Easier to Change*

Use comments and docstrings to explain ambigiuity, complexity,
or to avoid confusion

Don't comment *every* function, data structure, type declaration

Conda-forge docs: https://conda-forge.org/docs/user/introduction.html

Admin web services: https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services

E3SM Anaconda release guide: https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel

Be sure to run the entire integration test suite before making a production release.

This step should already have been completed during the release-candidate phase, however it is good practice to double check that the `E3SM Unified version tracking page <https://e3sm.atlassian.net/wiki/spaces/DOC/pages/129732419/Packages+in+the+E3SM+Unified+conda+environment>`_ has had the next E3SM Unified version updated with the new ``zstash`` version number.

This step should already have been completed during the release-candidate phase, however it is good practice to double check that the 

`E3SM Unified version tracking page <https://e3sm.atlassian.net/wiki/spaces/DOC/pages/129732419/Packages+in+the+E3SM+Unified+conda+environment>`_

Similar to the release-candidate directions, we'll use ``v1.2.3`` as an example version number here.

     cd zstash
     git status # Confirm there's no uncommitted changes
     git fetch upstream main # This assumes you've named your remote for the main repo as "upstream"
     git checkout -b v1.2.3 upstream/main
     git log --oneline | head -n 5
     # Check that the latest commits match what's on https://github.com/E3SM-Project/zstash/commits/main/
     conda activate env-name # Activate any zstash dev environment you have; we just need `tbump`
     tbump 1.2.3 --no-tag
     # This creates a commit, but won't push it (because the branch isn't named `main`)
     git push upstream v1.2.3
     # Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub

cd zstash
git status # Confirm there's no uncommitted changes
git fetch upstream main # This assumes you've named your remote for the main repo as "upstream"
git checkout -b v1.2.3 upstream/main
git log --oneline | head -n 5
# Check that the latest commits match what's on https://github.com/E3SM-Project/zstash/commits/main/
conda activate env-name # Activate any zstash dev environment you have; we just need `tbump`
tbump 1.2.3 --no-tag
# This creates a commit, but won't push it (because the branch isn't named `main`)
git push upstream v1.2.3
# Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub

.. figure:: /_static/figures/github_release.png
   :alt: GitHub Release Diagram

Draft a new release `here <https://github.com/E3SM-Project/zstash/releases>`_. Click "Draft a new release".

Set Tag version to ``v1.2.3``, including the "v". ``@Target`` should be ``main``. Click "Tag", then "Create new tag" and enter "v1.2.3".

Use "Describe this release" to summarize the changelog. Write two sections: "Summary of changes" (the high-level summary) & "Full list of changes" (the categorized list of commits, from reviewing the `zstash commits <https://github.com/E3SM-Project/zstash/commits/main>`_).

Use “Describe this release” to summarize the changelog. Write two sections: “Summary of changes” (the high-level summary) & “Full list of changes” (the categorized list of commits, from reviewing the 

`zstash commits <https://github.com/E3SM-Project/zstash/commits/main>`_

Click "Publish release". Unlike the RCs, ``v1.2.3`` should now appear on _both_ `Tags <https://github.com/E3SM-Project/zstash/tags>`_ and `Releases <https://github.com/E3SM-Project/zstash/releases>`_.

CI/CD release workflow will be automatically triggered. The docs workflow is just for the docs. Clicking "Publish release" is responsible for triggering the bot PR on conda-forge.

CI/CD release workflow will be automatically triggered. The docs workflow is just for the docs. Clicking “Publish release” is responsible for triggering the bot PR on conda-forge.

Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. Check https://github.com/conda-forge/zstash-feedstock/pulls. (Alternative: open an issue with the bot command: ``@conda-forge-admin, please update version`` and the PR will be opened.)

Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. Check 

Check the https://anaconda.org/conda-forge/zstash/files/manage page to view the newly updated package. Check it has the ``main`` label.

Wait until the CI/CD build is successful. You can view all workflows at `All Workflows <https://github.com/E3SM-Project/zstash/actions>`_.

Wait until the CI/CD build is successful. You can view all workflows at 

`All Workflows <https://github.com/E3SM-Project/zstash/actions>`_

Changes will be available on the `zstash documentation page <https://docs.e3sm.org/zstash/_build/html/main/index.html>`_.

`zstash documentation page <https://docs.e3sm.org/zstash/_build/html/main/index.html>`_

Be sure to run the entire integration test suite before making a release candidate.

Review `zstash commits <https://github.com/E3SM-Project/zstash/commits/main/>`_. Identify what changes since the last production release would require a patch, minor, or major version update (as in ``vMAJOR.MINOR.PATCH``).

`zstash commits <https://github.com/E3SM-Project/zstash/commits/main/>`_

. Identify what changes since the last production release would require a patch, minor, or major version update (as in 

The highest-level version update will be used. For example, if you found 1 commit requiring a patch update, 1 commit requiring a minor update, and 0 commits requiring a major update, then you'd want to increment the minor version.

The highest-level version update will be used. For example, if you found 1 commit requiring a patch update, 1 commit requiring a minor update, and 0 commits requiring a major update, then you’d want to increment the minor version.

Update the `E3SM Unified version tracking page <https://e3sm.atlassian.net/wiki/spaces/DOC/pages/129732419/Packages+in+the+E3SM+Unified+conda+environment>`_ with the new ``zstash`` version number. Be sure to update the section for the next E3SM Unified environment, not the current one.

 version number. Be sure to update the section for the next E3SM Unified environment, not the current one.

In this example, ``v1.2.3rc4`` is used -- so, the major version is 1, the minor version is 2, and the patch version is 3. The release candidate number is 4.

 is used – so, the major version is 1, the minor version is 2, and the patch version is 3. The release candidate number is 4.

     cd zstash
     git status # Make sure you don't have any uncommitted changes
     git checkout -b v1.2.3rc4 upstream/main
     tbump 1.2.3rc4 --no-tag
     git diff HEAD^ HEAD | cat
     # Review the change `tbump` added
     git push upstream v1.2.3rc4

cd zstash
git status # Make sure you don't have any uncommitted changes
git checkout -b v1.2.3rc4 upstream/main
tbump 1.2.3rc4 --no-tag
git diff HEAD^ HEAD | cat
# Review the change `tbump` added
git push upstream v1.2.3rc4

     git fetch upstream
     git reset --hard upstream/main
     git tag -a v1.2.3rc4 -m "v1.2.3rc4" # Add the tag for this RC
     # Delete the branch from the tbump step. Otherwise, the push command won't work.
     git branch -D v1.2.3rc4
     git push upstream v1.2.3rc4 # Push the new RC tag to GitHub

git fetch upstream
git reset --hard upstream/main
git tag -a v1.2.3rc4 -m "v1.2.3rc4" # Add the tag for this RC
# Delete the branch from the tbump step. Otherwise, the push command won't work.
git branch -D v1.2.3rc4
git push upstream v1.2.3rc4 # Push the new RC tag to GitHub

``v1.2.3rc4`` should now appear on `Tags <https://github.com/E3SM-Project/zstash/tags>`__, but _not_ on `Releases <https://github.com/E3SM-Project/zstash/releases>`_.

If you don't have a fork of zstash-feedstock, first go to the `zstash-feedstock repo <https://github.com/conda-forge/zstash-feedstock>`__, and click the "Fork" button in the top right.

`zstash-feedstock repo <https://github.com/conda-forge/zstash-feedstock>`__

     # If you don't have a clone of zstash-feedstock, first run:
     # Clone using the SSH from the green "Code" button in the top right of the repo home page.
     git clone git@github.com:conda-forge/zstash-feedstock.git
     cd zstash-feedstock
     git remote -v # See your remotes
     # Optional; this lets you use upstream to refer to the main repo:
     git remote add upstream git@github.com:conda-forge/zstash-feedstock.git
     # Required; this allows you to push to your fork:
     # Copy the SSH path from your fork's green "Code" button and run:
     git remote add your-fork-name git@github.com:your-fork-name/zstash-feedstock.git

     # If you already have your clone set up, just run:

     # In all cases:
     curl -sL https://github.com/E3SM-Project/zstash/archive/v1.2.3rc4.tar.gz | openssl sha256
     # SHA2-256(stdin)= long hex string
     git status # Check for uncommitted changes
     git fetch upstream dev # Make sure you fetch the dev branch, not the main branch!
     git checkout -b v1.2.3rc4 upstream/dev
     emacs recipe/meta.yaml
     # Update the version and sha256 (and the build number if needed):
     # {% set version = "1.2.3rc4" %}
     # sha256: ... # The sha256 from a few commands earlier
     # number: 0 # build >>> number should always be 0

     # Check the diff since the last release/RC. Examples:
     # - https://github.com/E3SM-Project/zstash/compare/v1.2.2...v1.2.3rc1
     # - https://github.com/E3SM-Project/zstash/compare/v1.2.3rc3...v1.2.3rc4
     # If there are changes in dependencies there (e.g., in `conda/dev.yml`),
     # you'll want to include them in this feedstock PR too.

     git add -A
     git commit -m "v1.2.3rc4"
     git push your-fork-name v1.2.3rc4

# If you don't have a clone of zstash-feedstock, first run:
# Clone using the SSH from the green "Code" button in the top right of the repo home page.
git clone git@github.com:conda-forge/zstash-feedstock.git
cd zstash-feedstock
git remote -v # See your remotes
# Optional; this lets you use upstream to refer to the main repo:
git remote add upstream git@github.com:conda-forge/zstash-feedstock.git
# Required; this allows you to push to your fork:
# Copy the SSH path from your fork's green "Code" button and run:
git remote add your-fork-name git@github.com:your-fork-name/zstash-feedstock.git

# If you already have your clone set up, just run:

# In all cases:
curl -sL https://github.com/E3SM-Project/zstash/archive/v1.2.3rc4.tar.gz | openssl sha256
# SHA2-256(stdin)= long hex string
git status # Check for uncommitted changes
git fetch upstream dev # Make sure you fetch the dev branch, not the main branch!
git checkout -b v1.2.3rc4 upstream/dev
emacs recipe/meta.yaml
# Update the version and sha256 (and the build number if needed):
# {% set version = "1.2.3rc4" %}
# sha256: ... # The sha256 from a few commands earlier
# number: 0 # build >>> number should always be 0

# Check the diff since the last release/RC. Examples:
# - https://github.com/E3SM-Project/zstash/compare/v1.2.2...v1.2.3rc1
# - https://github.com/E3SM-Project/zstash/compare/v1.2.3rc3...v1.2.3rc4
# If there are changes in dependencies there (e.g., in `conda/dev.yml`),
# you'll want to include them in this feedstock PR too.

git add -A
git commit -m "v1.2.3rc4"
git push your-fork-name v1.2.3rc4

Then, create a pull request to the ``dev`` branch. Do _not_ set the pull request to merge to ``main``, as that will create a production release! RC packages get the ``zstash_dev`` label.

, as that will create a production release! RC packages get the 

Follow any further directions given by the ``conda-forge/zstash-feedstock`` PR template. Once that PR is reviewed and merged and CI completes, check https://anaconda.org/conda-forge/zstash for the new package (allow ~15 minutes for mirroring).

 PR template. Once that PR is reviewed and merged and CI completes, check 

This page explains how zstash creates, tracks, transfers, and cleans up tar
files in the three supported ``--hpss`` modes:

files in the three supported 

Regardless of mode, ``zstash create`` and ``zstash update`` follow the same
high-level pattern:

Walk the source tree and decide which files belong in the next tar.

Record file-level metadata in ``files`` and, when enabled, tar-level
metadata in ``tars``.

The archive database is the source of truth for both file membership and tar
metadata.

In local-only mode, zstash still builds the archive exactly the same way, but
there is no remote transfer step.

tar files are created under the local cache, usually ``zstash/``

zstash removes write permission from completed tar files so they are less
likely to be changed accidentally

This is the simplest mode because the local cache is the archive.

When ``--hpss`` points to an HPSS directory, zstash uses ``hsi`` for transfers.

after a tar is closed, zstash uploads it to the target HPSS directory

if ``--keep`` is not set, the local tar can be deleted after a successful
transfer

 is not set, the local tar can be deleted after a successful
transfer

``index.db`` is uploaded after the tar work completes, but it is not tracked
for deletion like tar files are

 is uploaded after the tar work completes, but it is not tracked

This mode is the most direct path when the current machine already has HPSS
access.

When ``--hpss`` is a ``globus://`` URL, zstash still creates each tar locally
first, but transfer tracking becomes more explicit.

 URL, zstash still creates each tar locally

the batch is associated with Globus task state instead of an immediate
``hsi put``

local tar files are kept until Globus reports success for the corresponding
transfer task

``globus_finalize()`` submits any remaining pending transfer data, waits for
completion, and then removes successfully transferred tar files when
``--keep`` is not set

 submits any remaining pending transfer data, waits for

``index.db`` is transferred after the tar workflow, just as in the direct
HPSS mode

HPSS mode

 is transferred after the tar workflow, just as in the direct
HPSS mode

This mode is what allows zstash to archive to HPSS from machines that do not
have direct HPSS access.

This page summarizes the test process described in ``tests/README.md``.

The repository separates machine-independent tests from machine-specific
integration tests:

``tests/integration/python_tests/group_by_command/`` contains unittest-based
command-oriented integration tests

``tests/integration/python_tests/group_by_workflow/`` contains unittest-based
end-to-end workflow tests

``tests/integration/bash_tests/run_from_any/`` contains bash-driven tests
that can be run from any machine

``tests/integration/bash_tests/run_from_perlmutter/`` contains tests that
need Perlmutter or direct HPSS access

``tests/integration/bash_tests/run_from_chrysalis/`` contains tests that need
Chrysalis and Globus-related setup

For a normal development change, start with the machine-independent checks from
the repository root

  rm -rf build
  conda clean --all --y
  conda env create -f conda/dev.yml -n zstash_dev_test
  conda activate zstash_dev_test
  pre-commit run --all-files
  python -m pip install .
  pytest tests/unit/test_*.py
  python -m unittest tests/integration/python_tests/group_by_command/test_*.py
  python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py

rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash_dev_test
conda activate zstash_dev_test
pre-commit run --all-files
python -m pip install .
pytest tests/unit/test_*.py
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py

  # pytest tests/unit/test_*.py
  # 1 passed in 0.19s

  # python -m unittest tests/integration/python_tests/group_by_command/test_*.py
  # Ran 69 tests in 327.570s
  # OK

  # python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
  # Ran 4 tests in 2.666s
  # OK

# pytest tests/unit/test_*.py
# 1 passed in 0.19s

# python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 327.570s
# OK

# python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 2.666s
# OK

Some integration tests are skipped automatically on systems that do not have
``hsi`` or HPSS access.

The ``tests/unit/`` directory holds pytest-based tests targeting isolated,
pure-Python logic — no HPSS, Globus, or filesystem side-effects required.

 directory holds pytest-based tests targeting isolated,

``test_hpss.py``
Exercises the ``get_files_to_archive_with_stats`` helper and the logic that
compares on-disk files against the zstash SQLite database. Key scenarios
covered:

* Scanning a directory and returning size/mtime pairs for each file.
* Building the ``archived_files`` dictionary from database rows, keeping the
  entry with the most recent modification time when duplicates exist.
* Detecting **new** files (present on disk but absent from the database).
* Detecting **modified** files whose size or modification time differs from
  the archived record, using a configurable ``TIME_TOL`` tolerance (default
  1 second).
* Verifying that files within the tolerance window are *not* re-archived.

covered:

 helper and the logic that
covered:

Scanning a directory and returning size/mtime pairs for each file.

Building the ``archived_files`` dictionary from database rows, keeping the
entry with the most recent modification time when duplicates exist.

 dictionary from database rows, keeping the

Detecting **new** files (present on disk but absent from the database).

Detecting **modified** files whose size or modification time differs from
the archived record, using a configurable ``TIME_TOL`` tolerance (default
1 second).

 files whose size or modification time differs from
the archived record, using a configurable 

Verifying that files within the tolerance window are *not* re-archived.

``test_utils.py``
Covers ``zstash.utils.run_command`` subprocess behavior:

* When the command starts with ``hsi``, the function strips
  ``LD_LIBRARY_PATH`` and ``LD_PRELOAD`` from the child environment to
  avoid linker conflicts, while preserving other variables such as ``HOME``.
* For all other commands the loader variables are passed through unchanged.

When the command starts with ``hsi``, the function strips
``LD_LIBRARY_PATH`` and ``LD_PRELOAD`` from the child environment to
avoid linker conflicts, while preserving other variables such as ``HOME``.

 from the child environment to
avoid linker conflicts, while preserving other variables such as 

For all other commands the loader variables are passed through unchanged.

The ``tests/integration/python_tests/`` directory contains ``unittest``-based
tests that invoke zstash commands and inspect their output.  They are split
into two sub-directories:

-based

``group_by_command/``
Each test file focuses on a single zstash sub-command (e.g. ``create``,
``update``, ``extract``, ``check``).  This makes it straightforward to run
only the tests relevant to the command you have changed.

).  This makes it straightforward to run

``group_by_workflow/``
End-to-end tests that exercise multi-step workflows: create an archive,
update it with new or changed files, extract files, and verify integrity.

Tests that require ``hsi`` or HPSS are automatically skipped when those tools
are unavailable.

 or HPSS are automatically skipped when those tools

The bash tests are grouped by the machine or environment they require.

The ``run_from_any/`` directory contains bash tests that can be exercised
on any machine, although Globus authentication is part of the
workflow.

 directory contains bash tests that can be exercised
workflow.

Before running these tests, review the instructions at the bottom of
``globus_auth.bash``, then authenticate and run the Globus tar-deletion test

  cd tests/integration/bash_tests/run_from_any/
  # Review globus_auth.bash and run with the appropriate parameters:
  ./globus_auth.bash <parameters>
  ./test_globus_tar_deletion.bash <parameters>

cd tests/integration/bash_tests/run_from_any/
# Review globus_auth.bash and run with the appropriate parameters:
./globus_auth.bash <parameters>
./test_globus_tar_deletion.bash <parameters>

The ``run_from_perlmutter/`` directory contains tests that depend on direct
HPSS access and Perlmutter-specific paths. Update any hardcoded paths to
match your username before running.

 directory contains tests that depend on direct

  cd tests/integration/bash_tests/run_from_perlmutter/

  # Symlink-following test (edit paths for your username first)
  time ./follow_symlinks.sh
  # real 0m31.851s — No errors

  # HPSS update test
  time ./test_update_non_empty_hpss.bash
  # real 0m10.062s — No errors

  # Globus ls test
  # 1. Log into globus.org
  # 2. In File Manager, add both endpoints:
  #    - NERSC Perlmutter
  #    - Globus Tutorial Collection 1
  time ./test_ls_globus.bash   # You may be prompted to paste an auth-code
  # real 0m40.297s — No errors

cd tests/integration/bash_tests/run_from_perlmutter/

# Symlink-following test (edit paths for your username first)
time ./follow_symlinks.sh
# real 0m31.851s — No errors

# HPSS update test
time ./test_update_non_empty_hpss.bash
# real 0m10.062s — No errors

# Globus ls test
# 1. Log into globus.org
# 2. In File Manager, add both endpoints:
#    - NERSC Perlmutter
#    - Globus Tutorial Collection 1
time ./test_ls_globus.bash   # You may be prompted to paste an auth-code
# real 0m40.297s — No errors

The ``run_from_chrysalis/`` directory contains tests that depend on Chrysalis,
Globus setup, and in some cases explicit cleanup of previous authentication
state before rerunning.

 directory contains tests that depend on Chrysalis,

  cd tests/integration/bash_tests/run_from_chrysalis/

  # If not already done:
  # 2. In File Manager add both endpoints:
  #    - LCRC Improv DTN

  # --- database_corruption.bash ---
  # To reset completely before this test:
  #   Revoke consents: https://auth.globus.org/v2/web/consents
  #   > Globus Endpoint Performance Monitoring > rescind all
  #
  # Set up the required remote state (one-time):
  rm ~/.zstash_globus_tokens.json
  mkdir zstash_demo
  echo 'file0 stuff' > zstash_demo/file0.txt
  # NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
  zstash create \
    --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/<user>/zstash/tests/test_database_corruption_setup23 \
    zstash_demo
  # Paste the auth-code when prompted.  This pre-authentication means the
  # database_corruption test itself will not stop for user input.
  rm -rf zstash_demo/
  #
  # Pick a unique_id to avoid collisions with a previous run, or delete the
  # remote directory on Perlmutter first:
  #   rm -rf /global/homes/<user>/zstash/tests/test_database_corruption_<unique_id>
  #
  # Edit paths for your username, then run:
  time ./database_corruption.bash <unique_id>
  # Success count: 25
  # Fail count: 0
  # real 6m43.994s

  # --- symlinks.sh ---
  # Edit paths for your username first.
  time ./symlinks.sh
  # real 0m1.346s — No errors

cd tests/integration/bash_tests/run_from_chrysalis/

# If not already done:
# 2. In File Manager add both endpoints:
#    - LCRC Improv DTN

# --- database_corruption.bash ---
# To reset completely before this test:
#   Revoke consents: https://auth.globus.org/v2/web/consents
#   > Globus Endpoint Performance Monitoring > rescind all
#
# Set up the required remote state (one-time):
rm ~/.zstash_globus_tokens.json
mkdir zstash_demo
echo 'file0 stuff' > zstash_demo/file0.txt
# NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
zstash create \
  --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/<user>/zstash/tests/test_database_corruption_setup23 \
  zstash_demo
# Paste the auth-code when prompted.  This pre-authentication means the
# database_corruption test itself will not stop for user input.
rm -rf zstash_demo/
#
# Pick a unique_id to avoid collisions with a previous run, or delete the
# remote directory on Perlmutter first:
#   rm -rf /global/homes/<user>/zstash/tests/test_database_corruption_<unique_id>
#
# Edit paths for your username, then run:
time ./database_corruption.bash <unique_id>
# Success count: 25
# Fail count: 0
# real 6m43.994s

# --- symlinks.sh ---
# Edit paths for your username first.
time ./symlinks.sh
# real 0m1.346s — No errors

GitHub Actions runs the machine-independent test suite in
``.github/workflows/build_workflow.yml``:

``python -m unittest tests/integration/python_tests/group_by_command/test_*.py``

``python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py``

That workflow is the baseline CI safety net, while the machine-specific bash
tests remain primarily manual.

  cd zstash

  ./globus_auth.bash unique_id chrysalis path_to_repo chrysalis_dst_basedir perlmutter_dst_basedir hpss_dst_basedir compy_dst_basedir
  ./test_globus_tar_deletion.bash unique_id path_to_repo dst_basedir LCRC_IMPROV_DTN_ENDPOINT

  cd -
  # You should still have Globus set up from the globus auth test.
  time ./database_corruption.bash unique_id # NOTE: you will have to change out paths for your username
  time ./symlinks.sh # NOTE: you will have to change out paths for your username

cd zstash

./globus_auth.bash unique_id chrysalis path_to_repo chrysalis_dst_basedir perlmutter_dst_basedir hpss_dst_basedir compy_dst_basedir
./test_globus_tar_deletion.bash unique_id path_to_repo dst_basedir LCRC_IMPROV_DTN_ENDPOINT

cd -
# You should still have Globus set up from the globus auth test.
time ./database_corruption.bash unique_id # NOTE: you will have to change out paths for your username
time ./symlinks.sh # NOTE: you will have to change out paths for your username

  cd zstash

  cd -
  time ./follow_symlinks.sh # NOTE: you will have to change out paths for your username
  # Log into globus.org
  # Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
  time ./test_ls_globus.bash # NOTE: You may be asked to paste an auth-code

cd zstash

cd -
time ./follow_symlinks.sh # NOTE: you will have to change out paths for your username
# Log into globus.org
# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
time ./test_ls_globus.bash # NOTE: You may be asked to paste an auth-code

  cd zstash

cd zstash

zstash documentation master file, created by
sphinx-quickstart on Fri Jul 28 17:31:00 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Zstash is written entirely in Python using standard libraries.
Its design is intentionally minimalistic to provide an effective
long-term HPSS archiving solution without creating an overly complicated
(and hard to maintain) tool.

Files are archived into standard **tar files** with a user **specified maximum size**.

**Checksums (md5)** of input files are computed *on-the-fly* during
archiving. For large files, this saves a considerable amount of
time compared to separate checksumming and archiving steps.
Checksums are also computed *on-the-fly* for tars.

 during
Checksums are also computed 

Checksums and additional metadata (size, modification time, tar file and offset)
are stored in a sqlite3 **index database**.

are stored in a sqlite3 

**Database enables faster retrieval** of individual files by locating in which tar
file a specific file is stored, as well as its location (offset) within the
tar file.

 of individual files by locating in which tar
tar file.

**File integrity** is verified by computing checksums on-the-fly while **extracting**
files.

files.

Source code is available on Github: `<https://github.com/E3SM-Project/zstash>`_.

To change the documentation version, use the version selector in the bottom left-hand corner.

For documentation not included in the version selector (<= ``v1.0.1``):

:doc:`User Guide <user_guide/index>` for installation, day-to-day usage,
Globus setup, and archive management

 for installation, day-to-day usage,

:doc:`Developer Guide <dev_guide/index>` for contributing, testing, release
work, and internal implementation details

 for contributing, testing, release

The user-facing documentation is organized under :doc:`user_guide/index` and
includes:

:doc:`user_guide/design` for the high-level architecture and implementation overview

:doc:`user_guide/getting_started` for installation and first-time setup

:doc:`user_guide/globus/index` for Globus account setup and transfer workflows and ``.zstash.ini`` configuration details

:doc:`user_guide/best_practices` for archive management recommendations

:doc:`user_guide/database` for the archive index database layout

:doc:`user_guide/support` for where to ask questions or report issues

:doc:`user_guide/archived_documentation/index` for older documentation that may still be useful as a reference.

 for older documentation that may still be useful as a reference.

The contributor and maintainer documentation is organized under
:doc:`dev_guide/index` and includes:

:doc:`dev_guide/project_standards` for coding standards and conventions

:doc:`dev_guide/tar_tracking_modes` for tar tracking behavior in each storage
mode

:doc:`dev_guide/testing` for the test layout and execution guidance

:doc:`dev_guide/contributing_to_docs` for development environment setup and contribution
workflow

workflow

These pages document older versions of ``zstash`` but may still be useful as a reference.

This tutorial is specifically for ``zstash v0.4.2``.
Some features have been added/changed since.
Paths for E3SM Unified have also changed.

.

Some statements on this tutorial have been pulled from other pages of the
documentation. This is done to provide a comprehensive tutorial on a single page.
There is also an `accompanying video tutorial <https://youtu.be/kmdBdXa3rXo>`_.

There is also an 

(and hard to maintain) tool. For more information, see :ref:`index-label`,
`<https://e3sm.org/resources/tools/data-management/zstash-hpss-archive/>`_,
or `<https://e3sm.org/new-zstash-capabilities>`_.

(and hard to maintain) tool. For more information, see 

**zstash archive**:  A set of files {index.db, nnnnnn.tar where  nnnnnn=000000, 000001, …}

:  A set of files {index.db, nnnnnn.tar where  nnnnnn=000000, 000001, …}

**Local archive (or cache)**:  The zstash archive on our local file system. The default name is ``zstash``.

:  The zstash archive on our local file system. The default name is 

**Source directory**: the directory whose contents we are archiving.

**Create**: create a new cache (local archive), create a tar file of the source directory's contents,
and if using HPSS, then store it on the HPSS archive.

: create a new cache (local archive), create a tar file of the source directory’s contents,

**Check**: verify integrity of zstash archive (e.g., if using HPSS, check that files were uploaded on HPSS successfully.)

: verify integrity of zstash archive (e.g., if using HPSS, check that files were uploaded on HPSS successfully.)

**Update**: add new or modified files to an existing zstash archive
(ignoring unmodified files).

: add new or modified files to an existing zstash archive

**Extract**: extract files from an existing zstash archive into current directory.

: extract files from an existing zstash archive into current directory.

``--cache``, ``--hpss``, and ``--keep`` are covered in this tutorial.

A "x" indicates that the ``zstash`` command supports the given option.

Below is a basic guide to installation. For more information, see :ref:`getting-started`.

Below is a basic guide to installation. For more information, see 

Zstash is available in the E3SM unified environment.
This is the recommended method of installation.

$ source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified.sh

$ source /global/common/cori_cle7/software/python/3.7-anaconda-2019.10/etc/profile.d/conda.sh
$ conda create -n zstash_env -c e3sm -c conda-forge zstash
$ conda activate zstash_env

$ module load anaconda3/2019.03
$ source /share/apps/anaconda3/2019.03/etc/profile.d/conda.sh

NERSC machines have HPSS access. We can choose to use HPSS by setting
``--hpss`` (or ``--hpss=none`` if we do not wish to archive to HPSS).

Before using zstash with HPSS for the first time, run ``hsi`` on NERSC
and enter your credentials. Then, ``zstash`` will be able to access HPSS.
Compy and Anvil do not have HPSS access. Therefore, you’ll need to use a Globus URL
that points to a remote HPSS storage ``--hpss=globus://<Globus endpoint UUID/<path>``
or set ``--hpss=none``.

 will be able to access HPSS.
that points to a remote HPSS storage 

We will use NERSC so that we have access to HPSS. The commands remain the same on Compy,
with the exception that ``--hpss`` will have to be set to ``none`` in all cases.

with the exception that 

We'll create a ``tutorial`` directory in ``$CSCRATCH`` on NERSC. We can create our tutorial
directory anywhere but ``CSCRATCH`` is a useful workspace. ::

Each example that follows is independent of the others. Therefore, we'll need to set up the
directory structure before each example. We'll call the following script ``setup.sh``: ::

Each example that follows is independent of the others. Therefore, we’ll need to set up the
directory structure before each example. We’ll call the following script 

mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt

In one NERSC window, create this script and run it (``./setup.sh``).
That will create the following directory structure:

zstash_demo/

  * dir/

      * file1.txt (contains 'file1 stuff')

  * empty_dir/
  * file0.txt (contains 'file0 stuff')
  * file_empty.txt (empty)

* dir/

    * file1.txt (contains 'file1 stuff')

* empty_dir/
* file0.txt (contains 'file0 stuff')
* file_empty.txt (empty)

In some examples, we'll also want to add files after running some ``zstash`` commands.
We'll call the following script ``add_files.sh``: ::

In some examples, we’ll also want to add files after running some 

mkdir zstash_demo/dir2
echo 'file2 stuff' > zstash_demo/dir2/file2.txt
echo 'file1 stuff with changes' > zstash_demo/dir/file1.txt

If this is run (``./add_files.sh``) immediately after running ``setup.sh``, then we would have the
following directory structure:

  * dir/

      * file1.txt (contains 'file1 stuff with changes')

  * dir2/

      * file2.txt (contains 'file2 stuff')

* dir/

    * file1.txt (contains 'file1 stuff with changes')

* dir2/

    * file2.txt (contains 'file2 stuff')

If this is your first time using HPSS, you'll have to enter your credentials.
If you haven't used HPSS before, ``ls`` should print nothing.

If this is your first time using HPSS, you’ll have to enter your credentials.
If you haven’t used HPSS before, 

After every example, we'll want to remove the directories we created both in our workspace
(``$CSCRATCH/tutorial``) and on HPSS.

After every example, we’ll want to remove the directories we created both in our workspace
(

This simple case will illustrate ``create``, ``update``, and ``extract``.

Now, we will create the ``zstash`` archives -- both locally and on HPSS.
Note that the path passed to ``--hpss`` can be a **relative** path from ``/home/f/<username>`` or
an absolute path `on HPSS, not the local file system`.
``zstash_demo`` is the source directory whose contents we want to archive.
This is just a directory path; it can be a relative path and can contain ``..`` or
it can be an absolute path. ::

 archives – both locally and on HPSS.
Note that the path passed to 

 is the source directory whose contents we want to archive.
This is just a directory path; it can be a relative path and can contain 

INFO: Gathering list of files to archive
INFO: Creating new tar archive 000000.tar
INFO: Archiving file0.txt
INFO: Archiving file_empty.txt
INFO: Archiving dir/file1.txt
INFO: Archiving empty_dir
INFO: Transferring file to HPSS: zstash/000000.tar
INFO: Transferring file to HPSS: zstash/index.db

The "cache" is our local archive. HPSS is our long-term archive.
Our cache is given the default name ``zstash`` and can be found in the source directory,
``zstash_demo``. We've also created an archive in HPSS named ``zstash_archive``.

The “cache” is our local archive. HPSS is our long-term archive.
Our cache is given the default name 

The cache ``zstash_demo/zstash`` now contains ``index.db``,
which is the sqlite database used by ``zstash``.

In our HPSS window, ``ls zstash_archive`` will show ``000000.tar`` in addition to ``index.db``.
The former is the tar file created from the files and directories in ``zstash_demo``.

.
The former is the tar file created from the files and directories in 

Note that the maximum size of tar files will default to 256 GB, unless ``--maxsize`` is passed in
with a different value to ``zstash create``.

Note that the maximum size of tar files will default to 256 GB, unless 

Now, let's add some files to our source directory, ``zstash_demo``.
We want to update our HPSS archive with these
changes, so we will enter the directory ``zstash_demo`` and tell ``zstash`` to update it.
Note that we have to be inside the source directory to run ``zstash update``.
We also have to specify ``--hpss`` so that ``zstash`` will know which HPSS archive to update. ::

.
changes, so we will enter the directory 

 to update it.
Note that we have to be inside the source directory to run 

$ ./add_files.sh
$ cd zstash_demo
$ zstash update --hpss=zstash_archive

INFO: Creating new tar archive 000001.tar
INFO: Archiving dir2/file2.txt
INFO: Transferring file to HPSS: zstash/000001.tar

In our HPSS window, notice that ``ls zstash_archive`` now includes a new tar file ``000001.tar``.
``zstash update`` produces a new tar file for each update.
Nothing will have changed in ``zstash_demo/zstash``, however.

 produces a new tar file for each update.
Nothing will have changed in 

Now, we will extract the files from the HPSS archive. We will extract into a new directory
because ``zstash`` will not extract files that are already present. So, we will exit the
``zstash_demo`` directory and create a new directory. Extraction will always occur
in the current directory. ::

because 

 will not extract files that are already present. So, we will exit the

 directory and create a new directory. Extraction will always occur
in the current directory.

$ cd ..
$ mkdir zstash_extraction
$ cd zstash_extraction
$ zstash extract --hpss=zstash_archive

INFO: Transferring file from HPSS: zstash/index.db
INFO: Transferring file from HPSS: zstash/000000.tar
INFO: Opening tar archive zstash/000000.tar
INFO: Extracting file0.txt
INFO: Extracting file_empty.txt
INFO: Extracting empty_dir
INFO: Transferring file from HPSS: zstash/000001.tar
INFO: Opening tar archive zstash/000001.tar
INFO: Extracting dir/file1.txt
INFO: Extracting dir2/file2.txt
INFO: No failures detected when extracting the files.

The contents of ``zstash_extraction`` will be identical to those of ``zstash_demo``.

$ rm zstash_archive/000000.tar zstash_archive/000001.tar zstash_archive/index.db
$ rmdir zstash_archive

This example will again illustrate ``create``, ``update``, and ``extract``,
but this time without using HPSS.

Now, we will create the local ``zstash`` archive, but skip creating a HPSS archive. ::

INFO: put: HPSS is unavailable
INFO: put: Keeping tar files locally and removing write permissions
INFO: zstash/000000.tar original mode=b"'660'"
INFO: zstash/000000.tar new mode=b"'440'"

``zstash_demo/zstash`` (the cache, located in the source directory) is our only archive now.
It contains both ``index.db`` and ``000000.tar``.
In our HPSS window, ``ls`` will not show ``zstash_archive``.
Our cache has completely replaced the HPSS archive we used in the simple case.

 (the cache, located in the source directory) is our only archive now.
It contains both 

.

Now, we can add some more files and update this local archive. ::

INFO: zstash/000001.tar original mode=b"'660'"
INFO: zstash/000001.tar new mode=b"'440'"

``000001.tar`` has been added to ``zstash_demo/zstash``.
In our HPSS window, ``ls`` still shows no archive created from this example.

Now, we will extract the files from this local archive. Again, we will extract into a new directory
``zstash_demo`` directory and create a new directory. Unlike in the simple case however,
we will need to copy the cache ``zstash_demo/zstash`` into this new directory. We have
to do this because with ``--hpss=none``, ``zstash`` simply extracts from the cache
in the current directory, rather than extracting from an HPSS archive. We could also
pass in ``--cache=../zstash_demo/zstash``; ``--cache`` will be discussed in the next section. ::

because 

 directory and create a new directory. Unlike in the simple case however,
we will need to copy the cache 

 simply extracts from the cache
pass in 

$ cd ..
$ cp -r ../zstash_demo/zstash/ zstash
$ zstash extract --hpss=none

As with the simple case, the contents of ``zstash_extraction``
will be identical to those of ``zstash_demo``. We still had to specify ``--hpss=none`` because
if we had kept ``--hpss=hpss_archive``, then ``zstash`` would have tried to extract from a HPSS
archive that doesn't exist.

 would have tried to extract from a HPSS
archive that doesn’t exist.

In our NERSC window (use the ``-f`` option to force deletion of tar files): ::

but this time using a different cache.

This time, however, our local archive will be named ``my_cache`` instead ``zstash``.
``--cache`` just takes a directory path; it can
be a relative path and can contain ``..`` or it can be an absolute path. `However,` it
is recommended that however the path is written, it places the cache in the source
directory (in this case, ``zstash_demo``). ::

This time, however, our local archive will be named 

 just takes a directory path; it can
be a relative path and can contain 

 it
directory (in this case, 

$ zstash create --hpss=zstash_archive --cache=my_cache zstash_demo
zstash create --hpss=zstash_archive --cache=/global/cscratch1/sd/forsyth/tutorial/zstash_demo/my_cache zstash_demo

INFO: Transferring file to HPSS: my_cache/000000.tar
INFO: Transferring file to HPSS: my_cache/index.db

``zstash_demo/my_cache`` will now contain ``index.db`` just as ``zstash_demo/zstash``
did in the simple case.

As in the simple case, in our HPSS window, ``ls zstash_archive`` shows both
``000000.tar`` and ``index.db``.

$ zstash update --hpss=zstash_archive --cache=my_cache

INFO: Transferring file to HPSS: my_cache/000001.tar

As in the simple case, nothing will have changed in the cache
(named ``zstash_demo/my_cache`` here).

.
(named 

$ cd ..
$ zstash extract --hpss=zstash_archive --cache=my_cache

INFO: Transferring file from HPSS: my_cache/index.db
INFO: Transferring file from HPSS: my_cache/000000.tar
INFO: Opening tar archive my_cache/000000.tar
INFO: Transferring file from HPSS: my_cache/000001.tar
INFO: Opening tar archive my_cache/000001.tar

As in the simple case, the contents of ``zstash_extraction`` will be identical to those of
``zstash_demo``. The difference is that this time the cache will be named ``my_cache`` in both
``zstash_demo`` and ``zstash_extraction``.

but this time storing the tar files in the local archive (the cache) in addition to the
HPSS archive. In the ``--hpss=none`` example, we saw that when there was no HPSS archive
being used, the cache contained tar files. The ``--keep`` option allows us to store these files
locally even if we're using HPSS. If ``--keep`` is used with ``--hpss=none``, there won't be a
noticeable effect since the latter keeps tar files by default.

,
HPSS archive. In the 

 example, we saw that when there was no HPSS archive
being used, the cache contained tar files. The 

 option allows us to store these files
locally even if we’re using HPSS. If 

, there won’t be a

This time, however, we will specify that the cache should keep the tar file created
by ``zstash create``. ::

by 

``zstash_demo/zstash`` and our HPSS archive ``zstash_archive`` now have identical contents:
``000000.tar`` and ``index.db``. In the simple case, the former would not have contained
the tar file.

. In the simple case, the former would not have contained

$ zstash update --hpss=zstash_archive --keep

Again, ``zstash_demo/zstash`` and our HPSS archive ``zstash_archive``
will have identical contents: ``000001.tar`` in addition to ``000000.tar`` and ``index.db``.
In the simple case, the former would not have the tar file added.

.

$ cd ..
$ zstash extract --hpss=zstash_archive --keep

``zstash_demo``. The difference is that this time ``zstash_demo/zstash`` contains tar files
and thus so too does ``zstash_extraction/zstash``, whereas this was not the case in
the simple case.

This example will demonstrate how to check that files haven't been damaged during archive or transfer.

This example will demonstrate how to check that files haven’t been damaged during archive or transfer.

First, let's create local and HPSS archives as in the simple case: ::

First, let’s create local and HPSS archives as in the simple case:

Now, we will check the file integrity. Note that we'll want to enter the directory where our
local cache (``zstash``) is. If we run ``zstash check`` in another directory,
a local cache (named ``zstash``) will be created in that directory. ::

Now, we will check the file integrity. Note that we’ll want to enter the directory where our
local cache (

INFO: Checking file0.txt
INFO: Checking file_empty.txt
INFO: Checking dir/file1.txt
INFO: Checking empty_dir
INFO: No failures detected when checking the files.

No failures were detected, so the file integrity has been confirmed. This was done by verifying their md5 checksums
against the original ones stored in the database.

``zstash_demo/zstash`` will still only contain ``index.db`` and no files will have been added to
``zstash_demo``.

In our HPSS window, ``ls zstash archive`` will show ``000000.tar`` and ``index.db``.

$ rm zstash_archive/000000.tar zstash_archive/index.db

This example will demonstrate how to list the files contained in the tar files in the HPSS archive.

Recall from the simple case that the cache ``zstash_demo/zstash`` now contains ``index.db`` and
that ``zstash_archive`` will contain ``000000.tar`` in addition to ``index.db``.

We can check the contents of the tar file by running the following.
Note that it's good to be in the directory with the cache
(in this case ``zstash_demo`` contains the cache ``zstash``).
If we run ``zstash ls`` in another directory, a local cache (named ``zstash``) will be created
in that directory. ::

Note that it’s good to be in the directory with the cache
(in this case 

Compare to the output of ``ls -R .`` (i.e., list all files in ``zstash_demo``): ::

dir  empty_dir  file0.txt  file_empty.txt  zstash

./dir:
file1.txt

./empty_dir:

./zstash:
index.db

``zstash ls`` lists the same files but excludes the cache (in this case, named ``zstash``).

 lists the same files but excludes the cache (in this case, named 

Now that we have extracted files, let's list the files in the archive again. ::

Now that we have extracted files, let’s list the files in the archive again.

We can see that extraction did not alter what ``zstash ls`` displays, since extraction does not
change the contents of the HPSS archive.

 displays, since extraction does not

Now, instead of using placeholder data, let's archive actual data from E3SM. Archiving large amounts
of data can take several days, so we'll use a data transfer node to improve performance and
``screen`` so that we can detatch and reattach without stopping ``zstash``. For this example,
however, we'll use a small amount of data so ``zstash`` finishes in a reasonable amount of time.

Now, instead of using placeholder data, let’s archive actual data from E3SM. Archiving large amounts
of data can take several days, so we’ll use a data transfer node to improve performance and

. For this example,
however, we’ll use a small amount of data so 

We'll call our directory ``e3sm_output`` this time instead of ``zstash_demo``. We'll enter this
directory and copy over a couple ``nc`` files from the data repository on NERSC. ::

$ mkdir e3sm_output
$ cd e3sm_output
$ cp /global/cfs/cdirs/e3smpub/E3SM_simulations/20180215.DECKv1b_H1.ne30_oEC.edison/archive/atm/hist/20180215.DECKv1b_H1.ne30_oEC.edison.cam.h0.1850-01.nc .
$ cp /global/cfs/cdirs/e3smpub/E3SM_simulations/20180215.DECKv1b_H1.ne30_oEC.edison/archive/atm/hist/20180215.DECKv1b_H1.ne30_oEC.edison.cam.h0.1850-02.nc .

Now, let's enter the data transfer node, use ``screen``, install ``zstash``, and begin archiving. ::

$ ssh dtn01.nersc.gov
$ screen
$ bash
$ source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh
$ cd $CSCRATCH/tutorial
$ zstash create --hpss=zstash_archive e3sm_output

If the archiving is taking a long time, we could enter the keys ``CTRL-A`` and then ``CTRL-D`` to
detach and ``exit`` to get out of the data transfer node. We could reattach with: ::

If the archiving is taking a long time, we could enter the keys 

For more information on archiving E3SM output, see :ref:`best-practices`.

This page summarizes the recommended best practices for archiving
E3SM simulation output using zstash.

For best performance on NERSC, you should login to one
of the data transfer nodes (dtn<01..15>.nersc.gov). Also, because
archiving large amount of data with zstash can take several days,
it is recommended to invoke zstash within a UNIX `screen` session
to which you can detach and re-attach without killing zstash. You
can access zstash on the data transfer nodes by loading the E3SM unified environment:

it is recommended to invoke zstash within a UNIX 

 session

  ssh dtn01.nersc.gov
  screen
  bash
  source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

ssh dtn01.nersc.gov
screen
bash
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

To detach from the screen session, use CTRL-A followed by D (for detach).
You can then safely close your window. To re-attach to an existing session
later:

Typically, you should consider archiving the entire directory structure
of a simulation. The first time, this is accomplished with ``zstash create``.
For example:

of a simulation. The first time, this is accomplished with 

  screen -r
  cd /global/cscratch1/sd/golaz/E3SM/simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
  mkdir zstash
  zstash create --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \
    --maxsize 128 . 2>&1 | tee zstash/zstash_create_20190226.log

screen -r
cd /global/cscratch1/sd/golaz/E3SM/simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
mkdir zstash
zstash create --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \
  --maxsize 128 . 2>&1 | tee zstash/zstash_create_20190226.log

The command above will archive the entire directory structure under
`/global/cscratch1/sd/golaz/E3SM/simulations/20180129.DECKv1b_piControl.ne30_oEC.edison`.
The archive files will be stored on HPSS under `2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison`.
Individual tar files in the zstash archive will have a maximum size of 128 GB. NERSC typically
recommends file size between 100 and 500 GB for best performance.

.

If your model output has been reorganized using the CIME short-term archive utility, you can easily
archive only a subset of the restart files to conserve space. For example, to **archive
restart files every 5 years** only:

archive only a subset of the restart files to conserve space. For example, to 

  screen -r
    --exclude="archive/rest/???[!05]-*/" \

screen -r
  --exclude="archive/rest/???[!05]-*/" \

You can also add newly created files to an existing archive, or restart archiving after a
failure using the ``zstash update`` functionality:

failure using the 

  screen -r
  zstash update --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \
    --exclude="archive/rest/???[!05]-*/" 2>&1 | tee zstash/zstash_update_20190226.log

screen -r
zstash update --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \
  --exclude="archive/rest/???[!05]-*/" 2>&1 | tee zstash/zstash_update_20190226.log

After archiving or updating, it is **highly recommended** that you verify the integrity
of the tar files. The safest way to do so is go to a new, empty directory and run:

 that you verify the integrity

  zstash check --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison

``zstash check`` will download the tar archives to the local disk cache (under
the zstash/ subdirectory) and verify the md5 checksum of every file against the
checksum stored in the index database (index.db).

 will download the tar archives to the local disk cache (under

``zstash check`` can also be run in the original location. But if tar files
are present in the local cache (zstash/ sub-directory), the check will only
be performed against the local disk copy, not the one on HPSS.

 can also be run in the original location. But if tar files

To save time, you can check in parallel by passing in a number of processes with the ``--workers``
argument. Please see the full documentation for ``zstash check`` for more information.

To save time, you can check in parallel by passing in a number of processes with the 

If any corrupted file is found during the check, zstash will print a list of corrupted
files in the archive, along with the tar archive they belong to.

Checksum mismatch error: The file was extracted but the checksum doesn't
match the original one computed when the files was uploaded.

Checksum mismatch error: The file was extracted but the checksum doesn’t

General extraction error: The file can't be extracted due to another error.

General extraction error: The file can’t be extracted due to another error.

To see what kind of error you have, search the output for your filename
and you should be able to see what happened.

Regarding the first error, this seems to be caused by something on the HPSS end of this process.
If you have the original model data, please try uploading it again.

INFO: Checking archive/ocn/hist/mpaso.hist.am.timeSeriesStatsMonthly.1855-10-01.nc
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/zstash/extract.py", line 145, in extractFiles
    s = fin.read(BLOCK_SIZE)
  File "/global/homes/z/zshaheen/anaconda2/envs/zstash_env_v0.2.0/lib/python2.7/tarfile.py", line 831, in read
    buf += self.fileobj.read(size - len(buf))
  File "/global/homes/z/zshaheen/anaconda2/envs/zstash_env_v0.2.0/lib/python2.7/tarfile.py", line 743, in read
    return self.readnormal(size)
  File "/global/homes/z/zshaheen/anaconda2/envs/zstash_env_v0.2.0/lib/python2.7/tarfile.py", line 758, in readnormal
    return self.__read(size)
  File "/global/homes/z/zshaheen/anaconda2/envs/zstash_env_v0.2.0/lib/python2.7/tarfile.py", line 750, in __read
    raise ReadError("unexpected end of data")
ReadError: unexpected end of data

This seems to be caused by the filesystem. Simply run ``zstash check`` again.
To save time, like ``zstash extract``, you can check for specific files or tar archives:

  zstash check --hpss=/path/to/hpss/archive "archive/ocn/hist/mpaso.hist.am.timeSeriesStatsMonthly.1892-04-01.nc"
  zstash check --hpss=/path/to/hpss/archive "000012.tar"

zstash check --hpss=/path/to/hpss/archive "archive/ocn/hist/mpaso.hist.am.timeSeriesStatsMonthly.1892-04-01.nc"
zstash check --hpss=/path/to/hpss/archive "000012.tar"

There is no long-term HPSS storage attached to Compy or Anvil. To archive a new simulation,
we recommend the following:

While the instructions below are specific for Compy, adapting them for Anvil should
be straightforward.

Starting with v0.4, zstash supports the creation of local archives only (using the
``--hpss=none`` command line option). For example

  screen
  cd /compyfs/gola749/E3SM_simulations/20191216.alpha20.piControl.ne30_r05_oECv3_ICG.compy
  zstash create --hpss=none  --maxsize 128 . 2>&1 | tee zstash/zstash_create_20200224.log
  ctrl-a d # to disconnect from screen session

screen
cd /compyfs/gola749/E3SM_simulations/20191216.alpha20.piControl.ne30_r05_oECv3_ICG.compy
zstash create --hpss=none  --maxsize 128 . 2>&1 | tee zstash/zstash_create_20200224.log
ctrl-a d # to disconnect from screen session

Login to Globus web interface at https://www.globus.org/ using your NERSC credentials.

Search for 'NERSC HPSS'. Click on Green power button to activate endpoint.

Search for ‘NERSC HPSS’. Click on Green power button to activate endpoint.

.. image:: /_static/figures/Globus_Screenshot_1.png
   :scale: 50%
   :alt: Globus screenshot, NERSC HPSS endpoint

Search for 'compy-dtn'. Click on Green power button to activate endpoint. Login
using your compy credentials (username, PIN+RSA).

Search for ‘compy-dtn’. Click on Green power button to activate endpoint. Login

.. image:: /_static/figures/Globus_Screenshot_2.png
   :alt: Globus screenshot, compy-dtn endpoint

.. image:: /_static/figures/Globus_Screenshot_3.png
   :alt: Globus screenshot, file manager

Configure sync

- Select all source files in zstash folder.
- Select destination endpoint and folder (the Globus web interface cannot create new
  directories, so you will have to create the destination directory on NERSC HPSS is if
  doesn't already exist).
- Suggested options for sync

  - transfer files where the modification time is newer
  - preserve source file modification time

Select destination endpoint and folder (the Globus web interface cannot create new
directories, so you will have to create the destination directory on NERSC HPSS is if
doesn't already exist).

doesn’t already exist).

Suggested options for sync

- transfer files where the modification time is newer
- preserve source file modification time

.. image:: /_static/figures/Globus_Screenshot_4.png
   :alt: Globus screenshot, sync

Once archiving is complete, run ``zstash check`` on NERSC to verify integrity of the archive:

  cd <scratch directory>
  zstash check --hpss=<HPSS path>

cd <scratch directory>
zstash check --hpss=<HPSS path>

Directly interacting with the database is not recommened
unless you understand the consequences of what you are doing!

Metadata from zstash archives is stored in a sqlite3 database
in ``<cache>/index.db``, where ``<cache>`` is ``zstash`` by default.
The database schema consists of
two tables, one to store configuration parameters (``config``)
and the other one for metadata for each archived file (``files``). ::

in 

 by default.
two tables, one to store configuration parameters (

$ sqlite3 zstash/index.db
SQLite version 3.34.0 2020-12-01 16:14:00
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE config (
  arg text primary key,
  value text
);
CREATE TABLE files (
  id integer primary key,
  name text,
  size integer,
  mtime timestamp,
  md5 text,
  tar text,
  offset integer
);
CREATE TABLE tars (
id integer primary key,
name text,
size integer,
md5 text
);
sqlite> .quit

**offset**: offset in bytes where the file is located within its tar file.

: offset in bytes where the file is located within its tar file.

Direct interaction with the database can be useful to explore content
of an archive, beyond what might be available with :ref:`zstash list<zstash-list>`.

of an archive, beyond what might be available with 

    cd <mydir>
    sqlite3 zstash/index.db "select * from files;"

file # | path | size | modification time |md5 checksum |tar archive | offset (within tar)

To list **files matching a specified pattern** (for example \*/run/\*.nc):

    sqlite3 zstash/index.db "select * from files where name glob '*/run/*.nc';"

sqlite3 zstash/index.db "select * from files where name glob '*/run/*.nc';"

To list **all the files in a specific tar file** (for example 00000a.tar):

    sqlite3 zstash/index.db "select * from files where tar is '00000a.tar';"

sqlite3 zstash/index.db "select * from files where tar is '00000a.tar';"

NERSC HPSS and other similar tape storage systems are optimized
for file sizes of 100 to 500 GB. The retrieval of a large number
of small files that are spread across dozens or hundreds of tapes
is very inefficient because it requires multiple loads into tape
drives and positioning the tape.

The recommended practice is thus to bundle smaller files into larger
ones. zstash was created specifically for this purpose. While
the existing 'htar' utility offers a similar functionality, it also
suffers from limitations that make it not suitable for archiving
E3SM model output (see below).

the existing ‘htar’ utility offers a similar functionality, it also

Files are bundled **into standard tar files** of a specified maximum size.
The default maximum size is 256 GB, but it can be adjusted at
creation using the ``--maxsize`` command line argument.

 of a specified maximum size.
creation using the 

**Individual files are never split** between two separate tar
files. As a result, the size of tar files varies, but will
generally be smaller than the specified maximum size. The only
exception would be if individual files are larger than the target size.

 between two separate tar

**Tar files are named sequentially** using a six
digit hexadecimal number (`000000.tar`,  `000001.tar`, ...).

**Tar files are first created locally** on disk (under the
`zstash/` subdirectory) and **then uploaded to HPSS**. Similarly, when
extracting files, tar files will be first downloaded from HPSS into
the local cache (if they are not already present).

. Similarly, when

An sqlite3 **index database** is maintained (`zstash/index.db`) that contains
information about every archived files, including:

* full pathname relative to top-level archival directory.
* size
* modification time
* md5 checksum
* tar file in which the file is included
* offset within this tar file (for faster extraction).

) that contains

 during

files.

Htar is an HPSS utility with many nice features. From the NERSC `htar documentation page
<http://docs.nersc.gov/filesystems/archive_access/#htar>`_:

Htar is an HPSS utility with many nice features. From the NERSC 

`htar documentation page
<http://docs.nersc.gov/filesystems/archive_access/#htar>`_

*HTAR is a command line utility that creates and manipulates HPSS-resident
tar-format archive files. It is ideal for storing groups of files in HPSS.
Since the tar file is created directly in HPSS, it is generally faster and
uses less local space than creating a local tar file then storing that into
HPSS. Furthermore, HTAR creates an index file that (by default) is stored
along with the archive in HPSS. This allows you to list the contents of
an archive without retrieving it to local storage first. The index file is
only created if the HTAR bundle is successfully stored in the archive.*

HTAR is a command line utility that creates and manipulates HPSS-resident
only created if the HTAR bundle is successfully stored in the archive.

This sounds great. In fact, some of these htar functionalities served as inspiration for zstash.
However, htar also suffers from a number of limitations that were deemed to be **show stoppers**:

However, htar also suffers from a number of limitations that were deemed to be 

**Member File Path Length**

File path names within an HTAR aggregate of the form prefix/name are limited to
**154 characters** for the prefix and 99 characters for the file name. Link names
cannot exceed 99 characters.

**Member File Size**

The maximum file size the NERSC archive will support is approximately 20 TB.
However, we recommend you aim for HTAR aggregate sizes of several hundred GBs.
Member files within an HTAR aggregate are limited to approximately **68GB**.

 for the prefix and 99 characters for the file name. Link names

Member files within an HTAR aggregate are limited to approximately 

Both the member file path length and file size were simply too restrictive
for archiving output from E3SM simulations. Furthermore, htar lacks the ability
to compute checksums of member files.

This is the brute force approach: recursive conditional put. It will recursively
transfer all your files to tape and only do so if the file isn't already on tape.

transfer all your files to tape and only do so if the file isn’t already on tape.

While this approach *may* work, it is not recommended when dealing with very
large number of (relatively small) files. Instead, NERSC recommends to
`group small files together <https://docs.nersc.gov/filesystems/archive/#group-small-files-together>`_ :

 work, it is not recommended when dealing with very

`group small files together <https://docs.nersc.gov/filesystems/archive/#group-small-files-together>`_

*HPSS is a tape system and responds differently than a typical file system.
If you upload large numbers of small files they will be spread across
dozens or hundreds of tapes, requiring multiple loads into tape drives
and positioning the tape. Storing many small files in HPSS without
bundling them together will result in extremely long retrieval times
for these files and will slow down the HPSS system for all users.*

HPSS is a tape system and responds differently than a typical file system.
for these files and will slow down the HPSS system for all users.

If you have an account on one of the E3SM supported machines, you can access ``zstash`` by activating ``e3sm_unified``, which is
a conda environment that pulls together Python and other E3SM tools such as
``e3sm_diags`` and ``zppy``.

If you have an account on one of the E3SM supported machines, you can access 

, which is

The paths to ``e3sm_unified`` activation scripts are machine dependent. As of E3SM Unified 1.13.0, the supported machines and their corresponding activation scripts are:

 activation scripts are machine dependent. As of E3SM Unified 1.13.0, the supported machines and their corresponding activation scripts are:

**Andes**
::

 source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_andes.sh

**Aurora**
::

 source /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_aurora.sh

**Chrysalis**
::

 source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

**Compy**
::

 source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh

**Dane**
::

 source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_dane.sh

**Frontier**
::

 source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_frontier.sh

**Perlmutter (login or CPU nodes)**
::

 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

**ALCF Polaris**
::

 source /lus/grand/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_polaris.sh

``zstash`` development is largely synced with ``e3sm_unified``. The last several releases have been:

To use ``zstash`` features/fixes not yet in a production release, you can use a development environment.

 features/fixes not yet in a production release, you can use a development environment.

Nevertheless, it is possible that the version of ``zstash`` included in ``e3sm_unified`` may not be the latest. To install the latest stable release, refer to the following:

 may not be the latest. To install the latest stable release, refer to the following:

# Get the code ########################################################################

# Set up your fork:
# Go to https://github.com/E3SM-Project/zstash
# Click the green "Code" button
# Choose the SSH option and paste it here:
git clone git@github.com:E3SM-Project/zstash.git
cd zstash
git remote -v # You should see the main repo listed as `origin`

# A couple optional steps:
git remote add upstream git@github.com:E3SM-Project/zstash.git # Use the name "upstream" instead of origin
git remote add your-fork-name git@github.com:your-fork-name/zstash.git # Use your fork, if you have one

# To use the latest code:
git checkout -b main upstream/main
# To use code from a specific branch:
git checkout -b that-branch-name upstream/that-branch-name

# Set up the environment ##############################################################
# First, make sure you have conda activated. Then:
rm -rf build # Sometimes an existing `build` directory can cause problems.
conda clean --all --y # This makes sure conda will pick up the latest information.
conda env create -f conda/dev.yml -n env-name
conda activate env-name
pre-commit run --all-files # This is only necessary if you've made changes
python -m pip install . # Install the code into your development environment (env-name)

Note: if you'd like to contribute to ``zstash`` rather than just using the latest code, please refer to the Developer Guide instead.

 rather than just using the latest code, please refer to the Developer Guide instead.

zstash may create and read a configuration file at ``~/.zstash.ini`` when you
use Globus-backed archives.

The main setting currently used by zstash is the local Globus endpoint UUID.
The file uses an INI format such as:

[local]
globus_endpoint_uuid = 6bdc7956-fc0f-4ad2-989c-7aa5ee643a79 # NERSC_PERLMUTTER_ENDPOINT

If the file does not exist, zstash creates it the first time it needs Globus
configuration.

Most users do not need to edit ``~/.zstash.ini``. Manual updates are useful
when:

zstash cannot infer the local endpoint UUID from the machine hostname

the machine's Globus collection changed and the old UUID is still recorded

the machine’s Globus collection changed and the old UUID is still recorded

you are testing on a new system before hostname-based auto-detection has been
added to zstash

If the UUID is wrong or blank, Globus transfers may fail before they start.

Use the UUID for the local collection that should serve as the source or
destination of the transfer on the machine where you are running ``zstash``.
You can usually find that UUID from the Globus web interface for the
collection.

destination of the transfer on the machine where you are running 

.

After updating ``~/.zstash.ini``, rerun the zstash command that needs Globus.

``~/.zstash_globus_tokens.json`` stores refresh tokens from previous Globus
logins

``~/.globus-native-apps.cfg`` may be left over from older Globus-based
workflows

workflows

If zstash reports that a stored refresh token is invalid, deleting
``~/.zstash_globus_tokens.json`` is the usual way to force a fresh login.

This page is intended for users who want to run ``zstash`` on a machine that
does not have direct HPSS access, or who prefer to move zstash archives through
the `Globus <https://www.globus.org/>`_ transfer service.

 on a machine that
the 

Use a Globus destination when ``--hpss`` is set to a URL of the form::

``globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/~/my_archive`` (NERSC HPSS Globus endpoint)

The names ``nersc`` and ``alcf`` are built-in shortcuts for the NERSC HPSS and
ALCF HPSS Globus endpoints.

 are built-in shortcuts for the NERSC HPSS and

Identify the local Globus collection for the machine where you will run
``zstash``.

Activate both collections in the Globus web interface before you run

Start with a small archive so you can confirm that authentication, endpoint
activation, and path selection are correct.

If you are creating a new archive, a minimal first test looks like

    zstash create --hpss=globus://nersc/~/test/my_archive .

After the transfer completes, verify it with ``zstash check`` or retrieve a
small file with ``zstash extract``.

The first time zstash needs Globus credentials, it will print an authorization
URL and ask you to paste back the returned code. After a successful login,
zstash stores refresh-token state in ``~/.zstash_globus_tokens.json`` so future
Globus transfers between the same machines usually do not need another interactive login.

zstash stores refresh-token state in 

 so future

zstash also checks ``~/.zstash.ini`` for the local endpoint UUID. See
:doc:`configuration` for details on when that file needs to be created or
edited manually.

 for details on when that file needs to be created or

The destination portion of ``globus://<endpoint>/<path>`` should name the
remote directory that will hold the zstash archive contents:

 should name the

one or more tar files such as ``000000.tar``, ``000001.tar``, and so on

As with HPSS paths, use a destination directory that is dedicated to one
zstash archive.

``zstash create`` and ``zstash update`` create tar files locally first, then
transfer them through Globus.

 create tar files locally first, then

``zstash check``, ``zstash extract``, and ``zstash ls`` still rely on the
archive's ``index.db`` to locate files and tars.

``--non-blocking`` applies only to Globus transfers. Use it when you want
zstash to continue building later tar files before the current transfer has
finished.

finished.

 applies only to Globus transfers. Use it when you want
finished.

Re-activate the source and destination collections in the Globus web
interface.

Confirm that ``~/.zstash.ini`` points to the correct local endpoint UUID.

If zstash reports token problems after you switch endpoints or machines,
remove ``~/.zstash_globus_tokens.json`` and retry so zstash can request a new
login.

remove 

If the destination path does not exist, create it first and rerun the
command.

Use this guide for installing zstash, creating and maintaining archives, and
working with HPSS or Globus in day-to-day workflows.

If you run into **problems** using zstash, please open
a new `issue on Github <https://github.com/E3SM-Project/zstash/issues>`_.

`issue on Github <https://github.com/E3SM-Project/zstash/issues>`_

Suggestions for future **enhancements** can also be made by opening an
`issue on Github <https://github.com/E3SM-Project/zstash/issues>`_.

If running on Cori, it is preferable to run from ``$CSCRATCH`` rather than
``/global/homes``. Running from the latter may result in a
'Resource temporarily unavailable' error.

. Running from the latter may result in a
‘Resource temporarily unavailable’ error.

If you have not logged into HSI before, you will have to do so before running ``zstash`` with HPSS.
On NERSC machines, just run ``hsi`` on the command line and enter your credentials.
Note that ``hsi`` is only available on the log-in nodes, not the compute nodes.

If you have not logged into HSI before, you will have to do so before running 

If you set ``--hpss=none``, there is no need to set ``--keep``, as the former option includes the effects of the latter.

When specifying files, wildcards should be enclosed in double quotes (e.g., ``"a*"``).

When specifying files, wildcards should be enclosed in double quotes (e.g., 

Specifying a high number for ``--workers`` will result in slow downloads for each of the tars since your bandwidth
is limited. User discretion is advised.

 will result in slow downloads for each of the tars since your bandwidth

  zstash create --hpss=<path to HPSS> <local path>

``--hpss=<path to HPSS>`` specifies the destination path on the HPSS file
system where the archive files will be stored. This directory should be **unique** for each
zstash archive. If ``--hpss=none``, then files will be archived locally instead of being
transferred to HPSS. The ``none`` option should be used when running Zstash on a machine
without HPSS. The option also accepts a Globus URL, ``globus://<Globus endpoint UUID/<path to archive>``.
Then zstash will use `Globus <https://globus.org/>`_ to store a new zstash archive on a Globus endpoint.
Names ``alcf`` and ``nersc`` are recognized as referring to the ALCF HPSS and NERSC HPSS endpoints,
e.g. ``globus://nersc/~/my_archive``.
See :doc:`user_guide/globus/intro` and :doc:`user_guide/globus/configuration` for first-time Globus setup details.

 specifies the destination path on the HPSS file
system where the archive files will be stored. This directory should be 

, then files will be archived locally instead of being
transferred to HPSS. The 

 option should be used when running Zstash on a machine
without HPSS. The option also accepts a Globus URL, 

 are recognized as referring to the ALCF HPSS and NERSC HPSS endpoints,
e.g. 

``<local path>`` specifies the path to the local directory that should be archived.

 specifies the path to the local directory that should be archived.

``--cache`` to use a cache other than the default of ``zstash``. If hpss is ``--hpss=none``, then this will be the archive.

``--follow-symlinks`` Hard copy symlinks. This is useful for preventing broken links. Note that a broken link will result in a failed create.

 Hard copy symlinks. This is useful for preventing broken links. Note that a broken link will result in a failed create.

``--keep`` to keep a copy of the tar files on the local file system after
they have been transferred to HPSS. Normally, they are deleted after
successful transfer.

 to keep a copy of the tar files on the local file system after

``--maxsize MAXSIZE`` specifies the maximum size (in GB) for tar files.
The default is 256 GB. Zstash will create tar files that are smaller
than MAXSIZE except when individual input files exceed MAXSIZE (as
individual files are never split up between different tar files).

 specifies the maximum size (in GB) for tar files.

``--non-blocking`` Zstash will submit a Globus transfer and immediately create a subsequent tarball. That is, Zstash will not wait until the transfer completes to start creating a subsequent tarball. On machines where it takes more time to create a tarball than transfer it, each Globus transfer will have one file. On machines where it takes less time to create a tarball than transfer it, the first transfer will have one file, but the number of tarballs in subsequent transfers will grow finding dynamically the most optimal number of tarballs per transfer.

 Zstash will submit a Globus transfer and immediately create a subsequent tarball. That is, Zstash will not wait until the transfer completes to start creating a subsequent tarball. On machines where it takes more time to create a tarball than transfer it, each Globus transfer will have one file. On machines where it takes less time to create a tarball than transfer it, the first transfer will have one file, but the number of tarballs in subsequent transfers will grow finding dynamically the most optimal number of tarballs per transfer.

``--error-on-duplicate-tar`` FOR ADVANCED USERS ONLY: Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash's behavior will depend on whether or not the --overwrite-duplicate-tar flag is set.

 FOR ADVANCED USERS ONLY: Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash’s behavior will depend on whether or not the –overwrite-duplicate-tar flag is set.

``--overwrite-duplicate-tars`` FOR ADVANCED USERS ONLY: If a duplicate tar is encountered, overwrite the existing database record with the new one (i.e., it will assume the latest tar is the correct one). If this flag is not set, zstash will permit multiple entries for the same tar in its database.

 FOR ADVANCED USERS ONLY: If a duplicate tar is encountered, overwrite the existing database record with the new one (i.e., it will assume the latest tar is the correct one). If this flag is not set, zstash will permit multiple entries for the same tar in its database.

Local tar files as well as the sqlite3 index database (index.db) will be stored
under ``<local path>/zstash``.

under 

**After you run** ``zstash create`` **it's highly recommended that you
run** ``zstash check``, **detailed in the section below. This will allow you to check that your archival completed successfully. Do not delete any data until you've run** ``zstash check``.

**detailed in the section below. This will allow you to check that your archival completed successfully. Do not delete any data until you've run**

detailed in the section below. This will allow you to check that your archival completed successfully. Do not delete any data until you’ve run

To **archive** output from an E3SM simulation located
under `$CSCRATCH/ACME_simulations/20170731.F20TR.ne30_ne30.edison`

  cd $CSCRATCH/ACME_simulations/20170731.F20TR.ne30_ne30.edison
  zstash create --hpss=test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison .

cd $CSCRATCH/ACME_simulations/20170731.F20TR.ne30_ne30.edison
zstash create --hpss=test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison .

  hsi
  hsi> cd test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison
  hsi> ls
  # 000000.tar   index.db

hsi
hsi> cd test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison
hsi> ls
# 000000.tar   index.db

The data from this test simulation is small, so in this case there is only a single tar
file (000000.tar) and the index database (index.db).

You may decide that certain files do not need to be archived.
For example, if you want to **exclude \*.o and \*.mod files** under the build
subdirectory:

For example, if you want to 

  zstash create --hpss=test/ACME_simulations/20170731.F20TR.ne30_ne30.edison \
  --exclude="build/*/*.o","build/*/*.mod" .

zstash create --hpss=test/ACME_simulations/20170731.F20TR.ne30_ne30.edison \
--exclude="build/*/*.o","build/*/*.mod" .

Or you may decide that you only want to **archive restart files every 5 years**
to conserve storage space:

  --exclude="archive/rest/???[!05]-*/" .

--exclude="archive/rest/???[!05]-*/" .

This exclude pattern will skip all restart subdirectories under the short-term archive,
except for those with years ending in '0' or '5'.

except for those with years ending in ‘0’ or ‘5’.

If you run zstash on the system without the HPSS file system, but has a `Globus <https://app.globus.org/endpoints>`_ endpoint set up,
you can use a Globus URL:

If you run zstash on the system without the HPSS file system, but has a 

  cd $CSCRATCH/ACME_simulations/20170731.F20TR.ne30_ne30.anvil
  zstash create --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/~/test/E3SM_simulations/20170731.F20TR.ne30_ne30.anvil .

cd $CSCRATCH/ACME_simulations/20170731.F20TR.ne30_ne30.anvil
zstash create --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/~/test/E3SM_simulations/20170731.F20TR.ne30_ne30.anvil .

9cd89cfd-6d04-11e5-ba46-22000b92c6ec is the NERSC HPSS Globus endpoint UUID. Two names ``nersc`` and ``alcf``
are recognized by zstash and substituted internally with a corresponding Globus UUID
for the NERSC HPSS Globus endpoint (9cd89cfd-6d04-11e5-ba46-22000b92c6ec) and
the ALCF HPSS Globus endpoint (de463ec4-6d04-11e5-ba46-22000b92c6ec) endpoint.
If you want to store zstash archive on these two remote HPSS file systems, you can use the names instead of UUIDs:

9cd89cfd-6d04-11e5-ba46-22000b92c6ec is the NERSC HPSS Globus endpoint UUID. Two names 

  zstash create --hpss=globus://nersc/~/test/E3SM_simulations/20170731.F20TR.ne30_ne30.anvil .

zstash create --hpss=globus://nersc/~/test/E3SM_simulations/20170731.F20TR.ne30_ne30.anvil .

If you are a new Globus user, you should first do a small transfer to test functionality.

Always activate Globus endpoints via the Globus web interface before running ``zstash``.

Always activate Globus endpoints via the Globus web interface before running 

For a more complete first-time Globus workflow, including ``~/.zstash.ini``,
see :doc:`user_guide/globus/intro` and :doc:`user_guide/globus/configuration`.

Note: Most of the commands for this are the same for ``zstash extract`` and ``zstash ls``.

To verify that your files were uploaded on HPSS successfully,
go to a **new, empty directory** and run:

go to a 

  zstash check --hpss=<path to HPSS> [--workers=<num of processes>] [--cache=<cache>] [--keep] [-v] [files]

zstash check --hpss=<path to HPSS> [--workers=<num of processes>] [--cache=<cache>] [--keep] [-v] [files]

``--hpss=<path to HPSS>`` specifies the destination path on the HPSS file system. If ``--hpss=none``,
then ``zstash check`` will check the cache.

``--workers=<num of processes>`` an optional argument which specifies the number of
processes to use, resulting in checking being done in parallel.
**Using a high number will result in slow downloads for each of the tars since your bandwidth is limited.**
**User discretion is advised.**

 an optional argument which specifies the number of

Using a high number will result in slow downloads for each of the tars since your bandwidth is limited.

``--cache`` to use a cache other than the default of ``zstash``.

they have been extracted from the archive. Normally, they are deleted after

``--retries`` to set the number of times to retry ``hsi get`` if it is unsuccessful.
The default is 1 retry (2 tries total). Note: for a retry to occur automatically because of
an incomplete tar file, then the archive you're checking
must have been created using ``zstash >= v1.1.0``.

 if it is unsuccessful.
an incomplete tar file, then the archive you’re checking
must have been created using 

``--tars`` to specify specific tars to check. See below for example usage.

 to specify specific tars to check. See below for example usage.

``--error-on-duplicate-tar`` FOR ADVANCED USERS ONLY: Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash will check if the size matches the *most recent* entry.

 FOR ADVANCED USERS ONLY: Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash will check if the size matches the 

``[files]`` is a list of files to check (standard wildcards supported).

* Leave empty to check all the files.
* List of files with support for wildcards. Please note that any expression
  containing **wildcards should be enclosed in double quotes ("...")**
  to avoid shell substitution.
* Names of specific tar archives to check all files within these tar archives.

List of files with support for wildcards. Please note that any expression
containing **wildcards should be enclosed in double quotes ("...")**
to avoid shell substitution.

containing 

Names of specific tar archives to check all files within these tar archives.

the `zstash/` subdirectory) and verify the md5 checksum against the checksum
stored in the index database (`index.db`).

the 

 subdirectory) and verify the md5 checksum against the checksum
stored in the index database (

After the check is complete, a list of all corrupted files in the HPSS archive,
along with the tar archive they belong is listed. Below is an example:  ::

along with the tar archive they belong is listed. Below is an example:

INFO: Checking archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-01.nc
DEBUG: Valid md5: cfb388d9c4ffe3bf45985fa470855801 archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-01.nc
INFO: Checking archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-02.nc
DEBUG: Valid md5: ce9bb79fb60fdef2ca4c2c29afc54776 archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-02.nc
...
ERROR: Encountered an error for files:
ERROR: archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0214-06.nc in 00000a.tar
ERROR: archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0214-07.nc in 00000a.tar
ERROR: archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0214-08.nc in 00000a.tar
...
ERROR: archive/ocn/hist/mpaso.hist.am.timeSeriesStatsMonthly.0085-08-01.nc in 000029.tar
ERROR: archive/ocn/hist/mpaso.hist.am.timeSeriesStatsMonthly.0085-09-01.nc in 000029.tar
ERROR: The following tar archives had errors:
ERROR: 00000a.tar
ERROR: 000029.tar

If you encounter an error, **save your original data**.
You may need to reupload it via ``zstash create``.
Please contact the zstash development team, we're working on
identifying what causes these issues.

.
Please contact the zstash development team, we’re working on

  zstash create --hpss=none zstash_demo
  ls zstash_demo/
  # file0.txt  zstash
  ls zstash_demo/zstash/
  # 000000.tar  index.db
  cd zstash_demo
  zstash check --hpss=none
  # INFO: Opening tar archive zstash/000000.tar
  # INFO: Checking file0.txt
  # INFO: No failures detected when checking the files. If you have a log file, run "grep -i Exception <log-file>" to double check.

zstash create --hpss=none zstash_demo
ls zstash_demo/
# file0.txt  zstash
ls zstash_demo/zstash/
# 000000.tar  index.db
cd zstash_demo
zstash check --hpss=none
# INFO: Opening tar archive zstash/000000.tar
# INFO: Checking file0.txt
# INFO: No failures detected when checking the files. If you have a log file, run "grep -i Exception <log-file>" to double check.

  # Starting at 00005a until the end
  zstash check --tars=00005a-
  # Starting from the beginning to 00005a (included)
  zstash check --tars=-00005a
  # Specific range
  zstash check --tars=00005a-00005c
  # Selected tar files
  zstash check --tars=00003e,00004e,000059
  # Mix and match
  zstash check --tars=000030-00003e,00004e,00005a-

# Starting at 00005a until the end
zstash check --tars=00005a-
# Starting from the beginning to 00005a (included)
zstash check --tars=-00005a
# Specific range
zstash check --tars=00005a-00005c
# Selected tar files
zstash check --tars=00003e,00004e,000059
# Mix and match
zstash check --tars=000030-00003e,00004e,00005a-

An existing zstash archive can be updated to add new or modified files:

  cd <mydir>
  zstash update --hpss=<path to HPSS> [--cache=<cache>] [--dry-run] [--exclude] [--keep] [-v]

cd <mydir>
zstash update --hpss=<path to HPSS> [--cache=<cache>] [--dry-run] [--exclude] [--keep] [-v]

``--hpss=<path to HPSS>`` specifies the destination path on the HPSS file system,

``--dry-run`` an optional argument to specify a dry run, only lists files to be updated in archive.

 an optional argument to specify a dry run, only lists files to be updated in archive.

``--exclude`` an optional argument of comma separated list of file patterns to exclude

 an optional argument of comma separated list of file patterns to exclude

``--follow-symlinks`` Hard copy symlinks. This is useful for preventing broken links. Note that a broken link will result in a failed update.

 Hard copy symlinks. This is useful for preventing broken links. Note that a broken link will result in a failed update.

``--include`` an optional argument of comma separated list of file patterns to include

 an optional argument of comma separated list of file patterns to include

Note: in the event that an update includes revisions to files previously archived, ``zstash update``
will archive the new revisions. ``zstah extract`` will only extract the latest revision, but all
file versions will still be listed with the ``zstash ls`` and ``zstash ls -l`` commands.

Note: in the event that an update includes revisions to files previously archived, 

 will only extract the latest revision, but all
file versions will still be listed with the 

Starting with ``zstash v1.1.0`` the md5 hash for the tars will be computed on ``zstash create``.
If you're using an existing database, then ``zstash update`` will begin keeping track
of the tars automatically.

Following the '**zstash create**' example above, we now run zstash again with the
'**update**' functionality:

  zstash update --hpss=test/ACME_simulations/20170731.F20TR.ne30_ne30.edison

zstash update --hpss=test/ACME_simulations/20170731.F20TR.ne30_ne30.edison

  mkdir new
  echo "This is a new file..." > new/file.txt

Zstash recognizes the presence of a new file and adds it to the archive: ::

Zstash recognizes the presence of a new file and adds it to the archive:

INFO: Archiving new/file.txt
DEBUG: Closing tar archive 000001.tar

Note that the new file is added into a new archive tar file (000001.tar) even
though the first archive tar file (000000.tar) is smaller than the target size
and therefore could potentially hold more data. This is a design choice that
was made out of caution to avoid the risk of damaging an existing tar file by
appending to it.

Note: Most of the commands for this are the same for ``zstash check`` and ``zstash ls``.

To extract files from an existing zstash archive into current <mydir>:

  cd <mydir>
  zstash extract --hpss=<path to HPSS> [--workers=<num of processes>] [--cache=<cache>] [--keep] [-v] [files]

cd <mydir>
zstash extract --hpss=<path to HPSS> [--workers=<num of processes>] [--cache=<cache>] [--keep] [-v] [files]

``--hpss=<path to HPSS>`` specifies the destination path on the HPSS file system.
Note that if ``--hpss=none``, then ``--keep`` is automatically set to ``True``.
The option also accepts a Globus URL, ``globus://<Globus endpoint UUID/<path to archive>``.
Then zstash will use `Globus <https://globus.org/>`_ to extract from a zstash archive on a Globus endpoint.

 specifies the destination path on the HPSS file system.
Note that if 

processes to use, resulting in extracting being done in parallel.

an incomplete tar file, then the archive you're extracting from

an incomplete tar file, then the archive you’re extracting from

``--tars`` to specify specific tars to extract. See "Check" above for example usage.

 to specify specific tars to extract. See “Check” above for example usage.

``[files]`` is a list of files to be extracted (standard wildcards supported).

* Leave empty to extract all the files.
* Names of specific tar archives to extract all files within these tar archives.

 is a list of files to be extracted (standard wildcards supported).

Names of specific tar archives to extract all files within these tar archives.

You must pass in the **path relative to the top level** for the file(s). For help
finding path names, you can use ``zstash ls`` as documented below.

A few words about performance. All of the files are grouped into 256GB tar archives by default.
(See the ``--maxsize`` argument for ``zstash create`` for more information).
If the tar file is not already present in the local disk cache (under
the ``zstash/`` sub-directory), it must first be downloaded from HPSS before
the desired file can be extracted.

(See the 

 for more information).
the 

 sub-directory), it must first be downloaded from HPSS before

* Downloading a 256GB file on Cori/Edison takes about 30 mins (or more depending on load).
* Using NERSC data transfer nodes (DTN) may be about 3x faster, according to some users.
* Again, to see which of your files are in what tar archives, use ``zstash ls -l``.

  * Note the ``-l`` argument.
  * The sixth column is the tar archive that the file is in.
  * Please see the List documentation below for more information.

Downloading a 256GB file on Cori/Edison takes about 30 mins (or more depending on load).

Using NERSC data transfer nodes (DTN) may be about 3x faster, according to some users.

Again, to see which of your files are in what tar archives, use ``zstash ls -l``.

* Note the ``-l`` argument.
* The sixth column is the tar archive that the file is in.
* Please see the List documentation below for more information.

Again, to see which of your files are in what tar archives, use 

Extracting a single file by its full path ``archive/logs/atm.log.8229335.180130-143234.gz`` ::

$ zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison archive/logs/atm.log.8229335.180130-143234.gz
DEBUG: Opening index database
DEBUG: Running zstash extract
DEBUG: Local path : /global/cscratch1/sd/golaz/ACME_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
DEBUG: HPSS path  : /home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
DEBUG: Max size  : 274877906944
DEBUG: Keep local tar files  : False
INFO: Opening tar archive zstash/000018.tar
INFO: Extracting archive/logs/atm.log.8229335.180130-143234.gz
DEBUG: Valid md5: e8161bba53500848dc917258d1d8f56a archive/logs/atm.log.8229335.180130-143234.gz
DEBUG: Closing tar archive zstash/000018.tar
DEBUG: Closing index database

If the index database is already in the local disk cache (zstash/index.db), you can leave out the ``--hpss``
path. For example:

If the index database is already in the local disk cache (zstash/index.db), you can leave out the 

  zstash extract archive/logs/atm.log.8229335.180130-143234.gz

However, recall that wildcards are supported, so this full path isn't needed when using them.
Instead, you could download files matching ``"*atm.log.8229335.180130-143234.gz*"``. Note
the use of double quotes (") to avoid shell level substitution.

However, recall that wildcards are supported, so this full path isn’t needed when using them.
Instead, you could download files matching 

. Note
the use of double quotes (”) to avoid shell level substitution.

  zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison "*atm.log.8229335.180130-143234.gz*"
  # DEBUG: Opening index database
  # DEBUG: Running zstash extract
  # DEBUG: Local path : /global/cscratch1/sd/golaz/ACME_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
  # DEBUG: HPSS path  : /home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
  # DEBUG: Max size  : 274877906944
  # DEBUG: Keep local tar files  : False
  # INFO: Opening tar archive zstash/000018.tar
  # INFO: Extracting archive/logs/atm.log.8229335.180130-143234.gz
  # DEBUG: Valid md5: e8161bba53500848dc917258d1d8f56a archive/logs/atm.log.8229335.180130-143234.gz
  # DEBUG: Closing tar archive zstash/000018.tar
  # INFO: Opening tar archive zstash/000047.tar
  # INFO: Extracting case_scripts/logs/atm.log.8229335.180130-143234.gz
  # DEBUG: Valid md5: e8161bba53500848dc917258d1d8f56a case_scripts/logs/atm.log.8229335.180130-143234.gz
  # DEBUG: Closing tar archive zstash/000047.tar
  # DEBUG: Closing index database

zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison "*atm.log.8229335.180130-143234.gz*"
# DEBUG: Opening index database
# DEBUG: Running zstash extract
# DEBUG: Local path : /global/cscratch1/sd/golaz/ACME_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
# DEBUG: HPSS path  : /home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison
# DEBUG: Max size  : 274877906944
# DEBUG: Keep local tar files  : False
# INFO: Opening tar archive zstash/000018.tar
# INFO: Extracting archive/logs/atm.log.8229335.180130-143234.gz
# DEBUG: Valid md5: e8161bba53500848dc917258d1d8f56a archive/logs/atm.log.8229335.180130-143234.gz
# DEBUG: Closing tar archive zstash/000018.tar
# INFO: Opening tar archive zstash/000047.tar
# INFO: Extracting case_scripts/logs/atm.log.8229335.180130-143234.gz
# DEBUG: Valid md5: e8161bba53500848dc917258d1d8f56a case_scripts/logs/atm.log.8229335.180130-143234.gz
# DEBUG: Closing tar archive zstash/000047.tar
# DEBUG: Closing index database

In this particular example, the pattern matches two specific files, one under `archive/logs/`
and another one under `case_scripts/logs/`. If you didn't intend to retrieve both of them, a
more efficient approach would have been to first identify the desired files with 'zstash ls'.

In this particular example, the pattern matches two specific files, one under 

. If you didn’t intend to retrieve both of them, a
more efficient approach would have been to first identify the desired files with ‘zstash ls’.

Another example of wildcards would be to retrieve all **cam.h0** (monthly atmosphere output files)
between **years 0030 and 0069** for the DECKv1 piControl simulation. The zstash command would be:

 for the DECKv1 piControl simulation. The zstash command would be:

  zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison "*.cam.h0.00[3-6]?-??.nc"

You may specify the cache with the ``--cache`` option. Notice that there is no need to include
``--keep`` when not using HPSS

  zstash extract --hpss=none --cache=/p/user_pub/e3sm/archive/1_1/BGC-v1/20181217.BCRC_CNPCTC20TR_OIBGC.ne30_oECv3.edison "*cam.h3.1906-01-*-*.nc"

zstash extract --hpss=none --cache=/p/user_pub/e3sm/archive/1_1/BGC-v1/20181217.BCRC_CNPCTC20TR_OIBGC.ne30_oECv3.edison "*cam.h3.1906-01-*-*.nc"

To extract from the archive created with Globus in the ``zstash create`` example, you would run:

  zstash extract --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/~/test/E3SM_simulations/20170731.F20TR.ne30_ne30.anvil

Note: Most of the commands for this are the same for ``zstash extract`` and ``zstash check``.

  zstash ls --hpss=<path to HPSS> [-l] [--cache=<cache>] [--tars] [-v] [files]

zstash ls --hpss=<path to HPSS> [-l] [--cache=<cache>] [--tars] [-v] [files]

``[files]`` is a list of files to be listed (standard wildcards supported).

* Leave empty to list all the files.
* Names of specific tar archives to list all files within these tar archives.

 is a list of files to be listed (standard wildcards supported).

Names of specific tar archives to list all files within these tar archives.

  zstash ls -l --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison "*atm.log.8229335.180130-143234.gz*"
  # DEBUG: Running zstash ls
  # id        name    size    mtime   md5     tar     offset
  # 30482     archive/logs/atm.log.8229335.180130-143234.gz   20156521        2018-02-01 10:02:35     e8161bba53500848dc917258d1d8f56a        000018.tar      131697281536
  # 51608     case_scripts/logs/atm.log.8229335.180130-143234.gz      20156521        2018-02-01 10:02:52     e8161bba53500848dc917258d1d8f56a        000047.tar      202381473280

zstash ls -l --hpss=/home/g/golaz/2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison "*atm.log.8229335.180130-143234.gz*"
# DEBUG: Running zstash ls
# id        name    size    mtime   md5     tar     offset
# 30482     archive/logs/atm.log.8229335.180130-143234.gz   20156521        2018-02-01 10:02:35     e8161bba53500848dc917258d1d8f56a        000018.tar      131697281536
# 51608     case_scripts/logs/atm.log.8229335.180130-143234.gz      20156521        2018-02-01 10:02:52     e8161bba53500848dc917258d1d8f56a        000047.tar      202381473280

Below is an example of using ``ls`` to look at the tars in addition to the files:

  mkdir source_directory
  touch source_directory/file0.txt
  zstash create --hpss=hpss_archive source_directory
  # INFO: Gathering list of files to archive
  # INFO: Creating new tar archive 000000.tar
  # INFO: Archiving file0.txt
  # INFO: tar name=000000.tar, tar size=10240, tar md5=97d3e0ffaff4880251c77699d7438fe2
  # INFO: Transferring file to HPSS: zstash/000000.tar
  # INFO: Transferring file to HPSS: zstash/index.db

  zstash ls --hpss=hpss_archive --tars
  # INFO: Transferring file from HPSS: zstash/index.db
  # file0.txt

  # Tars:
  # 000000.tar

mkdir source_directory
touch source_directory/file0.txt
zstash create --hpss=hpss_archive source_directory
# INFO: Gathering list of files to archive
# INFO: Creating new tar archive 000000.tar
# INFO: Archiving file0.txt
# INFO: tar name=000000.tar, tar size=10240, tar md5=97d3e0ffaff4880251c77699d7438fe2
# INFO: Transferring file to HPSS: zstash/000000.tar
# INFO: Transferring file to HPSS: zstash/index.db

zstash ls --hpss=hpss_archive --tars
# INFO: Transferring file from HPSS: zstash/index.db
# file0.txt

# Tars:
# 000000.tar

Running ``zstash ls`` outside the source directory (the directory you're archiving)
is not advised. ``zstash`` will only retrieve ``index.db`` from the HPSS archive
if a local archive (cache) is not present.

 outside the source directory (the directory you’re archiving)
is not advised. 

 from the HPSS archive

  zstash create --hpss=hpss_archive source_directory           # Creates an HPSS archive named `hpss_archive` and a local archive (cache) `source_directory/zstash`.
  zstash ls --hpss=hpss_archive                                # List the contents of `hpss_archive` and creates a cache `zstash` at the same level of `source_directory`.
  # Add `source_directory/new_file.txt`
  zstash create --hpss=different_hpss_archive source_directory # Create a different HPSS archive of the source directory. This overwrites the local archive (cache) `source_directory/zstash`.
  zstash ls --hpss=different_hpss_archive                      # `new_file.txt` will NOT be shown. The existing cache `zstash` (same level as `source_directory`) is being used.
  rm -rf zstash                                                # Delete the cache. (You could instead change to another directory).
  zstash ls --hpss=different_hpss_archive                      # `new_file.txt` will be shown.

zstash create --hpss=hpss_archive source_directory           # Creates an HPSS archive named `hpss_archive` and a local archive (cache) `source_directory/zstash`.
zstash ls --hpss=hpss_archive                                # List the contents of `hpss_archive` and creates a cache `zstash` at the same level of `source_directory`.
# Add `source_directory/new_file.txt`
zstash create --hpss=different_hpss_archive source_directory # Create a different HPSS archive of the source directory. This overwrites the local archive (cache) `source_directory/zstash`.
zstash ls --hpss=different_hpss_archive                      # `new_file.txt` will NOT be shown. The existing cache `zstash` (same level as `source_directory`) is being used.
rm -rf zstash                                                # Delete the cache. (You could instead change to another directory).
zstash ls --hpss=different_hpss_archive                      # `new_file.txt` will be shown.

  zstash create --hpss=hpss_archive source_directory # Creates an HPSS archive named `hpss_archive` and a local archive (cache) `source_directory/zstash`.
  zstash ls --hpss=hpss_archive                      # List the contents of `hpss_archive` and creates a cache `zstash` at the same level of `source_directory`.
  cd source_directory
  zstash update --hpss=hpss_archive                  # Add `new_file.txt` to the HPSS archive. This updates the cache `zstash` (in `source_directory`).
  cd ..
  zstash ls --hpss=hpss_archive                      # `new_file.txt` will NOT be shown. The existing cache `zstash` (same level as `source_directory`) is being used.
  rm -rf zstash                                      # Delete the cache. (You could instead change to another directory).
  zstash ls --hpss=hpss_archive                      # `new_file.txt` will be shown.

zstash create --hpss=hpss_archive source_directory # Creates an HPSS archive named `hpss_archive` and a local archive (cache) `source_directory/zstash`.
zstash ls --hpss=hpss_archive                      # List the contents of `hpss_archive` and creates a cache `zstash` at the same level of `source_directory`.
cd source_directory
zstash update --hpss=hpss_archive                  # Add `new_file.txt` to the HPSS archive. This updates the cache `zstash` (in `source_directory`).
cd ..
zstash ls --hpss=hpss_archive                      # `new_file.txt` will NOT be shown. The existing cache `zstash` (same level as `source_directory`) is being used.
rm -rf zstash                                      # Delete the cache. (You could instead change to another directory).
zstash ls --hpss=hpss_archive                      # `new_file.txt` will be shown.

Example 3 -- changing the HPSS archive, running ``zstash ls`` from the source directory:

  cd source_directory                                          # This is the directory we are archiving.
  zstash ls --hpss=hpss_archive                                # List the contents of `hpss_archive` and uses the existing cache `zstash` (in `source_directory`).
  # Add `new_file.txt`
  cd ..
  zstash ls --hpss=different_hpss_archive                           # `new_file.txt` will be shown.

cd source_directory                                          # This is the directory we are archiving.
zstash ls --hpss=hpss_archive                                # List the contents of `hpss_archive` and uses the existing cache `zstash` (in `source_directory`).
# Add `new_file.txt`
cd ..
zstash ls --hpss=different_hpss_archive                           # `new_file.txt` will be shown.

Example 4 -- updating the HPSS archive, running ``zstash ls`` from the source directory:

  cd source_directory                                # This is the directory we are archiving.
  zstash ls --hpss=hpss_archive                      # List the contents of `hpss_archive` and uses the existing cache `zstash` (in `source_directory`).
  # Add new_file.txt

cd source_directory                                # This is the directory we are archiving.
zstash ls --hpss=hpss_archive                      # List the contents of `hpss_archive` and uses the existing cache `zstash` (in `source_directory`).
# Add new_file.txt

Starting with version 0.3, you can check the version of zstash from the command line: