Edit

Share via

Create an export job to export data from Azure Managed Lustre

  • Article

In this article, you learn how to copy data from your Azure Managed Lustre file system to long-term storage in Azure Blob Storage by creating an export job.

Export jobs are only available when you integrate Azure Blob Storage with your Azure Managed Lustre file system during file system creation. For more information, see Azure Blob Storage integration. If you didn't integrate a blob container when you created the file system, use client filesystem commands to copy the data without creating an export job.

Which files are exported during an export job?

When you export files from your Azure Managed Lustre system, not all of the files are copied into the blob container that you specified when you created the file system. The following rules apply to export jobs:

  • Export jobs only copy files that are new or whose contents are modified. If the file that you imported from the blob container during file system creation is unchanged, the export job doesn't export the file.
  • Files with metadata changes only aren't exported. Metadata changes include: owner, permissions, extended attributes, and name changes (renamed).
  • Files deleted in the Azure Managed Lustre file system aren't deleted in the original blob container during the export job. The export job doesn't delete files in the blob container.

Exporting data is a user-initiated process that you can do in the Azure portal or by using commands in the native Lustre client CLI. With both methods, you can monitor the state of the export job.

The following procedures tell how to:

Create an export job in the Azure portal

To create an export job to export changed data from an Azure Managed Lustre file system in the Azure portal, do the following steps:

  1. In the Azure portal, open your Azure Managed Lustre file system and navigate to the Blob integration pane under Settings.

  2. Select + Create new job.

  3. Specify what to export in the export job by entering a value in File system path.

    • All new or changed files whose filenames begin with this string in the Azure Managed Lustre file system are exported.
    • Files are written to the blob container with the same file path (or prefix) that they have in the Lustre system. If you want to avoid overwriting existing files in the blob container, make sure the path of the file in your Lustre system doesn't overlap the existing path of the file in the blob container.

The following screenshot shows the export job configuration settings in the Azure portal:

Screenshot showing portal setup for creating an export (archive) job.

Monitor or cancel an export job in the Azure portal

You can monitor or cancel export jobs you created through blob integration with your Azure Managed Lustre file system in the Azure portal. The Recent jobs section of the Blob integration page shows the status of each job.

Only one archive job runs at a time. To cancel the job that's in progress, select the Cancel link for that job in the Recent jobs table. The Cancel link isn't available for a completed job.

Create export job using the native Lustre client CLI

To create an export job to export changed data from an Azure Managed Lustre file system using native Lustre client CLI command, use one of the following commands:

  • To export a single file, run a command similar to this one:
sudo lfs hsm_archive path/to/export/file
  • To export all files in a directory, run a command similar to this one:
nohup find local/directory -type f -print0 | xargs -0 -n 1 sudo lfs hsm_archive &

Monitor export (archive) state using the native Lustre client CLI

To check on the status of an export job using the native Lustre client CLI, run the following command:

find path/to/export/file -type f -print0 | xargs -0 -n 1 -P 8 sudo lfs hsm_action | grep "ARCHIVE" | wc -l

Each file has an associated state, which indicates the relationship between the file data in the Lustre file system and the file data in Azure Blob Storage. To check the state of a file, run this command:

sudo lfs hsm_state path/to/export/file

The state command reports the state of changes to the file. The following table shows the possible file states:

State Description
(0x0000000d) released exists archived The file's contents (the data) exist in Blob Storage only. Only the metadata exists in Lustre. An export (archive) job doesn't update (overwrite) the file in Blob Storage.
(0x00000009) exists archived An export (archive) job doesn't export the file to Blob Storage because Blob Storage already has the latest copy.
(0x0000000b) exists dirty archived The file has changes that aren't archived. To send the changes in Lustre back to Blob Storage, run an export job. The export job overwrites the file in Blob Storage.
(0x00000000) The file is new and only exists in the Lustre file system. an export job creates a new file in the blob container. If the file is updated again in Lustre, run another export job to copy those changes to Blob Storage.
(0x00000001) exists The file is new and only exists in the Lustre file system. An export job has been started and hasn't completed for this file.

Next steps