Skip to content

MLE-31461: Expose zip archive read limits in Flux import commands #656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonmille wants to merge 6 commits into
base: develop
Choose a base branch
from
Open

MLE-31461: Expose zip archive read limits in Flux import commands #656

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
686df5e
MLE-31461: Expose zip archive read limits in Flux import commands
jonmille Jul 23, 2026
c1ba7ed
MLE-31461 Updated default values
jonmille Jul 23, 2026
1a993cb
MLE-31361 Refactor
jonmille Jul 23, 2026
506323f
MLE-31461 Added partitions JavaDoc
jonmille Jul 24, 2026
4583a96
MLE-31461 Changed --zip-max-entry-bytes to --zip-max-uncompressed-ent…
jonmille Jul 27, 2026
31d8713
MLE-31461 Updated Docs for new options
jonmille Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 28 additions & 8 deletions docs/import/import-files/aggregate-xml.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ grand_parent: Importing Data
nav_order: 5
---

Flux can split large XML files - called "aggregate XML files" - in the same fashion as
[MarkLogic Content Pump](https://docs.marklogic.com/11.0/guide/mlcp-guide/en/importing-content-into-marklogic-server/splitting-large-xml-files-into-multiple-documents.html).
Flux can split large XML files - called "aggregate XML files" - in the same fashion as
[MarkLogic Content Pump](https://docs.marklogic.com/11.0/guide/mlcp-guide/en/importing-content-into-marklogic-server/splitting-large-xml-files-into-multiple-documents.html).

## Table of contents
{: .no_toc .text-delta }
Expand All @@ -19,7 +19,7 @@ Flux can split large XML files - called "aggregate XML files" - in the same fash

The `import-aggregate-xml-files` command creates many XML documents based on a particular XML element name and
optional namespace. The command requires an instance of the `--element` option to identify the name of the XML element
in each file where each occurrence of that XML element will result in a separate document being written to MarkLogic.
in each file where each occurrence of that XML element will result in a separate document being written to MarkLogic.
The `--namespace` option must be used if that XML element has an associated namespace.

For example, consider the following notional XML file located at `/data/people.xml`:
Expand Down Expand Up @@ -68,9 +68,9 @@ bin\flux import-aggregate-xml-files ^

## Controlling document URIs

In addition to the options for controlling URIs described in the [common import features guide](../common-import-features.md),
In addition to the options for controlling URIs described in the [common import features guide](../common-import-features.md),
you can use the `--uri-element` and `--uri-namespace` options to identify an element in each XML document whose value should
be included in the URI. Using the example XML document in the above section, the following would construct a URI
be included in the URI. Using the example XML document in the above section, the following would construct a URI
based on the value of each `id` element in the `org:example` namespace:

{% tabs log %}
Expand Down Expand Up @@ -107,14 +107,14 @@ example, adding the following would result in URIs of `/person/1.xml` and `/pers

## Compressed XML files

Flux supports gzip and ZIP aggregate XML files. Simply include the `--compression` option with a value of `GZIP` or
Flux supports gzip and ZIP aggregate XML files. Simply include the `--compression` option with a value of `GZIP` or
`ZIP`.

## Specifying an encoding

MarkLogic stores all content
MarkLogic stores all content
[in the UTF-8 encoding](https://docs.marklogic.com/guide/search-dev/encodings_collations#id_87576).
If your aggregate XML files use a different encoding, you must specify that via the `--encoding` option so that
If your aggregate XML files use a different encoding, you must specify that via the `--encoding` option so that
the content can be correctly translated to UTF-8 when written to MarkLogic - e.g.:

{% tabs log %}
Expand All @@ -137,3 +137,23 @@ bin\flux import-aggregate-xml-files ^
```
{% endtab %}
{% endtabs %}

## Zip bomb protection

When importing compressed XML files from untrusted sources using `--compression ZIP`, you can protect against
[zip bombs](https://en.wikipedia.org/wiki/Zip_bomb) — archives crafted to expand to an enormous amount of data
and potentially exhaust the memory available to Flux. Both options below are disabled by default.

To limit the maximum number of uncompressed bytes read from any single ZIP entry:

--zip-max-uncompressed-entry-bytes 268435456

A value of `268435456` (256 MB) is a reasonable starting point for most use cases. If a single entry exceeds
this limit, an error is thrown. Set to `0` or any value less than `1` to disable.

To limit the maximum number of entries processed from a single ZIP file:

--zip-max-entry-count 100000

A value of `100000` is a reasonable starting point for most use cases. Set to `0` or any value less than `1` to
disable.
67 changes: 44 additions & 23 deletions docs/import/import-files/archives.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ grand_parent: Importing Data
nav_order: 7
---

Flux can import archive files containing documents and their associated metadata. This includes archives written via
the [`export-archive-files` command](../../export/export-archives.md) as well as archives written by
[MarkLogic Content Pump](https://docs.marklogic.com/11.0/guide/mlcp-guide/en/exporting-content-from-marklogic-server/exporting-to-an-archive.html),
Flux can import archive files containing documents and their associated metadata. This includes archives written via
the [`export-archive-files` command](../../export/export-archives.md) as well as archives written by
[MarkLogic Content Pump](https://docs.marklogic.com/11.0/guide/mlcp-guide/en/exporting-content-from-marklogic-server/exporting-to-an-archive.html),
which are hereafter referred to as "MLCP archives".

## Table of contents
Expand All @@ -19,7 +19,7 @@ which are hereafter referred to as "MLCP archives".

## Usage

The `import-archive-files` command will import the documents and metadata files in a ZIP file produced by the
The `import-archive-files` command will import the documents and metadata files in a ZIP file produced by the
`export-archive-files` command. You must specify at least one `--path` option along with connection information for the
MarkLogic database you wish to write to:

Expand All @@ -44,7 +44,7 @@ bin\flux import-archive-files ^

## Importing MLCP archives

You can also import
You can also import
[MLCP archives](https://docs.marklogic.com/11.0/guide/mlcp-guide/en/exporting-content-from-marklogic-server/exporting-to-an-archive.html)
that were produced via the `EXPORT` command in MLCP. The `import-mlcp-archive-files` command is used instead, and it also
requires at least one `--path` option along with connection information for the MarkLogic database you wish to write to:
Expand All @@ -71,9 +71,9 @@ bin\flux import-mlcp-archive-files ^

## Restricting metadata

By default, all metadata associated with a document in an archive will be included when the document is written to MarkLogic.
This is true for both the `import-archive-files` command and the `import-mlcp-archive-files` command. This is typically
desirable so that metadata like collections and permissions in the archive can be applied to the imported documents.
By default, all metadata associated with a document in an archive will be included when the document is written to MarkLogic.
This is true for both the `import-archive-files` command and the `import-mlcp-archive-files` command. This is typically
desirable so that metadata like collections and permissions in the archive can be applied to the imported documents.

You can instead restrict which types of metadata are included via the `--categories` option. This option accepts a comma-delimited
sequence of the following metadata types:
Expand All @@ -84,7 +84,7 @@ sequence of the following metadata types:
- `properties`
- `metadatavalues`

For example, the following option will only include the collections and properties found in each metadata entry in an
For example, the following option will only include the collections and properties found in each metadata entry in an
archive ZIP file or MLCP archive ZIP file:

--categories collections,properties
Expand Down Expand Up @@ -118,7 +118,7 @@ bin\flux import-archive-files ^

## Importing large binary files in archives

When [exporting archives](../../export/export-archives.md), you can use the `--streaming` option introduced in Flux
When [exporting archives](../../export/export-archives.md), you can use the `--streaming` option introduced in Flux
1.1.0 to ensure that large binary documents in MarkLogic can be streamed to an archive file. When importing archives
with large binary files, you should likewise use the `--streaming` option to ensure that each large binary can be read
into MarkLogic without exhausting the memory available to Flux or MarkLogic.
Expand All @@ -140,16 +140,16 @@ large binary document may exhaust the amount of memory available to MarkLogic.
In addition, when streaming documents to MarkLogic, URIs will be encoded. For example, an entry named `/my file.json`
will result in a URI of `/my%20file.json`. This is due to an
[issue in the MarkLogic REST API endpoint](https://docs.marklogic.com/REST/PUT/v1/documents) that will be resolved in
a future server release.
a future server release.

### Applying transforms selectively when streaming

When streaming archive entries to MarkLogic, you may need to apply a REST transform to certain documents but not
others. One reason for this is when an archive contains files that appear to be JSON, XML, or text based on their URI
When streaming archive entries to MarkLogic, you may need to apply a REST transform to certain documents but not
others. One reason for this is when an archive contains files that appear to be JSON, XML, or text based on their URI
extensions (such as `.json`, `.xml`, or `.txt`), but are stored in the archive with a binary format. As noted
in the [MarkLogic REST API documentation](https://docs.progress.com/bundle/marklogic-server-develop-rest-api-12/page/topics/intro.html#id_53367),
loading such documents as binaries requires the use of a REST transform that converts each document into a binary. This
approach prevents MarkLogic from automatically setting the document type based on the URI extension. The following
approach prevents MarkLogic from automatically setting the document type based on the URI extension. The following
REST transform is one example of how a JSON or XML document can be converted to a binary:

```
Expand All @@ -166,10 +166,10 @@ declare function transform($context as map:map, $params as map:map, $content as
};
```

The `--streaming-transform-binary-with-extension` option, introduced in Flux 2.1, allows you to specify which
documents should be sent to your transform during streaming. This allows you to stream as many documents as possible,
with only the documents that need to be converted into binaries being sent to the transform. The option accepts a
comma-delimited list of URI extensions. The transform specified via `--transform` will then only be applied if
The `--streaming-transform-binary-with-extension` option, introduced in Flux 2.1, allows you to specify which
documents should be sent to your transform during streaming. This allows you to stream as many documents as possible,
with only the documents that need to be converted into binaries being sent to the transform. The option accepts a
comma-delimited list of URI extensions. The transform specified via `--transform` will then only be applied if
both of the following conditions are met for a document in the archive:

1. The document format in the archive is `BINARY` (requires an archive created with Flux 2.1 or later).
Expand All @@ -185,14 +185,14 @@ For example, consider an archive containing:
The following set of options will result in the transform only being applied to `report.json` and `document.xml`:

```
--streaming
--streaming
--transform my-transform
--streaming-transform-binary-with-extension json,xml
```

This option only has an effect when `--streaming` and `--transform` are specified. Note that if `--streaming`
and `--transform` are specified without this option, then every document will be sent to the transform. This is
typically not desirable as it results in each document being read into memory in MarkLogic, which defeats some of the
and `--transform` are specified without this option, then every document will be sent to the transform. This is
typically not desirable as it results in each document being read into memory in MarkLogic, which defeats some of the
purpose of streaming the data into MarkLogic.

## Common errors
Expand All @@ -204,5 +204,26 @@ following message:
com.marklogic.spark: Could not find metadata entry for entry
```

To solve this, you should use Flux 1.1.0 or higher to import the archive. Flux 1.1.0 and higher can also import
archives created by Flux 1.0.x.
To solve this, you should use Flux 1.1.0 or higher to import the archive. Flux 1.1.0 and higher can also import
archives created by Flux 1.0.x.

## Zip bomb protection

When importing archive files from untrusted sources, you can protect against
[zip bombs](https://en.wikipedia.org/wiki/Zip_bomb) — archives crafted to expand to an enormous amount of data
and potentially exhaust the memory available to Flux. Both options below are disabled by default and apply to
both the `import-archive-files` and `import-mlcp-archive-files` commands.

To limit the maximum number of uncompressed bytes read from any single archive entry:

--zip-max-uncompressed-entry-bytes 268435456

A value of `268435456` (256 MB) is a reasonable starting point for most use cases. If a single entry exceeds
this limit, an error is thrown. Set to `0` or any value less than `1` to disable.

To limit the maximum number of entries processed from a single archive file:

--zip-max-entry-count 100000

A value of `100000` is a reasonable starting point for most use cases. Set to `0` or any value less than `1` to
disable.
Loading
Loading