diff --git a/.vscode/settings.json b/.vscode/settings.json index 568dbead8..143a61a0c 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -3,8 +3,15 @@ "*.md": "markdoc" }, "cSpell.words": [ + "cuno", "Gbps", "markdoc", - "Storj" - ] + "mdash", + "posix", + "Storj", + "Virtio", + "virtiofs" + ], + "editor.tabSize": 2, + "editor.detectIndentation": false } diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 02a3532b1..1ccdfb7b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -36,7 +36,12 @@ The front matter is a section at the beginning of each Markdown file (page.md) t **title:** The title of the article or page. This is typically displayed at the top of the page. -**docId:** A unique identifier for the document. This can be used for internal tracking and linking. See [Internal Linking](/CONTRIBUTING.md#internal-linking). If you're making a new page, generate a new unique ID with `pwgen -1 16` or something similar. +**docId:** A unique identifier for the document. This can be used for internal tracking and linking. See [Internal Linking](/CONTRIBUTING.md#internal-linking). If you're making a new page, generate a new unique ID with `pwgen -1 16` or something similar. + +`docid` Recommendations: +- Online password generators such as [https://pwgen.io/](https://pwgen.io/) work well. +- A length of 16 characters is preferred +- Avoid special characters or symbols. Use only UPPER, lower and numbers (0-9). **redirects:** A list of URLs that should redirect to this page. This is useful for maintaining links when a page's URL changes or for creating aliases for a page. diff --git a/app/(docs)/object-mount/appendix/POSIX/page.md b/app/(docs)/object-mount/appendix/POSIX/page.md new file mode 100644 index 000000000..1e31f3f59 --- /dev/null +++ b/app/(docs)/object-mount/appendix/POSIX/page.md @@ -0,0 +1,128 @@ +--- +title: POSIX Explained +hideTitle: false +docId: ySneAEd79CVewSSr +weight: 2 +redirects: + - /object-mount/faq/posix-mode +metadata: + title: POSIX Explained + description: + Understanding POSIX and using Object Mount in POSIX Mode. +hidden: false +--- + +## What is POSIX? + +The Portable Operating System Interface (🌐 [POSIX](https://en.wikipedia.org/wiki/POSIX)) is a set of IEEE specifications intended to encourage compatibility between disparate Linux/UNIX flavors through standardized definitions of APIs, shells, interfaces, commands, etc. + +Writing an application (or binary) which meets POSIX standards means that the program is “portable,” and is more likely to run correctly no matter what UNIX-based OS it is installed on. + +Some applications may (or may not) require _strict_ POSIX-compliance to work properly. + + +## Object Mount’s POSIX Mode + +To support those programs and applications that require POSIX-compliant interfaces, access-control, and metadata, Object Mount includes **POSIX Mode**. + +When activated, POSIX Mode enables the necessary additional features to support applications that rely on the rigid, traditional UNIX-style file permissions and metadata. + +Object storage, by default, lacks concepts like file ownership, permissions, and modification timestamps that are normal features of most local drives and file system calls. Object Mount’s POSIX Mode recreates these features and behaviors in the object storage world. + +When enabled, POSIX Mode generates, stores, and maintains the metadata that is typically expected in a local file system, including: +- File ownership (uid, gid) +- Read/Write/Execute permissions +- Timestamps (mtime, ctime, etc.) +- Symbolic links and directory flags (where supported) + +To retain this metadata, Object Mount creates and writes to a hidden index file that is stored at the root of your object storage bucket. + +{% callout type="info" %} + **POSIX Mode Requirements** + + To support POSIX Mode, ensure your S3 credentials allow, and your bucket supports, write access. Your credentials must allow `s3:PutObject` and `s3:DeleteObject`. +{% /callout %} + +{% callout type="warning" %} + **Accessing Content Outside of Object Mount** + + Using any non-Object Mount tool to rename, move, or copy your files will result in those objects _losing_ their POSIX metadata. You should only use Object Mount to manage POSIX-enabled files in order to preserve their metadata and attributes. +{% /callout %} + + +## When to Enable Object Mount’s POSIX Mode + +You should enable POSIX mode in Object Mount if: + +- You are working with software that checks for or enforces POSIX-style permissions +- You require symbolic link emulation or fine-grained permission mapping +- You are mounting object storage as a shared filesystem in team environments + +Enabling POSIX mode is recommended for workflows such as: + +- Media Production +- Backup Solutions +- Archive Systems +- File Synchronization +- File Manager/Tools + + +## When NOT to Enable Object Mount’s POSIX Mode + +You should _not_ enable POSIX Mode in Object Mount for: + +- **Read-only Bucket Credentials** + + POSIX Mode requires _write_ access to the object storage bucket in order to create and write to the hidden metadata file. If your credentials are read-only, POSIX mode will not function and may prevent the mount from working correctly. + +- **Lightweight Access** + + For read-only workflows or general content browsing, POSIX mode is typically unnecessary and can be left disabled. + + +## Enabling POSIX Mode: macOS and Windows + +POSIX mode is enabled on a mount-by-mount basis: You can choose which mounts require it and which don’t. + +You enable POSIX mode when **creating a new mount** within Object Mount for macOS or Windows. + +{% callout type="warning" %} + **Important:** You cannot change the POSIX Mode setting after a mount has been created in Object Mount for macOS or Windows. +{% /callout %} + +Enabling POSIX mode in **macOS** and **Windows** environments is done through Object Mount’s graphical user interface (GUI). + +See the associated page for your operating system: + +- Enabling POSIX Mode when [creating a mount in macOS](docId:QpBba8p4bMTXAkBK#step-2-configure-s3-credentials-and-create-a-mount) +- Enabling POSIX Mode [creating a mount in Windows](docId:khHGfZsyY9NJ2uGK#step-2-configure-s3-credentials-and-create-a-mount) + +{% callout type="info" %} + **POSIX Feature Support on Mac & Windows** + + User and Group identity features are not supported on Mac and Windows platforms: + - The owner of cloud objects is always reported to be the current user. + - The directory mode is reported as `0777` and the file mode is reported as `0666`. + + Also, POSIX access controls are not enforced by Object Mount on Mac and Windows. + + For a deeper understanding of the full suite of POSIX Mode Features see the Linux Getting Started Guide article: [](docId:cbm3PcQXmLpuYcbg) and the Linux User Guide article: [](docId:Eegoo1teiJ8eerae). +{% /callout %} + + +## Enabling POSIX Mode: Linux + +POSIX support in Object Mount for Linux includes more options, more modes, and offers more controls for the end user to configure their environment according to their unique needs and use cases. + +Configuration of POSIX options in Object Mount for Linux is achieved: + - Via the command line interface (the Object Mount CLI) + - Through the setting of environment variables + - Applying “tags” directly to cloud buckets using your object storage provider’s dashboard + +Use the links below to **learn** about the many POSIX Modes, options, and settings — and then **enable** the correct POSIX Mode for you use case: + +| **Linux Next Steps** | **Description** | +|----------------------|-----------------| +| **Learn about** [POSIX Options](docId:cbm3PcQXmLpuYcbg) | Understanding POSIX Modes in Object Mount for Linux | +| **Enable** [POSIX File Access](docId:cbm3PcQXmLpuYcbg#posix-file-access) | Enabling POSIX File Access Mode in Object Mount for Linux | +| **Enable** [Enforced POSIX File Access](docId:Eegoo1teiJ8eerae) | Enabling _Enforced_ POSIX File Access Mode in Object Mount for Linux | diff --git a/app/(docs)/object-mount/appendix/_meta.json b/app/(docs)/object-mount/appendix/_meta.json new file mode 100644 index 000000000..9e28b1342 --- /dev/null +++ b/app/(docs)/object-mount/appendix/_meta.json @@ -0,0 +1 @@ +{"title": "Appendix", "weight": 10} \ No newline at end of file diff --git a/app/(docs)/object-mount/appendix/core-concepts/page.md b/app/(docs)/object-mount/appendix/core-concepts/page.md new file mode 100644 index 000000000..be900577e --- /dev/null +++ b/app/(docs)/object-mount/appendix/core-concepts/page.md @@ -0,0 +1,59 @@ +--- +title: Core Concepts +hideTitle: false +docId: xzAqS9Um7xjQSbmD +weight: 1 +redirects: + - /object-mount/linux/user-guides/core-concepts +metadata: + title: Core Concepts + description: + Core concepts of Object Mount. +hidden: false +--- + +## Characteristics of Object Storage + +Object storage excels at high-throughput use-cases and is often considered best for “write-once, read-many” models. + +Direct file editing with object storage is traditionally seen as too slow and often requires gateways, proprietary translations, and each file often needs to be completely rewritten — even for the slightest modification. + +Additionally, object storage technology differs significantly from the conventional block-and-file I/O calls used by local drives and network-attached solutions — especially for applications that expect strict POSIX compliance. + +Storj’s Object Mount solves these shortcomings, redirecting and translating file access calls, simplifying cloud storage implementations, and delivering the highest possible performance from your object storage. POSIX compliance and unique optimizations ensure compatibility and improve performance enabling fast editing and file access directly from the cloud. + + +## Direct Use of Object Storage + +Object Mount enables the use of object storage as “hot storage” or as “first-class” storage. + +Prior to Object Mount, content workflows required you to “pre-stage” your files by copying content from the cloud down to local, faster drives. You would then run your tasks locally. When finished, you re-uploaded the resulting content back up to object storage to share with your team. + +Object Mount’s elegant simplicity and improved performance means you can now run your tasks _directly_ on your files in object storage — just mount and go. + +Object Mount overcomes the traditional inefficiencies of direct access to object storage by using prediction engines, partial writes, client-side write-caching and other optimizations to achieve much higher performance. Use your cloud storage like it was local storage. + + +## Read/Write Interception + +Once installed, Object Mount transparently inserts itself between your applications and your host operating system. Object Mount dynamically detects and intercepts relevant file and system calls made by your application and redirects those requests to your object storage provider. This is called **Direct Interception**. + +Direct Interception also facilitates the translation of your application’s standard file system calls (reads and writes) into the appropriate object storage API calls. + +This enables _any_ application to instantly access cloud objects as if they were local files — no code changes or workflow alterations required. Object Mount intercepts file commands from _any_ application, binary or EXE, including those you’ve written yourself. It even works inside unprivileged containerized environments like Docker and virtual environments such as Hyper-V and Virtualbox. + + +## What Object Mount Does + +- Provides applications high-speed access to object storage content as if it were a local drive +- Allows standard file system operations to list, copy, move, add, remove, and modify files directly on object storage +- Supports rich POSIX metadata including: users, groups, permissions, symbolic & hard links +- Preserves NFS-equivalent POSIX consistency guarantees + + +## What Object Mount Does NOT Do + +- Object Mount does _not_ interfere with how applications interact with local drives and any other non-S3 object data +- Object Mount does _not_ alter or change the representation of your files on object storage — file data is not modified. This means your content can be accessed directly from your object storage using standard S3 tools and dashboards, bypassing Object Mount at any time. + + **Note:** Content accessed _directly_ on your object storage (rather than through Object Mount) bypasses Object Mount’s consistency guarantees. diff --git a/app/(docs)/object-mount/appendix/faq/page.md b/app/(docs)/object-mount/appendix/faq/page.md new file mode 100644 index 000000000..4048dc55d --- /dev/null +++ b/app/(docs)/object-mount/appendix/faq/page.md @@ -0,0 +1,47 @@ +--- +title: FAQ +hideTitle: false +docId: dd27e33c-7804-4e15-bdc5-941f9a1a7e12 +weight: 10 +redirects: + - /object-mount/faq +metadata: + title: Frequently Asked Questions + description: + Resource on various FAQs on topics related to Object Mount. +hidden: false +--- + +## What S3 permissions are recommended to mount a bucket with Object Mount? +Short answer, for a generic cloud: read, write, and list permissions for the bucket intended to be mounted. + +## Are full permissions required to achieve maximum performance of Object Mount? +The maximum performance comes from Object Mount’s ability to prefetch data based on how it handle certain file types. It is not dependent on being able to write the POSIX-metadata file. With read-only permissions to a bucket, Object Mount will still achieve maximum performance — but it will not be able to update any POSIX-related metadata associated with the original files. + +## Why don’t read-only credentials work when POSIX is enabled in Object Mount? +The POSIX mode needs to write a hidden file at the root of the bucket that tracks all the POSIX permissions, etc. This is why the read-only use case works _so long as_ you mount the bucket without POSIX enabled. + +## How can I mount all buckets with Object Mount? +In order for Object Mount to “see” the buckets to be mounted when adding new credentials into Object Mount, the AWS equivalent of `ListAllMyBuckets` is required. In the absence of `ListAllMyBuckets`, you can specify what bucket to mount by manually pairing the bucket. + +## Where can I find the Object Mount logs? +For Windows, logs are available at `C:\Users\%username%\AppData\Local\Object Mount\cunofs.log`. + +For macOS the logs are located at `/Users/$USER/Library/Application Support/Object Mount/cunoFS.log`. + +The default logging level is “error”, so changing this to “access” or “trace” will provide more verbose logs to better understand what is going on. + +## Where are the configs and license files located? +For Windows: + - License key information is located along side the logs: `C:\Users\%username%\AppData\Local\Object Mount\license`. + - The `config.json` file is in the same directory where the mount/bucket information is kept + +For macOS: + - License key information is located along side the logs: `/Users/$USER/Library/Application Support/Object Mount/license`. + - The `config.json` file is in the same directory where the mount/bucket information is kept + +## I have multiple Mounts added but I can pnly activate one at a time? + + - Object Mount for both Windows and Mac can only have **one active** mount at a time. + - Activating a second mount, will automatically de-activate the previously active mount. + \ No newline at end of file diff --git a/app/(docs)/object-mount/appendix/release-notes/page.md b/app/(docs)/object-mount/appendix/release-notes/page.md new file mode 100644 index 000000000..e9fdb3182 --- /dev/null +++ b/app/(docs)/object-mount/appendix/release-notes/page.md @@ -0,0 +1,15 @@ +--- +title: Release Notes +hideTitle: false +docId: ScTuUGKGH5DHVqZF +weight: 11 +redirects: + - /object-mount/release-notes +metadata: + title: Object Mount Release Notes + description: + Access to Object Mount Release Notes. +hidden: false +--- + +Release notes for Object Mount can be found on our [GitHub page](https://github.com/storj/object-mount/releases). diff --git a/app/(docs)/object-mount/appendix/s3-compatibility/page.md b/app/(docs)/object-mount/appendix/s3-compatibility/page.md new file mode 100644 index 000000000..236181903 --- /dev/null +++ b/app/(docs)/object-mount/appendix/s3-compatibility/page.md @@ -0,0 +1,46 @@ +--- +title: Supported S3 Providers +hideTitle: false +docId: xCjeasn8SLQn2vfd +weight: 4 +metadata: + title: Supported S3 Providers + description: + List of tested and supported s3 Providers. +hidden: false +--- + +Object Mount can be used with any S3-compatible object storage provider, including these **Primary S3 Providers**: + + - 🌐 [Storj’s Object Storage](https://www.storj.io/cloud-object-storage) + - 🌐 [Amazon Web Services S3](https://aws.amazon.com/s3/) + - 🌐 [Microsoft Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) + - 🌐 [Google Cloud Platform](https://cloud.google.com/storage/) + - **Note:** For best GCP performance, we recommend using S3 with their 🌐 [S3 Access Point](https://cloud.google.com/storage/docs/interoperability), _not_ with their S3 Gateway. + + +## Tested Storage Providers + +In theory, Storj’s Object Mount works with _any_ S3-compatible object storage provider. + +In practice, each provider’s “S3-compatible API” implementation can have slightly different nuances and behaviors — some additional configuration may be necessary. + +Object Mount has been successfully tested with the **Compatible S3 Providers** below: + + - 🌐 [Oracle Cloud Infrastructure (OCI) Object Storage](https://www.oracle.com/cloud/storage/object-storage) + - 🌐 [Wasabi Cloud Object Storage](https://wasabi.com/cloud-object-storage) + - 🌐 [Min.IO AIStor](https://www.min.io/product/aistor) + - 🌐 [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid) + - 🌐 [Dell ECS Object Storage](https://www.dell.com/en-us/lp/dt/elastic-cloud-storage) + + +## Additional S3 Providers + +The following S3 storage providers have not yet been validated by Storj — however, users have reported success with the **Additional S3 Providers** below: + + - 🌐 [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage) + - 🌐 [Backblaze B2 Cloud Storage](https://www.backblaze.com/cloud-storage) + - 🌐 [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces) + - 🌐 [Cloudflare R2](https://www.cloudflare.com/en-gb/developer-platform/r2) + - 🌐 [Scality](https://www.scality.com) + - 🌐 [DataDirect Networks (DDN) Storage](https://www.ddn.com) diff --git a/app/(docs)/object-mount/appendix/s3-credentials/page.md b/app/(docs)/object-mount/appendix/s3-credentials/page.md new file mode 100644 index 000000000..922b25e08 --- /dev/null +++ b/app/(docs)/object-mount/appendix/s3-credentials/page.md @@ -0,0 +1,320 @@ +--- +title: S3 Credential Management +hideTitle: false +docId: E4NhE5kPdjURRajJ +weight: 3 +metadata: + title: Credentials & Authentication + description: + Understanding how to obtain credentials and use them to authenticate and mount object buckets. +hidden: false +--- + +Object Mount can connect to Storj’s Distributed Cloud Storage offering as well as other major S3-compatible cloud storage providers. + +You will need to create and/or obtain credentials from your S3 provider, and then configure Object Mount to use those credentials. + +**Note:** API access may also need to be enabled for some 3rd-party S3 providers. Consult your storage solution’s documentation to learn how to enable API access. + + +## Obtaining Your S3 Credentials + +You will need to obtain the following three items from your storage provider: + +| **Variable** | **Description** | +|-----------------------|-----------------| +| **Access Key ID** | Example: j4SQYkDxxxxxxxxxxsaSRYAahZgh +| **Access Key Secret** | Example: vMxJBmUxxxxxxxxxxxxxxxxxxxxxxxxxK4kKvwJnw8PwKpj4PmNvL +| **Endpoint** | An accessible IP address or domain/subdomain/host to use as an endpoint. Example: https://gateway.storjshare.io + +{% callout type="warning" %} +**Credential Requirements** + +The credentials you obtain need to have sufficient permissions in order for Object Mount to discover, read, write, and manage your data, including the permission to list buckets. + +If this is not possible (or desired), you can use `cuno creds pair` options. See the **Pairing Containers and Credentials** section in the Linux Advanced Configuration Guide article: [Advanced Credential Options](docId:aish4shuiheeZaig#pairing-containers-and-credentials). +{% /callout %} + +For instructions on how to create and/or obtain your S3 storage credentials, select the tab below that corresponds to your S3 provider: + +{% tabs %} +{% tab label="AWS S3" %} + **Amazon Web Services** + + **AWS EC2 with IAM:** + + If you are accessing S3 buckets through an Amazon AWS EC2 instance configured with an IAM role, no further configuration is needed and Object Mount will automatically authenticate using the AWS-managed configuration. + + **Using Existing Credentials:** + + You will need the Access Key ID and Access Key Secret for an AWS **IAM user** with permission to access your S3 buckets. These credentials would need to have been saved when the IAM user was _first created_. + + **Creating New Credentials:** + + Alternatively, create a new **IAM user** with “programmatic access” (access using keys), by following the AWS User Guide: 🌐 [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html). + + The steps should be similar to the following: + + 1. Log into the AWS S3 Console: 🌐 + + 2. Navigate to: **Services > IAM** + + 3. Open **Users** and click on the **Add user** button + + 1. Set a username + 2. Select the **Programmatic access** option + 3. Click on **Next** + + 4. On the **Set Permissions** page: + + 1. Select **Attach existing policies directly** + 2. Select an existing policy, e.g.: **AmazonS3ReadOnlyAccess** or **AmazonS3FullAccess** + 3. or Create a new policy that has the **s3:ListAllMyBuckets** permission and at least the **s3:ListBucket** and **s3:GetObject** permissions for each bucket available to the user + 4. Click **Next** + + 5. Review and confirm to create the new user + + 6. Download the CSV file by pressing the corresponding button +{% /tab %} + +{% tab label="Storj" %} + **Storj Object Storage** + + **Creating New Storj Credentials:** + + To create new Storj credentials to be used by Object Mount, follow these steps: + + 1. Log into the **Storj Dashboard** for your Satellite. + + For example: 🌐 + + 2. Open your **Project** and click on **Access Keys** in the left-hand menu. + + 3. Click the **New Access Key** button. + + 4. Enter a **Name**, select **S3 Credentials** and choose **Storj Object Mount** from the Application drop-down menu. Click Next. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/storj-create-new-access-keys-01.jpg) + + 5. Choose **Full Access** and click Next. + + 6. Click **Create Access**. + + 7. Click the **Copy All** button (or selectively click **Show** / **Copy** for each value). + + **Important:** These values will _only appear once_. Copy them now! + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/storj-create-new-access-keys-02.jpg) + + 8. (Optional) Click **Download All** to save the credentials to a text file. + + **Note:** This saved text file is _not_ in the correct format for importing into Object Mount for Linux. See below for details on creating a credential file with the correct syntax. +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + You will need the account name and the associated access key. No region or endpoint is required. + + To set up a new account, see 🌐 [Create an Azure storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create). + + If you already have an account and need the access key, see 🌐 [View account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys). + + The steps should be similar to: + + 1. Log into the Azure Portal: 🌐 + 2. From **services**, select **Storage Accounts** + 3. Click on the Storage Account you want the key for + 4. In the **Security + networking** section, click **Access keys** + 5. Click on the **Show keys** button + 6. Save the **Storage account name** and **Key** + + Alternatively, Object Mount for Linux supports using Shared Access Signatures (SAS) to access Azure containers. See the **Microsoft Azure Storage with Shared Access Signatures** section in the Linux Advanced Configuration Guide article: [Advanced Credential Options](docId:aish4shuiheeZaig#microsoft-azure-storage-with-shared-access-signatures). +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud Platform** + + You will need key-based access to be associated with your 🌐 [Google Cloud service account](https://developers.google.com/workspace/guides/create-credentials#service-account). + + If you need to set this up, you can have Google generate a new key by following Google’s guide at 🌐 [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete). + + The steps should be similar to: + + 1. Log into the GCS portal: 🌐 + 2. Navigate to the **Service Accounts** page + 3. Select the **project** you want to provide access to + 4. Click the **email address** of the service account that you want to create a key for + 5. Click the **Keys** tab + 6. Click **Add key** from the drop-down menu, then select **Create new key** + 7. Select **JSON** as the **Key type** and click **Create** + 8. Save the downloaded JSON file. + + Alternatively, you can generate your own key pair and upload it on the service account’s key management page, by following this guide 🌐 [Upload service account keys](https://cloud.google.com/iam/docs/keys-upload). +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **S3 Compatible Providers** + + Consult your object storage solution’s documentation to find out how to retrieve credentials for authentication using S3 APIs. + + You will need an **Access Key ID**, an **Access Key Secret**, and an accessible IP address/URL or domain/subdomain/host to use as an **endpoint**. +{% /tab %} + +{% /tabs %} + + + + + + + + + + + + + + +## Using & Saving Your S3 Credentials + +Depending on your S3 storage platform, you may only be able to see, copy, and save your access credentials _at the time you create them_. (For increased security, some platforms prevent the display of previously generated credentials.) + +Therefore, be sure to copy and record your **Access Key ID** and **Access Key Secret** at the time you _create them_. + +### Entering Credentials: Object Mount for macOS and Windows + +When activating Object Mount in **macOS** and **Windows**, you will need to copy/paste these keys with Object Mounts’s graphical interface. + +### Entering Credentials: Object Mount for Linux + +When activating Object Mount for Linux, you can copy/paste your your credentials, or you can import them from a previously saved text file. + +See below for instructions (and the syntax required) to save you credentials to a text file to be used for import. + +### Saving Credentials as a File + +A text file can be used to import your S3 credentials when activating a mount in Object Mount for Linux. + +Credential files are `plain-text` and must comply with the proper format defined for your specific cloud provider. + +Select your S3 storage provider below for details on syntax and variable naming: + +{% tabs %} + +{% tab label="AWS S3" %} + **Amazon Web Services** + + There are two equivalent formats for AWS S3 credential files. Use either one: + + ``` + aws_access_key_id = + aws_secret_access_key = + region = + ``` + + ``` + AWSAccessKeyId = + AWSSecretKey = + region = + ``` + + `region` is optional. + + Object Mount can also use an **AWS S3 Access Point** instead of a container. + + For more details, see the **AWS S3 Access Point Support** section in the User Guide article: [](docId:jieteeYeyievui9k#aws-s3-access-point-support) . + +{% /tab %} + +{% tab label="Storj" %} + **Storj Object Storage** + + Storj’s object storage is S3-compatible. + + Therefore, the syntax used for saving Storj credentials mimics the format used for AWS S3 storage: + + ``` + aws_access_key_id = + aws_secret_access_key = + endpoint = + ``` + + `endpoint` is required. + + The primary endpoint for Storj is: `endpoint = https://gateway.storjshare.io` +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + For Microsoft’s Azure cloud storage, the credential file should conform to the following format: + + ``` + AZURE_STORAGE_ACCOUNT= xxxxxxxxxxxxxxxxxx + AZURE_STORAGE_ACCESS_KEY= xxxxxxxxxxxxxxxxxx + ``` + + No region or endpoint is needed. +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud Platform** + + If you have an existing JSON/P12 (PCKS#12) file — that file is formatted correctly and will import properly. + + If you provided your own private key when creating the service account, you will need to create a JSON file which conforms to the following format: + + ``` + { + "type": "service_account", + "project_id": "PROJECT_ID", + "private_key_id": "KEY_ID", + "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n", + "client_email": "SERVICE_ACCOUNT_EMAIL", + "client_id": "CLIENT_ID", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://accounts.google.com/o/oauth2/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL" + } + ``` +{% /tab %} +{% tab label="Other S3 Compatible" %} + **S3 Compatible Providers** + + The file needs to contain at least: + + ``` + aws_access_key_id = + aws_secret_access_key = + region = + ``` +{% /tab %} + +{% /tabs %} + +_See the Advanced Guide article [](docId:aish4shuiheeZaig) for additional credential options and endpoint formatting details._ + + +## Securing Your S3 Credentials + +You can use any text editor to create the credential file — just remember to change the permissions on the file when you’re done to prevent other users from accessing this sensitive file: + +```shell +#terminal +chmod 0600 "" +``` + +{% callout type="info" %} + **Time Saving Tip** + + You can combine the **creation** of the file and securing **permissions** on the file in a single shell command (insert your credential details): + + ```shell + #terminal + touch credentials.txt ; chmod 0600 credentials.txt ; printf "aws_access_key_id = xxxxxxxxxxxxxxxxxx \naws_secret_access_key = xxxxxxxxxxxxxxxxx \nendpoint=A.B.C.D:PORT" > credentials.txt + ``` +{% /callout %} + +_SSee the Advanced Guide article [](docId:aish4shuiheeZaig) for additional credential options._ diff --git a/app/(docs)/object-mount/appendix/troubleshooting/page.md b/app/(docs)/object-mount/appendix/troubleshooting/page.md new file mode 100644 index 000000000..e71fa8055 --- /dev/null +++ b/app/(docs)/object-mount/appendix/troubleshooting/page.md @@ -0,0 +1,180 @@ +--- +title: Troubleshooting Guide +hideTitle: false +docId: troubleshooting +weight: 6 +metadata: + title: Troubleshooting Object Mount + description: + Common issues and solutions when using Object Mount, plus known limitations to be aware of. +hidden: false +--- + +This guide covers common issues encountered when using **Object Mount**, along with clear steps to resolve them. + +If you’re stuck, this is a good first place to check! + +Still stuck? + - Reach out to others in the Storj [Community Forum](docId:h0GeE0-z8ta1rOlKLL7lL) + - Or open a ticket with our Storj [Support Team](https://supportdcs.storj.io/hc/en-us) + + +## Common Issues + +Here are a few common issues to be aware of: + + - POSIX mode requires write access and may silently fail if credentials are read-only. + - Mac Finder & Windows Explorer progress bars may not reflect actual transfer status. + — Monitor network activity instead. + - Avid Media Composer has limited tolerance for long mount paths & non-local scratch files. + - Some S3-compatible providers require manual configuration (e.g. region or endpoint overrides). + - Mounts cannot currently be shared across users on the same machine (per-user context only). + + +## Mount Doesn’t Appear or Won’t Activate + +**Possible Cause(s):** + + - Incorrect credentials + - No accessible buckets + - Dependency not installed (e.g.: macFUSE) + - POSIX mode enabled without write access + +**Things to Check:** + + - Verify your license is active + - Windows and macOS: Click the **About** tab + - Linux: run `cuno creds list` + - Re-import your credentials and confirm bucket visibility + - Ensure dependencies are installed: + - **macOS**: [macFUSE](docId:MyV4Bv3fKJEZucAb#step-3-install-mac-fuse-required) + - For read-only mounts, disable POSIX mode + + +## Credentials Work Elsewhere, but Fail in Object Mount + +**Possible Cause(s):** + +Some S3-compatible providers require custom regions, endpoints, or flags. + +**Things to Check:** + + - Review the instructions on the **Other S3 Compatible** tab in the [S3 Credential Management](docId:E4NhE5kPdjURRajJ) article + - Be sure to select your provider from the dropdown + - Set an explicit region (e.g. `us-east-1`) + - Ensure the endpoint is reachable from your network + +**Note:** For Storj Object Storage, make sure the buckets are **lexicographically ordered** for best results. + + +## Mount Is Very Slow or Freezes + +**Possible Cause(s):** + + - High latency to cloud provider + - File system operations triggering too many API calls + - No caching or prefetch enabled + +**Things to Check:** + + - Enable **data cache** and **metadata cache** in Preferences + - Set `CUNO_OPTIONS = -filePrefetch` to disable file pre-fetching + - Increase S3 connection pool (e.g. 150-200 for 1Gbps+) + - Avoid using POSIX mode unless required + - Place cache directories on an internal SSD + + +## “Leave Files in Place” Doesn’t Work in Creative App + +**Possible Cause(s):** + +Some NLE Applications may try to verify write access or expect native macOS behavior. + +**Things to Check:** + + - Are you trying to import from a mounted Object Mount volume? + - Is POSIX mode conflicting with permission checks? + - Try relinking or using proxy workflows instead + + +## Logs Don’t Show Much Info + +**Possible Cause(s):** + +By default, logging is set to `error` only. + +**Things to Check:** + +Enable more detailed logging: + +1. Go to **Preferences > Advanced Settings** +2. Set **Log Level** to `debug` or `trace` +3. Reproduce the issue + +Check the logs at: + +- **macOS**: `~/Library/Application Support/Object Mount/cunoFS.log` +- **Windows**: `C:\Users\%username%\AppData\Local\Object Mount\cunofs.log` + + +## Files Are Stalling on Paste + +**Things to Check:** + +If files are stuck during paste: + + - Try using the **Fast Copy and Fast Move** on [Mac](docId:ehHbpq6KFndcVCgc#fast-copy-and-fast-move) or **Fast Paste** on [Windows](docId:xMKcaWrTVK6QMD8h#fast-paste-on-windows). + +These features bypasses the Mac and Windows native copy tools and uses Object Mount’s optimized transfer logic. + + +## Still Not Working? + +If you still need assistance in resolving your issue, follow the steps below to enable detailed logging and send the log files to the Storj Support team. + +**For macOS and Windows environments:** + +1. Run Object Mount, choose the **Settings** tab, and scroll down to **Advanced Settings**: + + - Change the “Object Mount log level” to `Trace`. + - Set “Filesystem debug logging” to `On`. + - Click the **Apply** button. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-01.jpg) + +2. Choose the **About** Tab: + + - Click **Report Issue**. + - Enter a **Title**. + - Enter details and/or **Steps to Reproduce** the issue. + - Ensure that **Include my system information** is checked. + - Click the **Continue** button. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-02.jpg) + +3. Save the Issue Report: + + - Name your `.zip` file. + - Click **Save** to store a `.zip` file on your local drive. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-03.jpg) + +4. Attach the `.zip` file and send the report: + + - Click the **Submit Via Email** button: + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-04.jpg) + + - Your default Email application will load with content preloaded. + - **IMPORTANT**: You must _manually_ attach the `.zip` file to the email! + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-05.jpg) + + - _Once attached_, click to **Send** the message to the Storj Object Mount support team. + +Alternatively, you can open a [support ticket](https://supportdcs.storj.io/hc/en-us/requests/new) and include: + +- OS and Object Mount version +- Cloud provider and bucket name +- Screenshots or log excerpts +- Steps to reproduce the issue diff --git a/app/(docs)/object-mount/appendix/tuning/page.md b/app/(docs)/object-mount/appendix/tuning/page.md new file mode 100644 index 000000000..09973584d --- /dev/null +++ b/app/(docs)/object-mount/appendix/tuning/page.md @@ -0,0 +1,129 @@ +--- +title: Performance Tuning +hideTitle: false +docId: tuning +weight: 5 +metadata: + title: Performance Tuning + description: + Guidance on how to optimize Object Mount performance across platforms and environments. +hidden: false +--- + +There are many variables to consider when optimizing **Object Mount** for the best performance, including: + - Operating System behavior + - Connection Speed + - Storage Provider Characteristics + +The sections below will provide tips on optimization for each of these three areas, tailored for your specific environment. + + +## Operating System Tips + +{% tabs %} +{% tab label="Linux" %} + - Use a recent kernel (5.10+ preferred) for improved FUSE performance. + - Ensure `fuse3` and required kernel modules are installed. + - Run Object Mount as a `systemd` user service if automatically mounting on login. + - Place cache directories (`data cache`, `metadata cache`) on a fast, local SSD or NVMe volume. + - Adjust `ulimit -n` if handling many open files (e.g. `ulimit -n 65535`). +{% /tab %} + +{% tab label=" macOS" %} + - Intel Processors: Use macOS Monterey (12.0) or later for best compatibility. + - Apple Silicon (M-Chips): Use macOS Ventura (13.0) or later for best compatibility. + - Make sure **macFUSE 4.8.2 or newer** is installed and authorized in **System Settings > Privacy & Security**. + - Disable Spotlight indexing on mount paths to reduce background load. + - Cache directories should be located on fast internal storage (not external or network drives). + - POSIX mode can add overhead. Enable only if required by your workflow. +{% /tab %} + +{% tab label="Windows" %} + - Use Windows 10 (21H2+) or Windows 11 for best driver support. + - Place cache folders on fast local drives (e.g. `C:\Temp\Object MountCache`). + - Disable antivirus scanning on mount points if performance is impacted. + - Run the app with administrator rights if mounting to system-level paths. +{% /tab %} +{% /tabs %} + + +## Connection Speed Tips + +Object Mount adapts well to different network conditions, but optimal performance depends on the speed and reliability of your connection to cloud storage. + +{% tabs %} +{% tab label="Fast" %} +**10 Gbps or faster** + + - Increase **S3 connections limit** (e.g. 200+). + - If Client and Storage are on the same **Local Area Network (LAN)**, disable **Data cache**. + - Consider setting **metadata cache timeout** to a lower value for frequent sync. +{% /tab %} + +{% tab label="Medium" %} +**1 Gbps to 10 Gbps** + + - Set **S3 connections limit** between 100-150. + - POSIX mode can add overhead. Enable only if required by your workflow. + - Use smart caching strategies if accessing large numbers of small files. +{% /tab %} + +{% tab label="Slow" %} +**100 Mbps to 1 Gbps** + + - Keep metadata **cache timeout** higher (e.g. 300-600 seconds). + - Lower **S3 connections limit** (e.g. 50 or fewer) to prevent throttling. + - Limit the number of simultaneous mounts or background operations. + - Use proxy editing in creative apps only if working with very large assets. +{% /tab %} +{% /tabs %} + + +## Storage Provider Tips + +Each storage backend behaves slightly differently depending on API responsiveness, region latency, and compatibility mode. + +{% tabs %} +{% tab label="AWS S3" %} + - Set an appropriate **region** to avoid cross-region latency. + - Use multipart uploads for large file writes (default behavior in Object Mount). + - Enable caching and proxy workflows for high-resolution media. + - Monitor AWS rate limits if using a high connection pool size. +{% /tab %} + +{% tab label="Storj" %} + - Use [Storj-managed encryption](docId:aitie6rohXai9uuv) projects. + - Use [**lexicographically ordered buckets**](docId:eZ4caegh9queuQuaazoo#list-objects) for faster directory listings. + - Enable `-filePrefetch` for large file reads. + - Ensure your satellite and uplink are in the same region as your client when possible. + - POSIX mode is supported but not required for most workflows. +{% /tab %} + +{% tab label="Microsoft Azure" %} + - Use a compatibility layer or gateway if not directly S3-compatible. + - Access tiers (e.g. Hot, Cool) may affect read latency. + - Monitor API throttling if using many simultaneous requests. + - Proxy editing recommended over direct edit for best responsiveness. +{% /tab %} + +{% tab label="Google Cloud" %} + - Use the XML S3 API compatibility layer (via interoperability settings). + - Latency-sensitive: caching strongly recommended. + - Consider using a regional GCS bucket near your compute node. + - POSIX emulation may be limited by object store behavior. +{% /tab %} + +{% tab label="Other S3" %} + - Use the **S3-compatible** provider tab when importing credentials. + - Always test the endpoint and region for compatibility. + - Providers like Wasabi may require explicitly setting the region. + - Avoid POSIX mode if write operations to root are not supported. +{% /tab %} +{% /tabs %} + + +## More Questions? + +Need help tuning for a specific workflow, app, or storage provider? + +Reach out via our Object Mount 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for additional assistance. diff --git a/app/(docs)/object-mount/faq/page.md b/app/(docs)/object-mount/faq/page.md deleted file mode 100644 index 21c073936..000000000 --- a/app/(docs)/object-mount/faq/page.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: FAQs -weight: 5 -docId: dd27e33c-7804-4e15-bdc5-941f9a1a7e12 -metadata: - title: Frequently Asked Questions - description: - Comprehensive resource to various FAQs covering topics related to Object Mount. ---- -## What S3 permissions are recommended to mount a bucket with Object Mount? -Short answer, for a generic cloud: read, write, and list permissions for the bucket intended to be mounted. - -## Are full permissions required to achieve maximum performance of Object Mount? -The maximum performance comes from Object Mount's ability to prefetch data based on how we handle certain file types. It is not dependent on being able to write the POSIX-metadata file. With read-only permissions to a bucket, Object Mount will still achieve maximum performance - but it will not be able to update any POSIX-related metadata associated with the original files. - -## Why don't read-only credentials work with POSIX enabled in Object Mount? -The POSIX mode needs to write a hidden file at the root of the bucket that tracks all the POSIX permissions, etc. This is why the read-only use case works so long as you mount the bucket without POSIX enabled. - -## How can I mount all buckets with Object Mount? -In order for Object Mount to "see" the bucket to be mounted when adding new credentials into Object Mount, the AWS equivalent of ListAllMyBuckets is required. In the absence of ListAllMyBuckets, you can specify what bucket to mount by manually pairing the bucket. - -## Where can I see logs of Object Mount on Windows? -Logs are available at `C:\Users\%username%\AppData\Local\Object Mount\cunofs.log` - the default logging level is "error" so changing this to "access" or "trace" will provide more verbose logs to better understand what is going on. - -## Where can I see logs of Object Mount on macOS? -For macOS the logs are located at `/Users/$USER/Library/Application Support/Object Mount/cunoFS.log` - -## Where are the configs and license files located on Windows? -For Windows they are located in [where logs are stored](#where-can-i-see-logs-of-object-mount-on-windows): `C:\Users\%username%\AppData\Local\Object Mount\license` and the `config.json` file is in the same directory, here the mount/bucket information is kept - -## Where configs and license files are located on macOS? -For macOS they are located in [where logs are stored](#where-can-i-see-logs-of-object-mount-on-mac-os): `/Users/$USER/Library/Application Support/Object Mount/license` and the `config.json` file is in the same directory, here the mount/bucket information is kept diff --git a/app/(docs)/object-mount/faq/posix-mode/page.md b/app/(docs)/object-mount/faq/posix-mode/page.md deleted file mode 100644 index 24369f0e3..000000000 --- a/app/(docs)/object-mount/faq/posix-mode/page.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: POSIX Mode Explained -docId: posix-mode -weight: 6 -metadata: - title: POSIX Mode in Object Mount - description: - Understand how POSIX mode works in Object Mount and when to enable or disable it. ---- - -**POSIX mode** in Object Mount provides additional compatibility for applications that rely on traditional UNIX-style file permissions and metadata. - -This feature emulates POSIX behaviour on top of S3 object storage, which by default lacks concepts like file ownership, permissions, and modification timestamps in the same way as a local file system. - ---- - -## What is POSIX Mode? - -When enabled, POSIX mode adds a virtual layer that stores and retrieves metadata typically expected in Linux or macOS environments — including: - -- File ownership (`uid`, `gid`) -- Read/write/execute permissions -- Timestamps (`mtime`, `ctime`, etc.) -- Symbolic links and directory flags (where supported) - -This metadata is stored in a hidden index file at the root of the bucket. - ---- - -## When to Enable POSIX Mode - -You should enable POSIX mode if: - -- You’re working with software that checks for or enforces POSIX-style permissions -- You require symbolic link emulation or fine-grained permission mapping -- You're mounting object storage as a shared filesystem in team environments - ---- - -## When Not to Enable POSIX Mode - -- **Read-only access**: - - POSIX mode requires **write access** to the bucket in order to store the hidden metadata file. If your credentials are read-only, POSIX mode will not function and may prevent the mount from working correctly. - -- **Lightweight access**: - - For read-only workflows or general browsing, POSIX mode is typically unnecessary and can be left disabled. - ---- - -## Why Write Access is Required - -The metadata file used for POSIX emulation needs to be written and maintained at the root of the bucket. This is why: - -- Buckets must be writable -- The credentials must allow `s3:PutObject` and `s3:DeleteObject` on that path -- You should **not** enable POSIX mode with read-only credentials — it will fail silently or generate errors - ---- - -## Enabling POSIX Mode - -You can enable POSIX mode when creating a new mount: - -- In the **Mounts** tab, select **Create new mount** -- Tick the checkbox for **Enable POSIX mode** -- Ensure your credentials and bucket support write access -- Save the mount and toggle it on - -> Note: POSIX mode applies on a **per-mount basis**, so you can choose which mounts require it and which don’t. - ---- diff --git a/app/(docs)/object-mount/faq/troubleshooting/page.md b/app/(docs)/object-mount/faq/troubleshooting/page.md deleted file mode 100644 index 6673a49cb..000000000 --- a/app/(docs)/object-mount/faq/troubleshooting/page.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: Troubleshooting Guide -docId: troubleshooting -weight: 8 -hideTitle: true -metadata: - title: Troubleshooting Object Mount - description: - Common issues and solutions when using Object Mount, plus known limitations to be aware of. ---- - -# Troubleshooting Guide - -This guide covers common issues encountered when using **Object Mount**, along with clear steps to resolve them. If you’re stuck, this is a good first place to check before reaching out to support. - ---- - -## Mount Won’t Appear or Activate - -**Possible Causes:** - -- Incorrect credentials -- No accessible buckets -- macFUSE or WinFsp not installed or authorized -- POSIX mode enabled without write access - -**What to check:** - -- Verify your license is active (`About` tab) -- Re-import your credentials and confirm bucket visibility -- Ensure dependencies are installed: - - **macOS**: [macFUSE](https://osxfuse.github.io/) - - **Windows**: [WinFsp](https://github.com/billziss-gh/winfsp) -- For read-only mounts, disable POSIX mode - ---- - -## Credentials Work Elsewhere but Fail in Object Mount - -**What’s happening:** - -Some S3-compatible providers require custom regions, endpoints, or flags. - -**What to try:** - -- Use the **S3-Compatible** tab when importing -- Select your provider from the dropdown -- Set an explicit region (e.g. `us-east-1`) -- Ensure the endpoint is reachable from your network - -> For Storj: make sure the bucket is **lexicographically ordered** for best results - ---- - -## Mount Is Very Slow or Freezes - -**Common causes:** - -- High latency to cloud provider -- File system operations triggering too many API calls -- No caching or prefetch enabled - -**Suggested steps:** - -- Enable **data cache** and **metadata cache** in Preferences -- Set `CUNO_OPTIONS = -filePrefetch` -- Increase S3 connection pool (e.g. 150-200 for 1Gbps+) -- Avoid using POSIX mode unless required -- Place cache directories on an internal SSD - ---- - -## “Leave Files in Place” Doesn’t Work in Creative App - -Some NLE Applications may try to verify write access or expect native macOS behavior. - -**Check:** - -- Are you trying to import from a mounted Object Mount volume? -- Is POSIX mode conflicting with permission checks? -- Try relinking or using proxy workflows instead - ---- - -## Logs Don’t Show Much Info - -By default, logging is set to `error` only. - -**To enable detailed logging:** - -1. Go to **Preferences > Advanced Settings** -2. Set **Log Level** to `debug` or `trace` -3. Reproduce the issue -4. Check the logs at: - -- **macOS**: `~/Library/Application Support/Object Mount/cunoFS.log` -- **Windows**: `C:\Users\%username%\AppData\Local\Object Mount\cunofs.log` - ---- - -## Files Are Stalling on Paste - -If files are stuck during paste, and you’re on **Windows**: - -- Try using the **Fast Paste Here** right-click option -- This bypasses Windows shell copy and uses Object Mount’s optimized transfer logic - ---- - -## Still Not Working? - -If you still need assistance in resolving your issue, follow the steps below to enable detailed logging and send the log files to the Storj Support team. - -For macOS and Windows environments: - -1. Run Object Mount, choose the **Settings** tab, and scroll down to **Advanced Settings**: - - - Change the “Object Mount log level” to `Trace`. - - Set “Filesystem debug logging” to `On`. - - Click the **Apply** button. - - ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-01.jpg) - -2. Choose the **About** Tab: - - - Click **Report Issue**. - - Enter a **Title**. - - Enter details and/or **Steps to Reproduce** the issue. - - Ensure that **Include my system information** is checked. - - Click the **Continue** button. - - ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-02.jpg) - -3. Save the Issue Report: - - - Name your `.zip` file. - - Click **Save** to store a `.zip` file on your local drive. - - ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-03.jpg) - -4. Attach the `.zip` file and send the report: - - - Click the **Submit Via Email** button: - - ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-04.jpg) - - - Your default Email application will load with content preloaded. - - **IMPORTANT**: You must _manually_ attach the `.zip` file to the email! - - ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-enable-trace-logs-05.jpg) - - - _Once attached_, click to **Send** the message to the Storj Object Mount support team. - -Alternatively you can open a [support ticket](https://supportdcs.storj.io/hc/en-us/requests/new) and include: - -- OS and Object Mount version -- Cloud provider and bucket name -- Screenshots or log excerpts -- Steps to reproduce the issue - ---- - -## Known Limitations - -Here are a few current limitations to be aware of: - -- POSIX mode requires write access and may silently fail if credentials are read-only -- Finder/Explorer progress bars may not reflect real transfer status — check network activity instead -- Avid Media Composer has limited tolerance for long mount paths or non-local scratch files -- Some S3-compatible providers require manual configuration (e.g. region or endpoint overrides) -- Mounts cannot currently be shared across users on the same machine (per-user context only) - ---- - -Let us know if you’d like to contribute improvements or suggest workarounds — we’re always refining based on real-world feedback. diff --git a/app/(docs)/object-mount/faq/tuning/page.md b/app/(docs)/object-mount/faq/tuning/page.md deleted file mode 100644 index 7253b37d5..000000000 --- a/app/(docs)/object-mount/faq/tuning/page.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: Performance Tuning -docId: tuning -weight: 1 -metadata: - title: Performance Tuning - description: - Guidance on how to optimise Object Mount performance across platforms and environments. ---- - -There are many variables to consider when tuning **Object Mount** for the best performance, including operating system behaviour, network speed, and the characteristics of your storage provider. - -Use the tabs below to explore optimisation tips for different platforms and configurations. - ---- - -## Operating System - -{% tabs %} -{% tab label="Linux" %} -- Use a recent kernel (5.10+ preferred) for improved FUSE performance -- Ensure `fuse3` and required kernel modules are installed -- Run Object Mount as a systemd user service if mounting on login -- Place cache directories (`data cache`, `metadata cache`) on a fast SSD or NVMe volume -- Adjust `ulimit -n` if handling many open files (e.g. `ulimit -n 65535`) -{% /tab %} - -{% tab label="MacOS" %} -- Use macOS Monterey (12.0) or later for best compatibility -- Make sure **macFUSE 4.8.2 or newer** is installed and authorised in **System Settings > Privacy & Security** -- Disable Spotlight indexing on mount paths to reduce background load -- Cache directories should be located on fast internal storage (not external or network drives) -- POSIX mode can add overhead. Enable only if required by your workflow -{% /tab %} - -{% tab label="Windows" %} -- Use Windows 10 (21H2+) or Windows 11 for best driver support -- Place cache folders on fast local drives (e.g. `C:\Temp\Object MountCache`) -- Disable antivirus scanning on mount points if performance is impacted -- Run the app with administrator rights if mounting to system-level paths -{% /tab %} -{% /tabs %} - ---- - -## Connection Speed - -Object Mount adapts well to different network conditions, but optimal performance depends on the speed and reliability of your connection to cloud storage. - -{% tabs %} -{% tab label="Fast" %} -**10 Gbps or faster** - -- Enable **file prefetching** via `CUNO_OPTIONS = -filePrefetch` -- Increase **S3 connections limit** (e.g. 200+) -- If Client and Storage are on same **Local Area Network (LAN)** switch off **Data cache** -- Consider setting **metadata cache timeout** to a lower value for frequent sync -{% /tab %} - -{% tab label="Medium" %} -**1 Gbps to 10 Gbps** - -- Set **S3 connections limit** between 100–150 -- Avoid enabling POSIX mode unless needed -- Use smart caching strategies if accessing large numbers of small files -{% /tab %} - -{% tab label="Slow" %} -**100 Mbps to 1 Gbps** - -- Keep metadata **cache timeout** higher (e.g. 300–600 seconds) -- Lower **S3 connections limit** (e.g. 50 or fewer) to prevent throttling -- Limit the number of simultaneous mounts or background operations -- Use proxy editing in creative apps only if working with very large assets -{% /tab %} -{% /tabs %} - ---- - -## Storage Provider Optimisation - -Each storage backend behaves slightly differently depending on API responsiveness, region latency, and compatibility mode. - -{% tabs %} -{% tab label="Storj" %} -- Use [Storj-managed encryption](docId:aitie6rohXai9uuv) projects (alternatively - [**lexicographically ordered buckets**](docId:eZ4caegh9queuQuaazoo#unencrypted-object-keys)) for faster directory listings -- Enable `-filePrefetch` for large file reads -- Ensure your satellite and uplink are in the same region as your client when possible -- POSIX mode is supported but not required for most workflows -{% /tab %} - -{% tab label="AWS S3" %} -- Set an appropriate **region** to avoid cross-region latency -- Use multipart uploads for large file writes (default behaviour in Object Mount) -- Enable caching and proxy workflows for high-resolution media -- Monitor AWS rate limits if using a high connection pool size -{% /tab %} - -{% tab label="Other S3" %} -- Use the **S3-compatible** provider tab when importing credentials -- Always test the endpoint and region for compatibility -- Providers like Wasabi may require explicitly setting the region -- Avoid POSIX mode if write operations to root are not supported -{% /tab %} - -{% tab label="Google Cloud" %} -- Use the XML S3 API compatibility layer (via interoperability settings) -- Latency-sensitive: caching strongly recommended -- Consider using a regional GCS bucket near your compute node -- POSIX emulation may be limited by object store behaviour -{% /tab %} - -{% tab label="Azure" %} -- Use a compatibility layer or gateway if not directly S3-compatible -- Access tiers (e.g. Hot, Cool) may affect read latency -- Monitor API throttling if using many simultaneous requests -- Proxy editing recommended over direct edit for best responsiveness -{% /tab %} -{% /tabs %} - ---- - -## Just Ask - -Need help tuning for a specific workflow, app, or storage provider? - -Reach out via our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) and our team will be happy to assist. - ---- diff --git a/app/(docs)/object-mount/linux/_meta.json b/app/(docs)/object-mount/linux/_meta.json deleted file mode 100644 index e12c22cac..000000000 --- a/app/(docs)/object-mount/linux/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Linux", "weight": 2} diff --git a/app/(docs)/object-mount/linux/advanced/config-options/page.md b/app/(docs)/object-mount/linux/advanced/config-options/page.md new file mode 100644 index 000000000..b8cc5006d --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/config-options/page.md @@ -0,0 +1,384 @@ +--- +title: Advanced Configuration Options +hideTitle: false +docId: phohPoowequie5ji +weight: 3 +redirects: + - /object-mount/linux/user-guides/configuration +metadata: + title: Advanced Configuration Options + description: Advanced Configuration Options +hidden: false +--- + +Setting the environment variable `CUNO_OPTIONS` can be used to further configure Object Mount. + +The `-o` option added to the `cuno` command can also be used to specify these options. + + +## Options Available for CUNO_OPTIONS + +{% callout type="note" %} + **Space Separation** + + Individual options must be separated by spaces. + + Don’t forget to enclose multiple options in quotes (`"` or `’`), or to escape spaces. +{% /callout %} + +### The static Option + +To use Object Mount with static binaries, the _parent process_ needs to be intercepted in static mode. + +Interception of static binaries is enabled by default. To disable it requires setting both `CUNO_OPTIONS="-static"` and starting a new Object Mount wrapped shell. + +### The uricompat Option + +Object Mount can override the handling of URI-style paths (`xx://`) in applications that attempt to provide internal handling of such URIs. Users may provide the `+uricompat` option to add similar patching for further applications. + +To specify any applications to be patched, specify them as a colon-separated list: + +```console ++uricompat=::< ... > +``` + +To limit the cases in which this patching occurs, you can filter interceptions to only occur when certain command-line arguments match specified keywords. This may be useful for certain code interpreters/executors (see the example below). + +To do this, in place of an executable name, use a slash-separated list of the names, a 1-based index of the argument to be matched (or any using `*`), and a value to match against. + +For example: + +```console ++uricompat=// +``` + +{% callout type="note" %} + **`uricompat`** + + The `+uricompat` option only works on URIs that have been paired. + + If you are working with a public access bucket, you will need to run the `cuno creds pair` command (see [Pairing Containers and Credentials](docId:aish4shuiheeZaig#pairing-containers-and-credentials)) against it for the `+uricompat` to take effect. +{% /callout %} + +**Applications With Patched URI Handling:** + +By default, Object Mount will attempt to override the internal URI handling of popular tools, such as: + +- `rsync` +- `ffmpeg` +- `tar` + +And some genomics tools: + +- `samtools` +- `igv` +- `fastQC` + +This means that when a URI-style path corresponding to a paired bucket or container is passed to any of the applications above, Object Mount will prevent the application from handling the path in a special way. + +For example, `ffmpeg` has special handling for some 🌐 [protocols](http://ffmpeg.org/ffmpeg-protocols.html#Protocols) which they specify in a similar URI format to Object Mount URI cloud paths (e.g. `ftp://example.foo`). + +As a result, Object Mount needs to prevent `ffmpeg` from failing (when given a path like `s3://bucket/file`) because `s3`, `az` and `gs` are not protocols it supports. + +**Example:** + +To override a Java application’s handling of URI-style paths, you can use: + +```console +export CUNO_OPTIONS="+uricompat=java/2/appname.jar:java/*/org.name.main.Class" +``` + +This will intercept any invocation of `java` with `appname.jar` as its second argument (e.g.: `java -jar appname.jar s3://bucketname/file.bam`). + +It will also intercept invocations of Java where any argument is equal to `org.name.main.Class` (e.g.: `java org.name.main.Class s3://bucketname/file.bam`). + + +### The cloudroot Option + +You may define a custom directory path prefix to replace the default `/cuno` path. Once defined, the path specified will be made available at the root of the file system. + +For example, if you configure the following: + +```console +export CUNO_OPTIONS='+cloudroot=my-object-storage' +``` + +You will then be able to list your configured S3 buckets with: + +```console +ls /my-object-storage/s3/ +``` + +{% callout type="warning" %} + **Using `cloudroot`** + + We strongly recommend that the cloudroot path _does not exist locally_ because it can confuse Object Mount’s ability to distinguish whether a path refers to the local file system or to the cloud. + + If you specify a cloudroot which conflicts with an existing directory, then it must not contain directories or files named `s3`, `az` or `gs`. +{% /callout %} + +{% callout type="note" %} + **`cd to cloud`** + + In order for the “cd to cloud” functionality to see this option, you will need to spawn a new shell after `CUNO_OPTIONS`. +{% /callout %} + +**FlexMount Usage:** + +Setting a cloudroot during the mounting process can affect the options required to be set when using it as a FlexMount. + +**Recommended option:** If cloudroot was not set when creating the mount (e.g. `cuno mount ~/my-object-storage`), then cloudroot needs to be set as the path to the mount point when launching the Object Mount CLI in order to use that mount as a FlexMount (e.g. `cuno -o cloudrootover -o cloudroot=""`). + +In some circumstances, you may need to set a cloudroot when creating the mount (for example, if a `/cuno` directory already exists on your system). In these cases, we strongly recommend migrating or removing any such physical directory so that you can continue to use the default settings. If this is not possible, there are a number of options available. + +**Recommended option if cloudroot _must_ be changed:** If a cloudroot needs to be set when creating the mount, then you should set the `--root` option to the same cloudroot to get expected behavior when using Object Mount FlexMount (e.g. `cuno -o cloudroot=/altcloudroot mount $HOME/my-object-storage --root /altcloudroot`). In this case, continue to use the same options when launching cuno (e.g. `cuno -o cloudrootover -o cloudroot=""`) to access the mount as a FlexMount. + +In very limited circumstances, it is not possible to choose a cloudroot that can also be used as the `--root` of the mount point (for example, if the cloudroot is chosen to be somewhere that is not mountable). In this case, we would advise you to choose a different cloudroot, one that can be mounted (such as a position within your user home directory e.g. `cuno -o cloudroot=$HOME/mycloudroot mount $HOME/my-object-storage --root $HOME/mycloudroot`). If this is not possible, then there are two options: + +**HIGHLY discouraged:** If a cloudroot needs to be set when creating the mount, and the `--root` option is set to a path within the cloudroot (e.g. `cuno -o cloudroot=/altcloudroot mount $HOME/my-object-storage --root /altcloudroot/s3/mybucket`), then, when using the FlexMount later, one must set the 'cloudrootover' path as so: `cuno -o cloudroot=/altcloudroot -o cloudrootover=$HOME/my-object-storage`. **Important:** always check that the mount point exposes the cloud at the same point when used directly as a Object Mount on FUSE (so outside of a Object Mount CLI instance) and when accessed as a FlexMount. + +**HIGHLY discouraged:** If a cloudroot needs to be set when creating the mount, and it is _not possible_ for `--root` to be set to a path within the cloudroot (i.e. `cuno -o cloudroot=/altroot mount $HOME/my-object-storage --root /`), then, when using the FlexMount later, one must set the cloudroot as follows: `cuno -o cloudrootover -o cloudroot=$HOME/my-object-storage/my/cloud/root`. + +{% callout type="warning" %} + **Critical Option Settings to Avoid Loss of Data** + + Object Mount behavior will be **broken and dangerous** if you do not set the correct options when launching Object Mount CLI with there types of FlexMounts. + + For example, in this case if the default parameters for launching a FlexMount are used, namely `cuno -o cloudrootover -o cloudroot=$HOME/my-object-storage`, then applications that are intercepted directly will see `$HOME/my-object-storage` as the cloudroot (so the immediate children are `az gs s3`) whereas any applications that fall through to the Object Mount on FUSE (static binaries, cloud binaries, SUID binaries, Snap apps, AppImage apps, and Flatpak apps) will see `$HOME/my-object-storage` as `/` - the root of your local filesystem. + + This could cause a **loss of data**, accidentally expose data in the wrong place or other misbehavior when running scripts. +{% /callout %} + + +## Ownership and Permissions + +### Core File Access + +In [Core File Access](docId:cbm3PcQXmLpuYcbg#core-file-access) Mode, and for files in object storage uploaded using tools _other_ than Object Mount, we assign some dynamic defaults for ownership and permissions. + +In these circumstances, the owner of cloud objects is always reported to be the current user, the directory mode is reported as `0777`, and the file mode is reported as `0666`. + +The defaults can be overridden by using the `uid`, `gid`, `filemode` and `dirmode` options within the `CUNO_OPTIONS` environment variable. + +By doing so, an administrator can set the default UID/GID and access mode permissions that apply to all files and directories of cloud storage accessed by Object Mount. This might be considered in circumstances where an application starts as one user, but mid-process switches context to run as another — services like web-servers often function in this way. + +To demonstrate this in action, you could run the following: + +```console +$ cuno -o 'uid=0 gid=0 filemode=0444 dirmode=0447' \ + mount --root s3://examplebucket/directory/ /mnt/s3-bucket + +$ ls -l /mnt/s3-bucket +dr--r--rwx 1 root root 0 Jan 1 1970 directory +-r--r--r-- 1 root root 0 Nov 10 11:16 file +``` + +While the default UID/GID and access mode permissions can be set ahead of time, they cannot be modified later. + +### POSIX File Access + +To **persist and modify** file system metadata, [POSIX File Access](docId:cbm3PcQXmLpuYcbg#posix-file-access) must be enabled. + +This allows the virtual files presented by Object Mount to have their system metadata modified by tools such as `chown`, `chmod`, and `touch`. + +{% callout type="note" %} + **POSIX Metadata File** + + The system metadata is stored as objects within your object storage system within hidden subdirectories where the data resides. The name of that subdirectory is a period followed by a unicode U+FBF80 `󻾀` character. + + From within Object Mount you will not see that directory or objects inside of it, as Object Mount filters them from view. However, if accessing your object storage using other tools, they will be visible. + + Non-Object Mount access which renames, moves or copies objects with Object Mount file attributes, will result in those objects **losing their system metadata**. Instead, you should always use Object Mount to manage those files and preserve their attributes. +{% /callout %} + +**Enabling POSIX for Direct Interception:** + +There are two ways to enable POSIX File Access when using [Direct Interception Mode](docId:bRnfbdNE6d5DaZzW) via the Object Mount CLI: + +1. **Setting a Bucket Tag** + + The `cuno creds setposix` command can be used to toggle POSIX mode `on` for the bucket and takes precedence over the `CUNO_POSIX` environment variable. This command writes a tag to the bucket’s metadata on the bucket store. + + The setting set by this command applies to all users using the bucket through Object Mount. + + ```console + cuno creds setposix s3://examplebucket true + ``` + + {% callout type="note" %} + **`cuno creds setposix`** + + The following permissions are required to use the `cuno creds setposix` command: + + - S3-compatible: `s3:PutBucketTagging` and `s3:GetBucketTagging` + - Google Cloud Storage: `storage.buckets.update` and `storage.buckets.get` IAM permissions + - Azure Blob Storage: Only the container owner may set labels + {% /callout %} + + Valid options are: + + - `true` - enable posix in enforcing mode, preventing unauthorized access + - `false` - disable posix + - `permissive` - enable posix in permissive mode, where object attributes like uid, gid, mode, timestamps are stored but user access privilege is not checked + - `metadata` - enable posix in enforcing mode and store posix attributes as object metadata (only on S3) + - `permissive-with-metadata` - enable posix in permissive mode and store posix attributes as object metadata + - `unset` - remove posix configuration from the bucket + + {% callout type="warning" %} + **Object Storage Compatibility with Tags** + + Some S3-compatible stores may not support setting custom metadata flags required for this feature to function. + + If this is the case, you can still use the `--posix` flag to enable POSIX enforcement when starting **Object Mount on FUSE**. + + You can also still use the `CUNO_POSIX` environment variable to enable it for a **Direct Interception** session. If these methods are not satisfactory or preferable, please get in contact with our 🌐 [Storj Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for additional assistance. + {% /callout %} + + To unset the bucket-level setting and allow controlling POSIX mode using either the `CUNO_POSIX` environment variable or the `--posix` command line flag, use `cuno creds setposix` with a setting of `unset`: + + ``` + cuno creds setposix s3://examplebucket unset + ``` + +2. Using the CUNO_POSIX Environment Variable + + If POSIX mode is not set at the bucket level, it can be enabled by setting the environment variable `CUNO_POSIX` to `1`. If POSIX is enabled for the bucket via a tag, it **cannot** be overridden with this environment variable. + + For example, to enable POSIX mode for a Direct Interception session via Object Mount CLI: + + ``` + $ export CUNO_POSIX=1 + $ cuno + ``` + + This mode is suitable for when you need deeper POSIX compatibility in order to correctly run applications or workflows. For example, a tool such as `rsync` may be checking `mtime` in order to check for updates to a file thus requiring you to preserve such attributes on the object storage side. + + {% callout type="note" %} + **Non-Object Mount Access & Security** + + This is not “secure” in the sense that Direct Interception mode requires the user to have access to object storage credentials. Consequently, they can work around any access limitations by using the credentials directly with other tools (such as `awscli`). + {% /callout %} + +**Enabling POSIX for a Object Mount on FUSE:** + +If a bucket doesn’t have POSIX mode already enabled, you can use the option `--posix` when mounting a [Object Mount on FUSE](docId:bRnfbdNE6d5DaZzW) to enable fine-grained control over ownership and permissions and have them enforced. + +If a bucket’s POSIX mode tag is already set but the `--posix` flag is not, the bucket’s setting will apply but the permissions will **not be enforced** by the mount. + +If you are mounting for yourself and others, mount the bucket in a location that is accessible to the users who need access to the files. This can be a shared location or a location that is only accessible to the user who needs access. + +If setting up the mount as an admin user, to allow for sudo access, please use the `-o allow_root` flag. + +To allow other users to access the mount as themselves, use `-o allow_other`. + +```console +$ cuno mount --posix -o allow_root -o allow_other --root s3://mybucket /mnt/cloud/bucket +``` + +Users will now be able to see the files in the bucket at `/mnt/cloud/bucket`. You should now treat this as any other POSIX file system, and set permissions as you need to control access. + +### Usage Examples + +Assuming a mount location of `/mnt/s3-bucket`: + +- Change permissions on an object/file: + + ```console + $ chmod 0770 /mnt/s3-bucket/file + $ ls -l /mnt/s3-bucket/file + -rwxrwx--- 1 root root 0 Nov 10 11:16 file + ``` + +- Change the owner of an object/file: + + ```console + $ chown alice /mnt/s3-bucket/file + $ ls -l /mnt/s3-bucket/file + -rwxrwx--- 1 alice root 0 Nov 10 11:16 file + ``` + +- Alter the modify time: + + ```console + $ echo "Hello" > /mnt/s3-bucket/file + $ ls -l /mnt/s3-bucket/file + -rwxrwx---+ 1 alice root 6 Nov 10 11:32 file + ``` + +- Alter the change time: + + ```console + $ chmod +x /mnt/s3-bucket/file + $ stat /mnt/s3-bucket/file + File: /mnt/s3-bucket/file + Size: 6 Blocks: 1 IO Block: 65536 regular file + Device: 49h/73d Inode: 4 Links: 1 + Access: (0770/-rwxrwx---) Uid: ( 1009/ alice) Gid: ( 0/ root) + Access: 2021-11-10 11:32:59.185000000 +0000 + Modify: 2021-11-10 11:32:59.185000000 +0000 + Change: 2021-11-10 11:47:21.846000000 +0000 + ``` + + +## Other Advanced Configuration Options + +### CUNO_CREDENTIALS Variable + +The environment variable `CUNO_CREDENTIALS` allows you to customize the location of the Object Mount credentials store. Full details can be found in the Advanced Guide article: [](docId:aish4shuiheeZaig). + +### Proxy Server Tunneling + +Object Mount supports HTTP and HTTPS proxying access through to your cloud object storage. + +Provide the environment variable `http_proxy` or `https_proxy` with your gateway and Object Mount will tunnel all transactions through your designated proxy. + +### Verbose Debug Output + +There are rare occasions where Object Mount outputs to the command line, and this will generally indicate a fault. + +All messages will be prefixed with `cuno:` and these will always be sent a to the standard error stream (`stderr`). + +The options available for `CUNO_LOG` are cumulative and build on the previous level: + +> ``` +> .. cssclass:: listtable43mm +> +> .. list-table:: +> :header-rows: 0 +> +> * - :code:`error` +> - At this level, only critical failures are output. (Default) +> * - :code:`warning` +> - At this level, errors and warnings are output. +> * - :code:`debug` +> - At this level, information relevant to debugging will be output. +> * - :code:`trace` +> - At this level, a verbose logging of all steps will be output. +> ``` + +For example, a directory listing with the trace level set, can produce output similar to the below: + +``` +$ CUNO_LOG=trace ls s3://example-bucket +cuno: [TRACE][01/02/21 08:39:07.39][Thread: 3817873920] Trying bucket store at: /home/user/.config/cuno/creds/... +cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Loading S3Client +cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Not On EC2 +cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Attempting to find CaCerts. +cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using Certs: /etc/ssl/certs/ca-certificates.crt +cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Attempting to load PGMAN +cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using custom Endpoint: http://127.0.0.1:8080 +cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using custom pathstyle +``` + +There is a separate option: `access` that enables detailed logging of the file access performed by Object Mount, which includes read, write, open, close, seek and similar. + +`access` can be combined with the other log levels to log both file access and the desired logs. + +For example: + +``` +$ CUNO_LOG=trace,access ls s3://example-bucket +cuno: [DEBUG][07/17/23 13:59:03.287][P/T: 296181:2897806400] Not On EC2 +cuno: [DEBUG][07/17/23 13:59:03.287][P/T: 296181:2897806400] Using Certs: /etc/ssl/certs/ca-certificates.crt +cuno: [ACCESS][07/17/23 13:59:03.355][P/T: 296181:2897806400] s3://example-bucket/file write() 12 @ 0 +``` diff --git a/app/(docs)/object-mount/linux/advanced/credentials/page.md b/app/(docs)/object-mount/linux/advanced/credentials/page.md new file mode 100644 index 000000000..ee51d6c3d --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/credentials/page.md @@ -0,0 +1,355 @@ +--- +title: Advanced Credential Options +hideTitle: false +docId: aish4shuiheeZaig +weight: 1 +redirects: + - /object-mount/credentials/linux/import +metadata: + title: Credential Management + description: + Credential Management +hidden: false +--- + +## Obtaining and Saving your S3 Credentials + +Users will need to obtain the credentials for their S3 cloud storage accounts and save those credentials to a text file. + +Refer to the Appendix article [](docId:E4NhE5kPdjURRajJ) for instructions on how to obtain and save your S3 credentials. + + +## Credential File Location + +Object Mount for Linux uses the `cuno creds` command to import and manage your S3 credentials. + +Linux credentials are stored in `"${XDG_CONFIG_HOME}"/cuno/creds` by default (or `~/.config/cuno/creds` if `XDG_CONFIG_HOME` if not set). + +You can override the defaults by setting the `CUNO_CREDENTIALS` environment variable. + +{% callout type="note" %} + **Credential Storage** + + `cuno creds` will determine the presence of the following paths to store credentials, in descending order of precedence: + + 1. `$CUNO_CREDENTIALS` + 2. `"${XDG_CONFIG_HOME}"/cuno/creds` + 3. `~/.config/cuno/creds` + 4. `"${CUNO_ROOT}"/cuno-config/creds` +{% /callout %} + + +## Endpoint Details + +Endpoints can be IP addresses or domain URIs (optionally with port numbers). + +Note that a schema is required (e.g. `http://` or `https://`). + +The following are examples of valid endpoint key-value pairs: + +``` +endpoint = http://127.0.0.1:8080 +endpoint = https://s3objectstorage.example.com +``` + +## Global S3 Configuration File Options + +Depending on the capabilities and behaviors of your object storage provider’s S3 API, you may find the additional keys useful: + +| **Option** | **Description** | +|-------------|-----------------| +| `skipssl` | Skip SSL certificate verification. Useful for private storage solutions with self-signed certificates. +| `nossl` | Use HTTP protocol instead of HTTPS. Use when the endpoint starts with `http://`. +| `pathstyle` | Switch from virtual host style to path style syntax for S3-compatible storage solutions. Enable for private storage solutions supporting only path style syntax (e.g. MinIO, Ceph). +| `no-delete-objects` | Do not use `DeleteObjects` to remove multiple objects. Instead, use `DeleteObject` for single object removal. +| `no-remote-copy` | Do not copy objects within the same bucket via the server — download and upload instead. +| `no-object-tagging` | Do not attempt to get/put object tags. +| `no-object-acl` | Do not attempt to get/put object ACLs. +| `no-multipart-upload` | Do not attempt to use 🌐 [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html). +| `no-copy-part` | Do not attempt to use 🌐 [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html). +| `no-range-download` | Do not attempt to use 🌐 [Byte-Range Fetches](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html#optimizing-performance-guidelines-get-range) — use a single request to download the whole object instead. +| `no-delete-objects` | Do not attempt to use 🌐 [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) (**plural**) when deleting multiple objects — use 🌐 [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) (**singular**) for each object to delete. +| `directory-marker` | Create a placeholder object when creating an empty prefix (e.g., `mkdir`). Remove the placeholder object when removing an empty directory. +| `ignore-empty-prefixes` | Ignore directories that were removed but still reported as existing prefixes by the server. +| `posix=` | Set POSIX mode for all buckets associated with this credential. This option takes precedence over the POSIX mode runtime environment variable, but may be overridden by a bucket-wide POSIX setting (set via `cuno creds`). + +{% callout type="note" %} + **Detect Features** + + Some of the above options can be automatically detected and populated by running `cuno creds detectfeatures ` (e.g. `cuno creds detectfeatures s3://test-bucket test_credentials.s3c`). + + This command needs a bucket to be specified that it can write temporary files to for testing purposes. The command will run a series of tests against the specified bucket, checking the availability of S3 features that Object Mount uses and updating the credentials file accordingly. + + **Note:** Running feature detection will use up to a few gigabytes of bandwidth and may take a few minutes to complete depending on the machine’s connection speed and the S3-compatible storage provider. +{% /callout %} + + +## Microsoft Azure Storage + +Azure Blob Storage credential files have the following format: + +``` +AZURE_STORAGE_ACCOUNT = +AZURE_STORAGE_ACCESS_KEY = +``` + +If you need the access key, see Microsoft’s 🌐 [View account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys) article. + +### Microsoft Azure Storage with Shared Access Signatures + +Object Mount also supports using **Shared Access Signatures (SAS)** to access remote containers in Azure. + +Shared Access Signatures permit access to storage resources _without_ sharing access keys with users. + +An SAS gives user’s access to a container for only specified intervals and with specified permissions. + +To provide a SAS to Object Mount use the following command: + +``` +cuno creds pair "?" +``` + +**Note:** The question mark `?` prefix and quotation marks `"` are required. + + +## Google Cloud Storage + +You will need key-based access to be associated with your [Google Cloud service account](https://developers.google.com/workspace/guides/create-credentials#service-account). + +If you need to set this up, you can have Google generate a new key by following Google’s guide, [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete). Alternatively, you can generate your own key pair and upload it on the service account’s key management page, by following the guide [Upload service account keys](https://cloud.google.com/iam/docs/keys-upload). + +If you’ve generated a new key, store the JSON file you have downloaded in a safe place with appropriate access permissions, ready for the next steps. + + +## Importing Credentials + +Every cloud account requires a unique credential file. + +Import a credential file into Object Mount using the following command: + +``` +cuno creds import +``` + +Object Mount performs the following actions when importing a credential file: + +1. Detect the credential file format, and connect to the remote server. +2. Discover all possible containers that can be accessed using the provided credentials. +3. Create pairings between the imported credential file and remote containers. + +{% callout type="note" %} + **Listing Permission are Required** + + Discovery only completes when the provided credentials include bucket listing permissions. + + If listing permissions cannot be provided, manually pair the imported credential file to remote buckets using `cuno creds pair` (see below)`. +{% /callout %} + + +## Pairing Containers and Credentials + +Manually pair additional containers with previously-imported credential files using `pair`: + +``` +cuno creds pair [imported_credential_file] +``` + +- `` is the remote URI of the container to be associated with the credential file. + +- `[imported_credential_file]` is optional and corresponds to the credential file that has already been imported. + +If the latter is not provided, all imported credential files are listed. + +To re-associate a previously paired remote container with a different credential file, use `pair` with the same remote URI and a different credential file. + +{% callout type="note" %} + **Automatic Verification** + + Object Mount verifies that a remote container is accessible when pairing it with a credential file. + + Object Mount auto-detects an AWS region if it is not specified, or is invalid. +{% /callout %} + + +## Public Access Buckets + +To pair a remote URI that has public access, use the following command: + +``` +cuno creds pair public +``` + +Notice `public` is used in the place of the imported credential file name. + + +## “Requester Pays” buckets + +Amazon AWS S3 and Google Cloud Storage offer “Requester Pays” features for containers. + +In these cases the requesters, not the bucket owners, are billed when remote objects are accessed. + +By default, Object Mount skips “Requester Pays” containers when importing a credential file to avoid unexpected charges. To enable access to such containers with a previously-imported credential file, run the following command: + +``` +cuno creds --interactive pair +``` + +The `--interactive` (`-i`) option will prompt the user to select from credentials in Object Mount’s database. + +### “Requester Pays” with Google Cloud Storage + +When you run the run the `cuno creds --interactive pair ` command, it will list all available project IDs associated with the selected credential file and available for billing. + +Select and confirm a “Requester Pays” container to pair as prompted. + +To run the command non-interactively, enter: + +``` +cuno creds pair --billing +``` + +### “Requester Pays” with Amazon AWS S3 + +To force a pairing between a “Requester Pays” container with an imported credential file use the following: + +``` +cuno creds --force pair +``` + + +## Listing Credentials + +`list` displays information about imported credential files and paired buckets. + +The command accepts additional options: + +| **Command** | **Description** | +|-------------|-----------------| +| `cuno creds list` | Display all imported credential files and all paired buckets | +| `cuno creds list creds` | Display all imported credential files **only** | +| `cuno creds list pairings` | Display all paired buckets **only** | +| `cuno creds list pairings ` | Display all paired buckets for the specified provider **only** | + +`pairings ` accepts a comma-separated list of providers. + +For example: `cuno creds list pairings gs,s3` will display information about all paired buckets in Google Cloud Storage and AWS S3. + + +## Unpair and Purge Credentials + +To dissociate a container from a credential file, use `unpair`: + +``` +cuno creds unpair [bucket_remote_uri] +``` + +If the second argument is missing, then existing pairings are listed. + +To completely remove a credential file and delete all its associations with remote containers, use `purge`: + +``` +cuno creds purge [imported_credential_file] +``` + +If the second argument is missing, then existing credential files are listed. + + +## Additional Flags + +Object Mount’’s `cuno creds` command accepts additional flags: + +``` +cuno creds [flags] +``` + +These flags are described in the following table: + +| **Flag** | **Description** | +|----------|-----------------| +| `--interactive/-i` | Asks for permission to replace existing credential files, pairings, SAS, etc. +| `--force/-f` | Always replaces existing credential files, pairings, SAS, etc. +| `--color/-c` | Displays using colored output. + + +## Alternative Authentication Methods + +{% callout type="warning" %} + **Native Object Mount Credential Management is Recommended** + + For improved security, it is highly recommended to use the `cuno cred` features as described above for all credential administration and management. + + Only use the following methods if additional flexibility is required. +{% /callout %} + +### Amazon AWS S3: Using Native Credentials + +Object Mount can also use the file supplied by Amazon AWS S3: + +``` +~/.aws/credentials +``` + +With the profile optionally specified with the environment variable: + +``` +AWS_PROFILE +``` + +Set the AWS region with the environment variable: + +``` +AWS_REGION +``` + +Alternatively, directly set the following two environment variables before launching Object Mount: + +``` +AWS_ACCESS_KEY_ID +AWS_SECRET_ACCESS_KEY +``` + +### Amazon AWS S3: Using IAM Roles for EC2 + +IAM Roles provide an alternative method to securely distribute AWS credentials across EC2 instances. +These credentials can be used for accessing AWS S3. Refer to the AWS article 🌐 [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) for more information. + +Object Mount will automatically detect IAM roles on an EC2 instance, so no further configuration is necessary. + +{% callout type="warning" %} + **Do Not Combine Methods** + + Combined use of Object Mount managed credentials with IAM Roles in EC2 is not currently supported. +{% /callout %} + +### Google Cloud Storage: Using Environment Variable + +Object Mount can directly use service account credentials, which are commonly used by applications to access Google Cloud Storage. + +To use a service account credential file (in JSON or PKCS12 format), specify a value for the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. + +For example: A credential file located in `/home/user/gckey.json` can be loaded to Object Mount by using: + +``` +export GOOGLE_APPLICATION_CREDENTIALS="/home/user/gckey.json" +``` + +### Azure Storage: Using Environment Variables + +To access Azure Storage without using Object Mount credentials management, provide a storage access key associated with a Microsoft Azure Storage account. + +To enable this access scheme, set the `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_ACCESS_KEY` environment variables: + +``` +export AZURE_STORAGE_ACCOUNT="" +export AZURE_STORAGE_ACCESS_KEY="" +``` + +{% callout type="warning" %} + **Not Recommended** + + Microsoft recommends that storage access keys **not** be shared. + + To permit access to storage resources _without_ sharing access keys, see above for how to use a [Shared Access Signature (SAS)](#microsoft-azure-storage-with-shared-access-signatures) instead. + + An SAS gives users access to a container for a limited time period with a fixed set of permissions. +{% /callout %} diff --git a/app/(docs)/object-mount/linux/user-guides/k8s/page.md b/app/(docs)/object-mount/linux/advanced/k8s/page.md similarity index 56% rename from app/(docs)/object-mount/linux/user-guides/k8s/page.md rename to app/(docs)/object-mount/linux/advanced/k8s/page.md index 13238519b..e5fbf2a76 100644 --- a/app/(docs)/object-mount/linux/user-guides/k8s/page.md +++ b/app/(docs)/object-mount/linux/advanced/k8s/page.md @@ -1,52 +1,66 @@ --- title: Kubernetes CSI Driver +hideTitle: false docId: zohm4zeXohpae9ga - +weight: 5 metadata: title: Kubernetes CSI Driver description: Kubernetes CSI Driver Guide - -weight: 10 +hidden: false --- +The Object Mount Kubernetes CSI Driver facilitates seamless integration of your S3 cloud storage services within a Kubernetes cluster. -The Object Mount CSI Driver facilitates seamless integration of your cloud storage services (Amazon S3, Google Cloud, and Azure Cloud) within a Kubernetes cluster. The driver is available through [Helm](https://helm.sh) under `oci://registry-1.docker.io/cunofs/cunofs-csi-chart`. More information can be found [on docker hub](https://hub.docker.com/r/cunofs/cunofs-csi-chart). +The driver is available through the [Helm](https://helm.sh) package manger under: -## Install + - `oci://registry-1.docker.io/cunofs/cunofs-csi-chart`. -1. Ensure that Helm is installed. If not, follow the [Helm installation guide](https://helm.sh/docs/intro/install/) -2. Deploy the Object Mount CSI Driver: +More information can be found on [Docker hub](https://hub.docker.com/r/cunofs/cunofs-csi-chart). -```shell -helm install cunofs-csi-chart oci://registry-1.docker.io/cunofs/cunofs-csi-chart \ - --set cunofsLicense.license="" \ - --set credsToImport="{,, ... ,}" -``` -- `--set cunofsLicense.license`: (required) Object Mount license -- `--set credsToImport`: (optional) cloud credentials [[more details](../getting-started/configuring-credentials)] +## Installing Helm & CSI Driver -For licensing, please schedule a [discovery call](https://www.storj.io/landing/get-in-touch). +1. Ensure that Helm is installed. + + - If not, follow the [Helm installation guide](https://helm.sh/docs/intro/install/) -3. Display the status of the Object Mount CSI Driver resources: +2. Deploy the Object Mount CSI Driver. -```shell -kubectl get all -l app.kubernetes.io/name=Object Mount-csi-driver -``` + ```shell + helm install cunofs-csi-chart oci://registry-1.docker.io/cunofs/cunofs-csi-chart \ + --set cunofsLicense.license="" \ + --set credsToImport="{,, ... ,}" + ``` -{% callout type="note" %} -For security reasons, helm doesn't allow access to files via paths. -Therefore, you need to provide the credential file contents in `credsToImport`, and not the paths. -To ensure that the cloud credentials are passed correctly, please provide them in `base64` encoding. For example: + - `--set cunofsLicense.license`: (required) Object Mount license key ([more details](docId:BN6yJrwasPnZsaf)) + - `--set credsToImport`: (optional) S3 cloud credentials ([more details](docId:E4NhE5kPdjURRajJ)) -```shell ---set credsToImport="{$(cat creds-1.txt | base64), $(cat creds-2.json | base64)}" -``` -{% /callout %} +3. Display the status of the Object Mount CSI Driver resources. + + ```shell + kubectl get all -l app.kubernetes.io/name=Object Mount-csi-driver + ``` -## Update + {% callout type="note" %} + **Base64 Encoding of Credentials** -Upgrade to the latest version: + For security reasons, `helm` does not allow access to files via paths. + + Therefore, you need to provide the credential file contents in `credsToImport`, and not via paths. + + To ensure that the cloud credentials are passed correctly, they should be `base64` encoded. + + For example: + + ```shell + --set credsToImport="{$(cat creds-1.txt | base64), $(cat creds-2.json | base64)}" + ``` + {% /callout %} + + +### Updating Helm + +To upgrade `helm` to the latest version, use: ```shell helm upgrade --reuse-values cunofs-csi-chart 'oci://registry-1.docker.io/cunofs/cunofs-csi-chart' @@ -54,22 +68,25 @@ helm upgrade --reuse-values cunofs-csi-chart 'oci://registry-1.docker.io/cunofs/ You can append the `--version ` to upgrade to a specific version. -## Uninstall + +### Uninstalling Helm + +To uninstall `helm`, use: ```shell helm uninstall cunofs-csi-chart ``` -# Storage allocation +## Storage Allocation -The Object Mount CSI Driver support the following strategies: +The Object Mount CSI Driver supports the following Storage Allocation strategies: -- [Static storage provisioning](#static_provisioning) -- [Dynamic storage provisioning](#dynamic_provisioning) +- Static Storage Provisioning +- Dynamic Storage Provisioning -## Static provisioning +### Static Storage Provisioning -To allocate storage statically, define one or more `PV` ([PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)) providing the bucket details and options: +To allocate storage **statically**, define one or more `PV` (🌐 [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)) providing the bucket details and options: ```yaml apiVersion: v1 @@ -93,7 +110,7 @@ spec: readonly: "true" # optional ``` -Then, define a `PVC` ([PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)): +Then, define a `PVC` (🌐 [Persistent Volume Claim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#lifecycle-of-a-volume-and-claim)): ```yaml apiVersion: v1 @@ -132,9 +149,9 @@ spec: claimName: cunofs-pvc # PVC metadata.name ``` -## Dynamic provisioning +### Dynamic Storage Provisioning -To allocate storage dynamically, define a [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) providing the bucket details and options: +To allocate storage **dynamically**, define a 🌐 [Storage Class](https://kubernetes.io/docs/concepts/storage/storage-classes/) providing the bucket details and options: ```yaml apiVersion: storage.K8s.io/v1 @@ -193,7 +210,7 @@ spec: claimName: cunofs-pvc # PVC metadata.name ``` -Alternatively, cluster users can create a generic inline volume which doesn't require a `PVC`: +Alternatively, cluster users can create a generic inline volume which doesn’t require a `PVC`: ```yaml apiVersion: v1 @@ -224,18 +241,20 @@ spec: storage: 16Ei # ignored but required ``` -{% callout type="note" %} -The current version of Object Mount CSI Driver does not support [CSI inline volumes](https://kubernetes.io/blog/2020/01/21/csi-ephemeral-inline-volumes/) +{% callout type="note" %} + **CSI Ephemeral Inline Volumes** + + Currently, Object Mount’s CSI Driver does not support 🌐 [CSI Ephemeral Inline Volumes](https://kubernetes.io/blog/2020/01/21/csi-ephemeral-inline-volumes/). {% /callout %} -# Configuration +## Configuring Helm & CSI Driver This section offers additional details about the configuration options for the Object Mount CSI driver. -## Helm chart +### Helm Chart -The Helm chart can be also installed, configured and deployed manually. +Helm chart can be also installed, configured and deployed manually. Download the chart manually: @@ -250,7 +269,7 @@ Set the `cunofsLicense.license` variable and import the cloud credentials: cunofsLicense: license: "" credsToImport: - - "" + - “" - "" - "<..>" - "" @@ -262,29 +281,32 @@ Finally, install the local chart by pointing to its directory: helm install cunofs-csi-chart ``` -Available options: +**Available Options:** -| Yaml Value | Description | Default Value | -|------------|-------------|---------------| +| **Yaml Value** | **Description** | **Default Value** | +|----------------|-----------------|-------------------| | `driverName` | Optionally change the name of the deployed driver. Only useful if you want to deploy several instances of the driver. | `cunofs.csi.com` | | `cunofsCSIimage.pullPolicy` | Specifies how the docker image is deployed onto the Node and Controller. Only useful to change if self-hosting the docker image. | `Always` | | `cunofsCSIimage.name` | Specifies the Object Mount CSI docker image. Only useful to change if self-hosting the docker image under a different name (Note: do not include the version here). | `cunofs/cunofs_csi` | -| `cunofsCSIimage.version` | Specifies the docker image's version. No need to change it unless you have a good reason to. | | -| `cunofsLicense.license` | The license used for activating Object Mount on the Driver. It needs to be a valid Professional or Enterprise license. | | +| `cunofsCSIimage.version` | Specifies the docker image’s version. No need to change it unless you have a good reason to. | | +| `cunofsLicense.license` | The license used for activating Object Mount on the Driver. It needs to be a valid Enterprise license. | | | `credsToImport` | Yaml array that you can populate with your s3/az/gs credential files. | | | `rbac.useRBAC` | Enables out of the box support for RBAC clusters (deploys the required ClusterRole/ClusterRoleBinding). | `true` | | `eks.iam_arn` | On Amazon EKS, associates IAM role to `ServiceAccount`. | | -## PersistentVolume options +**`PersistentVolume` Options:** + +{% callout type="warning" %} + **Boolean Parameters** -{% callout type="warning" %} -Note that due to K8s parameter passing design decisions, the boolean parameters require strings and not yaml booleans. -For this reason, please use `"true"` and `"false"` instead of `true` and `false`. + Due to K8s parameter passing design decisions, the boolean parameters require strings and not yaml booleans. + + For this reason, use `"true"` and `"false"` instead of `true` and `false`. {% /callout %} -| Yaml Value | Description | -|------------|-------------| +| **Yaml Value** | **Description** | +|----------------|-----------------| | `metadata.name` | Can be any legal, unique name | | `spec.capacity.storage` | This value is ignored, but is required to be set by the CSI specification | | `spec.csi.driver` | Set this to the name of the CSI driver you deployed which is `cunofs.csi.com` by default | @@ -300,35 +322,41 @@ For this reason, please use `"true"` and `"false"` instead of `true` and `false` | `spec.csi.volumeAttributes.CUNO_LOG` | Sets the `CUNO_LOG` of the `cuno mount` | | `spec.csi.volumeAttributes.fusion_pvc` | Enables Object Mount Fusion on the `PV` with the input as the backing `PVC` | -## StorageClass options +**`StorageClass` Options:** -| Yaml Value | Description | -|------------|-------------| -| `metadata.name` | Can be any name as long as it's unique | + **Yaml Value** | **Description** | +|----------------|-----------------| +| `metadata.name` | Can be any name as long as it’s unique | | `provisioner` | The name of the driver, by default: `cunofs.csi.com` | | `reclaimPolicy` | `Retain` will not delete the generated `PVs` and their storage when the `PVCs` go out of scope, `Delete` will | | `parameters.cloud-type` | Can be `s3`, `az` or `gs` | | `parameters.bucket` | The bucket used to create volumes | -| `parameters.bucket-subdir` | Optional. The subdirectory of the bucket where the `PVCs` will get generated. Can be nested subdirectories like "dir/other_dir/yet_another_dir" | +| `parameters.bucket-subdir` | Optional. The subdirectory of the bucket where the `PVCs` will get generated. Can be nested subdirectories like “dir/other_dir/yet_another_dir” | | `parameters.{posix, allow_root, allow_other, auto_restart, readonly, CUNO_OPTIONS, CUNO_LOG}` | These options will be passed down to the generated `PV` and behave the same way as described in the `PV` options | | `parameters.fusionStorageClass` | Tells Object Mount Fusion to use the given `StorageClass` to allocate backing `PVs` | -# Object Mount Fusion Support -The Object Mount CSI Driver, as of version `v1.0.2`, supports all access modes. -However, using a single `PV` on multiple pods at once with the `ReadWriteMany` access mode, on the same node or not, does not guarantee write consistency to the same file. -We offer a way around this limitation by using Object Mount Fusion. +## Object Mount Fusion Support + +The Object Mount CSI Driver, as of version `v1.0.2`, supports all Object Mount [Deployment Modes](docId:jqySXsBfj5WGUjgs) + +However, using a single `PV` on multiple pods at once with the `ReadWriteMany` access mode (on the same node or not) does not guarantee write consistency to the same file. + +We offer a way around this limitation by using [Object Mount Fusion](docId:GVT7eXEaMSZCWESj). + +Object Mount Fusion enables users to get the best out of object storage and traditional shared filesystems at the same time. In the case of the Object Mount CSI Driver, it gives you the ability of writing to a `PV` with `ReadWriteMany` without potential issues with multiple writers. + +Object Mount Fusion uses object storage with a supporting backing filesystem and will intelligently use the filesystem that has the best performance for large or small files, while ensuring that reads and writes remain ordered. -Object Mount Fusion enables users to get the best out of object storage and traditional shared filesystems at the same time. -In the case of the Object Mount CSI Driver, it gives you the ability of writing to a `PV` with `ReadWriteMany` without potential issues with multiple writers. -Object Mount Fusion uses object storage with a supporting backing filesystem and will intelligently use the filesystem that has the best performance for large or small files, while ensuring that read and writes remain ordered. -For more information, please refer to [the Object Mount Fusion documentation](../getting-started/object-mount-fusion). -## Static Allocation +### Static Allocation + +Using Object Mount Fusion under [static storage provisioning](#static-storage-provisioning), you need to deploy two `PVs`: one for Object Mount, and one for the backing mount. + +Choose a backing mount that offers write consistency (Amazon EFS, Amazon EBS, a local NFS server, etc.), and deploy it with a `PV` and `PVC`, as if it was used by a `Pod`. + +Then, refer to the `PVC`’s name in the `spec.csi.volumeAttributes.fusion_pvc` parameter of the Object Mount `PV`. -For static allocation, you need to deploy two `PVs`: one for Object Mount, and one for the backing mount. -Choose a backing mount that offers write consistency (Amazon EFS, Amazon EBS, a local NFS server, etc...), and deploy it with a `PV` and `PVC`, as if it was used by a `Pod`. -Then, refer to the `PVC`'s name in the `spec.csi.volumeAttributes.fusion_pvc` parameter of the Object Mount `PV`. The Object Mount CSI Driver will mount the `PV` to itself and bind the two filesystems. ```yaml @@ -364,16 +392,21 @@ spec: fusion_pvc: "backing-pvc" # gets the name of the pvc to try and mount to it ``` -{% callout type="warning" %} -Please ensure that the `PV`/`PVC` pair you create has the same access mode as the Object Mount `PV`/`PVC` pair and compatible parameters (readonly, etc...). +{% callout type="warning" %} + **Matching Access Modes** -If you have any issues deploying the Object Mount Fusion `PV`/`PVC` pair, pelase first ensure that the backing pair is correctly set up. + Please ensure that the `PV`/`PVC` pair you create has the same access mode as the Object Mount `PV`/`PVC` pair and compatible parameters (readonly, etc.). + + If you have any issues deploying the Object Mount Fusion `PV`/`PVC` pair, first ensure that the backing pair is correctly set up. {% /callout %} -## Dynamic Allocation -The Object Mount CSI Driver supports dynamic provisioning of Fusion `PV` pairs. -Simply deploy a backing `StorageClass` and refer to it in the Object Mount `StorageClasse's` `parameters.fusionStorageClass` parameter. +### Dynamic Allocation + +The Object Mount CSI Driver supports dynamic provisioning of Object Mount Fusion `PV` pairs. + +Simply deploy a backing `StorageClass` and refer to it in the Object Mount `StorageClass` `parameters.fusionStorageClass` parameter. + The Object Mount CSI Driver will use it to generate and delete backing `PVs` alongside Object Mount `PVs` and bind them as needed. ```yaml @@ -396,20 +429,25 @@ parameters: fusionStorageClass: "backing-sc" # Refer to the backing StorageClass ``` -{% callout type="warning" %} -Please ensure that the deployed `StorageClass` can create `PVs` that bind to `PVCs` with `ReadWriteMany` of size `8Gi`. -In a future release, you will be able to parameterise it. +{% callout type="warning" %} + Please ensure that the deployed `StorageClass` can create `PVs` that bind to `PVCs` with `ReadWriteMany` of size `8Gi`. + In a future release, you will be able to parameterize it. {% /callout %} -# RBAC Support + +## RBAC Support By default, the Object Mount CSI Driver deploys a `ServiceAccount`, `ClusterRole` and `ClusterRoleBinding` to support Role-Based Access Control (RBAC) out of the box. -They are respectively deployed under the following names: `-serviceaccount`, `-clusterrole` and `-clusterrolebinding`. +They are respectively deployed under the following names: + + - `-serviceaccount` + - `-clusterrole` + - `-clusterrolebinding`. This also means that the Object Mount CSI Driver supports Amazon EKS clusters out-of-the-box. -You can choose not to deploy the `ClusterRole` and `ClusterRoleBinding` by setting the `rbac.useRBAC` property to `false` in the `values.yaml` file: +You can choose _not_ to deploy the `ClusterRole` and `ClusterRoleBinding` by setting the `rbac.useRBAC` property to `false` in the `values.yaml` file: ```shell helm install cunofs-csi-chart oci://registry-1.docker.io/cunofs/cunofs-csi-chart \ @@ -418,50 +456,60 @@ helm install cunofs-csi-chart oci://registry-1.docker.io/cunofs/cunofs-csi-chart --set rbac.useRBAC=false ``` -# Technical Details +## Technical Details -The Object Mount CSI Driver abides by the [Kubernetes Container Storage Interace standard](https://github.com/container-storage-interface/spec/blob/master/spec.md). +The Object Mount CSI Driver abides by the 🌐 [Kubernetes Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) standard. It implements the `Node`, `Controller` and `Identity` plugins and uses sidecar containers for simplifying its deployment and maintenance. -The CSI Driver is shipped into one binary, which can act as the `Node` or the `Controller` depending on the context (how it's deployed and which sidecar containers are connected). -The helm chart deploys docker containers that have this binary preinstalled. -The `Node` plugin refers to the ability to mount and organise existing `PersistentVolumes` on a Kubernetes node. +The CSI Driver is shipped into one binary, which can act as the `Node` or the `Controller` depending on the context (how it is deployed and which sidecar containers are connected). + +The Helm chart deploys Docker containers that have this binary preinstalled. + +The `Node` plugin refers to the ability to mount and organize existing `PersistentVolumes` on a Kubernetes node. + The `Controller` plugin implements the ability to create `PersistentVolumes` dynamically through a `StorageClass`. The `Node` and the `Controller` need to handle logic at different levels: -- The `Node` plugin needs to be deployed on every K8s Node, since it handles mounting logic that's specific to each machine on which the application containers run. Therefore, it is deployed via a K8s [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). Additionally, these sidecar containers are shipped with the `Node`: +**Node:** - [Liveness Probe](https://kubernetes-csi.github.io/docs/livenessprobe.html) +- The `Node` plugin needs to be deployed on every K8s Node, since it handles mounting logic that’s specific to each machine on which the application containers run. Therefore, it is deployed via a K8s 🌐 [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). Additionally, these sidecar containers are shipped with the `Node`: - : This sidecar container ensures that the driver remains responsive, and replaces the driver container on crash. + 🌐 [Liveness Probe](https://kubernetes-csi.github.io/docs/livenessprobe.html) - [Node Driver Registrar](https://kubernetes-csi.github.io/docs/node-driver-registrar.html) + This sidecar container ensures that the driver remains responsive, and replaces the driver container on crash. - : This sidecar container registers the driver to the kubelet to simplify its discovery and communication. + 🌐 [Node Driver Registrar](https://kubernetes-csi.github.io/docs/node-driver-registrar.html) -- The `Controller` plugin needs to be unique across a Kubernetes cluster, since it handles the lifecycle of `PersistentVolumes`, which are K8s global objects. It is therefore managed through a K8s [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/): + This sidecar container registers the driver to the kubelet to simplify its discovery and communication. - [Liveness Probe](https://kubernetes-csi.github.io/docs/livenessprobe.html) +**Controller:** - : This sidecar container, like with the Node plugin, ensures that the driver remains responsive, and replaces the driver container on crash. +- The `Controller` plugin needs to be unique across a Kubernetes cluster, since it handles the lifecycle of `PersistentVolumes`, which are K8s global objects. It is therefore managed through a K8s 🌐 [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/): - [External Provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html) + 🌐 [Liveness Probe](https://kubernetes-csi.github.io/docs/livenessprobe.html) - : This sidecar container helps the driver interacting with the K8s API by listening to volume provisioning-related calls. + This sidecar container, like with the Node plugin, ensures that the driver remains responsive, and replaces the driver container on crash. + + 🌐 [External Provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html) + + This sidecar container helps the driver interacting with the K8s API by listening to volume provisioning-related calls. + +During the deployment, the Object Mount CSI Driver deploys the Object Mount **license** and cloud **credentials** as 🌐 [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). -During the deployment, the Object Mount CSI Driver deploys the Object Mount license and cloud credentials as [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). The license `Secret` is imported by the `Node` and the `Controller` through an environment variable. -The credential `Secret` is mounted to the `Node` and the `Controller` through a [Projected Volume](https://kubernetes.io/docs/concepts/storage/projected-volumes/) and sequentially imported by Object Mount. +The credentials `Secret` is mounted to the `Node` and the `Controller` through a 🌐 [Projected Volume](https://kubernetes.io/docs/concepts/storage/projected-volumes/) and sequentially imported by Object Mount. -# Limitations + +## Limitations Not every existing `K8s` optional feature is currently implemented in this driver. -Please contact us via our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) for specific feature inquiries. -- Due to the internals of K8s, the Object Mount CSI Driver makes use of `Object Mount on FUSE` as a backend instead of regular `Object Mount`. This means that performance will be high, but not always as high as a regular `Object Mount` installation. -- Not every `Object Mount` option is currently available for use in the driver. Please refer to the [configuration section](#configuration-section) for the available options. -- The `ReadWriteMany` access mode doesn't guarantee write consistency without Object Mount Fusion -- The `Object Mount` CSI Driver currently doesn't support CSI Ephemeral Volumes, raw block volumes, volume snapshotting, volume expansion, volume cloning and volume topology options. +Please contact our [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for specific feature inquiries. + + - Due to the internals of K8s, the Object Mount CSI Driver makes use of `Object Mount on FUSE` as a backend instead of regular `Object Mount`. This means that performance will be high, but not always as high as a regular `Object Mount` installation. + - Not every `Object Mount` option is currently available for use in the driver. Please refer to the [configuration section](#configuring-helm-and-csi-driver) for the available options. + - The `ReadWriteMany` access mode doesn’t guarantee write consistency without Object Mount Fusion. + - The `Object Mount` CSI Driver currently doesn’t support CSI Ephemeral Volumes, raw block volumes, volume snapshotting, volume expansion, volume cloning and volume topology options. diff --git a/app/(docs)/object-mount/linux/advanced/limitations/page.md b/app/(docs)/object-mount/linux/advanced/limitations/page.md new file mode 100644 index 000000000..b30e17d3f --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/limitations/page.md @@ -0,0 +1,81 @@ +--- +title: Technical Limitations +hideTitle: false +docId: Chiec3aS2eizieng +weight: 7 +metadata: + title: Technical Limitations + description: + Technical Limitations in Object Mount for Linux +hidden: false +--- + +{% callout type="note" %} + **Local File Access Unaffected** + + Object Mount imposes no restrictions on local file accesses. + + The limitations listed in this section only apply to data stored on object storage. +{% /callout %} + +{% callout type="note" %} + **Technical vs. License-based Limitations** + + This document contains technical limitations applying to all Object Mount for Linux users. + + There are additional limitations dictated by your license tier - set up a [discovery call](https://www.storj.io/landing/get-in-touch) for more information on licensing and pricing. +{% /callout %} + + +## Direct Interception + +Direct Interception Mode (using either Object Mount CLI or `LD_PRELOAD`) does not currently support SUID binaries, or certain packaged applications like 🌐 [Snap](https://snapcraft.io/about), 🌐 [AppImage](https://appimage.org/), or 🌐 [Flatpak](https://docs.flatpak.org/en/latest/introduction.html). + +Future updates are planned to improve this. + +If you need to use such applications you should deploy either [Object Mount on FUSE](docId:ZdvWLcm9uFmM5HLk) or [Object Mount FlexMount](docId:cFUt9zgCRFFDk5Sq). + + +## Maximum Object Size + +Depending on your Object Storage provider, Object Mount has a limitation on the maximum file size it can store on a remote location. + +The following table indicates the maximum file sizes per provider: + +| **Cloud Provider** | **Maximum File Size** | +|----------------------|-----------------------| +| AWS S3 | 5 TB +| Google Cloud Storage | 5 TB +| Azure Storage | 4.77 TB + + +## Ownership and Permissions + +If left at the default, file ownership, permissions and timestamps are set as follows when using POSIX [Core File Access](docId:cbm3PcQXmLpuYcbg#core-file-access) Mode: + + - The owner of the remote objects will be reported as the **current user**, and remote file permissions will be set to `777`. + - The creation time of directories is displayed as the Unix Epoch (00:00:00 UTC on 1 January 1970). + +These can be overridden using `CUNO_OPTIONS`. See the [Ownership and Permissions](docId:phohPoowequie5ji#ownership-and-permissions) section in the Advance Guide article: Advanced Configuration Options for details. + + +## Directories in Azure + +Creating a directory in Azure Storage (using `mkdir`) will result in a remote blob called `\` to be displayed inside the created directory when the user is using the GUI/File Explorer that the Azure Portal provides. + +However, `ls` and all CLI commands will behave as expected, showing the correct directory name. + + +## Tab Auto-Completion + +Tab-based auto-completion and wildcard characters are fully supported within Object Mount-wrapped shells. + +Wrapped-shell can be launched using either the `cuno` command or using `LD_PRELOAD` (e.g. `LD_PRELOAD=/usr/lib/cuno.so bash`). + +In the latter, paths containing colons (such as `s3://bucket`) on cloud paths will only succeed if `:` is removed from the separator list variable `COMP_WORDBREAKS`. + +For example: + +```bash +"$LD_PRELOAD" =~ cuno ]] && export COMP_WORDBREAKS=${COMP_WORDBREAKS/:/} +``` diff --git a/app/(docs)/object-mount/linux/advanced/loading/page.md b/app/(docs)/object-mount/linux/advanced/loading/page.md new file mode 100644 index 000000000..a1e8a4192 --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/loading/page.md @@ -0,0 +1,378 @@ +--- +title: Advanced Loading Options +hideTitle: false +docId: airoogh4Waengi8u +weight: 2 +metadata: + title: Advanced Loading + description: + Advanced Loading +hidden: false +--- + +## Loading Object Mount + +This section describes advanced usage of Object Mount, including the user-mode library, scripting, containerization, and high-performance computing (HPC) integration. + + +## Via CLI + +You can use the `cuno` command to launch a new default Linux shell that is “wrapped” by Object Mount running in Direct Interception Mode: + +``` +user:~$ cuno +``` + +You should see the `(cuno)` CLI prefix to the left of your user prompt. This indicates that a new shell has been launched, wrapped in Object Mount’s `cuno` wrapper: + +``` +user:~$ cuno +(cuno) user:~$ +``` + +The shell, and any command or application executed from within this shell, will be monitored by Object Mount. Any I/O calls will be intercepted, transformed and redirected to object storage as needed. + +Enter `exit` at any time to close the Object Mount CLI console and return to the unwrapped shell prompt: + +``` +(cuno) user:~$ exit +user:~$ +``` + +## Via User-Mode library: LD_PRELOAD + +As mentioned in the Object Mount for Linux [Introduction](docId:wxtofwqcb5f2), Object Mount provides a User-mode Library, `cuno.so`. This dynamic loader can be pre-loaded into a running process using the `LD_PRELOAD` environment variable. + +Set the environment variable `LD_PRELOAD="$CUNO_ROOT"/cuno.so` before executing a command and Object Mount will be enabled for that command. + +{% callout type="note" %} + **`CUNO_ROOT`** + + The `CUNO_ROOT` environment variable should be set to wherever your Object Mount installation is: + + - `/opt/cuno` for a system-wide installation + - `$HOME/.local/opt/cuno` for a user-local installation (generally) + + Display your `CUNO_ROOT` variable: + ```sh + echo $CUNO_ROOT + ``` + + If it is blank, first set the `CUNO_ROOT` environment variable: + + 1. Locate your deployment by running: + + ```sh + ls /opt/cuno + ls $HOME/.local/opt/cuno + ``` + + One of those directories should display multiple cuno-related files. + + 2. Set your CUNO_ROOT variable (for the current shell session) by running: + ```sh + export CUNO_ROOT="/opt/cuno" + -or- + export CUNO_ROOT="$HOME/.local/opt/cuno" + ``` + + See [Installation Locations](docId:ahWohd5eegh6eizi#installation-locations-scripted-installers) for more information on installation locations. +{% /callout %} + +After verifying that your `CUNO_ROOT` variable is set to the `cuno` installation directory, you can start a `bash` shell wrapped by `cuno.so` with the following command: + +```console +LD_PRELOAD="$CUNO_ROOT"/lib/cuno.so bash +``` + +The `bash` shell, and any command executed from within this shell instance, will run with Object Mount enabled. Other processes and applications will not be affected. + +**Note:** Loading the `cuno.so` library in this manner will **not** display the `(cuno)` prefix to the prompt. + +Example: Browse satellite images on Google Cloud Storage: +```sh +ls gs://gcp-public-data-landsat/ +``` + +### Additional Environment Variables + +In addition to `CUNO_ROOT`, you may consider setting the following environment variables for convenience: + +| **Environment Variable** | **Description** | **Value** | +|--------------------------|-----------------|-----------| +| `LD_PRELOAD`| Load the Object Mount library | "${CUNO_ROOT}"/lib/cuno.so${LD_PRELOAD+:$LD_PRELOAD} +| `PATH` | Path-free access to Object Mount executables | "${CUNO_ROOT}"/bin:${PATH} +| `MANPATH` | Easy access to Object Mount manual pages | "${CUNO_ROOT}"/share/man:${MANPATH} + + +## Via Shell Profiles + +To _always_ load Object Mount when starting a new interactive shell, append the following line to `~/.bashrc` or `~/.zshrc`: + +```console +export LD_PRELOAD="${CUNO_ROOT}"/lib/cuno.so` +``` + +`/etc/profile.d` contains application-specific startup scripts, which are executed at user login by their respective login shells. + +A sample script is provided in `${CUNO_ROOT}/etc/profile.d/cunorc.sh`. + +Copy this to `/etc/profile.d/` to enable Object Mount for all users. + + +## Via Docker + +There are multiple methods to use Object Mount from within Docker containers. + +### Automatic Docker Interception + +To enable automatic Docker interception, set the environment variable `CUNO_INTERCEPT_DOCKER=1` and load Object Mount. +Launching a Docker container via `docker run …` will make Object Mount available inside the container. + +{% callout type="warning" %} + **Automatic Interception within Docker** + + Support for automatic Docker interception is currently **experimental**. + + For ways of running Docker with Object Mount interception manually, see Manual Interception below. +{% /callout %} + +For instance, the following code will run a command inside an Ubuntu container with Object Mount enabled with the hosts Object Mount credentials: + +```console +export CUNO_INTERCEPT_DOCKER=1 +cuno run docker run --rm ubuntu:latest ls -l s3://bucket +``` + +Object Mount options will be forwarded into the container. + +To override Object Mount options just for Docker containers, use the `CUNO_DOCKER_OPTIONS` environment variable. If set, its value overrides that of `CUNO_OPTIONS` inside Docker containers. + +### Manual Docker Interception + +Besides automatic interception, the following manual methods can be used to enable Object Mount in Docker containers: + + - Install Object Mount as part of the creation of the Docker image. + - Inject Object Mount at launch into an existing Docker image. + +To inject Object Mount into a Docker container when it is launched: + + 1. Set `CUNO_CREDENTIALS` outside the container + + - See [Credential File Location](docId:aish4shuiheeZaig#credential-file-location) for details. + + 2. If desired, the `CUNO_OPTIONS` environment variable can be set using the Docker `--env` option when using `docker run` + + - See [Set environment variables (--env)](https://docs.docker.com/reference/cli/docker/container/run/#env) in the Docker documentation. + + 3. Use the following options when using `docker run`: + + ```console + docker run \ + --tmpfs /cunodb \ + -v $CUNO_ROOT:/opt/cuno:ro \ + -v /opt/cuno/etc/ld.so.preload:/etc/ld.so.preload:ro \ + -v $CUNO_CREDENTIALS:/opt/cuno-config/creds:ro \ + [container-commands] + ``` + + {% callout type="note" %} + This uses the volume mount option (`-v`) to make Object Mount and other directories available to the container. + + This command requires Object Mount already be installed and [activated](docId:BN6yJrwasPnZsaf), with a valid license key, on the host system. + {% /callout %} + +The Object Mount credentials directory is only readable by the current user for security reasons. + +However, credentials may need to be accessed by processes run by other users (e.g. NGINX worker processes) within the Docker container. + +To enable this, bind mount the nested `bindpoint` directory instead of the credentials directory: + +```console +docker run \ + --tmpfs /cunodb \ + -v $CUNO_ROOT:/opt/cuno:ro \ + -v /opt/cuno/etc/ld.so.preload:/etc/ld.so.preload:ro \ + -v $CUNO_CREDENTIALS/bindpoint:/opt/cuno-config/creds/bindpoint:ro \ + [container-commands] +``` + +## Within Kubernetes Clusters + +You can use Object Mount on Kubernetes clusters with the Object Mount Kubernetes CSI Driver. + +Refer to the Advanced Guide article: [Kubernetes CSI Driver](docId:zohm4zeXohpae9ga). + + +## With Sylabs Singularity + +There are multiple methods for enabling 🌐 [Sylabs Singularity](https://sylabs.io/) images with Object Mount, allowing transparent access to cloud hosted files. + +You can: + +- Install Object Mount as part of the creation of the Singularity image +- Inject the Object Mount library at launch into an existing Singularity image + +The following Singularity definition file will install Object Mount into an image: + +``` +Bootstrap: docker +From: rockylinux:8 + +%files + /home/admin/downloads/cuno-{FULL-VERSION}.x86_64.rpm /opt/src/ + +%post + yum update -y + yum install -y /opt/src/cuno-{FULL-VERSION}.x86_64.rpm + echo "YOUR LICENSE KEY HERE" | cuno creds activate + chmod og+r /opt/cuno/etc/license + +%environment + export LD_PRELOAD=/usr/lib/cuno.so + export CUNO_CREDENTIALS=/opt/cuno-config/creds + +%labels + Name object mount + URL storj.io + Email supportdcs@storj.io +``` + +The above will require that the Object Mount software package is available at the path’s starting `/home/admin/downloads` as listed in the `%files` section. + +Rather than modifying an existing Singularity image, another option is to pass-through the Object Mount library and binaries making them available inside the image. + +Optionally, the `CUNO_OPTIONS` environment variable can be set for a particular mode of operation. + +The following commands demonstrate injecting Object Mount into a Singularity image: + +``` +$ singularity exec \ + --bind $CUNO_ROOT:/opt/cuno \ + --bind $CUNO_ROOT/etc/ld.so.preload:/etc/ld.so.preload \ + --bind "$CUNO_CREDENTIALS":/opt/cuno-config/creds \ + ./singularity_image.sif ls s3://commoncrawl + +cc-index crawl-002 hive_analysis meanpath projects wikipedia +contrib crawl-analysis index2012 parse-output robots.txt +crawl-001 crawl-data mapred-temp parse-output-test stats-output +``` + +{% callout type="note" %} + **Activated License Required** + + This method requires Object Mount be installed and [activated](docId:BN6yJrwasPnZsaf), with a valid license key, on the system that is used to execute the Singularity images. +{% /callout %} + +Alternatively, rather than using command options, the `--bind` parameters can be specified as environment variables before the `singularity` binary is called, making it easier to modify existing pipelines without changing the command line call, like so: + +``` +export SINGULARITY_BIND="$CUNO_ROOT:/opt/cuno,$CUNO_CREDENTIALS:/opt/cuno-config/creds" +``` + +As another alternative, Object Mount can be used via a FUSE mount inside Singularity with the `--fusemount` parameter: + +``` +$ singularity exec \ + --bind "$CUNO_ROOT":/opt/cuno \ + --bind "$CUNO_CREDENTIALS":/opt/cuno-config/creds \ + --fusemount 'container:/opt/cuno/bin/cuno mount --root /cuno /cuno' \ + ./singularity_image.sif /cuno/s3/commoncrawl +``` + +This will allow software inside the container to access cloud buckets via the `/cuno` path prefix. + + +## Using the Lmod Environment Module System + +The [Lmod](https://lmod.readthedocs.io/en/latest/) Environment Module System is a tool to help users manage shell environments. It allows groups of related environment variable settings to be made or removed dynamically. + +Many HPC environments incorporate `module` as one of the means by which software can be loaded by their users. + +A sample script has been provided within your Object Mount installation directory here: `${CUNO_ROOT}/share/modulefiles/cuno/{FULL-VERSION}.lua`. + +It is built specifically for the **Lmod** implementation in **Lua**, and uses values set to example paths (see the `base` variable in particular) and default options that you might want to append to, modify, or remove to suit your environment and your installation choices. + +After adding the parent directory of the file to the `MODULEPATH` with `module use "${CUNO_ROOT}/share/modulefiles/cuno"`, a user should be able to run `module load cuno` to add Object Mount to their environment. + + +## With Spectrum LSF + +Depending on your exact requirements and setup, there are multiple ways of using IBM’s [Spectrum LSF](https://www.ibm.com/products/hpc-workload-management) HPC Workload Management suite with Object Mount. + +First, assuming that Object Mount is installed on all compute nodes, ensure that the `cuno` executable is available in the `PATH`, and that Object Mount credentials are configured for all nodes. + +Jobs can be submitted by prefixing the submitted command with `cuno run` (without quotes). + +For example: + +``` +bsub -Is cuno run ls -l s3://bucket +``` + +If further configuration of Object Mount is required, an LSF job starter can be created. + +The script should be created at the location `$LSF_ENVDIR/cuno-starter.sh` or some other location that is common to all LSF nodes. (**Note:** The queue template that ships with Object Mount assumes this location). + +The following script can be used as an outline for the job starter: + +```shell +#!/bin/sh + +cuno_basedir= +# e.g.: +# cuno_basedir=/opt/cuno + +# Further configuration as necessary, e.g.: +# export CUNO_CREDENTIALS=/usr/share/cuno-creds +# export CUNO_OPTIONS=+cloudroot=acme + +"$cuno_basedir"/bin/cuno run "$@" +``` + +Make the job starter executable: + +```console +chmod +x "$LSF_ENVDIR"/cuno-starter.sh +``` + +To configure a new LSF queue for Object Mount enabled jobs, fill in and append the LSF configuration template for Object Mount to `lsb.queues`. + +Use the following command: + +``` +cluster_name=$(lsid | grep 'cluster name' | awk '{print $NF}') +cuno run sh -c " + sed 's;@LSF_ENVDIR@;$LSF_ENVDIR;' \"\$CUNO_BASEDIR\"/share/lsf/lsb.queues \ + >>\"$LSF_ENVDIR\"/lsbatch/$cluster_name/configdir/lsb.queues +" +``` + +Afterwards, LSF needs to be reconfigured: + +``` +badmin reconfig +``` + +To check that the queue was set up correctly, run: + +``` +bqueues -l cuno +``` + +Regardless of whether an LSF queue was set up, the job starter can be used with the `lsrun` command (but **not** `bsub`). + +For example: + +```console +export LSF_JOB_STARTER=$LSF_ENVDIR/cuno-starter.sh +lsrun ls -l s3://bucket +``` + +To use the dedicated queue, specify it when invoking `bsub`. + +For example: + +```console +bsub -q cuno -Is ls -l s3://demo +``` diff --git a/app/(docs)/object-mount/linux/advanced/ml/page.md b/app/(docs)/object-mount/linux/advanced/ml/page.md new file mode 100644 index 000000000..f9d5941b2 --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/ml/page.md @@ -0,0 +1,112 @@ +--- +title: Machine Learning Accelerator +hideTitle: false +docId: aiwoo2Gohshah8Zo +weight: 6 +metadata: + title: Machine Learning Accelerator + description: + Machine Learning Accelerator Guide +hidden: false +--- + +The Object Mount **Machine Learning Accelerator** (Object Mount MLA) is a software library that accelerates existing machine learning models by leveraging Object Mount’s enhanced data retrieval and storage technology. + +{% callout type="note" %} + **Enterprise Feature** + + The Object Mount MLA library is only available to Object Mount for Linux users with an **Enterprise License Key**. +{% /callout %} + + +## Requirements + +Python 3.7 or higher is required to use the Object Mount Machine Learning Accelerator. + +If you require previous versions of Python 3, reach out to our [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new). + + +## Supported Functions + +The Object Mount MLA has been tested with the following data loading functions: + + - `open` + - `numpy.loadfile` + - `numpy.load` + - `numpy.loadtxt` + - `PIL.Image.open` + - `torchaudio.load` + +It may also work with additional data loading functions. + +Contact our [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for assistance with using additional functions. + + +## Installation and Usage + +1. **Installation:** + + To use Object Mount MLA you myst install the dependencies in your environment. + + Use `pip` to do so: + + ```console + pip install -r "/requirements.txt" + ``` + +2. **Set Environment Variables:** + + Then set the following environment variables: + + - `CUNO_SPEEDUP_PYTHON=1` + - `PYTHONPATH="${PYTHONPATH+:$PYTHONPATH}"` + + The `CUNO_SPEEDUP_PYTHON` variable signals Object Mount MLA to enable specific Machine Learning optimizations. + + The `PYTHONPATH` alteration causes Python to load the Object Mount MLA’s `sitecustomize.py` file first. + +When finished, you can begin using Object Mount. All standard Object Mount deployment modes are support, including Direct Interception and Object Mount on FUSE. + +Use the examples below for details on using Object Mount within your specific deployment mode. + +### Object Mount MLA with Direct Interception Mode + +Object Mount’s Direct Interception Mode is easily activate by running the `cuno` command to invoke a new Object Mount-wrapped shell. + + ```shell + # terminal + cuno + (cuno) $ + ``` + +If using the Object Mount CLI (`cuno`) is _not_ an option, you can use `cuno run` to enable Object Mount for a _single command_. + +You can also set the `LD_PRELOAD` environment variable directly: e.g.: `LD_PRELOAD=/lib/cuno.so`. + +See the “User-Mode library: LD_PRELOAD” section in the Advanced Guide article: [Advanced Loading Options](docId:airoogh4Waengi8u#via-user-mode-library-ld-preload) for additional details on using `LD_PRELOAD`. + +**Example Usage:** + +Activating a shell with Object Mount as an example: + +```console +$ export CUNO_SPEEDUP_PYTHON=1 +$ export PYTHONPATH="${PYTHONPATH+:$PYTHONPATH}" +$ cuno +(cuno) $ python3 demo/verify.py +``` + +### Object Mount MLA with Object Mount on FUSE + +To use Object Mount MLA with Object Mount on FUSE you will need to configure your Python scripts to look for data _inside_ the mount. + +**Example Usage:** + +Mount an S3 bucket named `dataset_bucket` at the location `$HOME/cloudmount`: + +```console +export CUNO_SPEEDUP_PYTHON=1 +export PYTHONPATH="/dataset_preload2${PYTHONPATH+:$PYTHONPATH}" +cuno mount --root s3://dataset_bucket "$HOME/cloudmount" +python3 dataset_preload2/demo/verify.py --location "$HOME/cloudmount/training_set_1" +``` diff --git a/app/(docs)/object-mount/linux/advanced/page.md b/app/(docs)/object-mount/linux/advanced/page.md new file mode 100644 index 000000000..c108770d1 --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/page.md @@ -0,0 +1,31 @@ +--- +title: Advanced Guides +hideTitle: true +docId: GeWpcQTuVpL6d3wU +weight: 4 +metadata: + title: Feature Guides for Linus + description: + Details on unique features of Object Mount for Linux +hidden: false +--- + +# Advanced Guides for Linux + +The articles in this Advanced Guides section provide expert-level configuration and deployment advise regarding Object Mount for Linux. + +**Articles include:** + + - Advanced Guides + - Advanced Credential Options + - Advanced Loading Options + - Advanced Configuration Options + - Server-Side Encryption + - Kubernetes CSI Driver + - Object Mount’s Machine Learning Accelerator + - Technical Limitations + + + + + diff --git a/app/(docs)/object-mount/linux/user-guides/extraopts/page.md b/app/(docs)/object-mount/linux/advanced/server-side-encryption/page.md similarity index 64% rename from app/(docs)/object-mount/linux/user-guides/extraopts/page.md rename to app/(docs)/object-mount/linux/advanced/server-side-encryption/page.md index 3e8292ac4..8036d8c0d 100644 --- a/app/(docs)/object-mount/linux/user-guides/extraopts/page.md +++ b/app/(docs)/object-mount/linux/advanced/server-side-encryption/page.md @@ -1,41 +1,50 @@ --- -title: Additional Options +title: Server-Side Encryption +hideTitle: false docId: aex8oakeudeepeRa - +weight: 4 +redirects: + - /object-mount/linux/user-guides/extraopts metadata: - title: Additional Options - description: Additional Options - -weight: 8 + title: Server-Side Encryption + description: + Server-Side Encryption +hidden: false --- -# Additional Options +Object Mount fully supports server-side encryption for both streaming to and from remote locations. -## Server-side encryption +That includes server-side encryption with customer-provided encryption keys (SSE-C) which allows users to set their own encryption keys. -Object Mount fully supports server-side encryption for both streaming to and from remote locations. -That includes server-side encryption with customer-provided encryption keys (SSE-C) that allows users to set their own encryption keys. Users are responsible for managing their encryption keys as well as providing them as part of every remote request. + As a result, users have to provide custom keys to Object Mount. -### SSE in AWS S3 + +## Server-Side Encryption in AWS S3 Object Mount fully supports the default behavior in AWS to always encrypt the user data on the server side, before it is written to disk. -Object Mount also supports customer-supplied encryption using AES-256 keys. For more information see the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html). + +Object Mount also supports customer-supplied encryption using AES-256 keys. + +For more information see the AWS article: 🌐 [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html). In AWS S3, users specify their encryption options by setting the `AWS_S3OPTIONS` environment variable. + The available encryption options are the same as those provided by `awscli`. -For example, the following: + +For example: ```console export AWS_S3OPTIONS="--sse-write-c AES256 --sse-write-c-key xxxxxxxxxxxxxxxxxx" ``` -enables AES256 server-side encryption with a customer-provided key. -All available options follow: +This enables AES256 server-side encryption with a customer-provided key. + +Available options include: -| Option | Description | -|-----------|-------------| +| **Option** | **Description** | +|------------|-----------------| | `--sse [string]` | Specifies server-side encryption of the object in S3. Valid values are `AES256` and `aws:kms`. The default keys are used in **encryption/decryption**. | | `--sse-read-c [string]` | This parameter should only be specified when **reading** an S3 object that was encrypted server-side with a customer-provided key. It specifies the algorithm to use when **decrypting** the source object. `AES256` is the only valid value. | | `--sse-read-c-key [string]` | This parameter should be specified when **reading** an S3 object that was encrypted server-side with a customer-provided key. Specifies the customer-provided encryption key for Amazon S3 to use to **decrypt** the source object. The encryption key provided must be one that was used when the source object was created. `--sse-read-c` must be specified as well. The key provided should not be base64 encoded. | @@ -43,30 +52,46 @@ All available options follow: | `-sse-write-c-key [string]` | The customer-provided encryption key to use to server-side **encrypt** the object in S3. If you provide this value, `--sse-c` must be specified as well. The key provided should not be base64 encoded. | | `--sse-kms-key-id [string]` | The AWS KMS key ID that should be used to server-side **encrypt** the object in S3. Note that you should only provide this parameter if the KMS key ID is different from the default S3 master KMS key. | -Note that AWS KMS keys are required only for writing encrypted objects. Reading of encrypted objects does not require the corresponding KMS key to be specified. Furthermore, only one of `--sse-write-c-key` and `--sse-kms-key-id` options can be enabled at a time. To use encryption with alternate paths the corresponding `ALT_AWS_S3OPTIONS` environment variable must be set. +{% callout type="note" %} + **AWS Key Management System** + + AWS KMS keys are required **only** for writing encrypted objects. Reading of encrypted objects does not require the corresponding KMS key to be specified. + + Furthermore, only one of `--sse-write-c-key` and `--sse-kms-key-id` options can be enabled at a time. To use encryption with alternate paths the corresponding `ALT_AWS_S3OPTIONS` environment variable must be set. +{% /callout %} + + +## Server-Side Encryption in Google Cloud Storage -### SSE in Google Cloud Storage +Object Mount fully supports the default behavior in Google Cloud Storage to always encrypt the user data on the server side, before it is written to disk. -Object Mount fully supports the default behavior in Google Cloud Storage to always encrypt the user data on the server side, before it is written to disk. In addition to this option, Object Mount also supports customer-supplied encryption using AES-256 keys. For more information see: +In addition to this option, Object Mount also supports customer-supplied encryption using AES-256 keys. -> [https://cloud.google.com/storage/docs/encryption/customer-supplied-keys](https://cloud.google.com/storage/docs/encryption/customer-supplied-keys) +For more information see the GCS article: 🌐 [Customer-supplied encryption keys](https://cloud.google.com/storage/docs/encryption/customer-supplied-keys) -To enable the SSE-C option the user must set the `GS_ENCRYPTION` environment variable. For example: +To enable the SSE-C option the user must set the `GS_ENCRYPTION` environment variable. + +For example: ``` export GS_ENCRYPTION="--sse-write-c-key xxxxxxxxxxxxxxxxxx" ``` -enables AES256 server-side encryption with the specified customer-provided key. +This enables AES256 server-side encryption with the specified customer-provided key. -Available options for encryption using customer-provided keys: +Available options include: -| Parameter | Description | -|-----------|-------------| +| **Parameter** | **Description** | +|---------------|-----------------| | `--sse-read-c-key [string]` | This parameter should be specified when **reading** a remote object on Google Cloud Storage that was encrypted server-side with a customer-provided key. Specifies the customer-provided encryption key for Google Storage to use to decrypt the source object. The encryption key provided must be one that was used when the source object was created. | -| `--sse-write-c-key [string]` | The customer-provided encryption key to use to server-side encrypt the object in Google Cloud Storage when **writing**. The key provided should not be base64 encoded.

Object Mount is also fully compatible with Google's cloud-hosted key management service (KMS). Customer-managed keys used in encryption are provided `GS_ENCRYPTION` environment variable using the `--sse-kms-key-id` option described below. | +| `--sse-write-c-key [string]` | The customer-provided encryption key to use to server-side encrypt the object in Google Cloud Storage when **writing**. The key provided should not be base64 encoded.

Object Mount is also fully compatible with Google’s cloud-hosted key management service (KMS). Customer-managed keys used in encryption are provided `GS_ENCRYPTION` environment variable using the `--sse-kms-key-id` option described below. | | `--sse-kms-key-id [string]` | The Cloud KMS key that should be used to server-side encrypt the object in Google Cloud Storage. Note that you should only provide this parameter if KMS key ID is different than the default KMS key associated with the remote bucket.Note that Cloud KMS keys are required only for **writing** encrypted objects. Reading of encrypted object does not require the corresponding Cloud KMS key to be specified. Furthermore, only one of `--sse-write-c-key` and `--sse-kms-key-id` options can be enabled at a time. All supported encryption schemes can be applied to alternate paths, e.g. `altgs://mybucket/location`. To that extent, the corresponding `ALT_GS_ENCRYPTION` environment variable must be set. | -### SSE in Microsoft Azure -Server-side encryption is always enabled at rest in Azure [for all containers](https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption). It is the default, and cannot be disabled. +## Server-Side Encryption in Microsoft Azure + +Server-side encryption is always enabled at rest in Microsoft Azure containers + +See the Microsoft Azure article: 🌐 [Azure Storage encryption for data at rest](https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption). + +It is the default, and cannot be disabled. diff --git a/app/(docs)/object-mount/linux/advanced/uninstall/page.md b/app/(docs)/object-mount/linux/advanced/uninstall/page.md new file mode 100644 index 000000000..72291a9e4 --- /dev/null +++ b/app/(docs)/object-mount/linux/advanced/uninstall/page.md @@ -0,0 +1,119 @@ +--- +title: Uninstalling Object Mount +hideTitle: false +docId: Oochi9lee7ookaih +weight: 13 +metadata: + title: Uninstallation + description: + Uninstallation Guide +hidden: false +--- + +To remove Object Mount for Linux from a workstation, follow one of these two uninstall procedures depending on how Object Mount was originally installed. + +For **Scripted Installations** (`glibc` and `musl`): + + 1. Determine the installation root folder + 2. Remove files for either one user or all users + 3. Remove any shell customizations + +For **Package Manager Installations** (Debian, Red Hat, Alpine): + + 1. Run the appropriate package manager uninstallation command (see below). + + +## Uninstall for: Scripted Installations + +If Object Mount was installed using a Scripted Installer, follow the steps below to uninstall Object Mount. + +### Step 1: Determine the Installation Root + +The `CUNO_ROOT` environment variable should be set to wherever your Object Mount installation is. + +This directory is usually: + + - `/opt/cuno` for a system-wide installation + - `$HOME/.local/opt/cuno` for a user-local installation (generally) + +Display your `CUNO_ROOT` variable: + +```sh +echo $CUNO_ROOT +``` + +If it is blank or not set, locate your deployment by running: + +```sh +ls /opt/cuno +ls $HOME/.local/opt/cuno +``` + +One of those directories should display multiple cuno-related files. + +**See:** [Installation Locations](docId:ahWohd5eegh6eizi#installation-locations-scripted-installers) for more information on installation locations. + +## Step 2: Remove the CUNO_ROOT Directory + +Delete the CUNO_ROOT directory and all files and subfolders within that CUNO_ROOT directory. + +In addition, you can remove the following added files: + +**For Single User/Local Uninstallation:** + +Remove the following additional files: + +``` +~/.local/lib/x86_64-linux-gnu/cuno.so +~/.local/bin/cuno +~/.local/share/man/man1/cuno.1 +~/.local/share/man/man1/cuno-creds.1 +~/.local/share/man/man1/cuno-mount.1 +~/.local/share/man/man8/cuno.so.8 +``` + +**For System-wide/Global Uninstallation:** + +Remove the following additional files: + +``` +/usr/lib/cuno.so +/usr/bin/cuno +/usr/share/man/man1/cuno.1 +/usr/share/man/man1/cuno-creds.1 +/usr/share/man/man1/cuno-mount.1 +/usr/share/man/man8/cuno.so.8 +``` + +### Step 3: Shell Customization Removal + +Remove any customizations made to shell initialization scripts such as + + - `~/.bashrc` + - `~/.profile` + - `.zshrc` + - `/etc/profile.d` + - etc. + + +## Uninstall for: Package Managers + +If Object Mount was installed using a Package Manager, use the commands in the following sections to uninstall Object Mount for specific Linux distributions. + +### Debian derivatives (e.g. Ubuntu) + +```console +sudo apt-get remove cuno\* +``` + +### RedHat derivatives (e.g. CentOS) + +```console +sudo yum remove cuno\* +``` + +### Alpine Linux derivatives + +```console +sudo apk del cuno +``` diff --git a/app/(docs)/object-mount/linux/getting-started/_meta.json b/app/(docs)/object-mount/linux/getting-started/_meta.json deleted file mode 100644 index 68b4567f0..000000000 --- a/app/(docs)/object-mount/linux/getting-started/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Getting Started", "weight": 2} diff --git a/app/(docs)/object-mount/linux/getting-started/configuration-modes/page.md b/app/(docs)/object-mount/linux/getting-started/configuration-modes/page.md deleted file mode 100644 index 74f1d19fc..000000000 --- a/app/(docs)/object-mount/linux/getting-started/configuration-modes/page.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Configure Object Mount -docId: hoeSh4ieSh6Ees9v -weight: 3 -metadata: - title: Configure Object Mount - description: - Each time you set up Object Mount in a new compute environment, you will need to assess the requirements of your workloads and your end goals. ---- -## Overview - -Each time you set up Object Mount in a new compute environment, you will need to assess the requirements of your workloads and your end goals. - -By default, Object Mount is configured for core object storage access in a LAN setting. If you have one of the following specialized use cases, follow the relevant quick start guide: - -- High-performance attached storage upgrade (EFS upgrade): [Object Mount Fusion](../getting-started/object-mount-fusion) -- Strict POSIX-controlled access to object storage: [Enforced Posix Access](../getting-started/enforced-posix-access) - -### Which mode is for me? - -When configuring Object Mount, it is important to understand the needs of the tools and workloads you wish to run. -There are many configuration options which can be reviewed in our user guide as well in the `cuno help` pages. You can change your settings for each workflow/tool, or you can set some options on the bucket-level to impose a usage mode for all users. - -There are two independent aspects you need to consider when configuring Object Mount: - -- The requirements of your tools in order to work *correctly*. -- The needs of your tools in different environments to work *efficiently*. - -For **correct operation**, you need to know what level of POSIX compatibility your tools require to function. -For **efficient operation** your primary concerns are the bandwidth/latency between the compute and object storage resources, as well as the behaviours of your tools. - -## Correct operation - Levels of POSIX compatibility - -Some applications have limited requirements from a POSIX perspective and only needs the filesystem for basic operations like renaming, reading, or writing new data. -Other applications may require a fully POSIX compatible interface, which requires additional work from the compatibility layer. Because full compatibility requires extra metadata to be written to object storage, and may slightly affect performance we make this optional. - -The main levels of POSIX compatibility are: - -- [Core File Access](#core-file-access) - no users/groups/symlinks/hardlinks/permissions control/file modes -- [POSIX File Access](#posix-file-access) - POSIX-compatible access with users/groups/symlinks/hardlinks/permission control/file modes -- [POSIX Enforced File Access](#posix-enforced-file-access) - as above but POSIX access control is enforced - -{% callout type="note" %} -How to check if you need POSIX File Access - -- Check the outputs correspond to the same workload when run on the local file system. If there are randomised elements to workload (such as sampling in machine learning use cases), then the seeds need to be fixed to make such a comparison. -- If the application running under Object Mount with Core File Access fails with `(134) ENOTSUP` (not supported) or `Operation not permitted` then it is likely that POSIX File Access needs to be enabled. -{% /callout %} - -### Core File Access - -Objects as files, files as objects; this is the **default mode**. Does not support the persistence or modification of POSIX users, groups, symlinks, hard links, permissions control or file modes attributes on objects. - -- Use this when you don’t have any metadata requirements and your tools don't need any POSIX metadata persistence to function correctly. -- Use this when you don't have any write access to the bucket in question, or you do not want to create any Object Mount-internal objects there. - -#### Example use-cases - -- Use this when interacting with data you already have on object storage and only require access to the names and data of those objects. For example, if you have machine learning datasets in the cloud and you have previously configured your libraries to read them directly from object storage. - -#### How to enable - -This is the default mode. - -### POSIX File Access - -This mode will maintain POSIX metadata for your objects, but it won't enforce any permissions or modes set on the objects. That means any user can use Object Mount to read or write any file that their object storage credentials give them access to. - -See [enforced posix](../installation/enforced-posix). - -#### Example use-cases - -- Use this when your applications are dependent on the preservation of POSIX metadata (owner/group permissions, change/modify times, etc.) or POSIX “links” (symlinks or hardlinks). -- Use this if you’re moving workflows from POSIX to object storage, such as workloads that were previously run on EC2 with EFS. - -{% callout type="note" %} -We don't currently support POSIX ACLs or extended attributes on the cloud. Please get in contact with us via our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) if you need these features. -{% /callout %} - -#### How to enable - -There are two main ways to enable this. If the object storage provider supports setting tags at the bucket level, then POSIX compatibility mode can be enabled using the command `cuno creds setposix s3://yourbucket true`. This will affect everyone using the bucket and force all Object Mount users of that bucket into POSIX compatability mode. Otherwise it can be enabled manually by a user setting the environment variable `export CUNO_POSIX=1` (valid per-session). - -### POSIX Enforced File Access - -See [enforced posix](../installation/enforced-posix) - -#### Example use-cases - -- Host a website using NGINX (or other server technologies) entirely backed by object storage, without any attached storage device (such as EFS). This mode lets you do this while controlling which files/directories the NGINX process (as the `nginx` user) can access. -- Host an organisation's user filesystem in the cloud. - -{% callout type="note" %} -We don't currently support POSIX ACLs or extended attributes on the cloud. Please get in contact with us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) if you need these features. -{% /callout %} - -#### How to enable - -Please refer to the getting started [guide](../getting-started/enforced-posix-access). - -## Efficient operation in your environments - -### Where is the client? - -#### Accessing object storage from within the same high-speed network - -If the client is in the same high-speed network as the object storage system, the connection is considered *LAN* (local-area network) access. Examples include: - -- The client is a cloud-hosted EC2 instance / virtual machine within the same **region** as the cloud data. -- The client is in the same LAN network as an on-premises object storage cluster and has a high-speed, low-latency connection linking the client to the storage. - -If this is the case, consider what the tools behaviours are: - -- If the workload requires *both* high IOPS (such as database operations), and high throughput, consider using [Object Mount Fusion](../getting-started/object-mount-fusion): combining the best of high-speed attached storage and object storage. -- Otherwise, no additional configuration is necessary. - -#### Accessing object storage from remote networks - -If the client is outside the network of the object storage system, the connection is considered *WAN* (wide-area network). Examples include: - -- Accessing buckets from a cloud-hosted EC2 instance / virtual machine in a *different* region to the bucket being accessed. -- A home computer accessing cloud-hosted object storage. -- Accessing cloud storage of a different cloud provider. - -If this is the case, consider: - -- If the client needs to repeatedly read the same parts of a file, consider using client-side caching on a fast local disk. - -{% callout type="note" %} -Client-side caching on disk for workloads requiring many reads of the same data, such as video editing, is coming soon. -Follow along [here](https://www.storj.io/fuse-filesystem). -{% /callout %} diff --git a/app/(docs)/object-mount/linux/getting-started/configuring-credentials/page.md b/app/(docs)/object-mount/linux/getting-started/configuring-credentials/page.md deleted file mode 100644 index 906f86709..000000000 --- a/app/(docs)/object-mount/linux/getting-started/configuring-credentials/page.md +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: Configuring Credentials -docId: heefureS2iubahpi -weight: 4 -metadata: - title: Configuring Credentials - description: - Object Mount can connect to any of the major cloud storage providers using the native storage credentials. ---- -## Authentication - -Object Mount can connect to any of the major cloud storage providers using the native storage credentials. - -If you have an S3-compatible object storage solution from another provider, you will first need to configure S3 API access. - -{% callout type="note" %} -If you are trying to access S3 through an EC2 instance configured with an IAM role, no further configuration is needed and Object Mount will automatically authenticate using the AWS-managed configuration. -You can skip to [Enabling Object Mount](./enabling-object-mount). -{% /callout %} - -## Getting your credentials - -{% callout type="warning" %} -Any credentials you use need to have sufficient permissions in order for Object Mount to discover and manage your data. This needs to include permission to list buckets. If this is not possible or desired, you must use the `cuno creds pair` options, for which instructions can be found [here](user-guide-pair-containers). -{% /callout %} - -{% tabs %} -{% tab label="AWS S3" %} - You will need the Access Key ID and Access Key Secret for an AWS IAM user with permission to access the S3 buckets you want to use. You will have stored these credentials somewhere when first creating the IAM user. - - Alternatively, create a new IAM user with "programmatic access" (access using keys), by following the AWS User Guide: [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html)`. - For further options and alternatives, consult our full guide on [accessing S3 object storage](../user-guides/credentials#amazon-web-services-s3). -{% /tab %} -{% tab label="Microsoft Azure" %} - You will need the account name, and the associated access key. - - To set up a new account, see [Create a storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create)`. - - If you already have an account and need the access key, see [View account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys). - - Alternatively, Object Mount supports using Shared Access Signatures (SAS) to Azure containers. To do this, see [Accessing Azure containers using SAS](../user-guides/credentials#microsoft-azure-blob-storage-with-shared-access-signatures). -{% /tab %} -{% tab label="Google Cloud Storage" %} - You will need key-based access to be associated with your [Google Cloud service account](https://developers.google.com/workspace/guides/create-credentials#service-account). - - If you need to set this up, you can have Google generate a new key by following Google's guide, [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete). Alternatively, you can generate your own key pair and upload it on the service account's key management page, by following the guide [Upload service account keys](https://cloud.google.com/iam/docs/keys-upload). - - If you've generated a new key, store the JSON file you have downloaded in a safe place with appropriate access permissions, ready for the next steps. -{% /tab %} -{% tab label="S3-compatible" %} - Consult your object storage solution's documentation to find out how to retrieve credentials for authenticating using the S3 API. You will need an Access Key ID, Access Key Secret, and an accessible IP address or domain/subdomain to use as an endpoint. -{% /tab %} -{% /tabs %} - - -## Saving credentials as a file -{% tabs %} -{% tab label="AWS S3" %} - The file needs to be of the form: - - ```shell - aws_access_key_id = xxxxxxxxxxxxxxxxxx - aws_secret_access_key = xxxxxxxxxxxxxxxxxx - ``` - - You can use any text editor to create the file, just remember to change the permissions on the file when you're done to prevent other users from reading it: - - ```shell - # terminal - chmod 0600 "" - ``` - - Alternatively, use our handy one-liner (after editing in your details): - - ```shell - # terminal - touch credentials.txt ; chmod 0600 credentials.txt ; printf "aws_access_key_id = xxxxxxxxxxxxxxxxxx \naws_secret_access_key = xxxxxxxxxxxxxxxxxx" > credentials.txt - ``` -{% /tab %} -{% tab label="Microsoft Azure" %} - The file needs to be of the form: - - ```shell - AZURE_STORAGE_ACCOUNT= xxxxxxxxxxxxxxxxxx - AZURE_STORAGE_ACCESS_KEY= xxxxxxxxxxxxxxxxxx - ``` - - You can use any text editor to create the file, just remember to change the permissions on the file when you're done to prevent other users from reading it: - - ```shell - # terminal - chmod 0600 "" - ``` - Alternatively, use our handy one-liner (after editing in your details): - - ```shell - # terminal - touch credentials.txt ; chmod 0600 credentials.txt ; printf "AZURE_STORAGE_ACCOUNT= xxxxxxxxxxxxxxxxxx \nAZURE_STORAGE_ACCESS_KEY= xxxxxxxxxxxxxxxxxx" > credentials.txt - ``` -{% /tab %} -{% tab label="Google Cloud Storage" %} - If you have an existing JSON/P12 (PCKS#12) file or you downloaded one in Step 1, that will work and you can move on to Step 3. - - If you provided you own private key when creating the service account, you will need to create a JSON file of the form: - - ```shell - { - "type": "service_account", - "project_id": "PROJECT_ID", - "private_key_id": "KEY_ID", - "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n", - "client_email": "SERVICE_ACCOUNT_EMAIL", - "client_id": "CLIENT_ID", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://accounts.google.com/o/oauth2/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL" - } - ``` -{% /tab %} -{% tab label="S3-compatible" %} - The file needs to contain at least: - - ```shell - aws_access_key_id = xxxxxxxxxxxxxxxxxx - aws_secret_access_key = xxxxxxxxxxxxxxxxxx - endpoint=http://127.0.0.1:8080 - ``` - - For more information on endpoint formatting, and additional options, consult the full user guide section for [Authenticating with S3-compatible solutions](../user-guides/credentials#s3-compatible-solutions). - - You can use any text editor to create the file, just remember to change the permissions on the file when you're done to prevent other users from reading it: - - ```shell - #terminal - chmod 0600 "" - ``` - - You can use any text editor to create the file, or use our handy one-liner (after editing in your details): - - ```shell - #terminal - touch credentials.txt ; chmod 0600 credentials.txt ; printf "aws_access_key_id = xxxxxxxxxxxxxxxxxx \naws_secret_access_key = xxxxxxxxxxxxxxxxxx\nendpoint=A.B.C.D:PORT" > credentials.txt - ``` -{% /tab %} -{% /tabs %} - - -## Importing the credentials into Object Mount - -{% callout type="note" %} -The default location for imported credentials is the directory `$XDG_CONFIG_HOME/cuno/creds` (if unset, `$XDG_CONFIG_HOME` defaults to `~/.config`). To use an alternative location, please set the `CUNO_CREDENTIALS` environment variable to point to this path. For example: `export CUNO_CREDENTIALS=/home/user/my-cloud-credentials`. - -Note that you should not insert your credentials directly into these locations because the `cuno creds import` command also creates corresponding bucket entries, and adds appropriate configuration settings (region, URL path style, etc.). -{% /callout %} - -Assuming you have saved your credentials in a file `credentials.txt`, run the following command to add these credentials to the local set of Object Mount managed credentials: - -```sh -cuno creds import credentials.txt -``` - -This command will attempt to discover all the buckets that these credentials have access to, as well as the settings, limitations, and compatibility of these buckets. This may take a while if you have many buckets associated with the credentials you are importing. - -{% callout type="note" %} -If you are using an S3-compatible service and are having problems, you can run a compatibility check: - -```sh -cuno creds detectfeatures s3://bucket-to-test credentials.txt -``` - -This command will test S3-compatibility, settings, and limitations, and then reconfigure credentials based on this. It needs a bucket to be specified that it can write temporary files to for testing purposes. - -{% callout type="warning" %} -Running feature detection will use up to a few gigabytes of bandwidth and may take a few minutes to complete depending on the machine's connection speed and the S3-compatible storage provider. -{% /callout %} -{% /callout %} - -## Testing that your credentials work - -You can immediately test that your credentials work using a private bucket that you are happy to use for these purposes (which we will assume is called `bucket1`). - -{% callout type="note" %} -Optional: after each command, you can confirm that the changes are reflect in your cloud or object storage provider's standard GUI interface. -{% /callout %} - -{% tabs %} -{% tab label="AWS S3" %} - First, ensure that Object Mount is enabled by calling ``cuno``. If you are using a ``bash`` or ``zsh`` terminal, this will prefix your prompt with ``(cuno)``. Otherwise, run ``cuno`` again to see if Object Mount has been successfully enabled (if it has, you will see the output ``INFO: CUNO already loaded``). - - Try listing your paired buckets: - - ```shell - #terminal - ls s3:// - ``` - - Try listing objects: - ```shell - #terminal - ls s3://bucket1/ - ``` - - Try writing an object: - ```shell - #terminal - echo "hello world" > s3://bucket1/helloworld.txt - ``` - - Try reading that file back: - - ```shell - #terminal - cat s3://bucket1/helloworld.txt - ``` - - Try deleting that file: - ```shell - #terminal - rm s3://bucket1/helloworld.txt - ``` -{% /tab %} -{% tab label="Microsoft Azure" %} - First, ensure that Object Mount is enabled by calling ``cuno``. If you are using a ``bash`` or ``zsh`` terminal, this will prefix your prompt with ``(cuno)``. Otherwise, run ``cuno`` again to see if Object Mount has been successfully enabled (if it has, you will see the output ``INFO: CUNO already loaded``). - - Try listing your paired buckets: - ```shell - #terminal - ls az:// - ``` - - Try listing objects: - ```shell - #terminal - ls az://your-azure-storage-account/bucket1/ - ``` - - Try writing an object: - - ```shell - #terminal - echo "hello world" > az://your-azure-storage-account/bucket1/helloworld.txt - ``` - - Try reading that file back: - - ```shell - #terminal - cat az://your-azure-storage-account/bucket1/helloworld.txt - ``` - - Try deleting that file: - - ```shell - #terminal - rm az://your-azure-storage-account/bucket1/helloworld.txt - ``` -{% /tab %} -{% tab label="Google Cloud Storage" %} - First, ensure that Object Mount is enabled by calling ``cuno``. If you are using a ``bash`` or ``zsh`` terminal, this will prefix your prompt with ``(cuno)``. Otherwise, run ``cuno`` again to see if Object Mount has been successfully enabled (if it has, you will see the output ``INFO: CUNO already loaded``). - - Try listing your paired buckets: - - ```shell - #terminal - ls gs:// - ``` - - Try listing objects: - - ```shell - #terminal - ls gs://bucket1/ - ``` - - Try writing an object: - ```shell - #terminal - echo "hello world" > gs://bucket1/helloworld.txt - ``` - - Try reading that file back: - ```shell - #terminal - cat gs://bucket1/helloworld.txt - ``` - - Try deleting that file: - - ```shell - #terminal - rm gs://bucket1/helloworld.txt - ``` -{% /tab %} -{% tab label="S3-compatible" %} - First, ensure that Object Mount is enabled by calling ``cuno``. If you are using a ``bash`` or ``zsh`` terminal, this will prefix your prompt with ``(cuno)``. Otherwise, run ``cuno`` again to see if Object Mount has been successfully enabled (if it has, you will see the output ``INFO: CUNO already loaded``). - - Try listing your paired buckets: - - ```shell - #terminal - ls s3:// - ``` - - Try listing objects: - - ```shell - #terminal - ls s3://bucket1/ - ``` - - Try writing an object: - - ```shell - #terminal - echo "hello world" > s3://bucket1/helloworld.txt - ``` - - Try reading that file back: - - ```shell - #terminal - cat s3://bucket1/helloworld.txt - ``` - - Try deleting that file: - - ```shell - #terminal - rm s3://bucket1/helloworld.txt - ``` -{% /tab %} -{% /tabs %} diff --git a/app/(docs)/object-mount/linux/getting-started/deployment-modes/page.md b/app/(docs)/object-mount/linux/getting-started/deployment-modes/page.md new file mode 100644 index 000000000..b99bebc3a --- /dev/null +++ b/app/(docs)/object-mount/linux/getting-started/deployment-modes/page.md @@ -0,0 +1,50 @@ +--- +title: Deployment Modes +hideTitle: false +docId: bRnfbdNE6d5DaZzW +weight: 3 +metadata: + title: Deployment Options for Linux + description: + Technical details on the multiple Deployment Options for Object Mount on Linux. +hidden: false +--- + +## Deployment Modes + +The purpose of Object Mount is to make working with object storage as fast and seamless as using a local drive. + +But each user’s environment is unique — comprising different applications, workflows, teams, geography, bandwidth, access to local and remote storage options, etc. + +Therefore, Object Mount for Linux is fully configurable and can be deployed in a variety of different modes to best fit your unique situation. + +These deployment modes include: + +1. **Direct Interception Mode** + + In Direct Interception Mode, Object Mount inserts itself between your standard Linux applications and the host Operating System. Object Mount detects and intercepts file I/O calls, then translates and redirects them, as needed, to your object storage provider. + + The Object Mount Command Line Interface (CLI), `cuno`, is used to configure Direct Interception Mode. + +2. **Object Mount on FUSE** + + If Object Mount’s CLI cannot be used, or you require broader support for non-standard Linux applications, Object Mount on FUSE uses Linux FUSE (Filesystem in Userspace) to mount object storage within the Linux file system hierarchy. + + This increases compatibility but can reduce performance. + +3. **Object Mount FlexMount** + + Object Mount FlexMount combines the speed and performance of Direct Interception (#1, above) with the widest compatibility and application support of Object Mount on FUSE (#2, above). + +4. **Object Mount Fusion** + + Object Mount Fusion is used to enhance high-performance, LAN-based _attached_ storage solutions (like Amazon’s Elastic File System - EFS) with the throughput of object storage. + + Examples include: + - The client is a cloud-hosted EC2 instance or virtual machine within the same **region** as the cloud data. + - The client is in the same LAN network as an on-premises object storage cluster and has a high-speed, low-latency connection linking the client to the storage. + + +## Configuring Deployment Modes + +_After you have installed_ Object Mount for Linux, read through the detailed configuration and usage guidelines for these different deployment modes in the Linux User Guide article: [](docId:jqySXsBfj5WGUjgs). diff --git a/app/(docs)/object-mount/linux/getting-started/download-install/page.md b/app/(docs)/object-mount/linux/getting-started/download-install/page.md deleted file mode 100644 index ebc7ac783..000000000 --- a/app/(docs)/object-mount/linux/getting-started/download-install/page.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: Download, Installation and Activation -docId: iethahkeeX5EiJoh -weight: 2 -metadata: - title: Download, Installation and Activation - description: - Object Mount supports the Linux operating system, and can run inside Docker and other virtual environments such as Hyper-V and Virtualbox. ---- -Object Mount supports the Linux operating system, and can run inside Docker and other virtual environments such as Hyper-V and Virtualbox. - -Object Mount offers a "Scripted Installer" which does not *require* root privileges to install. We also offer distributions that can be installed using popular Linux package managers. -All of our releases are [available on GitHub](https://github.com/cunoFS/cunoFS/releases). - -## Downloading and installing - -[Download and Install Instructions](../installation) - -## Licenses and activation -For licensing, please schedule a [discovery call](https://www.storj.io/landing/get-in-touch). - -### Activating your license - -If you have registered, you can activate your license by running the following command and following the interactive steps: - -{% tabs %} -{% tab label="Linux" %} - ```shell - # terminal - cuno creds activate - ``` -{% /tab %} -{% tab label="Mac" %} - ```shell - # terminal - cuno-mac creds activate - ``` - Note: Instructions for cuno-mac users: - - If any arguments are given to ``cuno-mac``, it starts a temporary container with your installation of cuno mounted into it, and runs the command inside the container. - - The first time ``cuno-mac`` is run, a Docker container will be created with Object Mount ready to use and a user set up within the container similar to your local user on your Mac. -{% /tab %} -{% /tabs %} - -## Testing your installation - -You should now be able to run Object Mount from the command line, which you can test out by running the following command: - -{% tabs %} -{% tab label="Linux" %} - ```shell - # terminal - cuno version - ``` -{% /tab %} -{% tab label="Mac" %} - ```shell - # terminal - cuno-mac version - ``` - Note: - - If you have installed Object Mount directly onto a Mac, you must choose between using ``cuno-mac`` and ``cuno`` depending on the environment you are currently in. - - If you're in a Mac Terminal session and have not run ``cuno-mac``, then you must replace ``cuno`` in any instructions with ``cuno-mac``. - - If you have already started a Object Mount container by calling ``cuno-mac``, you will see the ``(cuno)`` prefix on your command line so any instructions using ``cuno`` commands will work as-is. - - To return to macOS, run ``exit``. - - See [user-guide-cuno-mac](../installation/mac) for more information. - - If you have already started a ``cuno-mac`` session: - - ```shell - # terminal - cuno version - ``` - -{% /tab %} -{% /tabs %} - -To test that Object Mount is fully working before connecting your own object storage, you can use it to browse publicly available data. In the following example, we use Object Mount to browse space telescope data hosted on the `stpubdata` Amazon S3 bucket: - - -{% tabs %} -{% tab label="Linux" %} - ```shell - # terminal - cuno run sh -c "ls s3://stpubdata/" - galex hst jwst k2 kepler panstarrs tess - ``` - ```shell - # terminal - cuno run sh -c "ls s3://stpubdata/tess/public/*" - 's3://stpubdata/tess/public/ffi': - s0001 s0005 s0009 s0013 s0017 s0021 s0025 s0029 ... [truncated] - 's3://stpubdata/tess/public/mast': - tess-s0001-1-1-cube.fits tess-s0012-1-4-cube.fits tess-s0023-2-3-cube.fits ... [truncated] - ... [truncated] - ``` - Note that ``cuno run`` is a way to run a single command within a Object Mount context. Most of the time, and for interactive usage, - we recommend starting a "wrapped" shell using the command ``cuno`` - see the [Loading Object Mount](../getting-started/enabling-object-mount#direct-interception-with-object-mount-cli) section of this guide for more information, - or the [equivalent section](../user-guides/basic#direct-interception-with-object-mount-cli) from the full user guide for extra detail. -{% /tab %} -{% tab label="Mac" %} - ```shell - # terminal - cuno-mac run sh -c "ls s3://stpubdata/" - galex hst jwst k2 kepler panstarrs tess - ``` - ```shell - # terminal - cuno-mac run sh -c "ls s3://stpubdata/tess/public/*" - 's3://stpubdata/tess/public/ffi': - s0001 s0005 s0009 s0013 s0017 s0021 s0025 s0029 ... [truncated] - 's3://stpubdata/tess/public/mast': - tess-s0001-1-1-cube.fits tess-s0012-1-4-cube.fits tess-s0023-2-3-cube.fits ... [truncated] - ... [truncated] - ``` - Note that ``cuno-mac run`` is a way to run a single command within a Object Mount context. Most of the time, and for interactive usage, we recommend starting a container using the - command ``cuno-mac`` and running your commands directly inside it - see the [Loading Object Mount](../getting-started/enabling-object-mount#direct-interception-with-object-mount-cli) section of this guide for more information, or the [equivalent section](../user-guides/basic#direct-interception-with-object-mount-cli) from the full user guide for extra detail. - -{% /tab %} -{% /tabs %} - diff --git a/app/(docs)/object-mount/linux/getting-started/enabling-object-mount/page.md b/app/(docs)/object-mount/linux/getting-started/enabling-object-mount/page.md deleted file mode 100644 index 074c13848..000000000 --- a/app/(docs)/object-mount/linux/getting-started/enabling-object-mount/page.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -title: Enabling Object Mount -docId: ooceeNge9iig6uSe -weight: 5 -metadata: - title: Enabling Object Mount - description: - Object Mount Fusion is way to upgrade high-performance attached storage solutions like Amazon Elastic File System (EFS) with the throughput of object storage. It is a **cheaper** and **faster** solution compared to using EFS alone. ---- - -## How to use Object Mount - -By default, Object Mount is configured for [Core File Access](../getting-started/configuration-modes#core-file-access) which does not write additional Object Mount metadata to your buckets. Once you’re up and running, you can change the mode of operation to better suit your requirements. - -For program calls to be routed through Object Mount, they must be either [directly intercepted](#direct-interception-with-object-mount-cli) using the `cuno` command line; or pointed at a [Object Mount on FUSE](#object-mount-on-fuse) (a custom FUSE mount) set up using `cuno mount`. -There is also [Object Mount FlexMount](#object-mount-flex-mount), which makes use of both Direct Interception and a Object Mount on FUSE as a fallback. - -{% callout type="note" %} -If you have a specialised use case, follow the relevant quick start guide: - -- High-performance attached storage upgrade (EFS upgrade): [Object Mount Fusion](../getting-started/object-mount-fusion) -- Strict POSIX-controlled access to object storage: [Enforced Posix Access](../getting-started/enforced-posix-access) - -{% /callout %} - - - -## Direct Interception with Object Mount CLI - -When running in Direct Interception mode, Object Mount will insert itself between your applications and the operating system. It will dynamically redirect relevant filesystem calls made by the application through the API of your object storage solution. As a result, our mapping between filesystem concepts and object storage gives any application instant access to objects as files. - -{% callout type="note" %} -This includes applications you've written yourself - just treat the paths to object storage as local filesystem paths, and Object Mount handles the rest. -{% /callout %} - - -Direct Interception offers the highest performance access to object storage that Object Mount provides. - -{% callout type="warning" %} -Direct interception does not currently support SUID binaries, or certain packaged apps like [Snap](https://ubuntu.com/core/services/guide/snaps-intro), [AppImage](https://appimage.org/), or [Flatpak](https://docs.flatpak.org/en/latest/introduction.html) applications. Future updates are planned to address this. -{% /callout %} - -This mode is best suited for situations in which installed app compatibility is not a concern - for example, when setting up a fixed workflow with Object Mount that can be tested and verified before putting it into production. - -### Usage - -Direct Interception can be enabled per-session or per-command. Enable it for a session by calling `cuno` on the command line: - -```shell -# terminal -cuno -``` - -This will launch a new interactive "wrapped" shell with Object Mount acting in Direct Interception mode. The shell itself has Object Mount intercepting its calls, so every application launched from within it can be intercepted as well as the command line arguments being used. - -{% callout type="note" %} -When the Object Mount CLI is used to launch a new shell, the primary purpose is to start a new shell (whether that's bash, zsh, or whatever else) with `LD_PRELOAD` set to point at `cuno.so`. There is no Object Mount shell binary - it only wraps existing shells with Object Mount pre-loaded. -{% /callout %} - -Inside the shell, object storage is accessed either in path or URI formats: - -{% tabs %} -{% tab label="AWS S3" %} - ```shell - # focus(1) - # terminal - cuno - (cuno)$ ls s3:/// - (cuno)$ ls /cuno/s3// - ``` -{% /tab %} -{% tab label="Microsoft Azure" %} - ```shell - # terminal - # focus(1) - cuno - (cuno)$ ls az://// - (cuno)$ ls /cuno/az/// - ``` -{% /tab %} -{% tab label="Google Cloud Storage" %} - ```shell - # terminal - # focus(1) - cuno - (cuno)$ ls gs:/// - (cuno)$ ls /cuno/gs// - ``` -{% /tab %} -{% tab label="S3-compatible" %} - ```shell - # focus(1) - # terminal - cuno - (cuno)$ ls s3:/// - (cuno)$ ls /cuno/s3// - ``` -{% /tab %} -{% /tabs %} - -See [user-guide-cloud-paths](../user-guides/access) for more information, options and examples of specfiying cloud paths using Object Mount CLI. - -See [User Guide: Direct Interception with Object Mount CLI](../user-guides/basic#direct-interception-with-object-mount-cli) for additional information, including [how it works](../user-guides/basic#how-it-works), -as well as the [benefits and drawbacks](../user-guides/basic#advantages-and-disadvantages) of using Object Mount in this way. - -## Object Mount on FUSE - -Object Mount on FUSE allows you to mount an object storage path in a directory within the local file system hierarchy. This allows you (and any other user with access to the mount) to access object storage as if it were just another local directory. - -Object Mount on FUSE is a FUSE file system that routes calls through the Object Mount object storage back-end. Due to the nature of FUSE file systems, this is usually less performant than Direct Interception. However, you can still expect great performance. - -### Usage - -A default mount can be achieved using: - -```shell -# terminal -mkdir ~/my-object-storage -# terminal -cuno mount ~/my-object-storage -``` - -Any paired object storage buckets will now be accessible through the mount: - -{% tabs %} -{% tab label="Storj" %} -```shell -# terminal -ls ~/my-object-storage/sj// -``` -{% /tab %} -{% tab label="AWS S3" %} -```shell -# terminal -ls ~/my-object-storage/s3// -``` -{% /tab %} -{% tab label="Microsoft Azure" %} -```shell -# terminal -ls ~/my-object-storage/az/// -``` -{% /tab %} -{% tab label="Google Cloud Storage" %} -```shell -# terminal -ls ~/my-object-storage/gs// -``` -{% /tab %} -{% tab label="S3-compatible" %} -```shell -# terminal -ls ~/my-object-storage/s3// -``` -{% /tab %} -{% /tabs %} - - -For directions on unmounting, configuration options and further information, see the [user guide section on Object Mount on FUSE](../user-guides/basic#object-mount-on-fuse). - -## Object Mount FlexMount - -Use Object Mount FlexMount for the widest compatibility and support (including SUID binaries, Snap, AppImage and FlatPak applications using FUSE), combined with the speed of Direct Interception when possible. - -FlexMount is used by setting up a Object Mount on FUSE and then accessing the cloud using Object Mount CLI always "through" the mount path. Object Mount will recognise that the path is a mount and use direct interception where possible for faster access. It will naturally fall back to the Object Mount on FUSE for anything that cannot be directly intercepted. - -### Usage - -A FlexMount is set up as follows: - -1. First, set up a Object Mount on FUSE: - - ```console - $ mkdir "$HOME/my-object-storage" - $ cuno mount "$HOME/my-object-storage" - ``` - -2. Each time the Object Mount CLI is launched, use the `--flex "` parameter, for example: - - ```console - $ cuno --flex "$HOME/my-object-storage" - (cuno) $ ls $HOME/my-object-storage/s3// - ``` - - {% callout type="warning" %} - You cannot use a tilde `~` in your `CUNO_OPTIONS` or `cuno -o` CLI options when setting up a FlexMount as this is something that the shell needs to resolve. You may still use it for your `cuno mount` commands, and subsequent FlexMount operations (such as `ls ~/my-object-storage`). - {% /callout %} - -The same FlexMount can be re-used across multiple Object Mount wrapped shells. - -The `--flex` option can also be used with `cuno run` to run a single command/script with Object Mount enabled, for example: - -```shell -# terminal -cuno run --flex "$HOME/my-object-storage" bash -c "touch $HOME/object_storage_mount/s3//newfile" -``` - -For more information and configuration options, see the [user guide section on Object Mount FlexMount](../user-guides/basic#object-mount-flex-mount). - -## Enabling Object Mount in other environments - -For information on setting up Docker containers with Object Mount pre-loaded, automatically setting up a Object Mount on FUSE at boot, and other containerisation and HPC use-cases, see [Advanced Loading](../user-guides/advanced). diff --git a/app/(docs)/object-mount/linux/getting-started/enforced-posix-access/page.md b/app/(docs)/object-mount/linux/getting-started/enforced-posix-access/page.md deleted file mode 100644 index 6763ba9de..000000000 --- a/app/(docs)/object-mount/linux/getting-started/enforced-posix-access/page.md +++ /dev/null @@ -1,164 +0,0 @@ ---- -title: Setting up Enforced POSIX Access -docId: Eegoo1teiJ8eerae -weight: 6 -metadata: - title: Setting up Enforced POSIX Access - description: - This guide represents the general process for setting up enforced POSIX access. The specific steps may vary depending on your object storage provider. ---- - -This guide represents the general process for setting up enforced POSIX access. The specific steps may vary depending on your object storage provider. Please contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) if this is a use-case that you are interested in and these instructions don't match your expectations or preferences. - -This mode will maintain POSIX metadata for your objects, and will enforce POSIX access controls on those objects. Use this when you want to manage what users have access to based on the UID/GID of their UNIX user and the corresponding POSIX metadata (owner, group, mode) on files. - -Users will encounter `access denied` errors if they try to read or write to a file/directory they haven't been given permission to (by a suitably privileged user doing `chown`, `chgrp` or `chmod`). Note that this is client-side rather than server-side enforcement, and is not enforced using server-side IAM roles or ACL lists. - -See [metadata warning](../installation/enforced-posix) - - -{% callout type="note" %} -We can also provide a hybrid-approach, which does both client-side access management and server-side access management through ACLs. If you need server-side ACL policies to reflect POSIX access controls, contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). -{% /callout %} - -## Key steps - -1. Through your object storage provider, generate access credentials with the highest level of permissions that any user or admin could need, including permissions to [edit bucket tags](../user-guides/configuration#using-a-bucket-tag). On a public cloud supporting IAM, you set up an `admin` IAM user with such credentials. -2. The admin credentials are stored privately and are used to set up a Object Mount on FUSE in an accessible location. -3. Users are only told the path to the mount; they are not given access to `cuno` nor to the admin credentials. - -{% callout type="warning" %} -The approach given here is not suitable if users have the ability to create their own VMs/instances where they can set any POSIX uid/gid. In that case, consider using server-side IAM roles or ACL lists to enforce access controls, -and contacting us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) for advice on how this integrates with Object Mount. -{% /callout %} - -## Key considerations - -- If multiple machines or locations are being used, it is important that user and group IDs are controlled and consistent across all machines. This is because the POSIX metadata is stored in the object storage, and Object Mount will use the same user and group IDs to present the files on all machines. If you require assistance with managing user and group IDs across multiple machines, contact our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). -- Independent IAM roles or credentials for each user are not required. The admin credentials are used to set up the mount, and the users are only given access to the mount. If the user has access to object storage credentials with server-side privileges beyond this, then the user can potentially access or modify objects outside of these POSIX access controls. -- This guide assumes that the bucket to use is empty initially. If you are switching from an ACL-based approach to access management, then you will need to map existing rules to POSIX uids/gids and "apply" those ACLs through POSIX commands like `chown` and `chmod`. - -{% callout type="note" %} -We don't currently support POSIX ACLs or extended attributes on the cloud. Please get in contact with us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) if you need these features. -{% /callout %} - -## How to set up enforced POSIX access - -Assuming you have already [installed](../getting-started/download-install) Object Mount, the following steps will guide you through setting up enforced POSIX access for the users in a generic POSIX-based system. - -### Import your admin credentials - -Follow the steps in [user-guide-import-credentials](../user-guides/credentials#import-credentials) to import your admin credentials. - -You will need to have run `cuno creds import `. After importing, you should ensure that any credentials files have been created (usually in `${XDG_CONFIG_HOME}"/cuno/creds` or `~/.config/cuno/creds`) with appropriately strict permissions so that non-admin cannot read them. - -### Tag the bucket - -We check a [bucket tag](../user-guides/configuration#using-a-bucket-tag) when Object Mount tries to access a bucket to check the POSIX enforcement settings. If your bucket or object storage provider does not support tags on buckets, -you can skip this step and proceed to [enforced-posix-mounting](../getting-started/enforced-posix-access#mount-the-bucket). - -NB: You will need to set the ``--posix`` flag every time you run ``cuno mount``. - -Use `cuno creds setposix true` to set the bucket tag which will automatically enable POSIX File Access whenever this bucket is accessed by any Object Mount installation. This will affect everyone using the bucket and force all Object Mount users of that bucket into enforced POSIX compatability mode. Example: - -```shell -# terminal -cuno creds setposix s3://mybucket true -``` - -On the `true` setting, this mode stores hidden subdirectories inside your object storage directories describing the POSIX metadata (owner/group permissions, change/modify times, etc.) for each file in the directory. This allows Object Mount to present the files in a way that is compatible with POSIX file access semantics. - -If you are on S3, to additionally store the POSIX metadata as object metadata on each individual file in the bucket, use `cuno creds setposix metadata`. We don't normally recommend this, -as it will slow down Object Mount. For more information, see [user-guide-posix-file-access](../user-guides/configuration#posix-file-access) - -{% callout type="note" %} -When a bucket tag enabling POSIX File Access is set, Object Mount Direct Interception, along with any Object Mount on FUSEs/FlexMounts are compelled to operate in POSIX mode while accessing the bucket. However, the mounts will not be able to enfroce POSIX access without the additional `--posix` flag at mount time. - -However, Object Mount Direct Interception necessitates read access to object storage credentials, or some other user-accessible way to authenticate with the object storage (e.g. an IAM role on EC2). This implies that Direct Interception is not appropriate for enforcing POSIX, as users can easily circumvent this by using the accessible credentials or IAM role with a tool other than Object Mount (such as awscli or directly through an S3 API library). Since FlexMounts are dependent on Object Mount Direct Interception, they are also unsuitable for use-cases that require POSIX enforcement. -{% /callout %} - -### Mount the bucket - -Mount a bucket in a location that is accessible to the users who need access to the files. This can be a shared location, or a location that is only accessible to the user who needs access. In this guide, we will use a shared location, `/mnt/cloud/bucket`, as the mount point. - -When setting up the mount as an admin user, to allow for sudo access, please use the `-o allow_root` flag, and to allow other users to access the mount as themselves plase use `-o allow_other`. - -```console -$ cuno mount --posix -o allow_root -o allow_other --root s3://mybucket /mnt/cloud/bucket -``` - -Users will now be able to see the files in the bucket at `/mnt/cloud/bucket`, and POSIX permissions persisted by Object Mount for files in that bucket will be respected. You should now treat this as any other POSIX file system, and set permissions as you need to control access. - -## Common patterns - -Below are some common behaviours and patterns that you might want to implement when using enforced POSIX access. For more examples, see [user-guide-posix-examples](../user-guides/configuration#usage-examples). - -### Setting default permissions for new files - -On POSIX systems, you would normally do this using `umask`. To determine the umask value you want to set, subtract the value of the permissions you want from `666` (for a file) or `777` (for a directory). - -For example, if `user1` wants the default permissions on new files to be `rw-r--r-- 644`, they would subtract `644` from `666`, giving `022`. - -```console -$ umask 022 -$ touch /mnt/cloud/bucket/user1/newfile -$ ls -l /mnt/cloud/bucket/user1/newfile --rw-r--r-- 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/user1/newfile -``` - -If you want to set the umask for all users, you can set it in the system-wide profile, or in the user's profile. For example, to set it for all users, you could add the umask command to `/etc/profile`. - -### Converting a bucket to POSIX enforced mode - -If you have an existing storage location, for which you now need to implement POSIX-based access controls, you can simply set the bucket tag as in [enforced-posix-guide-tag-bucket](../getting-started/enforced-posix-access#tag-the-bucket). This will not affect any existing files until they or their directories are accessed, and will ensure that any new files are stored with POSIX metadata. - -To control access to any particular files or directories, proceed with `chown`, `chgrp` and `chmod` as you would on a POSIX file system. For example, to limit access to `/mnt/cloud/bucket/directory` to only allow `user1`, you could do: - -```console -$ chown -R user1 /mnt/cloud/bucket/directory -$ chgrp -R group1 /mnt/cloud/bucket/directory -$ chmod -R 700 /mnt/cloud/bucket/directory -``` - -As a result of these commands, only `user1` will be able to access the directory and its contents across all systems using Object Mount. - -You can check the POSIX metadata for a file or directory using `ls -l` or `stat`. For example: - -```console -$ ls -l /mnt/cloud/bucket/directory -drwx------ 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/directory -$ stat /mnt/cloud/bucket/directory -File: /mnt/cloud/bucket/directory -Size: 0 Blocks: 0 IO Block: 4096 directory -Device: 1ch/28d Inode: 123456 Links: 1 -Access: (0700/drwx------) Uid: ( 1000/ user1) Gid: ( 1000/ group1) -Access: 2022-03-01 12:00:00.000000000 +0000 -Modify: 2022-03-01 12:00:00.000000000 +0000 -Change: 2022-03-01 12:00:00.000000000 +0000 - Birth: - -``` - -### Setting inherited permissions on a directory - -{% callout type="warning" %} -Support for POSIX ACLs is coming soon. If this is a feature you need, please contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). -{% /callout %} - - -Use [setfacl](https://linux.die.net/man/1/setfacl) to define additional default permissions to be applied to files in a directory when they are created. - -For example, an admin could create a shared space, `/mnt/cloud/bucket/shared`, for users to collaborate, and set the default permissions to allow all users and groups to read and write files in the directory: - -```console -$ setfacl -m u::rwX,g::rwX,o::- /mnt/cloud/bucket/shared -``` - -This will result in the following behaviour: - -```console -$ touch /mnt/cloud/bucket/shared/newfile -$ mkdir /mnt/cloud/bucket/shared/newdir -$ ls -l /mnt/cloud/bucket/shared/ -drwxrwxr-x 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/shared/newdir --rw-rw-r-- 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/shared/newfile -``` diff --git a/app/(docs)/object-mount/linux/getting-started/intro/page.md b/app/(docs)/object-mount/linux/getting-started/intro/page.md deleted file mode 100644 index 9268a6980..000000000 --- a/app/(docs)/object-mount/linux/getting-started/intro/page.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: Core Concepts -docId: Xaegoh6iedietahf -weight: 1 -metadata: - title: Core Concepts - description: - The ideas presented here are intended to familiarise you with our approach to object storage. ---- -The ideas presented here are intended to familiarise you with our approach to object storage. If you're already familiar, feel free to skip to the [next section](../getting-started/download-install). - -## Characteristics of object storage - -Object storage excels at high-throughput use-cases - usually in a "write-once-read-many" model. Object Mount, by itself, excels at unlocking the possible highest performance from object storage in many scenarios. - -However, object storage technology significantly differs from conventional block and file storage solutions that POSIX applications typically expect. On object storage, file editing is typically slow and requires the file to be rewritten even for slight modifications. For this, Object Mount has some specific optimisations. - -## Direct use of object storage - -Object Mount is intended to enable the usage of object storage as "hot storage" or as "first-class" storage. Whilst it is possible to use Object Mount to stage data from object storage locally, then run the task, and then upload the result back to object storage - Object Mount' performance and POSIX-compliance mean that you can now just *run the task* directly on object storage. - -Due to the "write-once-read-many" model of object storage, it does not always allow *efficient* partial modification in remote files or appending of new content. For many cases, our prediction engines, partial writes, client-side write-caching and other optimizations are enough to keep things running with great performance. -For truly random, repeated writes our solution is [Object Mount Fusion](../getting-started/object-mount-fusion): combining the best of high-speed attached storage and object storage. - -## Interception - -When running in Direct Interception mode, Object Mount will insert itself between your applications and the operating system. It will dynamically redirect relevant filesystem calls made by the application through the API of your object storage solution. As a result, our mapping between file system concepts and object storage gives any application instant access to objects as files. - -## What actually is Object Mount? - -Object Mount is the common back-end system of both our Direct Interception and Object Mount on FUSE (a modified FUSE mount). It is an architecture and implementation for mapping POSIX file-system concepts to object storage concepts. - -When the Object Mount CLI is used to launch a new shell, the primary purpose is to start a new shell (whether that's bash, zsh, or whatever else) with `LD_PRELOAD` set to point at `cuno.so`. There is no Object Mount shell binary - it only wraps existing shells with Object Mount pre-loaded. - -As a result, any applications that can access file storage (including ones you’ve written yourself) are intercepted so that they can work with object storage as well. It intercepts any user binaries including those written in C/C++, Rust, Go, Java, Python, etc. It’s built to work everywhere. It works with dynamic binaries and static binaries. It works inside unprivileged containerised environments. - -## What Object Mount does - -- Provides any POSIX-compliant application high-speed access to object storage. -- Allows you to use filesystem operations to list, copy, move, add, remove, and modify files stored on object storage. -- Supports rich POSIX metadata including: users, groups, permissions, symbolic/hard links. -- Preserves POSIX consistency guarantees equivalent to NFS. [Object Mount Fusion](../getting-started/object-mount-fusion) can provide even stronger consistency guarantees if needed) - -## What Object Mount does not do - -- Change the representation of files on object storage. No lock-in: file data is stored in a single object, and is not modified. This means data can be accessed directly on the object storage, bypassing Object Mount. -- Affect the way that processes interact with non-cloud data. -- Provide consistency guarantees for objects accessed directly on the object store rather than through Object Mount. - diff --git a/app/(docs)/object-mount/linux/getting-started/object-mount-fusion/page.md b/app/(docs)/object-mount/linux/getting-started/object-mount-fusion/page.md deleted file mode 100644 index bba61bc48..000000000 --- a/app/(docs)/object-mount/linux/getting-started/object-mount-fusion/page.md +++ /dev/null @@ -1,424 +0,0 @@ ---- -title: Object Mount Fusion -docId: duac8aixa9oPiw4u -weight: 7 -metadata: - title: Object Mount Fusion - description: Getting started with Object Mount Fusion ---- - -## Overview -{% callout type="warning" %} -Object Mount Fusion is currently in `BETA`. Your feedback via our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) will help us improve it. -{% /callout %} - -Object Mount Fusion is way to upgrade high-performance attached storage solutions like Amazon Elastic File System (EFS) with the throughput of object storage. -It is a **cheaper** and **faster** solution compared to using EFS alone. - -Object Mount Fusion takes an attached storage filesystem and an initially empty object storage bucket/directory and exposes a single interface for both. -Object Mount Fusion will migrate files between object storage and local filesystem depending on where they are best for performance/cost. - - -{% callout type="note" %} -This feature is available only to Professional and Enterprise Tier customers. -For licensing, please schedule a [discovery call](https://www.storj.io/landing/get-in-touch). -{% /callout %} - -## How it works - -Object Mount Fusion combines both into a virtual mount or a FUSE mount. The files on object storage are represented as hidden links from the host filesystem to the object store. -Unlike other solutions, the object store is a first-class high throughput tier, rather than a slow archival tier. -Object Mount Fusion automatically migrates files between the two according to application behaviour on such files. -New files may be written in either tier, depending on predicted and observed access properties. -Object Mount Fusion supports multiple users simultaneously accessing files on multiple nodes by sharing the attached storage and mount location. - - -## Setting up Object Mount Fusion - -Object Mount Fusion is expected to be used within the same high-speed LAN as your object storage. For example, if you use AWS S3, Object Mount Fusion should only be set up on EC2 nodes within the same -[AWS Region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) as the bucket to accessed. If you are using an S3-compatible on-premises object storage solution, -then Object Mount Fusion should be set up on a computer on the same high-speed local network. - -If you're already set up and using an attached storage system, such as EFS, you may skip ahead to [Mounting a Object Mount Fusion filesystem](#mounting-a-object-mount-fusion-filesystem). - -### Set up an empty bucket or directory on object storage - -For use as the object storage half of a running Object Mount Fusion mount, you will need a location on object storage that is empty. -This will be used as the location that the Fusion filesystem will migrate data to when it's more suitably stored on object storage. -This can either be an entirely empty bucket, or an empty directory on an existing bucket. This location must not be modified by anything but Object Mount Fusion filesystems. - -{% tabs %} -{% tab label="AWS S3" %} - For a new empty bucket, follow the instructions for [setting up a new bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) -{% /tab %} -{% tab label="Microsoft Azure" %} - For a new empty container, follow the instructions for [setting up a new storage container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container). -{% /tab %} -{% tab label="Google Cloud Storage" %} - For a new empty bucket, follow the instructions for [setting up a new GCS bucket](https://cloud.google.com/storage/docs/creating-buckets). -{% /tab %} -{% tab label="S3-compatible" %} - For a new empty bucket, follow your storage provider's instructions for setting up a new bucket. -{% /tab %} -{% /tabs %} - - -### Set up a new compute node - -Object Mount Fusion needs to be set up on a compute node in the same LAN/region as the object storage bucket to be used. - -You will also need to attach a high-speed file storage solution to the instance. If convenient or if this is not possible later, please do it while creating the new instance/image. - -To set up a compute node in the same region as your bucket, follow the relevant steps: - -{% tabs %} -{% tab label="AWS S3" %} - Follow the guide [Tutorial: Get started with Amazon EC2 Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html). -{% /tab %} -{% tab label="Microsoft Azure" %} - Follow the guide [Quickstart: Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu). -{% /tab %} -{% tab label="Google Cloud Storage" %} - Follow the instructions for creating [Instance templates](https://cloud.google.com/compute/docs/instance-templates). -{% /tab %} -{% tab label="S3-compatible" %} - Typically, you will need a computer connected to the same LAN as your object storage and your file storage device. -{% /tab %} -{% /tabs %} - -### Attach a file storage device - -You will need an empty directory on your attached file storage that is writable by your user. If don't have this set up already, follow the relevant steps to attach a writable storage device to your compute node: - - -{% tabs %} -{% tab label="AWS S3" %} - Follow the guide [Getting started with Amazon Elastic File System (EFS)](https://docs.aws.amazon.com/efs/latest/ug/getting-started.html). -{% /tab %} -{% tab label="Microsoft Azure" %} - Follow the documentation to [Use the portal to attach a data disk to a linux VM](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal). - Follow the Microsoft documentation for [setting up Azure Files](https://learn.microsoft.com/en-us/azure/storage/files/storage-files-introduction). -{% /tab %} -{% tab label="Google Cloud Storage" %} - Use the information available on the [Storage options](https://cloud.google.com/compute/docs/disks) to choose an appropriate attached disk option. Then, attach your instance to it using the relevant guide. - Follow the instructions for [setting up Filestore](https://cloud.google.com/filestore/docs/install) -{% /tab %} -{% tab label="S3-compatible" %} - You can use local storage, or attach an NFS/SMB mount to a file storage device in the same LAN. -{% /tab %} -{% /tabs %} - -## Mounting a Object Mount Fusion filesystem - -To set up Object Mount Fusion, you need Object Mount [installed](../getting-started/download-install) on the compute node, and you need to set up [credentials](../getting-started/configuring-credentials) so that the bucket/container is accessible. - -Your next steps depend on the filesystem you are using and the options used when mounting/attaching it. Please see the relevant section below: -- [Filesystems without extended attribute support (e.g. EFS)](#filesystems-without-extended-attribute-support-e-g-efs) -- [Filesystems with extended attribute support (e.g. NFSv4 with Linux Kernel 5.9+)](#filesystems-without-extended-attribute-support-e-g-efs) - -### Filesystems without extended attribute support (e.g. EFS) - -Object Mount Fusion is set up and accessed through a Object Mount on FUSE. To do this, additional options must be passed to `cuno mount` when it is run. You will need to use the `--fusion` option to specify the path to the attached storage backing directory, and the `--root` option to specify the path to the object storage: - -```sh -cuno mount \ - --fusion "" \ - --root "" \ - "" -``` - -Example: `cuno mount --fusion /dev/sdf/fusion-store --root s3://bucket/fusion-store $HOME/my-fusion-filesystem` - -By running the above command, the `` becomes the way to access the Fusion filesystem. All operations, workflows, pipelines, etc. should therefore be pointed to ``. - -#### cuno mount options relevant to Object Mount Fusion -| Argument/Option | Description | Example | -| ----------- | ----------- | ----------- | -| ```` | The location from which the Fusion filesystem will be accessed. | ``$HOME/my-fusion-filesystem`` | -| ``--fusion ""`` | Enables Fusion for this mount, and sets it to use the location specified to store file-storage data. This is where files that are migrated off the object store into local storage go. This can happen when access patterns suggest the file would benefit from higher IOPS, or when a file doesn't meet the minimum size or age thresholds. | ``/dev/sdf/fusion-store`` | -| ``--root ""`` | Sets the object storage location to be used as the place to store data that is better suited to be stored on object storage rather than on the file storage. | ``/cuno/s3/bucket/fusion-store`` | -| ``--fusion-size-threshold `` (optional) | Defines the minimum size that a file needs to be for it to be migrated from file storage to object storage. The argument value is of the form ``[UNIT]``. If no unit is given, the value is assumed to be in *bytes*. Valid units are ``K`` (Kilobytes), ``M`` (Megabytes), ``G`` (Gigabytes), ``T`` (Terabytes). | | -| ``--fusion-age-threshold `` (optional) | Defines the minimum age requirement for files to be considered for migration from file storage to object storage. The age is measured as the time since the most recent of the POSIX creation time, modify time, access time, and change time - in other words, the time since each of these must be greater than the set threshold. The argument value is of the form ``[UNIT]``. If no unit is given, the value is assumed to be in seconds. Valid units are ``s`` (seconds), ``m`` (minutes), ``h`` (hours), ``d`` (days). | | - -{% tabs %} -{% tab label="AWS S3" %} -If the bucket is not empty, create a new empty directory on your bucket: - -```shell -#terminal -cuno run mkdir "/cuno/s3//fusion-store" -``` - -Create a directory on the attached-storage device (assuming you've mounted it at ``/dev/sdf``) to use for this purpose: - -```shell -#terminal -mkdir "/dev/sdf/fusion-store" -``` - -Create the Object Mount Fusion mount: - -```shell -#terminal -cuno mount --fusion "/dev/sdf/fusion-store" --root "/cuno/s3//fusion-store" "~/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="Microsoft Azure" %} -If the bucket is not empty, create a new empty directory on your bucket: - -```shell -#terminal -cuno run mkdir "/cuno/az///fusion-store" -``` - -Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: - -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - -Create the Object Mount Fusion mount: - -```shell -#terminal -cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/az///fusion-store" "~/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="Google Cloud Storage" %} -If the bucket is not empty, create a new empty directory on your bucket: - -```shell -#terminal -cuno run mkdir "/cuno/gs//fusion-store" -``` - -Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: - -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - -Create the Object Mount Fusion mount: - -```shell -#terminal -cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/gs//fusion-store" "~/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="S3-compatible" %} -If the bucket is not empty, create a new empty directory on your bucket: - -```shell -#terminal -cuno run mkdir "/cuno/s3//fusion-store" -``` - -Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: - -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - -Create the Object Mount Fusion mount: - -```shell -#terminal -cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/s3//fusion-store" "~/my-fusion-filesystem" -``` -{% /tab %} -{% /tabs %} - - -### Filesystems with extended attribute support (e.g. NFSv4 with Linux Kernel 5.9+) - -When using a filesystem that supports extended attributes please "bind" it to a cloud location first as this does additional error checking and makes for a simpler mount/unmount procedure. - -#### Bind a directory to a cloud location - -To bind a local Fusion directory to a cloud location, use `cuno fusion`. -This command will make a note locally of where the mount we make later will point to. -It will also save any relevant options used at this stage as metadata, to be used by default by any mounts that use this fusion binding. - -```shell -#terminal -cuno fusion "" "" -``` - -Example: `cuno fusion /dev/sdf/fusion-store s3://bucket/fusion-store` - -#### cuno fusion arguments -| Argument | Description | -| ----------- | ----------- | -| ```` | Sets the location specified as the place to store file-storage data. This is where files that are migrated off the object store into local storage go. This can happen when access patterns suggest the file would benefit from higher IOPS, or when a file doesn't meet the minimum size or age thresholds. | -| ```` | Text | - - -#### cuno fusion options -| Option | Description | -| ----------- | ----------- | -| ``--fusion-size-threshold `` (optional) | Define the minimum size that a file needs to be for it to be migrated from file storage to object storage. The argument value is of the form ``[UNIT]``. If no unit is given, the value is assumed to be in *bytes*. Valid units are ``K`` (Kilobytes), ``M`` (Megabytes), ``G`` (Gigabytes), ``T`` (Terabytes). | -| ``--fusion-age-threshold `` (optional) | Defines the minimum age requirement for files to be considered for migration from file storage to object storage. The age is measured as the time since the most recent of the POSIX creation time, modify time, access time, and change time - in other words, the time since each of these must be greater than the set threshold. The argument value is of the form ``[UNIT]``. If no unit is given, the value is assumed to be in seconds. Valid units are ``s`` (seconds), ``m`` (minutes), ``h`` (hours), ``d`` (days). | - - -#### cuno fusion subcommands -| Subcommand | Description | -| ----------- | ----------- | -| ``cuno fusion bind `` | Bind a local directory to a cloud directory. | -| ``cuno fusion unbind `` | Unbind a local directory from a cloud directory. | -| ``cuno fusion rebind `` | Rebind a local directory to a cloud directory. | -| ``cuno fusion info `` | Get information about the binding at the location. | - - -#### Mount the Object Mount Fusion filesystem after binding - -Object Mount Fusion is set up and accessed through a Object Mount on FUSE. To do this, additional options must be used when creating a [Object Mount on FUSE](../user-guides/basic#object-mount-on-fuse). - -If you have already bound a directory to a cloud location, you only need to use the `--root` option to specify the path to the attached-storage backing directory in order to use the mount as a Fusion mount: - -```shell -# terminal -cuno mount \ - --root "" \ - "" -``` - -#### cuno mount arguments/options relevant to Object Mount Fusion -| Argument | Description | -| ----------- | ----------- | -| ```` | The location from which the Fusion filesystem will be accessed. | -| ``--root ""`` | Sets the path that the mount will look for a directory that has been set up for Object Mount Fusion. This means that you must previously have run ``cuno fusion`` with the same path, so that Fusion Mount can see which object storage path it is bound to. If your filesystem does not support extended attributes, you should use --root as specified. Example: ``/dev/sdf/fusion-store`` | - - -{% tabs %} -{% tab label="AWS S3" %} - If the bucket is not empty, create a new empty directory on your bucket: -```shell -# terminal -cuno run mkdir "s3:///fusion-store" -``` - - Create a directory on the attached-storage device (assuming you've mounted it at ``/dev/sdf``) to use for this purpose: -```shell -#terminal -mkdir "/dev/sdf/fusion-store" -``` - - Create the Object Mount Fusion mount: -```shell -#terminal -cuno fusion "/dev/sdf/fusion-store" "s3:///fusion-store" -#terminal -cuno mount --root "/dev/sdf/fusion-store" "$HOME/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="Microsoft Azure" %} - If the bucket is not empty, create a new empty directory on your bucket: -```shell -#terminal -cuno run mkdir "az://///fusion-store" -``` -Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: - -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - - Create the Object Mount Fusion mount: -```shell -#terminal -cuno fusion "/mnt/fast/fusion-store" "az://///fusion-store" -#terminal -cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="Google Cloud Storage" %} - If the bucket is not empty, create a new empty directory on your bucket: -```shell -#terminal -cuno run mkdir "gs:////fusion-store" -``` - - Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - - Create the Object Mount Fusion mount: - -```shell -#terminal -cuno fusion "/mnt/fast/fusion-store" "gs:////fusion-store" -#terminal -cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" -``` -{% /tab %} -{% tab label="S3-compatible" %} - If the bucket is not empty, create a new empty directory on your bucket: -```shell -#terminal -cuno run mkdir "s3:///fusion-store" -``` - - Create a directory on the attached-storage device (assuming you've mounted it at ``/mnt/fast``) to use for this purpose: -```shell -#terminal -mkdir "/mnt/fast/fusion-store" -``` - Create the Object Mount Fusion mount: -```shell -#terminal -cuno fusion "/mnt/fast/fusion-store" "s3:///fusion-store" -#terminal -cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" -``` - -{% /tab %} -{% /tabs %} - -For instructions on unmounting, see [user-guide-unmount](../user-guides/basic#object-mount-on-fuse-commands). - -## Using the Object Mount Fusion filesystem - -All operations, workflows, pipelines, etc. should be pointed to your mount location. - -### Examples - -The following examples assume you have mounted a Fusion filesystem at `$HOME/my-fusion-filesystem`. - -- Download data from the web into the Fusion filesystem: - - ```shell - #terminal - cd $HOME/my-fusion-filesystem - #terminal - wget http://vision.stanford.edu/aditya86/ImageNetDogs/images.tar - ``` - -- List the files in the Fusion filesystem: - - ```shell - #terminal - ls $HOME/my-fusion-filesystem/ - ``` - -- Unpack a tar archive: - - ```shell - #terminal - cd $HOME/my-fusion-filesystem - #terminal - tar -xf images.tar - ``` - -### Testing - -Check that access, modification and migration are working as expected. We recommend measuring cost differences, and the overall time taken for equivalent jobs run on attached storage alone. - -NB: You can set the ``CUNO_DEBUG`` environment variable to ``trace``, or ``debug`` at the time of mounting. - -Not seeing what you expect? Contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). diff --git a/app/(docs)/object-mount/linux/getting-started/page.md b/app/(docs)/object-mount/linux/getting-started/page.md index 8776b796f..f63e52dbc 100644 --- a/app/(docs)/object-mount/linux/getting-started/page.md +++ b/app/(docs)/object-mount/linux/getting-started/page.md @@ -1,37 +1,39 @@ --- -title: Linux +title: Getting Started Guides hideTitle: true -docID: wxtofwqcb5f2 -weight: 2 +docId: xhNvtETAA6UBZVNH +weight: 1 metadata: - title: Linux Installation and Usage + title: Getting Started description: - How to install and use the Object Mount Linux package across different distributions. + Getting started with Object Mount for Linux. +hidden: false --- -# Linux Installation and Usage +# Getting Started Guides for Linux -This section covers everything you need to install and use the **Object Mount Linux package** across different distributions. +The pages in these **Getting Started Guides** for Linux will introduce you to topics that should be understood _before_ installing and using Object Mount. -Whether you're setting up on Ubuntu, Debian, CentOS, or another system, you'll find step-by-step instructions, package details, and best practices to help you get up and running quickly. ---- +## Deployment & Configuration Options on Linux -## What’s Covered +When configuring Object Mount on Linux, it is important to understand the needs of the **tools** and **workloads** you plan to run. -- Installation guides tailored for popular Linux distributions -- System requirements and dependency notes -- Instructions for mounting, configuration, and permissions -- Optional performance and logging tips +You can adapt settings for each workflow or tool, or you can set options at the bucket-level to impose a particular usage mode for all users. ---- -## Related Pages +## Deployment Considerations -- [Debian/Ubuntu Installation](../linux/installation/debian/) -- [RHEL/CentOS Installation](../linux/installation/redhat/) -- [Core Concepts](../linux/getting-started/intro/) +There are two important, but independent, aspects to consider when configuring Object Mount for Linux: -If you're just getting started, we recommend beginning with your OS-specific guide. Each includes exactly what you need to install Object Mount safely and effectively. +1. The requirements for your tools and applications to work _correctly_: ---- + - For **correct operation**, you need to know what level of **POSIX Compatibility** your tools require in order to function. + - Object Mount for Linux offers multiple POSIX compatibility options giving the end user a wide array of tools to fine-tune and tailor the functionality and performance of Object Mount for their specific use case and environment. + - See the article on [POSIX Options](docId:cbm3PcQXmLpuYcbg) to understand the many different POSIX compatibility choices available to you. + +2. Ensuring your specific tools and environment work _efficiently_: + + - For **efficient operation**, the bandwidth and latency between your compute resources and your object storage resources will help guide your selection of the best **Deployment Mode** for Object Mount. + - Object Mount for Linux supports multiple Deployment Modes supporting a variety of unique bandwidth, user requirements, and environmental situations. + - See the [Deployment Modes](docId:bRnfbdNE6d5DaZzW) article to understand the different deployment choices available to you. diff --git a/app/(docs)/object-mount/linux/getting-started/posix-options/page.md b/app/(docs)/object-mount/linux/getting-started/posix-options/page.md new file mode 100644 index 000000000..e412debc9 --- /dev/null +++ b/app/(docs)/object-mount/linux/getting-started/posix-options/page.md @@ -0,0 +1,97 @@ +--- +title: POSIX Options +hideTitle: false +docId: cbm3PcQXmLpuYcbg +weight: 2 +metadata: + title: POSIX Options + description: + Technical Details on the POSIX Options for Object Mount for Linux. +hidden: false +--- + +## POSIX Compatibility + +For Object Mount to function correctly, it needs to comply with the level of POSIX compatibility that your tools require. + +Some Linux applications have only limited or minimal requirements from a POSIX perspective, and may only use the file system for basic operations like renaming, reading, or writing new data. + +Other applications may require a fully POSIX-compatible interface, which invokes additional services within Object Mount’s compatibility layer. + + - **Note:** Enabling full POSIX compatibility requires extra metadata to be written to object storage and may have a slight effect on performance. + + +## POSIX Modes + +The levels of POSIX compatibility provided in Object Mount for Linux are: + +- [Core File Access](#core-file-access) — no users/groups/symlinks/hardlinks/permissions control/file modes. +- [POSIX File Access](#posix-file-access) — POSIX-compatible access with users/groups/symlinks/hardlinks/permission control/file modes. +- [POSIX Enforced File Access](#posix-enforced-file-access) — Same as “POSIX File Access” above, but POSIX access control _is enforced_. + +{% callout type="note" %} +**How to check if you need to enable “POSIX File Access”** + +- Run your workload using a local file system volume. Then run the same workload using an Object Mount volume. Compare the outputs and results. If you find discrepancies, it is likely that POSIX File Access needs to be enabled. +- If Object Mount running with only **Core File Access** fails with an error `(134) ENOTSUP` (not supported) or `Operation not permitted` then it is likely that POSIX File Access needs to be enabled. +{% /callout %} + + +## Core File Access + +**Core File Access** mode disables all POSIX compatibility features in Object Mount for linux. Core File Access mode does not support the persistence or modification of POSIX users, groups, symlinks, hard links, permissions control or file modes attributes on objects. This is the default mode. + + - Use “Core File Access” when you don’t have any metadata requirements and your tools don’t need any POSIX metadata persistence to function correctly. + + - Use “Core File Access” when you don’t have write access to the bucket in question, or you don’t want to create any Object Mount-generated objects there. + +**Example use-cases:** + + - Use “Core File Access” when interacting with data you have already moved up to object storage and when you only require read-only access to the filenames and content of those objects. For example, if you have machine learning datasets in the cloud and you have previously configured your libraries to read them directly from object storage. + +**How to enable:** + + - “Core File Access” is enabled as the default mode. No additional action is required. + + +## POSIX File Access + +**POSIX File Access** mode will generate and maintain POSIX metadata for your objects, but it _will not_ enforce any permissions or modes set on the objects. This means any user can use Object Mount to read or write any file that their object storage credentials give them access to. + + - Use “POSIX File Access” when you need POSIX metadata but do _not_ need to enforce POSIX read/write/execute file permissions. + +**Example use-cases:** + + - Use “POSIX File Access” when your applications are dependent on the preservation of POSIX metadata (owner/group permissions, change/modify times, etc.) or POSIX “links” (symlinks or hardlinks). + + - Use “POSIX File Access” if you are moving workflows from POSIX to object storage, such as workloads that were previously run on AWS EC2 with EFS. + + **Note:** Object Mount does not currently support POSIX ACLs or extended attributes in cloud object storage. Contact the 🌐 [Storj Support](https://supportdcs.storj.io/hc/en-us) team if you need assistance in providing these features. + +**How to enable:** + +There are two main ways to enable “POSIX File Access”: + + - If the object storage provider supports setting **tags** at the bucket level, then you can enable POSIX File Access mode for _all users_ using the command `cuno creds setposix s3://your_bucket_name true` from a command line/shell interface. This will affect _everyone_ using the bucket and will force all Object Mount users of that bucket into POSIX File Access mode. + + - Otherwise, to enable POSIX File Access on an _individual workstation-basis_ you can set a local environment variable. Enter `export CUNO_POSIX=1` from a command line/shell interface. This setting is valid per-session. + + +## POSIX Enforced File Access + +**POSIX _Enforced_ File Access** mode will generate and maintain POSIX metadata for your objects (like “POSIX File Access” mode, above) but _will enforce_ POSIX access controls on those objects. + + - Use “POSIX Enforced File Access” when you want to manage what users have access to, based on the UID/GID of their UNIX user and the corresponding POSIX metadata (owner, group, mode) on files. + - This means users will encounter `access denied` errors if they try to read or write to a file/directory they haven’t been given permission to (by a suitably privileged user/admin using: `chown`, `chgrp` or `chmod`). + +**Example use-cases:** + + - Use “POSIX Enforced File Access” to host a website (using NGINX or other server technologies) entirely from object storage, without any attached storage device(s) (such as EFS). “POSIX Enforced File Access” mode lets you maintain control over which files/directories the web server process (e.g.: the `nginx` user) can access. + + - Use “POSIX Enforced File Access” to host an organization’s user filesystem in the cloud. + + **Note:** Object Mount does not currently support POSIX ACLs or extended attributes in cloud object storage. Contact the 🌐 [Storj Support](https://supportdcs.storj.io/hc/en-us) team if you need assistance in providing these features. + +**How to enable:** + + - To enable and configure “POSIX Enforced File Access” see the detailed steps in the Configuration Guide article: [](docId:Eegoo1teiJ8eerae). diff --git a/app/(docs)/object-mount/linux/getting-started/profile-mode/page.md b/app/(docs)/object-mount/linux/getting-started/profile-mode/page.md deleted file mode 100644 index 6e05bf6d5..000000000 --- a/app/(docs)/object-mount/linux/getting-started/profile-mode/page.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: Linux Profile Mode (CUNO_PROFILE) -docId: linux-profile-mode -hideTitle: true -weight: 9 -metadata: - title: Object Mount on Linux – Profile Modes - description: - Explains the CUNO_PROFILE environment variable for Linux builds and how to choose the appropriate mode (M&E vs HPC). ---- - -# Linux Profile Mode: `CUNO_PROFILE` - -Object Mount for Linux includes an environment variable called **`CUNO_PROFILE`** that configures internal behaviour for different types of workloads. This setting is **only applicable on Linux**, and helps Object Mount optimise itself for either: - -- **Media & Entertainment workflows** (`CUNO_PROFILE=M&E`) -- **High-performance computing environments** (`CUNO_PROFILE=hpc`) - -Understanding which mode to use is important, the wrong setting can lead to suboptimal behaviour, especially for large-scale automated environments. - ---- - -## What Does `CUNO_PROFILE` Do? - -The `CUNO_PROFILE` variable adjusts how Object Mount behaves internally, including: - -- How aggressively metadata is cached -- Which filesystem features are exposed -- Logging, debugging, and memory handling profiles -- Compatibility expectations with the Linux userland and toolchains - -{% callout type="info" %} -If unset, the default is: -`CUNO_PROFILE=M&E` -{% /callout %} - -This is well-suited to desktop users, editors, and creative professionals using Object Mount interactively, but **not always suitable** for headless systems or scripted automation. - ---- - -## Choosing the Right Profile - -Before shipping or deploying the Linux version of Object Mount, ask the following: - -### 1. **Is the user working interactively, or via headless automation?** -- `M&E` is tuned for interactive, graphical workflows -- `hpc` is preferred for background tasks, CLI tools, and scripts - -### 2. **Will the mount be used by creative software (e.g. Resolve, Premiere, FCP)?** -- If yes: `M&E` -- If not (e.g. media transformation pipelines, backups): `hpc` - -### 3. **Is the system a personal workstation or shared server node?** -- Workstation → `M&E` -- Server node / render farm → `hpc` - -### 4. **Is the mount expected to be long-lived and static, or frequently toggled?** -- Session-based, user-initiated mounts → `M&E` -- Long-lived, persistent mounts → `hpc` - -### 5. **Will users be working via GUI, or purely via CLI/API?** -- GUI or desktop usage → `M&E` -- CLI tools, automation, batch jobs → `hpc` - -### 6. **Is latency more important than throughput, or vice versa?** -- Low-latency I/O (desktop feel) → `M&E` -- Bulk, high-throughput tasks (e.g. rendering, transcoding) → `hpc` - ---- - -## Setting the Profile - -To explicitly set the profile, define the environment variable before launching Object Mount: - -``` -bash -export CUNO_PROFILE=hpc -``` - -For permanent use, add this to: -- Your shell profile (`~/.bashrc`, `~/.zshrc`) -- A systemd service or login script -- Docker or container entrypoints (if applicable) - ---- - -## Quick Reference - -| Use Case | Recommended Profile | -|----------------------------|---------------------| -| Desktop video editing | `M&E` | -| Automated ingest pipeline | `hpc` | -| Interactive audio work | `M&E` | -| Render farm node | `hpc` | -| Remote headless access | `hpc` | -| Local workstation previews | `M&E` | - ---- - -## Summary - -Choosing the right `CUNO_PROFILE` ensures Object Mount behaves appropriately for the workload. If in doubt: - -- Start with `M&E` for creative users -- Use `hpc` for background, server, or scripted automation - -{% callout type="info" %} -Still unsure which profile to use? [Reach out to our support team](https://supportdcs.storj.io/hc/en-us/requests/new) and we’ll help you pick the right configuration. -{% /callout %} diff --git a/app/(docs)/object-mount/linux/installation/_meta.json b/app/(docs)/object-mount/linux/installation/_meta.json deleted file mode 100644 index 594f8c010..000000000 --- a/app/(docs)/object-mount/linux/installation/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Installation Instructions", "weight": 3} diff --git a/app/(docs)/object-mount/linux/installation/alpine/page.md b/app/(docs)/object-mount/linux/installation/alpine/page.md index aa37a5640..492124560 100644 --- a/app/(docs)/object-mount/linux/installation/alpine/page.md +++ b/app/(docs)/object-mount/linux/installation/alpine/page.md @@ -1,74 +1,99 @@ --- title: Alpine +hideTitle: true docId: MeiPie8EDuo7eise - +weight: 5 metadata: title: Alpine description: - Alpine Installation Instructions + Alpine Package Installer Instructions +hidden: false --- -1. Download the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the latest version of the [self-extracting package for Alpine](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_x86_64_musl_apk.run) from your browser, or run the command: +# Alpine Package Installer + +Storj provides an Object Mount package installer for the Alpine Linux distribution. + +This installer can be used with any musl-based Linux distribution. + +{% callout type="info" %} + **Optional: Scripted Installer** + + If you prefer to use a musl-based **scripted installer**, refer to the [](docId:ao0yaeng2Aitheel) installation instructions. +{% /callout %} + +Follow the steps below to install and configure Object Mount for Alpine-based Linux platforms. + + +## Step 1. Download the Installer Package + +Download the latest package installer. + +- **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + + +## Step 2. Unpack the Archive + +Use the installer’s `.run` script to unpack the archive by entering the following command from a terminal window: + +```console +sh cuno_mne_x86_64_musl_apk.run +``` + +{% callout type="info" %} + **Processor Compatibility** + + Although the Alpine distro file naming convention is to only specify `X86_64` (ex: `cuno_mne_x86_64_musl_apk.run`), the Object Mount APK Installer Package is designed to run on _both_ AMD64 and Intel x86_64 processors (although not ARM-based CPUs). + + **Note:** ARM-based processors are not supported at this time. +{% /callout %} + +The unpacking process will prompt you to read and agree to the Object Mount **end-user license agreement (EULA)**. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-eula.jpg) + +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"`. (This is equivalent to accepting the EULA). + +The archive and its contents will be unpacked into a directory named +`cuno_{VERSION}_x86_64_musl_apk` containing the package itself plus some additional files. - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_x86_64_musl_apk.run - ``` +View the unpacked contents with the following `ls` command (substituting your version number): -2. Download the public key for package verification `cunoFS.rsa.pub`: +```console +$ ls cuno_1.2.8.8_x86_64_musl_apk +cuno_1.2.8.8_x86_64_musl.apk +CUNO-Installation-and-Usage-Guide.pdf +``` - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cunoFS.rsa.pub - ``` - or create a file called `cunoFS.rsa.pub` with our public key using this "one-liner" (copy the whole thing, paste into your terminal, hit enter/return): +## Step 3. Install Object Mount and its dependencies using `apk` - ```console - cat >cunoFS.rsa.pub < | `. - ```console - $ ls cuno_{FULL-VERSION}_x86_64_musl - cuno_{FULL-VERSION}_x84_64_.apk - CUNO-Installation-and-User-Guide.pdf - ``` +{% callout type="note" %} + **Access for: All-Users** -7. Install Object Mount and its dependencies using `apk` (may need `sudo`): + By using `sudo`, all package manager installations of Object Mount install the app for use by _all users_ of the system, not just the account that initiated the install process. +{% /callout %} - ```console - apk add ./cuno_{FULL-VERSION}_x86_64_musl/cuno_{FULL-VERSION}_x86_64_musl.apk - ``` -8. The installation will prompt you to activate Object Mount by starting a trial or entering a license key. Follow the steps interactively, or automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. +## Next Steps -9. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. +_Once Object Mount is installed_, proceed to the Linux User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. diff --git a/app/(docs)/object-mount/linux/installation/debian/page.md b/app/(docs)/object-mount/linux/installation/debian/page.md index bc498e0ca..fcbcf045d 100644 --- a/app/(docs)/object-mount/linux/installation/debian/page.md +++ b/app/(docs)/object-mount/linux/installation/debian/page.md @@ -1,46 +1,104 @@ --- title: Debian +hideTitle: true docId: aemie9zeiP9Nie2k - +weight: 3 metadata: title: Debian description: - Debian Installation Instructions + Debian Package Installer Instructions +hidden: false --- -1. Download the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the latest version of the [self-extracting package for Debian](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_amd64_glibc_deb.run) from your browser, or run the command: +# Debian Package Installer + +Storj provides an Object Mount package installer for Debian (and Debian-based) Linux distributions. + +This installer can be used with 🌐 [Debian-based distributions](https://en.wikipedia.org/wiki/Category:Debian-based_distributions) such as Ubuntu, etc. + +{% callout type="info" %} + **Optional: Scripted Installer** + + If you prefer to use a glibc-based **scripted installer**, refer to the [](docId:ahWohd5eegh6eizi) installation instructions. +{% /callout %} + +Follow the steps below to install and configure Object Mount for Debian-based Linux platforms. + + +## Step 1. Download the Installer Package + +Download the latest package installer. + +- **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + + +## Step 2. Unpack the Archive + +Use the installer’s `.run` script to unpack the archive by entering the following command from a terminal window: + +```console +sh cuno_mne_amd64_glibc_deb.run +``` + +{% callout type="info" %} + **Processor Compatibility** + + Although the Debian distro file naming convention is to only specify `AMD64` (ex: `cuno_mne_amd64_glibc_deb.run`), the Object Mount DEB/APT Installer Package is designed to run on _both_ AMD64 and Intel x86_64 processors (although not ARM-based CPUs). + + **Note:** ARM-based processors are not supported at this time. +{% /callout %} + +The unpacking process will prompt you to read and agree to the Object Mount **end-user license agreement (EULA)**. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-eula.jpg) + +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"`. (This is equivalent to accepting the EULA). + +The archive and its contents will be unpacked into a directory named `cuno_{VERSION}_amd64_glibc_deb` containing the package itself plus some additional files. + +View the unpacked contents with the following `ls` command (substituting your version number): + +```console +$ ls cuno_1.2.8.8_amd64_glibc_deb +cuno_1.2.8.8_amd64_glibc.deb +CUNO-Installation-and-Usage-Guide.pdf +``` + + +## Step 3. Install Object Mount and its dependencies using apt + +**3a.** Use `sudo` to update `apt`: + +```console +sudo apt update +``` + +**3b.** Then install the Object Mount package (substituting your version number): - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_amd64_glibc_deb.run - ``` +```console +sudo apt install ./cuno_1.2.8.8_amd64_glibc_deb/cuno_1.2.8.8_amd64_glibc.deb +``` -2. Unpack the archive by running: +The installer will fetch and install any necessary additional packages (such as `libfuse`, etc.). - ```console - sh cuno_amd64_glibc_deb.run - ``` +**3c.** Next, the installation process will prompt you to **activate Object Mount**. -3. Please follow the interactive steps, read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the extraction. To automate this, you can set the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"` (this is equivalent to accepting the EULA). +- Press 1 to activate a **free trial**: + - This will activate a fully-featured 14-day free trial. +- Press 2 if you already have a **license key**: + - Paste your license key or enter a full path to a license key file. -4. The archive and its contents will be unpacked into a directory named `cuno_{FULL-VERSION}_amd64_glibc` containing the package itself and additional documents: + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-enter-license-key.jpg) - ```console - $ ls cuno_{FULL-VERSION}_amd64_glibc - cuno_{FULL-VERSION}_amd64.deb - CUNO-Installation-and-User-Guide.pdf - ``` +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. -5. Install Object Mount and its dependencies using `apt`: +{% callout type="note" %} + **Access for: All-Users** - {% callout type="note" %} - Be sure to use a relative path here, `./`, to instruct `apt` not to search its repositories. - {% /callout %} + By using `sudo`, all package manager installations of Object Mount install the app for use by _all users_ of the system, not just the account that initiated the install process. +{% /callout %} - ```console - sudo apt update - sudo apt install ./cuno_{FULL-VERSION}_amd64_glibc/cuno_{FULL-VERSION}_amd64_glibc.deb - ``` -6. The installation will prompt you to activate Object Mount by starting a trial or entering a license key. Follow the steps interactively, or automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. +## Next Steps -7. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. +_Once Object Mount is installed_, proceed to the Linux User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. diff --git a/app/(docs)/object-mount/linux/installation/enforced-posix/page.md b/app/(docs)/object-mount/linux/installation/enforced-posix/page.md deleted file mode 100644 index 3f6a87acf..000000000 --- a/app/(docs)/object-mount/linux/installation/enforced-posix/page.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Enforced Posix -docId: IX6thohVi6kaiY7i - -metadata: - title: Enforced Posix - description: - Enforced Posix - ---- - -This mode will maintain POSIX metadata for your objects, and will enforce POSIX access controls on those objects. Use this when you want to manage what users have access to based on the UID/GID of their UNIX user and the corresponding POSIX metadata (owner, group, mode) on files. That means users will encounter `access denied` errors if they try to read or write to a file/directory they haven't been given permission to (by a suitably privileged user doing `chown`, `chgrp` or `chmod`). - -Note that this is client-side rather than server-side enforcement. If the user has access to object storage credentials with server-side privileges beyond this, then the user can potentially access or modify objects outside of these POSIX access controls. Contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) for how to setup ACL Policies to enforce server-side access control that reflects POSIX access controls. - -{% callout type="warning" %} -This mode stores POSIX metadata as objects in a "hidden" directory in *your* buckets alongside *your* data. You cannot see these directories when using Object Mount to list objects, but you will see them if you use other tools (such as your storage provider's web console). Non-Object Mount access which renames, moves, or copies objects with Object Mount-stored POSIX file attributes will result in those objects losing their metadata. You will need to use Object Mount to manage those files while preserving their attributes. -{% /callout %} diff --git a/app/(docs)/object-mount/linux/installation/glibc/page.md b/app/(docs)/object-mount/linux/installation/glibc/page.md index adb36ae5a..4aeddfc33 100644 --- a/app/(docs)/object-mount/linux/installation/glibc/page.md +++ b/app/(docs)/object-mount/linux/installation/glibc/page.md @@ -1,36 +1,126 @@ --- -title: Glibc +title: glibc +hideTitle: true docId: ahWohd5eegh6eizi - +weight: 1 metadata: - title: Glibc + title: glibc description: - Glibc Installation Instructions - + glibc Scripted Installer Instructions +hidden: false --- -The Scripted Installer allows you to install Object Mount without root privileges (no sudo). +# glibc Scripted Installer + +Many Linux distributions use the GNU Project’s C standard library: 🌐 [glibc](https://en.wikipedia.org/wiki/glibc). + +This includes: Debian, Ubuntu, RedHat and their derivatives. + +Storj’s Object Mount **glibc scripted installer** (a `.run` file) is compatible with all such Linux distributions. + +This installer shell script allows you to install Object Mount (for the current user) _without_ needing root privileges — no `sudo` required. + +{% callout type="info" %} + **Optional: Package Installer** + + If you prefer a **full installer** for use with package managers, refer to the [](docId:aemie9zeiP9Nie2k) (Debian, Ubuntu, etc.) or [](docId:woosaugaiNohree9) installation instructions. +{% /callout %} + +Follow the steps below to install and configure Object Mount for **glibc** platforms. + + +## Step 1. Download the Install Script + +Download the latest scripted installer. + +- **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + +Be sure to download the correct script file for your specific chip architecture: + +| **Chip Type** | **Scripted Installer Filename** (Example) | +|---------------|-------------------------------------------| +| AMD x64 | cuno-mne-amd64-glibc-installer.run +| Intel x86 | cuno-mne-x86_64-glibc-installer.run + +- **Note:** ARM-based processors are not supported at this time. + + +## Step 2. Install & Activate Object Mount + +- **Note:** If you do not yet have a license key, you can install Object Mount for Linux in a fully-featured **Free Trial** mode. Reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) for more details. + +**2a.** To install Object Mount for use _only_ by the **current user**, run the following command from a terminal window (select your processor type): + +{% tabs %} + +{% tab label="AMD x64" %} + ```shell + # terminal + sh cuno-mne-amd64-glibc-installer.run + ``` +{% /tab %} + +{% tab label="Intel x86" %} + ```shell + # terminal + sh cuno-mne-x86_64-glibc-installer.run + ``` +{% /tab %} + +{% /tabs %} + +**2b.** To install Object Mount to be used by **all users** system-wide, add `sudo` to the _front_ of the command: + +{% tabs %} + +{% tab label="AMD x64" %} + ```shell + # terminal + sudo sh cuno-mne-amd64-glibc-installer.run + ``` +{% /tab %} + +{% tab label="Intel x86" %} + ```shell + # terminal + sudo sh cuno-mne-x86_64-glibc-installer.run + ``` +{% /tab %} + +{% /tabs %} + +**2c.** The installation process will prompt you to read and agree to the Object Mount **end-user license agreement (EULA)**. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-eula.jpg) + +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"`. (This is equivalent to accepting the EULA). + +**2d.** The installation process will then prompt you to **activate Object Mount**. + +- Press 1 to activate a **free trial**: + - This will activate a fully-featured 14-day free trial. +- Press 2 if you already have a **license key**: + - Paste your license key or enter a full path to a license key file. -1. Download the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the [latest version](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno-glibc-installer.run) from your browser, or run the command: +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-enter-license-key.jpg) - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cuno-glibc-installer.run - ``` +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. -2. To install only for the current user, run the following terminal command: - ```console - sh cuno-glibc-installer.run - ``` +## Installation Locations: Scripted Installers - To install system-wide, run: +When Object Mount is installed via a **Scripted Installer** (e.g.: glibc or musl), components are installed to different locations depending on whether it was installed by a root, or by a non-root user. - ```console - sudo sh cuno-glibc-installer.run - ``` +| Install Method | Location (`CUNO_ROOT`) | Symlinks created in | +|--------------------|------------------------|--------------------------------| +| Non-Root User | `~/.local/opt/cuno/` | `~/.local/bin/` +| | | `~/.local/lib/x86_64-linux-gnu` +| | | `~/.local/share/man/` +| Root User (`sudo`) | `/opt/cuno/` | `/usr/bin/` +| | | `/usr/lib/` +| | | `/usr/share/man/` -3. The installation will prompt you to read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the installation. To automate this, you can set the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"` (this is equivalent to accepting the EULA). -4. The installation will prompt you to activate Object Mount by starting a trial or entering a license key. Follow the steps interactively, or automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. +## Next Steps -5. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. +_Once Object Mount is installed_, proceed to the Linux User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. diff --git a/app/(docs)/object-mount/linux/installation/mac/page.md b/app/(docs)/object-mount/linux/installation/mac/page.md deleted file mode 100644 index eb6877741..000000000 --- a/app/(docs)/object-mount/linux/installation/mac/page.md +++ /dev/null @@ -1,191 +0,0 @@ ---- -title: MacOS -docId: yoopieyewevei1Eo - -metadata: - title: MacOS - description: - MacOS Installation Instructions - ---- -## MacOS - Using Linux Containers - -A fully native Mac client is in private beta, with a public beta available soon! - -However, if you would like to run the Linux version of Object Mount on your Mac, you can do so inside a container. - -{% callout type="note" %} -For those on Apple Silicon (ARM) Macs, macOS 13.0 Ventura or later is required. -{% /callout %} - -1. *All* Mac users require a virtualisation/containerisation software to be installed and set up. -{% tabs %} -{% tab label="Intel" %} -## Intel -We fully support usage of [Docker](https://docs.docker.com/desktop/install/mac-install/) and [Rancher Desktop](https://rancherdesktop.io/). Docker is assumed in these guides. - -Some settings should be changed to use Object Mount: -{% tabs %} -{% tab label="Docker" %} -## Docker - -We recommend changing the file sharing implementation to VirtioFS: -- Open Docker Desktop's ``Settings`` panel -- Open the ``General`` section. -- For the setting ``Choose file sharing implementation for your containers`` select ``VirtioFS``. -- Use the ``Apply & Restart`` button to save the change. -{% /tab %} -{% tab label="Rancher" %} -## Rancher - -We recommend using the Apple Virtualization Framework (VZ), and the VirtioFS file sharing implementation: -- Open Rancher Desktop's ``Preferences`` panel. -- Open the ``Virtual Machine`` section. -- Open the ``Emulation`` tab, and set ``Virtual Machine Type`` to ``VZ``. -- Open ``Volumes`` tab, and set the ``Mount Type`` to ``virtiofs``. -- Use the ``Apply`` button to save the changes. - -In order to use the ``cuno-mac`` utility with Rancher, the "dockerd" (moby) container engine must be used: -- Open Rancher Desktop's ``Preferences`` panel. -- Open the ``Container Engine`` section. -- In the ``General`` tab, set the ``Container Engine`` to ``dockerd (moby)``. -- Use the ``Apply`` button to save the change. -{% /tab %} -{% tab label="OrbStack" %} -## OrbStack - -You may also use [OrbStack for Mac](https://orbstack.dev/) and find it to be faster, but ``cuno-mac`` native support is not available in this case. - -Unfortunately, [colima](https://github.com/abiosoft/colima) is not currently supported. -{% /tab %} -{% /tabs %} -{% /tab %} - -{% tab label="Apple Silicon (ARM)" %} -## Apple Silicon (ARM) -We assume usage of Docker in these guides, but any of following containerisation technologies is supported with some additional set up required for ARM Macs: -{% tabs %} -{% tab label="Docker" %} -## Docker - -[Docker Desktop](https://docs.docker.com/desktop/install/mac-install/) version ``4.16.0`` or later is required. - -Use Rosetta for hardware emulation: -- Open Docker Desktop's ``Settings`` panel -- Open the ``Features in development`` section. -- Open the ``Beta features`` tab, and enable the setting :code:`Use Rosetta for x86/amd64 emulation on Apple Silicon`. -- We recommend changing the file sharing implementation to VirtioFS: -- Open Docker Desktop's ``Settings`` panel -- Open the ``General`` section. -- For the setting ``Choose file sharing implementation for your containers`` select ``VirtioFS``. -- Use the ``Apply & Restart`` button to save the change. -- Restart the Docker Engine by clicking on the Docker icon in the menu bar and selecting :code:`Restart`. -{% /tab %} -{% tab label="Rancher" %} -## Rancher - -Use Rosetta for emulation, and we recommend changing the file sharing implementation to VirtioFS: -- Open Rancher Desktop's ``Preferences`` panel. -- Open the ``Virtual Machine`` section. -- Open the ``Emulation`` tab -- Set ``Virtual Machine Type`` to ``VZ``. -- In the ``VZ`` suboption, enable ``Rosetta``. -- Open ``Volumes`` tab, and set the ``Mount Type`` to ``virtiofs``. -- Use the ``Apply`` button to save the changes. -- In order to use the ``cuno-mac`` utility with Rancher, the "dockerd" (moby) container engine must be used: -- Open Rancher Desktop's ``Preferences`` panel. -- Open the ``Container Engine`` section. -- In the ``General`` tab, set the ``Container Engine`` to ``dockerd (moby)``. -- Use the ``Apply`` button to save the change. -{% /tab %} -{% tab label="OrbStack" %} -## OrbStack - -Usage of Object Mount on [OrbStack for Mac](https://orbstack.dev/) containers works without any additonal set up required. - -OrbStack does not currently work with the native/host ``cuno-mac`` utility. -{% /tab %} -{% /tabs %} -{% /tab %} -{% /tabs %} - -2. Download the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the [Scripted Installer](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno-glibc-installer.run). - -3. Install the package onto your Mac directly, or inside a Linux container: -{% tabs %} -{% tab label="Install on macOS" %} -## Install on macOS -Installing Object Mount on to your Mac directly will allow you to use ``cuno-mac``, our handy tool for launching Linux containers that automatically have your local installation of Object Mount installed inside them. - -NB: To use ``cuno-mac``, you must also have Python 3 installed and available as ``python3`` in Terminal. - -In a Terminal on your Mac, navigate to your download directory. If this was the default ``Downloads`` folder, you can run: -```shell -cd /Users/$USER/Downloads -``` -Run the installer (without ``sudo``): -```shell -sh cuno-glibc-installer.run -``` - -Please follow the interactive steps, read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the installation. -- Export the ``cuno-mac`` location to your ``PATH`` variable: -```shell -export PATH=$PATH:~/.local/opt/cuno/share/macos -``` -- To have this persist and apply to new Terminal sessions, add this as a new line at the bottom of the file ``/Users//.bash_profile`` using a text editor, or using the following command in a Terminal session: -```shell -echo "PATH=\$PATH:~/.local/opt/cuno/share/macos" >> ~/.bash_profile -``` -**WARNING:** Do not miss the ``\`` because otherwise your current ``PATH`` will be written literally into the profile and break future changes. -{% /tab %} -{% tab label="Install in Linux containers" %} -## Install in Linux containers -If installing on your Mac's local user is not preferred, you can install Object Mount within a container of your choice. - -We provide instructions here for installing within Docker containers using Ubuntu images. These instructions can be adapted to other Linux distributions, as well other virtualisation technologies such as Rancher, and OrbStack. - -These instructions depend on the architecture of you Mac, please follow the appropriate guide: -{% tabs %} -{% tab label="Intel" %} -## Intel -- Run the following command to run a Docker container with the Object Mount installer made available at ``/tmp/cuno_install``: - -**WARNING:** This command will download the latest stable Ubuntu image, which could be large. -```shell -docker run -it --rm --entrypoint /bin/bash \ --v /cuno-glibc-installer:/tmp/cuno_install \ ---name cuno-container \ -ubuntu:latest -``` -- Within the Docker container (using ``docker exec -it cuno-container /bin/bash``), install Object Mount: -```shell -sh /tmp/cuno_install -``` -- Please follow the interactive steps, read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the installation. -{% /tab%} -{% tab label="Apple Silicon (ARM)" %} -## Apple Silicon (ARM) -- Run the following command to run a Docker container with the Object Mount installer made available at ``/tmp/cuno_install``: - -**WARNING:** This command will download the latest stable Ubuntu image, which could be large. -```shell -docker run -it --rm --entrypoint /bin/bash \ --v :/tmp/cuno_install \ ---name cuno-container \ ---platform linux/amd64 ubuntu:latest -``` - -NB: The :code:`--platform linux/amd64` flag is *required* to run Object Mount on Macs with Apple Silicon. - -- Within the Docker container (using ``docker exec -it cuno-container /bin/bash``), install Object Mount: -```shell -sh /tmp/cuno_install -``` -- Please follow the interactive steps, read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the installation. -{% /tab%} -{% /tabs %} -{% /tab %} -{% /tabs %} - -4. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. diff --git a/app/(docs)/object-mount/linux/installation/microsoft-wsl/page.md b/app/(docs)/object-mount/linux/installation/microsoft-wsl/page.md new file mode 100644 index 000000000..20e6ca9ff --- /dev/null +++ b/app/(docs)/object-mount/linux/installation/microsoft-wsl/page.md @@ -0,0 +1,32 @@ +--- +title: Windows (WSL) +hideTitle: true +docId: bekoo5aenePoo7Oh +weight: 7 +redirects: + - /object-mount/linux/installation/windows +metadata: + title: Windows (WSL) + description: + Installation Instructions for running Object Mount for Linux using Windows Subsystem for Linux. +hidden: false +--- + +# Windows Subsystem for Linux Installation + +Storj provides a full-featured, native version of [Object Mount for Windows](docId:TSMB5yXSc4FcqwXj). + +You are **strongly encouraged** to use that native Windows version. + +However, if you would like to install the Linux version of Object Mount within your Windows environment, you can leverage 🌐 [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/install). + +Installation of Object Mount for Linux within a Linux virtual machine or container is also possible. + + +## Windows Subsystem for Linux + +WSL installs Ubuntu by default. + +For Ubuntu, follow either the [glibc Scripted Installer](docId:ahWohd5eegh6eizi) instructions or the [Debian Package Installer](docId:aemie9zeiP9Nie2k) instructions. + +Once completed, you will be able to use Object Mount with any Linux applications also installed within the Linux subsystem. diff --git a/app/(docs)/object-mount/linux/installation/musl/page.md b/app/(docs)/object-mount/linux/installation/musl/page.md index 462d46835..b7071f67d 100644 --- a/app/(docs)/object-mount/linux/installation/musl/page.md +++ b/app/(docs)/object-mount/linux/installation/musl/page.md @@ -1,45 +1,103 @@ --- -title: Musl +title: musl +hideTitle: true docId: ao0yaeng2Aitheel - +weight: 2 metadata: - title: Musl + title: musl description: - Musl Installation Instructions - + musl Scripted Installer Instructions +hidden: false --- +# musl Scripted Installer -The Scripted Installer allows you to install Object Mount without root privileges (no sudo). +Some Linux distributions (including Alpine) use the musl C standard library: 🌐 [musl](https://en.wikipedia.org/wiki/musl). -{% callout type="note" %} -This installation method assumes you have the `libgcc` package installed (due to a dependency on `libgcc_s.so`). On Alpine you can do (may require `sudo`): +Storj’s Object Mount **musl scripted installer** (a `.run` file) is compatible with all such Linux distributions. -```console -apk add libgcc -``` +This installer shell script allows you to install Object Mount (for the current user) _without_ needing root privileges — no `sudo` required. + +{% callout type="info" %} + **Optional: Package Installer** + + If you prefer a **full installer** for use with package managers (i.e.: Alpine, etc.) refer to the [](docId:MeiPie8EDuo7eise) installation instructions. {% /callout %} -1. Download the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the [latest musl version](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno-musl-installer.run) from your browser, or run the command: +Follow the steps below to install and configure Object Mount for **musl** platforms. + +{% callout type="warning" %} + **`libgcc` Dependency** + + The `musl` Scripted Installer instructions below assume you have the `libgcc` package installed (due to a dependency on `libgcc_s.so`). + + If necessary, run the following command on your Alpine distro to add the `libgcc` package (may require `sudo`): + + ```console + apk add libgcc + ``` +{% /callout %} + + +## Step 1. Download the Install Script + +Download the latest scripted installer. + +- **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + +- **Note:** Although the Object Mount `musl` scripted installer file is named `cuno-mne-amd64-musl-installer.run`, it is designed to run on _both_ AMD64 and Intel x86_64 processors (although not ARM-based CPUs). + + +## Step 2. Install & Activate Object Mount + +- **Note:** If you do not yet have a license key, you can install Object Mount for Linux in a fully-featured **Free Trial** mode. Reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) for more details. + +**2a.** To install Object Mount for use _only_ by the **current user**, run the following command from a terminal window: + +```shell +# terminal +sh cuno-mne-amd64-musl-installer.run +``` + +**2b.** To install Object Mount to be used by **all users** system-wide, add `sudo` to the _front_ of the command: + +```shell +# terminal +sudo sh cuno-mne-amd64-musl-installer.run +``` + +**2c.** The installation process will prompt you to read and agree to the Object Mount **end-user license agreement (EULA)**. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-eula.jpg) + +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"`. (This is equivalent to accepting the EULA). + +**2d.** The installation process will then prompt you to **activate Object Mount**. + +- Press 1 to activate a **free trial**: + - This will activate a fully-featured 14-day free trial. +- Press 2 if you already have a **license key**: + - Paste your license key or enter a full path to a license key file. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-enter-license-key.jpg) - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cuno-musl-installer.run - ``` +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. -2. To install only for the current user, run the following terminal command: - ```console - sh cuno-musl-installer.run - ``` +## Installation Locations: Scripted Installers - To install system-wide, run: +When Object Mount is installed via a **Scripted Installer** (e.g.: glibc or musl), components are installed to different locations depending on whether it was installed by a root, or by a non-root user. - ```console - sudo sh cuno-musl-installer.run - ``` +| Install Method | Location (`CUNO_ROOT`) | Symlinks created in | +|--------------------|------------------------|--------------------------------| +| Non-Root User | `~/.local/opt/cuno/` | `~/.local/bin/` +| | | `~/.local/lib/x86_64-linux-gnu` +| | | `~/.local/share/man/` +| Root User (`sudo`) | `/opt/cuno/` | `/usr/bin/` +| | | `/usr/lib/` +| | | `/usr/share/man/` -3. The installation will prompt you to read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the installation. To automate this, you can set the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"` (this is equivalent to accepting the EULA). -4. The installation will prompt you to activate Object Mount by starting a trial or entering a license key. Follow the steps interactively, or automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. +## Next Steps -5. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. +_Once Object Mount is installed_, proceed to the Linux User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. diff --git a/app/(docs)/object-mount/linux/installation/page.md b/app/(docs)/object-mount/linux/installation/page.md index ca8933a08..aa09d2edc 100644 --- a/app/(docs)/object-mount/linux/installation/page.md +++ b/app/(docs)/object-mount/linux/installation/page.md @@ -1,67 +1,65 @@ --- -title: Installation Instructions -docId: doh9ma3vidoo4Eef -weight: 3 +title: Installation Guides +hideTitle: true +docId: iethahkeeX5EiJoh +weight: 2 +redirects: + - /object-mount/linux/installation + - /object-mount/linux/installation/mac metadata: - title: Installation + title: Linux Installation Overview description: - Object Mount Installation Instructions + Overview of the installation next steps for Linux. +hidden: false --- -Use the selector to guide your installation: -{% tabs %} -{% tab label="Linux" %} -{% tabs %} -{% tab label="Debian/Ubuntu" %} -{% tabs %} -{% tab label="Scripted Installer" %} -[glibc](./installation/glibc) -{% /tab %} -{% tab label="Package Manager" %} -[Debian](./installation/debian) -{% /tab %} -{% /tabs %} -{% /tab %} -{% tab label="Red Hat & derivatives" %} -{% tabs %} -{% tab label="Scripted Installer" %} -[glibc](./installation/glibc) -{% /tab %} -{% tab label="Package Manager" %} -[RedHat](./installation/redhat) -{% /tab %} -{% /tabs %} -{% /tab %} -{% tab label="Alpine" %} -{% tabs %} -{% tab label="Scripted Installer" %} -[musl](./installation/musl) -{% /tab %} -{% tab label="Package Manager" %} -[Alpine](./installation/alpine) -{% /tab %} -{% /tabs %} -{% /tab %} -{% tab label="Any other distribution" %} -{% tabs %} -{% tab label="Scripted Installer (glibc)" %} -Most Linux distributions use the GNU Project's C standard library (glibc). This installer/version is compatible with all such Linux distributions. - -[glibc](./installation/glibc) -{% /tab %} -{% tab label="Scripted Installer (musl)" %} -Some Linux distributions use the musl C standard library. This installer/version is compatible with all such Linux distributions. These instructions will assume you are using Alpine or a derivative. - -[musl](./installation/musl) -{% /tab %} -{% /tabs %} -{% /tab %} -{% /tabs %} -{% /tab %} -{% tab label="MacOS" %} -[MacOS Installation Instructions](./installation/mac) -{% /tab %} -{% tab label="Windows" %} -[Windows Installation Instructions](./installation/windows) -{% /tab %} -{% /tabs %} +# Linux Installation Overview + +Storj’s Object Mount supports multiple flavors of Linux and can also run inside Docker and other virtual environments, such as Microsoft’s Hyper-V and Oracle’s VirtualBox. + +Object Mount offers both **Scripted Installers** (which do _not_ require root privileges to install for a single user) as well as **Package Manager** installation files for different Linux platforms. + + +## General Installation Procedure + +The general steps to install and activate Object Mount are the _same_ regardless of your specific Linux distribution. + +A outline of these steps include: + +1. **Obtain** an Object Mount license key. + + - **Note:** If you do not yet have a license key, you can install Object Mount for Linux in a fully-featured **Free Trial** mode. Reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) for more details. + +2. **Download** the latest scripted installer or installation package for your Linux distribution. + + - **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + +3. **Install & Activate** Object Mount. + +_Once Object Mount is installed_, proceed to the User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. + + +## OS-Specific Installation Procedures + +Equipped with your installation download link and your Object Mount license key, proceed to the installation instructions for your specific OS and preferred installation method: + +| **Scripted Installers** | **Installation Guide** | +|-------------------------------------------|------------------------| +| **glibc** (C standard library) | [glibc Scripted Installer](docId:ahWohd5eegh6eizi) +| **musl** (C standard library) | [musl Scripted Installer](docId:ao0yaeng2Aitheel) + +| **Package Managers** | **Installation Guide** | +|-------------------------------------------|------------------------| +| **Alpine** (and APM, etc.) | [APK Package Installer](docId:MeiPie8EDuo7eise) +| **Debian** (and Ubuntu, etc.) | [Debian Package Installer](docId:aemie9zeiP9Nie2k) +| **Red Hat** (and RPM, RHEL, CentOS, etc.) | [Red Hat Package Installer](docId:woosaugaiNohree9) + +| **Other** | **Installation Guide** | +|-------------------------------------------|------------------------| +| **Windows** (WSL) | [Windows (via Windows Subsystem for Linux [WSL])](docId:bekoo5aenePoo7Oh) + +**Other Distro? Not Sure?** + +Most Linux distributions use either the GNU Project’s C standard library 🌐 [glibc](https://en.wikipedia.org/wiki/glibc) or the 🌐 [musl](https://en.wikipedia.org/wiki/musl) C standard library. + +Check your distribution, then follow instructions for either our [glibc Scripted Installer](docId:ahWohd5eegh6eizi) or our [musl Scripted Installer](docId:ao0yaeng2Aitheel). diff --git a/app/(docs)/object-mount/linux/installation/redhat/page.md b/app/(docs)/object-mount/linux/installation/redhat/page.md index 1f275ee5f..aeed77c5b 100644 --- a/app/(docs)/object-mount/linux/installation/redhat/page.md +++ b/app/(docs)/object-mount/linux/installation/redhat/page.md @@ -1,43 +1,97 @@ --- title: Red Hat +hideTitle: true docId: woosaugaiNohree9 - +weight: 4 metadata: title: Red Hat description: - Red Hat Installation Instructions - + Red Hat Package Installer Instructions +hidden: false --- -1. Donwload the installer. By downloading you are agreeing to the terms of the [End User License Agreement](https://www.storj.io/legal/terms-of-use). Click to download the latest version of the [self-extracting package for RHEL](https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_x86_64_glibc_rpm.run) from your browser, or run the command: +# Red Hat Package Installer + +Storj provides an Object Mount package installer for Red Hat Linux (and 🌐 [RHEL-based Linux distributions](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux_derivatives)) + +{% callout type="info" %} + **Optional: Scripted Installer** + + If you prefer to use a glibc-based **scripted installer**, refer to the [](docId:ahWohd5eegh6eizi) installation instructions. +{% /callout %} + +Follow the steps below to install and configure Object Mount for Red Hat-based Linux platforms. + + +## Step 1. Download the Installer Package + +Download the latest package installer. + +- **Note:** If you do not have a download link, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + + +## Step 2. Unpack the Archive + +Use the installer’s `.run` script to unpack the archive by entering the following command from a terminal window: + +```console +sh cuno_mne_x86_64_glibc_rpm.run +``` + +{% callout type="info" %} + **Processor Compatibility** + + Although the Red Hat distro file naming convention is to only specify `X86_64` (ex: `cuno_mne_x86_64_glibc_rpm.run`), the Object Mount YUM/RPM Installer Package is designed to run on _both_ AMD64 and Intel x86_64 processors (although not ARM-based CPUs). + + **Note:** ARM-based processors are not supported at this time. +{% /callout %} + +The unpacking process will prompt you to read and agree to the Object Mount **end-user license agreement (EULA)**. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-eula.jpg) + +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"`. (This is equivalent to accepting the EULA). + +The archive and its contents will be unpacked into a directory named `cuno_{VERSION}_x86_64_glibc_rpm` containing the package itself plus some additional files. + +View the unpacked contents with the following `ls` command (substituting your version number): + +```console +$ ls cuno_1.2.8.8_x86_64_glibc_rpm +cuno_1.2.8.8_x86_64_glibc.rpm +CUNO-Installation-and-Usage-Guide.pdf +``` + + +## Step 3. Install Object Mount and its dependencies using yum + +**3a.** Use the following commands to extract and install Object Mount (substituting your version number): + +```console +sudo yum makecache +sudo yum install ./cuno_1.2.8.8_x86_64_glibc_rpm/cuno_1.2.8.8_x86_64_glibc.rpm +``` - ```console - wget https://github.com/cunoFS/cunoFS/releases/latest/download/cuno_x86_64_glibc_rpm.run - ``` +The installer will fetch and install any necessary additional packages (such as `libfuse`, etc.). -2. Unpack the archive by running: +**3b.** Next, the installation process will prompt you to **activate Object Mount**. - ```console - sh cuno_x86_64_glibc_rpm.run - ``` +- Press 1 to activate a **free trial**: + - This will activate a fully-featured 14-day free trial. +- Press 2 if you already have a **license key**: + - Paste your license key or enter a full path to a license key file. -3. Please follow the interactive steps, read the displayed end-user licence agreement (EULA) and agree to the terms to continue with the extraction. To automate this, you can set the environment variable `CUNO_INSTALL_ACCEPT_EULA="yes"` (this is equivalent to accepting the EULA). + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-enter-license-key.jpg) -4. The archive and its contents will be unpacked into a directory named `cuno_{FULL-VERSION}_x86_64_glibc` containing the package itself and additional documents: +- **Note:** You can automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. - ```console - $ ls cuno_{FULL-VERSION}_x86_64_glibc - cuno_{FULL-VERSION}_x84_64_.rpm - CUNO-Installation-and-User-Guide.pdf - ``` +{% callout type="note" %} + **Access for: All-Users** -5. Install Object Mount and its dependencies using `yum`: + By using `sudo`, all package manager installations of Object Mount install the app for use by _all users_ of the system, not just the account that initiated the install process. +{% /callout %} - ```console - sudo yum makecache - sudo yum install ./cuno_{FULL-VERSION}_x86_64_glibc/cuno_{FULL-VERSION}_x86_64_glibc.rpm - ``` -6. The installation will prompt you to activate Object Mount by starting a trial or entering a license key. Follow the steps interactively, or automate this step by setting the environment variable `CUNO_INSTALL_LICENSE` to one of `trial | none | | `. +## Next Steps -7. The installation will prompt you to set the `CUNO_ROOT` environment variable to the installation directory. This is not always necessary when using Object Mount, but it will make it easier to follow the steps in this guide. +_Once Object Mount is installed_, proceed to the Linux User Guide article: [](docId:JDK2ED8HGFmyaxk) to ensure you can access both public and private Object Storage buckets. diff --git a/app/(docs)/object-mount/linux/installation/windows/page.md b/app/(docs)/object-mount/linux/installation/windows/page.md deleted file mode 100644 index 5e3c7d9e1..000000000 --- a/app/(docs)/object-mount/linux/installation/windows/page.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: Windows -docId: bekoo5aenePoo7Oh - -metadata: - title: Windows - description: - Windows Installation Instructions - ---- - -A fully native Windows client is in private beta, with a public beta available soon! - -However, if you would like to install the Linux version of Object Mount on Windows, you can leverage Windows Subsystem for Linux 2 (WSL2): [Install WSL on Windows](https://learn.microsoft.com/en-us/windows/wsl/install) or inside a Linux virtual machine. - -WSL installs Ubuntu by default, so the installation instructions for [Debian](../debian) can then be followed. You will be able to use Object Mount with any Linux applications also installed in the Linux subsystem. - diff --git a/app/(docs)/object-mount/linux/page.md b/app/(docs)/object-mount/linux/page.md index a0a189033..51c991184 100644 --- a/app/(docs)/object-mount/linux/page.md +++ b/app/(docs)/object-mount/linux/page.md @@ -1,14 +1,132 @@ --- title: Linux hideTitle: true -docID: wxtofwqcb5f2 -weight: 2 +docId: wxtofwqcb5f2 +weight: 5 metadata: title: Linux Installation and Usage description: - Details about installing and using Object Mount Linux Package on various OS + Home Page for guidance on installing and using Object Mount on various Linux OS versions. +hidden: false --- # Linux Installation and Usage -The pages within this set are all dedicated to installation of the Linux Package on various operating systems. +This section contains everything you need to understand, install and use **Object Mount** on a Linux machine. + +Whether you’re setting up for creative workflows, remote storage access, or team-based collaboration, these pages will walk you through installation, setup, and day-to-day use. + + +## Linux Package + +Object Mount is a scalable, high-performance POSIX compatibility layer that lets you interact with files stored on object storage such as Amazon S3, Storj Object Storage, Azure Blob Storage, Google Cloud Storage, or any S3-compatible object store hosted in the cloud or locally. + +**Object Mount for Linux** retains much of its original look and feel from its **cunoFS** heritage. + +Object Mount for Linux includes an Object Mount Command Line Interface (CLI), `cuno`, providing the highest performance and most straightforward way to interact with object storage. You will encounter many `cuno`-related commands, settings, and custom variables throughout your configuration and daily use of Object Mount for Linux. + +Object Mount functions via a user-mode library, `cuno.so`, which acts as a 🌐 [dynamic linker/loader](https://man7.org/linux/man-pages/man8/ld.so.8.html) intercepting applications (both dynamic and static) via fast, dynamic binary instrumentation. + + +## Object Mount: OS Version Differences + +Storj’s Object Mount for Linux, with its CLI-heavy user interface, is in stark contrast to the Graphical User Interface (GUI) experience that Object Mount presents to Windows and macOS users. + +Other OS differences include: + +| **Object Mount Feature** | **For Linux** | **For Windows & macOS** | +|--------------------------|---------------------|-------------------------| +| CLI-based | 100% | Limited CLI +| GUI-based | No GUI available | 100% +| POSIX Mode Features | Wide array of settings | Either `on` or `off` +| `cuno`-based commands, settings, and variables | Large quantity of options | Limited options + + +## What’s Covered + +The many articles in this Linux section will cover the installation, configuration, and use of Object Mount on multiple flavors of Linux. Navigate to what you need, when you need it. + +Object Mount’s Linux-specific content includes: +- Getting Started Guides +- Installation Guides +- User Guides +- Advanced Guides + +Click below to access the content you are looking for: + +### Getting Started Guides + +_Prior to installing_ Object Mount for Linux, view the Linux-specific [Getting Started Guide](docId:xhNvtETAA6UBZVNH) articles to better understand the capabilities, options, and deployment choices available. + +Getting Started topics include: + - Understanding POSIX Compatibility and Options in Object Mount for Linux + - Understanding Deployment Modes for Object Mount for Linux + +### Installation Guides + +When _ready to install_, view the Object Mount for Linux [Installation Guides](docId:iethahkeeX5EiJoh) for your particular Linux variant, including: + + - Alpine + - Debian & Ubuntu + - Red Hat + - RHEL & CentOS + - glibc (standard C library) + - musl (standard C library) + - Windows - via Windows Subsystem for Linux (WSL) + +### User and Configuration Guides + +_After installation_, read through the Linux [User Guides](docId:ohs0ailohSh0Vie3) to learn more about using and configuring Object Mount for Linux, including details on: + + - Validating Functionality & Access Object Storage + - Activating License Keys + - The different Linux Deployment Modes + - Direct Interception Mode + - Object Mount on FUSE + - Object Mount FlexMount + - Object Mount Fusion + - Enabling Enforced POSIX File Access + - Cloud Path Details + - Application Tips + - Media & Entertainment and High Performance COmputing Modes + +### Advanced Guides + +Once up and running, explore the Linux-specific _advanced configuration tips_ found in the Object Mount for Linux [Advanced Guides](docId:GeWpcQTuVpL6d3wU). These articles will help you fine-tune your deployment, improve Object Mount’s performance, and increase your productivity. + +Topics include: + + - Advanced Credential Options + - Advanced Loading Options + - Advanced Configuration Options + - Server-Side Encryption + - Kubernetes CSI Driver + - Machine Learning Accelerator + - Technical Limitations + - Uninstalling Object Mount + + +## Additional Reference Material + +Once your Linux deployment of Object Mount is up and running, there are several **Global Features** and **Media Workflow Guides** that you can reference, including: + +**Global Features & Concepts:** + +See the many articles in the [Appendix](docId:xzAqS9Um7xjQSbmD) for additional details on features that apply to Object Mount on _all_ operating systems (macOS, Windows, and Linux): + +- Object Storage Core Concepts +- POSIX explained +- S3 Credential Management +- Supported S3 Providers +- Performance Tuning +- Troubleshooting Guide +- FAQ +- Release Notes + +**Media Application & Workflow Guides:** + +Detailed usage tips and recommendations for [Media Workflows](docId:JXF4_NLsV@nh9QcT), including: + + - Using Object Mount with Adobe Premiere Pro + - Using Object Mount with Avid Media Composer + - Using Object Mount with DaVinci Resolve diff --git a/app/(docs)/object-mount/linux/user-guides/_meta.json b/app/(docs)/object-mount/linux/user-guides/_meta.json deleted file mode 100644 index 8a7f3055a..000000000 --- a/app/(docs)/object-mount/linux/user-guides/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "User Guides", "weight": 4} diff --git a/app/(docs)/object-mount/linux/user-guides/access/page.md b/app/(docs)/object-mount/linux/user-guides/access/page.md deleted file mode 100644 index 42a931e00..000000000 --- a/app/(docs)/object-mount/linux/user-guides/access/page.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -title: Cloud Paths -docId: jieteeYeyievui9k - -metadata: - title: Cloud Paths - description: Cloud Paths Guide - -weight: 9 ---- -## Transparent access to object storage - -Object Mount provides transparent access to remote cloud objects. -Users can continue using their favorite software tools, pipelines and filesystem commands without any modifications. -Object Mount offers virtual versions of the original files that are streamed in real-time. -Object Mount also supports random access reads and writes of any length, minimizing network bandwidth usage and processing time. - -## Specifying cloud paths - -There are two ways of interacting with the remote files and directories: - -- the URI format, e.g. `s3://my-bucket/dir/file.txt` -- the directory format, e.g. `/cuno/s3/my-bucket/dir/file.txt` - -The following table describes URI prefixes for supported cloud providers. - -| Cloud provider | URI format | -| -------------------- | -------------------------------------------- | -| AWS S3 | `s3://` | -| Google Cloud Storage | `gs://` | -| Azure Blob Storage | `az:///` | - -{% callout type="warning" %} -Some software tools cannot process the URI format. -Use the directory format instead in such cases. -{% /callout %} - -### Examples - -Some examples of using Object Mount are given below. - -1. List the contents of a remote container `my-bucket` on AWS S3: - - ```console - ls s3://my-bucket - ``` - - or - - ```console - ls /cuno/s3/my-bucket - ``` - -2. Move a text file `sample.txt` from the local filesystem to a remote Google Cloud Storage container `my-bucket`: - - ```console - mv ./sample.txt gs://my-bucket/dir/sample.txt - ``` - - or - - ```console - mv ./sample.txt /cuno/gs/my-bucket/dir/sample.txt - ``` - -3. List the contents of a subdirectory `dir` in a remote Google Cloud Storage container `my-bucket`: - - ```console - ls gs://my-bucket/dir - ``` - - or - - ```console - ls /cuno/gs/my-bucket/dir - ``` - -4. Use UNIX coreutils (e.g. `head`) on a remote file `sample.txt` on AWS S3: - - ```console - head s3://my-bucket/dir/sample.txt - ``` - -## Microsoft Azure Blob Storage usage - -The associated Account Name must be provided in the corresponding URI: - -```console -ls az://my-account-name/my-container -``` - -or, equally: - -```console -ls /cuno/az/my-account-name/my-container -``` - -## AWS S3 Access Point support - -Object Mount supports using an AWS S3 Access Point instead of a bucket as an endpoint. Access points are unique addresses that customers can create to enforce distinct permissions and network controls for any request made specifically through that access point. - -To use an Access Point instead of a container, provide the full Amazon Resource Name (ARN) in place of the bucket name: - -``` -s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name -``` - -This applies to all places a container can be specified, such as within `cuno creds`, or on the command line: - -```console -$ cuno creds -i pair s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name -$ ls s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name/ -``` - -## File system behaviours when using Object Mount - -### Symbolic links - -By default, symbolic links can `only` be created (using `ln --symbolic` or `ln -s`) to point `from` a local system `to` a cloud location using either the URI or directory format. - -### Hard links and Server-side copy - -Hard links cannot be created from the local system to a remote cloud location. - -Use hard-linking to trigger a more efficient cloud-only copy mechanism within a cloud region. -`mv` or `cp` within a single cloud vendor will trigger a server-side copy, which is more efficient than a download and upload. - -### POSIX mode - -Symbolic links can be created (using `ln --symbolic` or `ln -s`) between local and cloud objects in either direction under POSIX mode (see [user-guide-object-mount-on-fuse](../user-guides/basic#object-mount-on-fuse)). - -Hard links can be created (using `ln`) between cloud objects in the `same` bucket under POSIX mode. - -{% callout type="warning" %} -If the target is deleted or moved, then both symbolic and hard links will break. -{% /callout %} - -### File metadata - -`ls --long` (or `ls -l`) will list the current user as the owner of the remote objects, and the permissions will be `777`. -The creation date of a remote directory is not always available to system calls. - -### Performance - -Object Mount uses multiple concurrent connections to remote objects to achieve the highest performance. -It can also execute operations on objects entirely remotely where possible. -Some optimised operations are listed below. - -- `mv` between locations within the same object storage provider uses remote-copy operations to avoid streaming through the client. -- `ln` hard links between locations within the same object storage provider use remote-copy operations to avoid streaming through the client. - : Hard links between objects in the same bucket will work like local hard links when in POSIX mode. diff --git a/app/(docs)/object-mount/linux/user-guides/activate/page.md b/app/(docs)/object-mount/linux/user-guides/activate/page.md new file mode 100644 index 000000000..da20be5f1 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/activate/page.md @@ -0,0 +1,62 @@ +--- +title: Activate a License Key +hideTitle: false +docId: BN6yJrwasPnZsaf +weight: 2 +redirects: + - /object-mount/linux/user-guides/activation +metadata: + title: Activate a License Key for Object Mount for Linux + description: + Steps to activate Object Mount on Linux. +hidden: false +--- + +## Activate using an Object Mount License Key + +When you first [installed Object Mount](docId:wxtofwqcb5f2) for Linux, you may have already added your license key. + + - If not, you may have selected the option to start a Free Trial. + - At the end of your free trial period you will need to register and obtain an Object Mount License Key. + - Or you may have renewed an expired license key. + +**Note:** If you do not yet have a license key, reach out to your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch). + +In any of the above situations, you can activate (or re-activate) Object Mount with your new license key by entering the `cuno creds activate` command and following the interactive steps: + +```shell +# terminal +cuno creds activate +``` + + +## Non-Interactive Activation + +Alternatively, for non-interactive license key activation, you can point the command to a license key file: + +```shell +# terminal +cuno creds activate "" +``` + +You can also pipe the license key into the command as its input: + +```shell +# terminal +echo "" | cuno creds activate +``` + +{% callout type="note" %} + **License File Location** + + Once activated, the location of the license file is: `$CUNO_ROOT/etc/license`. + + By default, the permissions on this file are set to `0600`. If needed, Administrators can manually modify the group and permissions to allow other system users access to the `cuno` CLI command. +{% /callout %} + + +## Activation for Multiple Users + +Normally, activation means that access to Object Mount is **limited to the one user** who runs the `cuno creds activate` command. + +However, if the user that runs `cuno creds activate` is `root`, then access can be optionally given to all users. diff --git a/app/(docs)/object-mount/linux/user-guides/activation/page.md b/app/(docs)/object-mount/linux/user-guides/activation/page.md deleted file mode 100644 index e07ca1a71..000000000 --- a/app/(docs)/object-mount/linux/user-guides/activation/page.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: Activate License -docId: Yee3Kun6axeeFeuh - -metadata: - title: Activate License - description: Activate License - -weight: 4 ---- - -## Licenses and activation -For licensing information, please schedule a [discovery call](https://www.storj.io/landing/get-in-touch). - -### Interactive - -Activate your licence by running the command and following the interactive steps: - -```console -cuno creds activate -``` - -{% callout type="note" %} -Normally, activation means that access to Object Mount is limited to the user who runs `cuno creds activate`. -However, if the user that runs `cuno creds activate` is `root`, then access can be optionally given to other users. -{% /callout %} - -### Non-interactive - -For non-interactive activation, you can supply a file to the command: - -```console -cuno creds activate "" -``` - -You can also pipe the licence in as input: - -```console -echo "" | cuno creds activate -``` - -{% callout type="note" %} -The location of the license file after activation is `$CUNO_ROOT/etc/license`. By default, the permissions on this file are set to `0600`. Administrators can manually modify the group and permissions to allow other system users accessing `cuno`. -{% /callout %} diff --git a/app/(docs)/object-mount/linux/user-guides/advanced/page.md b/app/(docs)/object-mount/linux/user-guides/advanced/page.md deleted file mode 100644 index 0d0f74cdd..000000000 --- a/app/(docs)/object-mount/linux/user-guides/advanced/page.md +++ /dev/null @@ -1,281 +0,0 @@ ---- -title: Advanced Loading -docId: airoogh4Waengi8u - -metadata: - title: Advanced Loading - description: Advanced Loading - -weight: 7 ---- - -This section describes advanced usage of Object Mount, including the user-mode library, scripting, containerisation, and high-performance computing (HPC) integration. - -## User-mode library - -{% callout type="note" %} -The `$CUNO_ROOT` environment variable should be set to wherever your Object Mount installation is: `/opt/cuno` for a system-wide installation, and usually `$HOME/.local/opt/cuno` for a user-local installation. See [user-guide-install-locations](../user-guides/install#install-locations) for more information on installation locations. -{% /callout %} - -As mentioned in [user-guide-overview](../user-guides), Object Mount provides `cuno.so` which may be loaded into a running process using `LD_PRELOAD`. -Set the environment variable `LD_PRELOAD="$CUNO_ROOT"/cuno.so` before executing a command, and Object Mount will be enabled for that command. -To start a `bash` instance with Object Mount loaded, run: - -```console -LD_PRELOAD="$CUNO_ROOT"/cuno.so bash -``` - -`bash` and any command executed from within this instance will run with Object Mount enabled. -However, other processes are not affected; this could be useful for unattended scripts. - -{% callout type="note" %} -Loading the library in this manner will {emphasis}`not` apply the `(cuno)` prefix to the prompt. -{% /callout %} - -## Environment variables - -If Object Mount was not installed to a default location, first set the `CUNO_ROOT` environment variable as described in [user-guide-install-locations](../user-guides/install#install-locations). Then, set the following environment variables for convenience: - -| Description | Environment Variable | Value | -|-------------|---------------------|-------| -| Load the Object Mount library | `LD_PRELOAD` | `"${CUNO_ROOT}"/lib/cuno.so${LD_PRELOAD+:$LD_PRELOAD}` | -| Path-free access to Object Mount executables | `PATH` | `"${CUNO_ROOT}"/bin:${PATH}` | -| Easy access to Object Mount manual pages | `MANPATH` | `"${CUNO_ROOT}"/share/man:${MANPATH}` | - -## Shell profiles - -To always load Object Mount when starting a new interactive shell, append the following line to `~/.bashrc` or `~/.zshrc`: - -```console -export LD_PRELOAD="${CUNO_ROOT}"/lib/cuno.so` -``` - -`/etc/profile.d` contains application-specific startup scripts, which are executed at user login by their respective login shells. -A sample script is provided in `${CUNO_ROOT}/etc/profile.d/cunorc.sh`. -Copy this to `/etc/profile.d/` to enable Object Mount for all users. - -## Containerisation and HPC - -### Docker - -There are multiple methods to use Object Mount from within Docker containers. - -#### Automatic interception - -{% callout type="note" %} -Support for automatic Docker interception is currently **experimental**. -For ways of running Docker with Object Mount interception manually, see [user-guide-manual-docker-interception](#manual-interception). -{% /callout %} - -To enable automatic Docker interception, set the environment variable `CUNO_INTERCEPT_DOCKER=1` and load Object Mount. -Launching a Docker container via `docker run …` will make Object Mount available inside the container. -For instance, the following code will run a command inside an Ubuntu container with Object Mount enabled with the host's Object Mount credentials: - -```console -export CUNO_INTERCEPT_DOCKER=1 -cuno run docker run --rm ubuntu:latest ls -l s3://bucket -``` - -Object Mount options will be forwarded into the container. -To override Object Mount options just for Docker containers, use the `CUNO_DOCKER_OPTIONS` environment variable: if set, its value overrides that of `CUNO_OPTIONS` inside Docker containers. - -#### Manual interception - -Besides automatic interception, the following methods can be used to enable Object Mount in Docker containers: - -- Install Object Mount as part of the creation of the Docker image (see [user-guide-download-and-install](../user-guides/install)); -- Inject Object Mount at launch into an existing Docker image. - -To inject Object Mount into a Docker container when it is launched: - -1. Set `CUNO_CREDENTIALS` outside the container; see [user-guide-credentials-management](../user-guides/credentials) for details. -2. Optionally, the `CUNO_OPTIONS` environment variable can be set using the `--env` [option](https://docs.docker.com/engine/reference/commandline/run/#env) of `docker run`. -3. Use the following options when using `docker run`: - ```console - docker run \ - --tmpfs /cunodb \ - -v $CUNO_ROOT:/opt/cuno:ro \ - -v /opt/cuno/etc/ld.so.preload:/etc/ld.so.preload:ro \ - -v $CUNO_CREDENTIALS:/opt/cuno-config/creds:ro \ - [container-commands] - ``` - - {% callout type="note" %} - This uses the volume mount option (`-v`) to make Object Mount and other directories available to the container. - - This command requires Object Mount to already be installed and [activated](../user-guides/activation) on the host system. - - The `$CUNO_ROOT` environment variable should be set to wherever your Object Mount installation is: `/opt/cuno` for a system-wide installation, and usually `$HOME/.local/opt/cuno` for a user-local installation. See [user-guide-install-locations](../user-guides/install#install-locations) for more information on installation locations. - {% /callout %} - -The Object Mount credentials directory is only readable by the current user for security reasons. -However, credentials may need to be accessed by processes run by other users (e.g. NGINX worker processes) within the Docker container. -To enable this, bind mount the nested `bindpoint` directory instead of the credentials directory: - -```console -docker run \ - --tmpfs /cunodb \ - -v $CUNO_ROOT:/opt/cuno:ro \ - -v /opt/cuno/etc/ld.so.preload:/etc/ld.so.preload:ro \ - -v $CUNO_CREDENTIALS/bindpoint:/opt/cuno-config/creds/bindpoint:ro \ - [container-commands] -``` - -## Kubernetes - -You can use Object Mount on Kubernetes clusters with the Object Mount Kubernetes CSI Driver. -Please refer to [the Object Mount K8s CSI Driver page](../user-guides/k8s). - -## Singularity - -There are multiple methods for enabling [Singularity] images with Object Mount, allowing transparent access to cloud hosted files: - -- Install Object Mount as part of the creation of the Singularity image -- Inject the Object Mount library at launch into an existing Singularity image - -The following Singularity definition file will install Object Mount into an image: - -``` -Bootstrap: docker -From: rockylinux:8 - -%files - /home/admin/downloads/cuno-{FULL-VERSION}.x86_64.rpm /opt/src/ - -%post - yum update -y - yum install -y /opt/src/cuno-{FULL-VERSION}.x86_64.rpm - echo "YOUR LICENCE KEY HERE" | cuno creds activate - chmod og+r /opt/cuno/etc/license - -%environment - export LD_PRELOAD=/usr/lib/cuno.so - export CUNO_CREDENTIALS=/opt/cuno-config/creds - -%labels - Name object mount - URL storj.io - Email supportdcs@storj.io -``` - -The above will require that the Object Mount software package is available at the paths starting `/home/admin/downloads` as listed in the `%files` section. - -Rather than modifying an existing Singularity image, another option is to pass-through the Object Mount library and binaries making them available inside the image. Optionally, the `CUNO_OPTIONS` environment variable can also be set for a particular mode of operation. - -The following commands demonstrate injecting Object Mount into a Singularity image: - -``` -$ singularity exec \ - --bind $CUNO_ROOT:/opt/cuno \ - --bind $CUNO_ROOT/etc/ld.so.preload:/etc/ld.so.preload \ - --bind "$CUNO_CREDENTIALS":/opt/cuno-config/creds \ - ./singularity_image.sif ls s3://commoncrawl - -cc-index crawl-002 hive_analysis meanpath projects wikipedia -contrib crawl-analysis index2012 parse-output robots.txt -crawl-001 crawl-data mapred-temp parse-output-test stats-output -``` - -{% callout type="note" %} -This method requires Object Mount be installed and [activated](../user-guides/activation) on the system that is used to execute the Singularity images. - -The `$CUNO_ROOT` environment variable should be set to wherever your Object Mount installation is: usually `/opt/cuno` for a system-wide installation, and usually `$HOME/.local/opt/cuno` for user-local installion. See [user-guide-install-locations](../user-guides/install#install-locations) for more information on installation locations. -{% /callout %} - -Alternatively, rather than using command options, the `--bind` parameters can be specified as environment variables before the `singularity` binary is called, making it easier to modify existing pipelines without changing the command line call, like so: - -``` -export SINGULARITY_BIND="$CUNO_ROOT:/opt/cuno,$CUNO_CREDENTIALS:/opt/cuno-config/creds" -``` - -As another alternative, Object Mount can be used via a FUSE mount inside Singularity with the `--fusemount` parameter: - -``` -$ singularity exec \ - --bind "$CUNO_ROOT":/opt/cuno \ - --bind "$CUNO_CREDENTIALS":/opt/cuno-config/creds \ - --fusemount 'container:/opt/cuno/bin/cuno mount --root /cuno /cuno' \ - ./singularity_image.sif /cuno/s3/commoncrawl -``` - -This will allow software inside the container to access cloud buckets via the `/cuno` path prefix. - -## Lmod - -The Environment Modules system is a tool to help users manage shell environments, by allowing groups of related environment variable settings to be made or removed dynamically. - -Many HPC environments incorporate `module` as one of the means by which software can be loaded by their users. - -A sample script has been provided within your Object Mount installation directory here: `${CUNO_ROOT}/share/modulefiles/cuno/{FULL-VERSION}.lua`, it is built specifically for the Lmod implementation in Lua, and uses values set to example paths (see the `base` variable in particular) and default options that you might want to append to, modify or remove to suit your environment and your installation choices. - -After adding the parent directory of the file to the `MODULEPATH` with `module use "${CUNO_ROOT}/share/modulefiles/cuno"`, a user should be able to run `module load cuno` to add Object Mount to their environment. - -## Spectrum LSF - -Depending on the exact requirements and setup, there are multiple ways of using Spectrum LSF with Object Mount. - -First, assuming that Object Mount is installed on all compute nodes, that the `cuno` executable is available in the `PATH`, and that Object Mount credentials are configured for all nodes, jobs can be submitted by prefixing the submitted command with `cuno run` (without quotes!), e.g.: - -``` -bsub -Is cuno run ls -l s3://bucket -``` - -If further configuration of Object Mount is required, an LSF job starter can be created. The script should be created at the location `$LSF_ENVDIR/cuno-starter.sh`, or some other location that is common to all LSF nodes (however, the queue template that ships with Object Mount assumes this location). The following script can be used as an outline for the job starter: - -```shell -#!/bin/sh - -cuno_basedir= -# e.g.: -# cuno_basedir=/opt/cuno - -# Further configuration as necessary, e.g.: -# export CUNO_CREDENTIALS=/usr/share/cuno-creds -# export CUNO_OPTIONS=+cloudroot=acme - -"$cuno_basedir"/bin/cuno run "$@" -``` - -Make the job starter executable: - -```console -chmod +x "$LSF_ENVDIR"/cuno-starter.sh -``` - -To configure a new LSF queue for Object Mount enabled jobs, fill in and append the LSF configuration template for Object Mount to `lsb.queues`. -Use the following command: - -``` -cluster_name=$(lsid | grep 'cluster name' | awk '{print $NF}') -cuno run sh -c " - sed 's;@LSF_ENVDIR@;$LSF_ENVDIR;' \"\$CUNO_BASEDIR\"/share/lsf/lsb.queues \ - >>\"$LSF_ENVDIR\"/lsbatch/$cluster_name/configdir/lsb.queues -" -``` - -Afterwards, LSF needs to be reconfigured: - -``` -badmin reconfig -``` - -To check that the queue was set up correctly, run: - -``` -bqueues -l cuno -``` - -Regardless of whether an LSF queue was set up, the job starter can be used with the `lsrun` command (but not `bsub`!), e.g.: - -```console -export LSF_JOB_STARTER=$LSF_ENVDIR/cuno-starter.sh -lsrun ls -l s3://bucket -``` - -To use the dedicated queue, specify it when invoking `bsub`, e.g.: - -```console -bsub -q cuno -Is ls -l s3://demo -``` - -[singularity]: https://sylabs.io diff --git a/app/(docs)/object-mount/linux/user-guides/app-tips/page.md b/app/(docs)/object-mount/linux/user-guides/app-tips/page.md new file mode 100644 index 000000000..5c8074b8f --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/app-tips/page.md @@ -0,0 +1,159 @@ +--- +title: Application Tips +hideTitle: false +docId: lawaDooteicha9li +weight: 6 +redirects: + - /object-mount/linux/user-guides/tips +metadata: + title: Application Tips + description: + Application Tips when using Object Mount for Linux. +hidden: false +--- + +This article provides additional recommendations and guidance when using Object Mount for Linux with the following applications: + + +## Python + +Python’s `os.path.realpath(path)` does _not_ support [URI formatted](docId:jieteeYeyievui9k) cloud paths (e.g.: `s3:///path`). + +Use directory formatted cloud paths instead (e.g.: `/cuno/s3//path`). + + +## Rsync + +We strongly recommend running `rsync` with the options `--inplace -W`. This makes Rsync work more efficiently with object storage. + +To use Rsync options that preserve permissions (`-p`) and modification times (`-t`), such as when you want to update files _only_ when the source has been updated, you must enable POSIX File Access. + +See the Getting Started Guide: [POSIX File Access](docId:cbm3PcQXmLpuYcbg#posix-file-access) for details. + + +## Fpsync + +When using `fpsync`, use the `-o` option to pass the options recommended for `rsync` down to the worker processes (e.g.: `-o "--inplace -W"`). + +Further, because Object Mount is already parallelized, we recommend limiting the number of Fpsync worker processes using the `-n` option. + + +## FFmpeg + +While `ffmpeg` works, it may be slow in the following use-cases: + + - Writing to object storage: If the task requires many randomly-placed writes during an upload. For example, for a large mp4 file being written with the flags `-movflags +faststart`. + + - Reading from object storage: A complex filter chain, where multiple subtitle streams are being read from the same input file. + - Improvements in this area are being developed. + + - Reading from object storage: Files that contain title screen and extra credit scenes can be slow to start. + + +## Using sudo with Direct Interception + +Using [Direct Interception](docId:UHsd5HnesueQyhnZ) requires the `LD_PRELOAD` environment variable to be set and maintained for all executed child processes. + +Since `sudo` usage, by default, does not preserve the environment variables set, the following requirements apply: + + - `sudo` needs to be run with `--preserve-env` to preserve `CUNO_OPTIONS`. + - `sudo` needs to launch a child shell that will then run the command, so that the `LD_PRELOAD` environment variable can be set _before_ running the command to be intercepted. + - `LD_PRELOAD` needs to be manually set inside the child shell launched. + +To use `sudo` with Direct Interception, please do the following: + + 1. Start a wrapped-shell using the Object Mount CLI: `cuno` + + 2. Run `sudo` in the following way: + + ```console + sudo --preserve-env /bin/bash -c "export LD_PRELOAD=$LD_PRELOAD && " + ``` + + +## Locate and UpdateDB + +The `locate` command requires some heightened privileges to create the file path database. It also has some incompatibilities with Object Mount [Direct Interception](docId:UHsd5HnesueQyhnZ) Mode. + +When using Direct Interception Mode, both `locate` and `updatedb` will not work with [URI formatted](docId:jieteeYeyievui9k) cloud paths (e.g.: `s3:///path`). + +Use directory formatted cloud paths instead (e.g.: `/cuno/s3//path`). + +Furthermore, Direct Interception (even when using the Object Mount CLI) requires the `LD_PRELOAD` environment variable to be set and maintained for all executed child processes. + +**Note:** Since `updatedb` typically needs to be run with `sudo`, the limitations specified for **`sudo` with Direct Interception** (above) apply here as well. + +### Instructions for Running UpdateDB + +To help work around these limitations, follow the guidance below when using `locate` and `updatedb`: + +1. **Create a New Database** + + {% tabs %} + {% tab label="Direct Interception Mode" %} + Run the Object Mount CLI by entering the `cuno` command: + + ```console + # terminal + cuno + ``` + + Then run `updatedb` inside the Object Mount-wrapped shell, using the `sudo` recommendations with cloud paths specified in the directory format. (We use an example database name of: `cunoloc.db`). + + ```console + (cuno) $ sudo --preserve-env /bin/bash -c "export LD_PRELOAD=$LD_PRELOAD && updatedb -U /cuno// -o cunoloc.db" + ``` + {% /tab %} + + {% tab label="Object Mount on FUSE" %} + Assuming you have abject Mount on FUSE set up at `~/my-object-storage`, you can use `updatedb` _directly_ to crawl all paired buckets from all your object storage providers: + + ```console + sudo updatedb -U ~/my-object-storage -o cunoloc.db + ``` + + {% callout type="note" %} + **Mount Location Consistency** + + The mount location (`~/my-object-storage` in the example above) should not be change after running `updatedb` as the mount details are written into the file path database. + {% /callout %} + {% /tab %} + {% /tabs %} + +2. **Change the Database Ownership** + + The newly created database needs to have its ownership changed back to the current user: + + ```console + sudo chown $(whoami):$(whoami) cunoloc.db + ``` + +3. **Add the Database to Your `LOCATE_PATH`** + + Update the `LOCATE_PATH` environment variable to include the database. + + Then use locate normally, or search within the database, as follows: + + ```console + locate -d cunoloc.db myfile + ``` + +### Modify the Locate cron Job + +By default, the global locate database is periodically updated by a cron job. + +To setup the cron job to work properly with Object Mount, you need to edit the file `/etc/cron.daily/mlocate`. + +The last line of that file is used to update the global database: + +```console +flock --nonblock /run/mlocate.daily.lock $NOCACHE $IONICE nice /usr/bin/updatedb.mlocate +``` + +Modify that line, replacing it with something similar to the line below: + +```console +LD_PRELOAD='/usr/lib/cuno.so' CUNO_OPTIONS='' CUNO_CREDENTIALS='' flock --nonblock /run/mlocate.daily.lock $NOCACHE $IONICE nice /usr/bin/updatedb.mlocate +``` + +If you prefer not to index _all_ of your object storage files, you can specify where `updatedb` does _not_ look for files by adding paths to `PRUNEPATHS` in the file `/etc/updatedb.conf`. diff --git a/app/(docs)/object-mount/linux/user-guides/appendix/page.md b/app/(docs)/object-mount/linux/user-guides/appendix/page.md deleted file mode 100644 index f45f1d04a..000000000 --- a/app/(docs)/object-mount/linux/user-guides/appendix/page.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Appendix -docId: vood1Lai4OoV3Ooz - -metadata: - title: Appendix - description: Appendix - -weight: 14 ---- - -## Getting S3 credentials using the AWS console - -1. **Visit** - -2. **Navigate to** *Services* > *IAM* - -3. **Open** *Users* and **click** on the *Add user* button - - 1. **Set** a username - 2. **Select** the *Programmatic access* option - 3. **Click** on *Next* - -4. In the *Set Permissions* page **select** *Attach existing policies directly* - - 1. **Select** an existing policy, e.g., *AmazonS3ReadOnlyAccess* or *AmazonS3FullAccess* - 2. or **Create** a new policy that has the *s3:ListAllMyBuckets* permission and at least the *s3:ListBucket* and *s3:GetObject* permissions for each bucket available to the user - 3. **Click** on *Next* - -5. **Review** and confirm to create the new user - -6. **Download** the CSV file by pressing the corresponding button - -7. **Use** the Access key ID and Secret Access key as described in [user-guide-import-credentials](../user-guides/credentials#import-credentials) - -## Getting Azure credentials using Azure portal - -1. **Visit** -2. **From services, select** *Storage Accounts* -3. **Click on the** *Storage Account* **you want the key for** -4. **In the** *Security + networking* section **click** *Access keys* -5. **Click on the** *Show keys* button -6. **Use** the *Storage account name* and *Key* as described in [user-guide-azure-native-credentials](../user-guides/credentials#azure-storage-using-environment-variables) - -## Getting Google Cloud credentials using the console - -1. **Visit** -2. **Go to the** *Service Accounts* page -3. **Select the** *project* **you want to provide access to** -4. **Click the** *email address* **of the service account that you want to create a key for** -5. **Click the** *Keys* tab -6. **Click the** *Add key* **drop-down menu, then select** *Create new key* -7. **Select** *JSON* **as the** *Key type* **and click** *Create* -8. **Use** the *downloaded JSON file* as described in [user-guide-gcs-native-credentials](../user-guides/credentials#google-cloud-storage-using-environment-variable) diff --git a/app/(docs)/object-mount/linux/user-guides/basic/page.md b/app/(docs)/object-mount/linux/user-guides/basic/page.md deleted file mode 100644 index 5e282f4f3..000000000 --- a/app/(docs)/object-mount/linux/user-guides/basic/page.md +++ /dev/null @@ -1,317 +0,0 @@ ---- -title: Basic Loading -docId: aiShu0aaruaceidi - -metadata: - title: Basic Loading - description: Basic Loading - -weight: 6 ---- -Object Mount may be loaded in a number of ways, depending on user requirements and environment. - -For program calls to be routed through Object Mount, they must use - -- [Directly Intercepted](#direct-interception-with-object-mount-cli) using the `cuno` command line or `LD_PRELOAD` environment variable; -- or the program must be pointed to a [Object Mount on FUSE](#object-mount-on-fuse) (a custom FUSE mount) set up using `cuno mount`. - -There is also [Object Mount FlexMount](#object-mount-flex-mount), which makes use of both Direct Interception and a Object Mount on FUSE as a fallback. - -## Direct Interception with Object Mount CLI - -When running in Direct Interception mode, Object Mount will insert itself between your applications and the operating system. It will dynamically redirect relevant filesystem calls made by the application through the API of your object storage solution. As a result, our mapping between filesystem concepts and object storage gives any application instant access to objects as files. - -{% callout type="note" %} -This includes applications you've written yourself - just treat the paths to object storage as local filesystem paths, and Object Mount handles the rest. -{% /callout %} - -Direct Interception offers the highest performance access to object storage that Object Mount provides. - -{% callout type="warning" %} -Direct interception does not currently support SUID binaries, or certain packaged apps like [Snap](https://ubuntu.com/core/services/guide/snaps-intro), [AppImage](https://appimage.org/), or [Flatpak](https://docs.flatpak.org/en/latest/introduction.html) applications. Future updates are planned to address this. -{% /callout %} - -This mode is best suited for situations in which installed app compatibility is not a concern - for example, when setting up a fixed workflow with Object Mount that can be tested and verified before putting it into production. - -Direct Interception can be enabled per-session or per-command. Enable it for a session by calling `cuno` on the command line: - -```console -cuno -``` - -This will launch a new interactive "wrapped" shell with Object Mount acting in Direct Interception mode. The shell itself has Object Mount intercepting its calls, so every application launched from within it can be intercepted as well as the command line arguments being used. - -{% callout type="note" %} -When the Object Mount CLI is used to launch a new shell, the primary purpose is to start a new shell (whether that's bash, zsh, or whatever else) with `LD_PRELOAD` set to point at `cuno.so`. There is no Object Mount shell binary - it only wraps existing shells with Object Mount pre-loaded. -{% /callout %} - -To enable direct interception of a single command, use: - -```console -cuno run bash -c "" -``` - -{% callout type="note" %} -Always execute one-time commands this way to maintain support for wildcard expansion (`*`) . -{% /callout %} - -### How it works - -Direct interception uses the `LD_PRELOAD` environment variable so that Object Mount can capture and redirect storage access library and system calls through object storage APIs. If a static binary is intercepted, a JIT ELF binary translator will redirect relevant calls when the binary is loaded into memory. - -### Advantages and Disadvantages - -| Advantages | Disadvantages | -|------------|---------------| -| **Speed**: Dynamic interception offers the best performance. | **Compatibility**: Direct interception does not currently support SUID binaries, [Snaps](https://ubuntu.com/core/services/guide/snaps-intro), [AppArmor](https://ubuntu.com/server/docs/security-apparmor), or [Flatpak](https://docs.flatpak.org/en/latest/introduction.html) applications. | -| **Set up time**: This is the default; all you have to do is launch a Object Mount shell with the `cuno` command line utility. No FUSE mount set up is required. | **Activation**: In some cases, it is difficult to enable direct interception or keep it enabled (because of environment variable clobbering or lack of privileges to set use LD_PRELOAD). | -| | **Non-technical users**: If Object Mount is being used by many people in an organization, teaching people how to use the Object Mount shell and diagnose if it's working correctly may be less desirable than setting up a Object Mount on FUSE on their behalf. | - -### Usage - -When Direct Interception is enabled using `cuno` or `cuno run `, object storage is accessed either in path or URI formats: - -```console -$ cuno -(cuno) $ ls s3:/// -(cuno) $ ls /cuno/s3// -``` - -See [user-guide-cloud-paths](../user-guides/access) for more information, options and examples of using Direct Interception with the Object Mount CLI. - -#### Fully supported shells - -`bash` and `zsh` are fully supported when using Object Mount CLI. -This includes support for: - -- tab auto-completion of remote paths; -- wildcard expansion (`*`) of filesystem and remote paths. - -In either of these fully-supported shells, the prompt will be prefixed with `(cuno)` to indicate that Direct Interception is enabled and that you are using a Object Mount CLI shell, like so: - -```console -(cuno) user@host:~$ -``` - -{% callout type="note" %} -To select between the two shells, Object Mount detects if either is present in one of the following, {emphasis}`in order`: - -1. the shell used to launch Object Mount; -2. the user's preferred login shell; -3. all installed shells. - -If neither `bash` nor `zsh` are found, then the first shell in this list is used. -If no shell is set, then `/bin/sh` is used. - -% warning: - -% The behaviour described here relies on ``ps``. If you installed with the [user guide](../user-guides/install/scripted-installer), you may need to install `ps` manually (RHEL derivatives install package ``procps``). -{% /callout %} - -To launch a specific shell with Object Mount enabled, use `cuno run`: - -```console -cuno run -``` - -## Object Mount on FUSE - -Object Mount on FUSE allows you to mount an object storage path in a directory within the local file system hierarchy. This allows you and any other user of the mount to access object storage as if it were just another directory. - -Object Mount on FUSE uses Linux FUSE (Filesystem in Userspace) v3 to mount an object storage path in a directory within the file system hierarchy. -Due to the nature of FUSE file systems, Object Mount on FUSE is usually less performant than Direct Interception using Object Mount CLI. Consider using Object Mount CLI or the userspace library (described in [user-guide-ldpreload](../user-guides/advanced#user-mode-library)) if speed is the primary objective. - -{% callout type="note" %} -The `--posix` option requires that a FUSE package be installed on the system. -{% /callout %} - -### Advantages - -1. **Compatibility:** use a Object Mount on FUSE when extensively using Object Mount with a variety of operating systems, applications or when compatibility is a particular concern. -2. **Simple usage:** after a mount is set up, scripts only need to be changed to point at it, without any further changes to the workflow. No environment variables need to be set, and no Object Mount-wrapped shell has to be launched. -3. **Greater admin control:** a centralised Object Mount on FUSE can be set up by an admin, without giving users any credentials. This allows the admin to abstract away the concepts of object storage entirely from users. -4. When launching the `cuno` binary is not possible (limited cases, e.g. an automation engine without the required features). - -### Disadvantages - -1. You cannot use URI paths like "s3://bucket1/foo" directly when using a Object Mount on FUSE. To convert such a path to be usable, you need to replace the URI prefix (`s3://`) with the path to your mount. - -### Usage and options - -```console -cuno [cuno-options] mount [mount-options] -``` - -There are various options that can be specified, including - -- [Object Mount on FUSE options](#usage-and-options) -- [Object Mount options for the mount's Object Mount subsystem](#options-to-configure-the-mount-s-object-mount-subsystem) -- [FUSE options](#fuse-options) - -#### Mount options - -On the right hand-side of the `mount` verb, you can specify options that are specific to the mount operation. - -``` -cuno mount [option] ... -``` - -{% callout type="note" %} -These options must come AFTER the verb `mount`. -{% /callout %} - -| Option | Description | -|--------|-------------| -| `--root ` | Specifies the root object storage path to be mounted. For example, to mount a single bucket, you would use `--root s3://`. | -| `--posix` | Enables setting and enforcing access permissions, symbolic and hard links, users, groups, etc. Can be used with default permissions. Implicitly sets `-o allow_other`. This option is ideally used with a mount that is only on a single bucket, e.g. `cuno mount --root s3:// --posix `. You should also be using `cuno creds setposix s3://examplebucket true` to enable POSIX handling of the bucket consistently when Direct Interception or a FlexMount is used. | -| `` | Options that are specific to the FUSE mount operation. These options are passed on to `fum` ([fusermount3](https://www.man7.org/linux/man-pages/man8/mount.fuse3.8.html)). Not to be confused with Object Mount subsystem options which use the same `-o` syntax. See below for some of these. See user-guide-cuno-mount-fuse-options. | -| `--no-allow-root` | Do not allow root to access the mount (allowed by default). Disables support for SUID binaries, Snap, AppArmor, and Flatpak applications. | -| `--auto-restart` | Automatically restarts Object Mount on FUSE if problems occur during execution. | -| `--mkdir` | Automatically creates the mount point directory if it does not exist. | -| `--debug` | Enables debug output (same as `cuno mount -o debug`). | -| `--dev-logs` | Enabled debug logging to `/tmp/fuse.logs` | -| `--verbose` | Enables verbose output. | - -#### Options to configure the mount's Object Mount subsystem - -On the left hand-side of the `mount` verb, you can specify options that are specific to the Object Mount subsystem. - -``` -cuno -o mount ... -``` - -``` -export CUNO_OPTIONS="" -``` - -Some relevant Object Mount options are given below. Refer to [user-guide-config-options](../user-guides/configuration) for more information. - -{% callout type="note" %} -These options must come BEFORE the verb `mount`. -{% /callout %} - -| Object Mount option | Description | -|---------------------|-------------| -| `cachehome=` (default: `/cunodb;/dev/shm;/tmp/cache/cuno`) | This option allows you to set a semicolon-delimited list of directories to consider for caching. This is for the internal metadata cache of the Object Mount process behind the mount. Note that the FUSE cache is separate from this cache. Example: `cuno -o cachehome=/mnt/cache;/cunodb;/dev/shm mount ` | -| `uid=` | Define the default user ownership of files and directories within a bucket. These are the UIDs that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. Example: `cuno -o uid=$(id -u ) mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| `gid=` | Define the default group ownership of files and directories within a bucket. These are the GIDs that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, prefer to use POSIX file access. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. Example: `cuno -o gid=$(id -g ) mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| `filemode=` | Define the default file access permission bits of files within a bucket. Supply the octal (numeric) representation of the permissions you want to apply. These are the permissions that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. For example, to set the default file access permission bits to 0770 (`-rwxrwx---`), use the following command: `cuno -o filemode=0770 mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| `dirmode=` | Define the default file access permission bits of directories within a bucket. Supply the octal (numeric) representation of the permissions you want to apply. These are the permissions that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. For example, to set the default file access permission bits to 0770 (`drwx - - -#### FUSE options - -On the right hand-side of the `mount` verb, you can also specify options that are specific to the FUSE mount operation. These options are passed on to `fum` ( [fusermount3](https://www.man7.org/linux/man-pages/man8/mount.fuse3.8.html)). - -``` -cuno mount [FUSE option] ... -``` - -{% callout type="note" %} -These options must come AFTER the verb `mount`. - -Some of these are provided using the same `-o` syntax as the `cuno -o` options, but they are not the same. -{% /callout %} - -| FUSE option | Description | -|-------------|-------------| -| `-o allow_root` | Allows root access to the mount; required for [SUID](https://www.redhat.com/sysadmin/suid-sgid-sticky-bit) permissions. | -| `-o allow_other` | Allows other users to access the mount; requires `user_allow_other` in `/etc/fuse.conf`. | -| `-o default_permissions` | Enable permission checking by the kernel. To use Enforced POSIX, use `--posix`, which will set this internally. | -| `-o auto_unmount` | Automatically unmounts the mount when the process terminates. | -| `-o ro` | Mounts the file system read-only. To achieve a read-only mount through Object Mount, you can alternatively use `cuno mount --posix` and change the permissions on dirs/files in the mount manually to read-only using `chmod`. See [here](https://www.man7.org/linux/man-pages/man8/mount.8.html) for more on standard mount options like `ro`, `rw`, etc. | -| `-o rw` (default) | Mounts the file system read-write. | -| `-o exec` (default) | Allow execution of binaries on the file system. | -| `-o noexec` | Disallow execution of binaries on the file system. | -| `-o clone_fd` | Uses a separate fuse device fd for each thread (may improve performance). | -| `-o max_idle_threads` | The maximum number of idle worker threads allowed (default: 10). | -| `-s` | Run in single-threaded mode. | -| `-f` | Run in the foreground. | -| `-o debug` | Enable debug output (implies `-f`). | -| **FUSE kernel default attributes** | | -| `-o uid=N` | Sets the file owner of all mounted files/dirs to the specified user ID. These are the UIDs that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| `-o gid=N` | Sets the file group of all mounted files/dirs to the specified group ID. These are the GIDs that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| `-o umask=M` | Sets the file permissions (octal) of all mounted files/dirs. These are the permissions that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | -| **FUSE kernel cache configuration** | | -| `-o kernel_cache` | This method caches file data across `open`; i.e. disables flushing of the file data cache on every `open`. Without this option (and neither `direct_io`), data is cached before the next `open` so a `read` syscall may not initiate a `read` operation. | -| `-o auto_cache` | Enables automatic flushing of the data cache on open(). The cache is only flushed if the modification based on modification times (off). | -| `-o entry_timeout=T` | Sets the kernel cache timeout for names (1.0s). | -| `-o negative_timeout=T` | Sets the kernel cache timeout for a negative lookup (file not found) (0.0s). | -| `-o attr_timeout=T` | Sets the kernel cache timeout for attributes (1.0s). | -| `-o ac_attr_timeout=T` | Sets the kernel cache timeout for checking if `-o auto_cache` should flush file data on `open`. | - - -#### Object Mount on FUSE commands - -Once a Object Mount on FUSE is set up, you can use the following commands to manage it. - -| Command | Description | -|---------|-------------| -| `--list` | Lists your active Object Mount on FUSEs. You can also use the Linux utility `mount` directly.

Example usage:
```console
cuno mount --list
``` | -| `--unmount` / `-u` | Unmounts an Object Mount on FUSE from the specified mount path. You can also use the Linux utility `umount` directly.

Example usage:
```console
cuno mount --unmount
``` | -| `--unmount-kill` / `-U` | Unmounts an Object Mount on FUSE from the specified mount path and kills the fum process. You can also use the Linux utility `umount` directly.

Example usage:
```console
cuno mount --unmount-kill
``` | - -#### Mount on boot - -You can add mount commands to `/etc/fstab` to automount on start up. - -## Object Mount FlexMount - -Object Mount FlexMount is for when the speed of direct interception is preferred whenever it is possible to use, but wider-support for different contexts and types of applications is also required. - -FlexMount is used by setting up a Object Mount on FUSE and then accessing the cloud using Object Mount CLI always "through" the mount path. Object Mount will recognise that the path is a mount and use direct interception where possible for faster access. It will naturally fall back to the Object Mount on FUSE for anything that cannot be directly intercepted. - -### Usage - -A FlexMount is set up as follows: - -1. First, set up a Object Mount on FUSE, with a `cloudroot` set: - - ```console - $ mkdir "$HOME/my-object-storage" - $ cuno mount "$HOME/my-object-storage" - ``` - -2. Each time a Object Mount CLI is launched, use the parameters `--flex =""`, for example: - - ```console - $ cuno --flex "$HOME/my-object-storage" - (cuno) $ ls $HOME/object_storage_mount/s3// - ``` - - {% callout type="warning" %} - You cannot use a tilde `~` in your `CUNO_OPTIONS` or `cuno -o` CLI options when setting up a FlexMount as this is something that the shell needs to resolve. You may still use it for your `cuno mount` commands, and subsequent FlexMount operations (such as `ls ~/my-object-storage`). - {% /callout %} - -The same FlexMount can be re-used across multiple Object Mount wrapped shells. - -The `-flex` option can also be used with `cuno run` to run a single command/script with Object Mount enabled, for example: - -```console -cuno run --flex "$HOME/my-object-storage" bash -c "touch $HOME/object_storage_mount/s3//newfile" -``` - -The `-flex` option is synonymous with the `-o cloudrootover=exact -o cloudroot=""` option, which is used to tell Object Mount to intercept paths that exactly match the cloudroot setting, and to handle them using the Object Mount on FUSE. - -When using Object Mount Direct Inteception in FlexMount mode, most "local" paths are intercepted but not actioned upon, because they can be handled by the local file system. The `cloudrootover` setting is telling Object Mount to intercept local paths that match the cloudroot setting, and to prioritise itself handling them over the mount. This means that whenever a path is recognised as the cloudroot, it can be more efficiently handled in user-space by the Direct Inteception/`LD_PRELOAD` library. That path recognition can be done in two ways: - -- (default) Exact string matching (`exact`) will match the cloudroot setting exactly. This is faster, and will end up relying on the FUSE mount whenver the paths don't match the cloudroot - for example with symbolic links located outside the mount pointing into the mount. - -- Resolved path matching (`resolve`) will resolve the full path given in a file system call including symbolic links to check if the file is ultimately located inside the mount point. This requires more calls for every path-based file system call, so is slower when many files are being accessed. It is useful when symbolic links are used to point into the mount, and other cases where the path ultimately is inside the mount. This can be more efficient in some special cases (few files, large transfers, complex relationships between files) as `cuno.so` will intercept more calls without requiring them to go through the slower FUSE mount. - - To use resolve mode use the parameters `-o cloudrootover=resolve -o cloudroot=""`, for example: - - ```console - $ cuno -o cloudrootover=resolve -o cloudroot="$HOME/my-object-storage" - (cuno) $ ls $HOME/object_storage_mount/s3// - ``` - -### Advantages - -1. Speed: where interception is possible, Object Mount in FlexMount mode will be as fast as Object Mount -2. Support: support for all POSIX applications, as anything that cannot be directly intercepted falls through to the Object Mount on FUSE. - -### Disadvantages - -1. Set up time: a Object Mount on FUSE needs to be set up. Consider [setting up the mount at boot](#mount-on-boot). -2. Launch is more complicated: each time a Object Mount shell is launched it must be configured to use the mount. This can be worked around by setting up a Object Mount on FUSE on boot, and setting an alias to launch a Object Mount shell with the correct parameters. diff --git a/app/(docs)/object-mount/linux/user-guides/cloud-paths/page.md b/app/(docs)/object-mount/linux/user-guides/cloud-paths/page.md new file mode 100644 index 000000000..f55020d78 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/cloud-paths/page.md @@ -0,0 +1,180 @@ +--- +title: Cloud Path Details +hideTitle: false +docId: jieteeYeyievui9k +weight: 5 +redirects: + - /object-mount/linux/user-guides/access +metadata: + title: Cloud Path Details + description: Cloud Paths Guide +hidden: false +--- + +When using Object Mount for linux from the Object Mount CLI + +## Specifying Cloud Paths + +When accessing cloud-based, object storage volumes using the Object Mount CLI, there are two primary ways of interacting with the remote files and directories: + +- The URI format, e.g.: `s3://my-bucket/dir/file.txt` +- The directory format, e.g.: `/cuno/s3/my-bucket/dir/file.txt` + +### URI Format + +The following table describes **URI prefixes** for supported cloud providers: + +| **Cloud Provider** | **URI Format** | +|----------------------|----------------------------------------| +| AWS S3 | `s3://` | +| Storj | `s3://` | +| Google Cloud Storage | `gs://` | +| Azure Blob Storage | `az:///` | +| Other S3 Compatible | `s3://` | + +### Directory Format + +The following table describes **directory format** for supported cloud providers: + +| **Cloud Provider** | **Directory Format** | +|----------------------|--------------------------------------------| +| AWS S3 | `/cuno/s3/` | +| Storj | `/cuno/s3/` | +| Google Cloud Storage | `/cuno/gs/` | +| Azure Blob Storage | `/cuno/az//` | +| Other S3 Compatible | `/cuno/s3/` | + +{% callout type="warning" %} + **URI Format Support** + + Some software tools cannot process the URI format. Use the directory format instead in such cases. +{% /callout %} + +### Examples + +Some examples of using Object Mount are given below: + +1. List the contents of a remote container `my-bucket` on AWS S3: + + ```sh + (cuno) $ ls s3://my-bucket + ``` + + or + + ```sh + (cuno) $ ls /cuno/s3/my-bucket + ``` + +2. Move a text file `sample.txt` from the local filesystem to a remote Google Cloud Storage container `my-bucket`: + + ```sh + (cuno) $ mv ./sample.txt gs://my-bucket/dir/sample.txt + ``` + + or + + ```sh + (cuno) $ mv ./sample.txt /cuno/gs/my-bucket/dir/sample.txt + ``` + +3. List the contents of a subdirectory `dir` in a remote Google Cloud Storage container `my-bucket`: + + ```sh + (cuno) $ ls gs://my-bucket/dir + ``` + + or + + ```sh + (cuno) $ ls /cuno/gs/my-bucket/dir + ``` + +4. Use UNIX coreutils (e.g. `head`) on a remote file `sample.txt` on AWS S3: + + ```sh + (cuno) $ head s3://my-bucket/dir/sample.txt + ``` + + +## Microsoft Azure Blob Storage Usage + +When using Microsoft’s Azure Storage, the associated **Account Name** must be included in both the URI and directory format: + +```sh +(cuno) $ ls az://my-account-name/my-container +``` + +or: + +```sh +(cuno) $ ls /cuno/az/my-account-name/my-container +``` + + +## AWS S3 Access Point Support + +Object Mount supports using an **AWS S3 Access Point** instead of a bucket as an endpoint. + +Access Points are unique addresses that customers can create to enforce distinct permissions and network controls for requests made through that Access Point. + +To use an Access Point instead of a container, provide the full **Amazon Resource Name (ARN)** in place of the bucket name: + +``` +s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name +``` + +This applies to all places a container can be specified, such as within `cuno creds` or console commands: + +```sh +(cuno) $ cuno creds -i pair s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name +(cuno) $ ls s3://arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name/ +(cuno) $ ls /cuno/s3/arn:aws:s3:us-east-1:999999999999:accesspoint:my-access-point-name/ +``` + + +## File System Behaviors When Using Object Mount + +### Symbolic Links + +WHen not using POSIX Mode, symbolic links can **only** be created (via `ln --symbolic` or `ln -s`) to point **from** a local system **to** a cloud location. Either URI or directory format can be used. + +### Hard Links and Server-Side Copy + +Conversely, hard links **cannot** be created from the local system to a remote cloud location. + +Use hard-linking to trigger a more efficient cloud-only copy mechanism within a cloud region. + +`mv` or `cp` within a single cloud vendor will trigger a **server-side copy**, which is more efficient than a download followed by an upload. + +### POSIX Mode + +**Symbolic links** can be created (via `ln --symbolic` or `ln -s`) between local and cloud objects in **either direction** when POSIX mode in enabled. + +**Hard links** can be created (using `ln`) between cloud objects in the `same` bucket when POSIX mode in enabled. + +See the Getting Started Guide article [POSIX File Access](docId:cbm3PcQXmLpuYcbg#posix-file-access) for more details. + +{% callout type="warning" %} + **HardLinks and SymLink on File Deletion** + + If the target file/object is deleted or moved, both symbolic and hard links will break. +{% /callout %} + +### File Metadata + +`ls --long` (or `ls -l`) will list **the current user** as the owner of the remote objects, and the permissions will be `777`. + +The creation date of a remote directory is not always available to system calls. + +### Performance Optimizations + +Object Mount uses multiple concurrent connections to remote objects to achieve the highest performance. + +It can also execute operations on objects entirely remotely where possible. + +Some optimized operations provided by Object Mount are listed below: + + - `mv` commands between locations **within the same object storage provider** will invoke a server-side remote-copy operation to avoid streaming through the client. + - `ln` hard links between locations **within the same object storage provider** will invoke a server-side remote-copy operation to avoid streaming through the client. + - Hard links between objects **in the same bucket** will work like local hard links when POSIX mode in enabled. diff --git a/app/(docs)/object-mount/linux/user-guides/configuration/page.md b/app/(docs)/object-mount/linux/user-guides/configuration/page.md deleted file mode 100644 index 622b225e5..000000000 --- a/app/(docs)/object-mount/linux/user-guides/configuration/page.md +++ /dev/null @@ -1,330 +0,0 @@ ---- -title: Configuration Options -docId: phohPoowequie5ji - -metadata: - title: Configuration Options - description: Configuration Options - -weight: 3 ---- -## Overview -Setting the environment variable `CUNO_OPTIONS` can be used to further configure Object Mount. The -o option on the `cuno` command can also be used to specify these options. - -Options available for `CUNO_OPTIONS`: - -{% callout type="note" %} -Individual options must be separated by spaces. Don't forget to enclose multiple options in quotes (`"` or `'`), or to escape spaces. -{% /callout %} - - -### The static option - -To use Object Mount with static binaries, the *parent process* needs to be intercepted in static mode. Interception of static binaries is enabled by default; to disable it requires setting both `CUNO_OPTIONS="-static"` and starting a new Object Mount wrapped shell. - - -### The uricompat option - -Object Mount can override the handling of URI-style paths (`xx://`) in applications that attempt to provide internal handling of such URIs. Users may provide the `+uricompat` option to add similar patching for further applications. - -To specify any applications to be patched, specify them as a colon-separated list: - -```console -+uricompat=::< ... > -``` - -To limit the cases in which this patching occurs, you can filter interceptions to only occur when certain command-line arguments match specified keywords. This may be useful for certain code interpreters/executors (see the [example](../user-guides/configuration#example) below). To do this, in place of an executable name, use a slash-separated list of the name, a 1-based index of the argument to be matched (or any using `*`), and a value to match against: - -```console -+uricompat=// -``` - -{% callout type="note" %} -The `+uricompat` option only works on URIs that have been paired. - -If you are working with a public access bucket, you will need to run the `cuno creds pair` command (see [user-guide-pair-containers](../user-guides/credentials#pair-containers-and-credentials)) against it for the `+uricompat` to take effect. -{% /callout %} - -#### Applications with patched URI handling - -By default, Object Mount will attempt to override the internal URI handling of popular tools, such as: - -- `rsync` -- `ffmpeg` -- `tar` - -And some genomics tools: - -- `samtools` -- `igv` -- `fastQC` - -This means that when a URI-style path (corresponding to a [paired bucket or container](../user-guides/credentials#pair-containers-and-credentials)) is passed to any of the applications above, Object Mount will prevent the application from handling the path in a special way. - -For example, `fmpeg` has special handling for some [protocols](http://ffmpeg.org/ffmpeg-protocols.html#Protocols) which they specify in a similar URI format to Object Mount URI cloud paths (e.g. `ftp://example.foo`). As a result, Object Mount needs to prevent `ffmpeg` from failing (when given a path like `s3://bucket/file`) because `s3`, `az` and `gs` are not protocols it supports. - -#### Example - -To override a Java application’s handling of URI-style paths, we may use: - -```console -export CUNO_OPTIONS="+uricompat=java/2/appname.jar:java/*/org.name.main.Class" -``` - -This will intercept any invocation of `java` with `appname.jar` as its second argument, e.g. `java -jar appname.jar s3://bucketname/file.bam`. It will also intercept invocations of Java where any argument is equal to `org.name.main.Class`, e.g. `java org.name.main.Class s3://bucketname/file.bam`. - - -### The cloudroot option - -You may define a custom directory path prefix to replace the default `/cuno` path (see :ref:\`\` ). Once defined, the path specified will be made available at the root of the file system. For example, if you configure the following: - -```console -export CUNO_OPTIONS='+cloudroot=my-object-storage' -``` - -You will then be able to list your configured S3 buckets with: - -```console -ls /my-object-storage/s3/ -``` - -{% callout type="warning" %} -We strongly recommend that the cloudroot path does not exist locally, because it can confuse Object Mount's ability to distinguish whether a path refers to the local file system or to the cloud. - -If you specify a cloudroot which conflicts with an existing directory, then it must not contain directories or files named `s3`, `az` or `gs`. -{% /callout %} - -{% callout type="note" %} -In order for the "cd to cloud" functionality to see this option, you will need to spawn a new shell after `CUNO_OPTIONS`. -{% /callout %} - -#### FlexMount usage - -Setting a cloudroot during the mounting process can affect the options required to be set when using it as a FlexMount. - -**Recommended option** If cloudroot was not set when creating the mount (e.g. `cuno mount ~/my-object-storage`), then cloudroot needs to be set as the path to the mountpoint when launching Object Mount CLI in order to use that mount as a FlexMount (e.g. `cuno -o cloudrootover -o cloudroot=""`). - -In some circumstances, you may need to set a cloudroot when creating the mount (for example, if a `/cuno` directory already exists on your systems). In these cases, we strongly recommend migrating or removing any such physical directory so that you can continue to use the default settings. If this is not possible, there are a number of options available. - -**Recommended option if cloudroot must be changed** If a cloudroot needs to be set when creating the mount, then you should set the `--root` option to the same cloudroot to get expected behaviour when using Object Mount FlexMount (e.g. `cuno -o cloudroot=/altcloudroot mount $HOME/my-object-storage --root /altcloudroot`). In this case, continue to use the same options when launching cuno (e.g. `cuno -o cloudrootover -o cloudroot=""`) to access the mount as a FlexMount. - -In very limited circumstances, it is not possible to choose a cloudroot that can also be used as the `--root` of the mount point (for example, if the cloudroot is chosen to be somewhere that is not mountable). In this case, we would advise you to choose a different cloudroot, one that can be mounted (such as a position within your user home directory e.g. `cuno -o cloudroot=$HOME/mycloudroot mount $HOME/my-object-storage --root $HOME/mycloudroot`). If this is not possible, then there are two options: - -**Highly discouraged** If a cloudroot needs to be set when creating the mount, and the `--root` option is set to a path within the cloudroot (e.g. `cuno -o cloudroot=/altcloudroot mount $HOME/my-object-storage --root /altcloudroot/s3/mybucket`), then, when using the FlexMount later, one must set the 'cloudrootover' path as so: `cuno -o cloudroot=/altcloudroot -o cloudrootover=$HOME/my-object-storage`. Important: always check that the mount point exposes the cloud at the same point when used directly as a Object Mount on FUSE (so outside of a Object Mount CLI instance) and when accessed as a FlexMount. - -{% callout type="warning" %} -Object Mount behaviour will be broken and dangerous if you do not set the correct options when launching Object Mount CLI to this type of FlexMount. For example, in this case if the default parameters for launching a FlexMount are used, namely `cuno -o cloudrootover -o cloudroot=$HOME/my-object-storage`, then applications that are intercepted directly will see `$HOME/my-object-storage` as the cloudroot (so the immediate children are `az gs s3`) whereas any applications that fall through to the Object Mount on FUSE (static binaries, cloud binaries, SUID binaries, Snap apps, AppImage apps, and Flatpak apps) will see `$HOME/my-object-storage` as the bucket or directory specified as the `--root`. This could cause a loss of data, accidentally expose data in the wrong place or other misbehaviour when running scripts. -{% /callout %} - -**HIGHLY discouraged** If a cloudroot needs to be set when creating the mount, and it is not possible for '--root' to be set to a path within the cloudroot (i.e. `cuno -o cloudroot=/altroot mount $HOME/my-object-storage --root /`), then, when using the FlexMount later, one must set the cloudroot as so: `cuno -o cloudrootover -o cloudroot=$HOME/my-object-storage/my/cloud/root`. - -{% callout type="warning" %} -Object Mount behaviour will be broken and dangerous if you do not set the correct options when launching Object Mount CLI to this type of FlexMount. For example, in this case if the default parameters for launching a FlexMount are used, namely `cuno -o cloudrootover -o cloudroot=$HOME/my-object-storage`, then applications that are intercepted directly will see `$HOME/my-object-storage` as the cloudroot (so the immediate children are `az gs s3`) whereas any applications that fall through to the Object Mount on FUSE (static binaries, cloud binaries, SUID binaries, Snap apps, AppImage apps, and Flatpak apps) will see `$HOME/my-object-storage` as `/` - the root of your local filesystem. This could cause a loss of data, accidentally expose data in the wrong place or other misbehaviour when running scripts. -{% /callout %} - - -## Ownership and Permissions - -### Core File Access - -In [Core File Access](../getting-started/configuration-modes#core-file-access) mode, and for files in object storage uploaded using tools other than Object Mount, we have some dynamic defaults set for ownership and permissions. In these circumstances, the owner of cloud objects is always reported to be the current user, the directory mode is reported as `0777`, and the file mode is reported as `0666`. - -The defaults can be overridden by using the `uid`, `gid`, `filemode` and `dirmode` options within the `CUNO_OPTIONS` environment variable. By doing so, an administrator can set the default UID/GID and access mode permissions that apply to all files and directories of cloud storage accessed by Object Mount. This might be considered in circumstances where an application starts as one user, but mid-process switches context to run as another; services like web-servers often function in this way. - -To demonstrate this in action, you could run the following: - -```console -$ cuno -o 'uid=0 gid=0 filemode=0444 dirmode=0447' \ - mount --root s3://examplebucket/directory/ /mnt/s3-bucket - -$ ls -l /mnt/s3-bucket -dr--r--rwx 1 root root 0 Jan 1 1970 directory --r--r--r-- 1 root root 0 Nov 10 11:16 file -``` - -While the default UID/GID and access mode permissions can be set ahead of time, they cannot be later modified. - -### POSIX File Access - -To persist and modify file system metadata, [POSIX File Access](../getting-started/enforced-posix-access) must be enabled. This allows the virtual files presented by Object Mount to have their system metadata modified by tools such as `chown`, `chmod`, and `touch`. - -{% callout type="note" %} -The system metadata is stored as objects within your object storage system within hidden subdirectories where the data resides. The name of that subdirectory is a period followed by a unicode U+FBF80 `󻾀` character. - -From within Object Mount you will not see that directory or objects inside of it, as Object Mount filters them from view; however, accessing your object storage using other tools, they will be visible. - -Non-Object Mount access which renames, moves or copies objects with Object Mount file attributes, will result in those objects losing their system metadata. You will need to use Object Mount to manage those files and preserve their attributes. -{% /callout %} - -#### Enabling for Direct Interception - -There are two ways to enable POSIX File Access when using [Direct Interception via Object Mount CLI](user-guide-direct-interception): - -##### Using a bucket tag - -{% callout type="note" %} -The following permissions are required to use the `cuno creds setposix` command: - -- S3-compatible: `s3:PutBucketTagging` and `s3:GetBucketTagging` -- Google Cloud Storage: `storage.buckets.update` and `storage.buckets.get` IAM permissions -- Azure Blob Storage: Only the container owner may set labels -{% /callout %} - -This setting toggles POSIX mode on for the bucket and takes precedence over the `CUNO_POSIX` environment variable. - -```console -cuno creds setposix s3://examplebucket true -``` - -Valid options are: - -- `true` - enable posix in enforcing mode, preventing unauthorized access. -- `false` - disable posix -- `permissive` - enable posix in permissive mode, where object attributes like uid, gid, mode, timestamps are stored but user access privilege is not checked -- `metadata` - enable posix in enforcing mode and store posix attributes as object metadata (only on S3) -- `permissive-with-metadata` - enable posix in permissive mode and store posix attributes as object metadata -- `unset` - remove posix configuration from the bucket - -The `cuno creds setposix` command writes a tag to the bucket's metadata on the bucket store. The setting set by this command applies to all users using the bucket through Object Mount. - -{% callout type="warning" %} -Some S3-compatible stores may not support setting custom metadata flags required for this feature to function. If this is the case for you, you can still use the `--posix` flag to enable POSIX enforcement when starting a Object Mount on FUSE. You can also still use the `CUNO_POSIX` environment variable to enable it for a Direct Interception session. If these methods are not satisfactory or preferable, please get in contact at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). -{% /callout %} - -To unset the bucket-level setting and allow controlling POSIX mode using either the `CUNO_POSIX` environment variable or the `--posix` command line flag, use `cuno creds setposix` with a setting of `unset`: - -``` -cuno creds setposix s3://examplebucket unset -``` - -##### Using the CUNO_POSIX environment variable - -If POSIX mode is not set at the bucket level, it may be enabled by setting the environment variable `CUNO_POSIX` to `1`. If POSIX is enabled for the bucket via a tag, it cannot be overridden with this environment variable. - -For example, to enable POSIX mode for a Direct Interception session via Object Mount CLI, do: - -``` -$ export CUNO_POSIX=1 -$ cuno -``` - -This mode is suitable for when you need deeper POSIX compatibility in order to correctly run applications or workflows. For example, a tool such as `rsync` may be checking `mtime` in order to check for updates to a file so you will need to be preserving such attributes on the object storage side. - -{% callout type="note" %} -This is not "secure" in the sense that Direct Interception mode requires the user to have access to object storage credentials so they can work around any access limitations by using the credentials directly with other tools (such as awscli). -{% /callout %} - -#### Enabling POSIX for a Object Mount on FUSE - -If a bucket doesn't have POSIX mode already set, you can use the option `--posix` when mounting a [Object Mount on FUSE](../user-guides/basic#object-mount-on-fuse) to enable fine-grained control over ownership and permissions and have them enforced. If a bucket's POSIX mode tag is set, but the `--posix` flag is not, the bucket's setting will apply but the permissions will not be enforced by the mount. - -If you are mounting for yourself and others, mount the bucket in a location that is accessible to the users who need access to the files. This can be a shared location, or a location that is only accessible to the user who needs access. - -If setting up the mount as an admin user, to allow for sudo access, please use the `-o allow_root` flag, and to allow other users to access the mount as themselves plase use `-o allow_other`. - -```console -$ cuno mount --posix -o allow_root -o allow_other --root s3://mybucket /mnt/cloud/bucket -``` - -Users will now be able to see the files in the bucket at `/mnt/cloud/bucket`. You should now treat this as any other POSIX file system, and set permissions as you need to control access. - -#### Usage examples - -Assuming a mount location of `/mnt/s3-bucket`: - -- Change permissions on the object/file: - - ```console - $ chmod 0770 /mnt/s3-bucket/file - $ ls -l /mnt/s3-bucket/file - -rwxrwx--- 1 root root 0 Nov 10 11:16 file - ``` - -- Change the owner of the object/file: - - ```console - $ chown alice /mnt/s3-bucket/file - $ ls -l /mnt/s3-bucket/file - -rwxrwx--- 1 alice root 0 Nov 10 11:16 file - ``` - -- Alter the modify time: - - ```console - $ echo "Hello" > /mnt/s3-bucket/file - $ ls -l /mnt/s3-bucket/file - -rwxrwx---+ 1 alice root 6 Nov 10 11:32 file - ``` - -- Alter the change time: - - ```console - $ chmod +x /mnt/s3-bucket/file - $ stat /mnt/s3-bucket/file - File: /mnt/s3-bucket/file - Size: 6 Blocks: 1 IO Block: 65536 regular file - Device: 49h/73d Inode: 4 Links: 1 - Access: (0770/-rwxrwx---) Uid: ( 1009/ alice) Gid: ( 0/ root) - Access: 2021-11-10 11:32:59.185000000 +0000 - Modify: 2021-11-10 11:32:59.185000000 +0000 - Change: 2021-11-10 11:47:21.846000000 +0000 - ``` - -## Other Environment Variables - -### CUNO_CREDENTIALS - -The environment variable `CUNO_CREDENTIALS` allows you to customise the location of the Object Mount credentials store, and is elaborated upon in [user-guide-credentials-management](../user-guides/credentials). - -### Proxy server tunneling - -Object Mount supports HTTP and HTTPS proxying access through to your cloud object storage. - -Provide the environment variable `http_proxy` or `https_proxy` with your gateway and Object Mount will tunnel all transactions through your designated proxy. - -### Verbose Debug Output - -There are rare occasions where Object Mount will output to the command line, and this will generally indicate a fault. -All messages will be prefixed with `cuno:` and these will always be sent a to the standard error stream (`stderr`). - -The options available for `CUNO_LOG` are cumulative and build on the previous level: - -> ``` -> .. cssclass:: listtable43mm -> -> .. list-table:: -> :header-rows: 0 -> -> * - :code:`error` -> - At this level, only critical failures are output. (Default) -> * - :code:`warning` -> - At this level, errors and warnings are output. -> * - :code:`debug` -> - At this level, information relevant to debugging will be output. -> * - :code:`trace` -> - At this level, a verbose logging of all steps will be output. -> ``` - -For example, a directory listing with the trace level set, can output similar to the below: - -``` -$ CUNO_LOG=trace ls s3://example-bucket -cuno: [TRACE][01/02/21 08:39:07.39][Thread: 3817873920] Trying bucket store at: /home/user/.config/cuno/creds/... -cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Loading S3Client -cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Not On EC2 -cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Attempting to find CaCerts. -cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using Certs: /etc/ssl/certs/ca-certificates.crt -cuno: [TRACE][01/02/21 08:39:07.40][Thread: 3817873920] Attempting to load PGMAN -cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using custom Endpoint: http://127.0.0.1:8080 -cuno: [DEBUG][01/02/21 08:39:07.40][Thread: 3817873920] Using custom pathstyle -``` - -There is a separate option: `access` that enables detailed logging of the file access performed by Object Mount, which includes read, write, open, close, seek and similiar. -`access` can be combined with the other log levels to log both file access and the desired logs. For example: - -``` -$ CUNO_LOG=trace,access ls s3://example-bucket -cuno: [DEBUG][07/17/23 13:59:03.287][P/T: 296181:2897806400] Not On EC2 -cuno: [DEBUG][07/17/23 13:59:03.287][P/T: 296181:2897806400] Using Certs: /etc/ssl/certs/ca-certificates.crt -cuno: [ACCESS][07/17/23 13:59:03.355][P/T: 296181:2897806400] s3://example-bucket/file write() 12 @ 0 -``` diff --git a/app/(docs)/object-mount/linux/user-guides/core-concepts/page.md b/app/(docs)/object-mount/linux/user-guides/core-concepts/page.md deleted file mode 100644 index c137574b3..000000000 --- a/app/(docs)/object-mount/linux/user-guides/core-concepts/page.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Core Concepts -docId: shiequai7tooNgoo - -metadata: - title: Core Concepts - description: Core Concepts - -weight: 1 ---- - -The ideas presented here are intended to familiarise you with our approach to object storage. - -See [Getting Started Core Concepts](../getting-started/intro) diff --git a/app/(docs)/object-mount/linux/user-guides/credentials/page.md b/app/(docs)/object-mount/linux/user-guides/credentials/page.md deleted file mode 100644 index 5aadd524b..000000000 --- a/app/(docs)/object-mount/linux/user-guides/credentials/page.md +++ /dev/null @@ -1,340 +0,0 @@ ---- -title: Credential Management -docId: aish4shuiheeZaig -redirects: - - /object-mount/credentials/linux/import -metadata: - title: Credential Management - description: Credential Management - -weight: 5 ---- - -## Overview -Users will need to specify their credentials for their cloud storage accounts in order to fully use Object Mount. -`cuno creds` allows users to import and manage their credentials for supported cloud providers. -Credentials are stored in `"${XDG_CONFIG_HOME}"/cuno/creds` by default (or `~/.config/cuno/creds` if `XDG_CONFIG_HOME` is not set). - -Override the defaults by setting the `CUNO_CREDENTIALS` environment variable. - -{% callout type="note" %} -`cuno creds` will determine the presence of the following paths to store credentials, in {emphasis}`descending` order of precedence: - -1. `$CUNO_CREDENTIALS` -2. `"${XDG_CONFIG_HOME}"/cuno/creds` -3. `~/.config/cuno/creds` -4. `"${CUNO_ROOT}"/cuno-config/creds` -{% /callout %} - -## Create and retrieve credentials - -### Credential file formats - -Credential files are `plain-text`, with a specific format for each cloud provider. - -### Amazon Web Services S3 - -There are two (equivalent) formats for AWS S3 credential files: - -``` -aws_access_key_id = -aws_secret_access_key = -region = [optional] -``` - -``` -AWSAccessKeyId = -AWSSecretKey = -region = [optional] -``` - -More information on S3 access keys is available in (user-guide-getting-s3-credentials)[../user-guides/appendix#getting-s3-credentials-using-the-aws-console]. - -Object Mount can also use an AWS S3 Access Point instead of a container; see [user-guide-s3-access-points](../user-guides/access#aws-s3-access-point-support) for more details. - -### S3-compatible solutions - -To define a custom endpoint for S3-compatible storage solutions, create a file containing at least the following: - -> ```text -> aws_access_key_id = xxxxxxxxxxxxxxxxxx -> aws_secret_access_key = xxxxxxxxxxxxxxxxxx -> endpoint=http://127.0.0.1:8080 -> ``` - -Endpoints could be IP addresses or domain URIs (optionally with port numbers). Note that a schema is required (e.g. `http://` or `https://`). -The following are examples of valid endpoint key-value pairs: - -``` -endpoint = http://127.0.0.1:8080 -endpoint = https://s3objectstorage.example.com -``` - -Depending on the capabilities and behaviours of your object storage provider's S3 API, you may find the additional keys useful: - -| Option | Description | -|--------|-------------| -| `skipssl` | Skip SSL certificate verification. Useful for private storage solutions with self-signed certificates. | -| `nossl` | Use HTTP protocol instead of HTTPS. Use when the endpoint starts with `http://`. | -| `pathstyle` | Switch from virtual host style to path style syntax for S3-compatible storage solutions. Enable for private storage solutions supporting only path style syntax (e.g. MinIO, Ceph). | -| `no-delete-objects` | Do not use `DeleteObjects` to remove multiple objects. Instead, use `DeleteObject` for single object removal. | -| `no-remote-copy` | Do not copy objects within the same bucket via the server; download and upload instead. | -| `no-object-tagging` | Do not attempt to get/put object tags. | -| `no-object-acl` | Do not attempt to get/put object ACLs. | -| `no-multipart-upload` | Do not attempt to use [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html). | -| `no-copy-part` | Do not attempt to use [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html). | -| `no-range-download` | Do not attempt to use [Byte-Range Fetches](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html); use a single request to download the whole object instead. | -| `no-delete-objects` | Do not attempt to use [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) when deleting multiple objects; use [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) for each object to delete. | -| `directory-marker` | Create a placeholder object when creating an empty prefix (e.g., `mkdir`). Remove the placeholder object when removing an empty directory. | -| `ignore-empty-prefixes` | Ignore directories that were removed but still reported as existing prefixes by the server. | -| `posix=` | Set POSIX mode for all buckets associated with this credential. This option takes precedence over the POSIX mode runtime environment variable, but may be overridden by a bucket-wide POSIX setting (set via `cuno creds`). | - - -{% callout type="note" %} -Some of the above options can be automatically detected and populated by running `cuno creds detectfeatures ` (e.g. `cuno creds detectfeatures s3://test-bucket test_credentials.s3c`). -This command needs a bucket to be specified that it can write temporary files to for testing purposes. The commmand will run a series of tests against the specified bucket, checking the availability of S3 features that Object Mount uses and updating the credentials file accordingly. -{% /callout %} - -{% callout type="warning" %} -Running feature detection will use up to a few gigabytes of bandwidth and may take a few minutes to complete depending on the machine's connection speed and the S3-compatible storage provider. -{% /callout %} - -## Microsoft Azure Blob Storage - -Azure Blob Storage credential files have the following format: - -``` -AZURE_STORAGE_ACCOUNT = -AZURE_STORAGE_ACCESS_KEY = -``` - -If you need the access key, see [View account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys). - -Object Mount also supports using Shared Access Signatures (SAS) to access remote containers in Azure. -Refer to [user-guide-azure-sas]() to configure SAS. - -## Google Cloud Storage - -You will need key-based access to be associated with your [Google Cloud service account](https://developers.google.com/workspace/guides/create-credentials#service-account). - -If you need to set this up, you can have Google generate a new key by following Google's guide, [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete). Alternatively, you can generate your own key pair and upload it on the service account's key management page, by following the guide [Upload service account keys](https://cloud.google.com/iam/docs/keys-upload). - -If you've generated a new key, store the JSON file you have downloaded in a safe place with appropriate access permissions, ready for the next steps. - - -## Import credentials - -Every cloud account requires a unique credential file. -Import a credential file into Object Mount using the following command: - -```console -cuno creds import -``` - -Object Mount performs the following actions when importing a credential file: - -1. Detect the credential file format, and connect to the remote server. -2. Discover all possible containers that can be accessed using the provided credentials. -3. Create pairings between the imported credential file and remote containers. - -{% callout type="note" %} -Discovery only completes when the provided credentials include bucket listing permissions. -If listing permissions cannot be provided, manually pair the imported credential file to remote buckets using `cuno creds pair` (refer to [#pair-containers-and-credentials]()`). -{% /callout %} - -## Pair containers and credentials - -Manually pair additional containers with previously-imported credential files using `pair`: - -```console -cuno creds pair [imported_credential_file] -``` - -`` is the remote URI of the container to be associated with the credential file. -`[imported_credential_file]` is optional and corresponds to the credential file that has already been imported. -If the latter is not provided, all imported credential files are listed. - -To re-associate a previously paired remote container with a different credential file, use `pair` with the same remote URI and a different credential file. - -{% callout type="note" %} -Object Mount verifies that a remote container is accessible when pairing it with a credential file. -Object Mount auto-detects an AWS region if it is not specified, or is invalid. -{% /callout %} - -### Microsoft Azure Blob Storage with Shared Access Signatures - -To permit access to storage resources without sharing access keys, use a Shared Access Signature (SAS). -An SAS gives users access to a container for only specified intervals and with specified permissions. -A remote Microsoft Azure Blob Storage container can be accessed using a Shared Access Signature (SAS). -Provide an SAS to Object Mount using the following command: - -``` -cuno creds pair "?" -``` - -{% callout type="note" %} -The question mark `?` prefix and quotation marks `"` are `required`. -{% /callout %} - -### Public Access buckets - -To pair a remote URI that has public access, use the following command: - -``` -cuno creds pair public -``` - -Notice `public` is used in the place of the imported credential file name. - -### Requester Pays buckets - -Amazon AWS S3 and Google Cloud Storage offer 'Requester Pays' features for containers. -In this case, requesters—and `not` bucket owners—are billed when remote objects are accessed. - -By default, Object Mount skips 'Requester Pays' containers when importing a credential file to avoid unexpected charges. -To enable access to such containers with a previously-imported credential file, run the following command: - -``` -cuno creds --interactive pair -``` - -The `--interactive`/`-i` option will prompt the user to select from credentials in Object Mount's database. - -#### Amazon AWS S3 - -To force a pairing between a 'Requester Pays' container with an imported credential file: - -``` -cuno creds --force pair -``` - -#### Google Cloud Storage - -Run the command in [#requester-pays-buckets](); this will list all available project IDs associated with the selected credential file and available for billing. -Then, select and confirm a 'Requester Pays' container to pair as prompted. - -To run the command non-interactively: - -``` -cuno creds pair --billing -``` - -## List credentials - -`list` displays information about imported credential files and paired buckets. -The command accepts additional options: - -| Command | Description | -|---------|-------------| -| `cuno creds list` | Display all imported credential files and all paired buckets | -| `cuno creds list creds` | Display all imported credential files **only** | -| `cuno creds list pairings` | Display all paired buckets **only** | -| `cuno creds list pairings ` | Display all paired buckets for the specified provider **only** | - -`pairings ` accepts a comma-separated list of providers. -For example, `cuno creds list pairings gs,s3` will display information about all paired buckets in Google Cloud Storage and AWS S3. - -## Unpair and purge credentials - -To dissociate a container from a credential file, use `unpair`: - -```console -cuno creds unpair [bucket_remote_uri] -``` - -If the second argument is missing, then existing pairings are listed. - -To completely remove a credential file and delete all its associations with remote containers, use `purge`: - -``` -cuno creds purge [imported_credential_file] -``` - -If the second argument is missing, then existing credential files are listed. - -## Additional flags - -Object Mount accepts additional flags: - -``` -cuno creds [flags] -``` - -These flags are described in the following table: - -| Flag | Description | -|---------|-------------| -| `--interactive/-i` | Asks for permission to replace existing credential files, pairings, SAS etc. | -| `--force/-f` | Always replaces existing credential files, pairings, SAS etc. | -| `--color/-c` | Displays coloured output. | - - -## Alternative methods of authenticating - -{% callout type="note" %} -Prefer using the built-in credential management to administer credentials; only use the following methods if additional flexibility is required. -{% /callout %} - -### Amazon AWS S3 using native credentials - -Object Mount can also use the file supplied by Amazon AWS S3: - -``` -~/.aws/credentials -``` - -with the profile optionally specified with the environment variable: - -``` -AWS_PROFILE -``` - -Set the AWS region with the environment variable: - -``` -AWS_REGION -``` - -Alternatively, directly set the following two environment variables before launching Object Mount: - -``` -AWS_ACCESS_KEY_ID -AWS_SECRET_ACCESS_KEY -``` - -### AWS S3 using IAM Roles for EC2 - -IAM Roles provide an alternative method to securely distribute AWS credentials across EC2 instances. -These credentials can be used for accessing AWS S3. Refer to the [Amazon AWS EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) for more information. - -Object Mount will automatically detect IAM roles on an EC2 instance, so no further configuration is necessary. - -{% callout type="note" %} -Combined use of Object Mount managed credentials with IAM Roles in EC2 is `not currently supported`. -{% /callout %} - -### Google Cloud Storage using environment variable - -Object Mount can directly use service account credentials, which are commonly used by applications to access Google Cloud Storage. To use a service account credential file (in JSON or PKCS12 format), specify a value for the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. -For example, a credential file located in `/home/user/gckey.json` can be loaded to Object Mount by using: - -``` -export GOOGLE_APPLICATION_CREDENTIALS="/home/user/gckey.json" -``` - -### Azure Storage using environment variables - -To access Azure Storage without using Object Mount credentials management, provide a storage access key associated with a Microsoft Azure Storage account. -To enable this access scheme, set the `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_ACCESS_KEY` environment variables: - -```console -export AZURE_STORAGE_ACCOUNT="" -export AZURE_STORAGE_ACCESS_KEY="" -``` - -{% callout type="warning" %} -Microsoft recommends that storage access keys are not shared with anyone else. -To permit access to storage resources without sharing access keys, use a Shared Access Signature (SAS). -An SAS gives users access to a container for only specified time period with a fixed set of permissions. -Refer to [user-guide-azure-sas](#microsoft-azure-blob-storage-with-shared-access-signatures) for more information. -{% /callout %} diff --git a/app/(docs)/object-mount/linux/user-guides/deployment-modes/direct-interception/page.md b/app/(docs)/object-mount/linux/user-guides/deployment-modes/direct-interception/page.md new file mode 100644 index 000000000..661221cfd --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/deployment-modes/direct-interception/page.md @@ -0,0 +1,206 @@ +--- +title: Direct Interception +hideTitle: false +docId: UHsd5HnesueQyhnZ +weight: 1 +metadata: + title: Configuring Direct Interception Modes + description: + Details on the use and configuration of Direct Interception Mode in Object Mount for Linux. +hidden: false +--- + +This article dives into the configuration and use of **Direct Interception Mode** in Object Mount for Linux. + + +## Overview + +When running in **Direct Interception Mode**, Object Mount will insert itself between your applications and the operating system. It will dynamically redirect relevant file system calls made by shell commands and applications to the appropriate APIs of your object storage provider. This dynamic “mapping” between _file system concepts_ and _object storage interfaces_ gives your tools instant access to objects as files. + +This direct interception means _any_ applications that can access file storage can now work with object storage. Object Mount intercepts I/O calls from any user binaries, including those written in C/C++, Rust, Go, Java, Python, etc. + +It’s built to work everywhere: It works with dynamic binaries and static binaries. It works inside unprivileged containerized environments. + +{% callout type="note" %} + **Support for Any Application** + + This includes applications you’ve written yourself — just treat the paths to object storage as local file system paths. Object Mount handles the rest. +{% /callout %} + +Direct interception Mode uses the `LD_PRELOAD` environment variable so that Object Mount can capture and redirect storage access library and system calls to object storage APIs. + + - If a static binary is intercepted, a JIT ELF binary translator will redirect relevant calls when the binary is loaded into memory. + + +## Highlights + + - Direct Interception offers the highest performance access to object storage that Object Mount provides. + - Direct Interception Mode is the _default mode_ and the most common mode for Object Mount to be deployed in. + - Unless otherwise noted, all documentation pages and installation instructions are written assuming you are using “Direct Interception Mode”. + + +## Example Use Cases + + - Direct Interception Mode is best suited for situations in which installed app compatibility is not an issue. + - For example: When setting up a fixed workflow with Object Mount that can be tested and verified before putting it into production. + +{% callout type="warning" %} + **Direct Interception vs. Object Mount on Fuse** + + Direct Interception Mode does _not_ currently support SUID binaries, or certain packaged apps like 🌐 [Snap](https://snapcraft.io/about), 🌐 [AppImage](https://appimage.org/), or 🌐 [Flatpak](https://docs.flatpak.org/en/latest/introduction.html) applications. + + See [Object Mount on FUSE](docId:ZdvWLcm9uFmM5HLk) and/or [Object Mount FlexMount](docId:cFUt9zgCRFFDk5Sq) to support these tools. +{% /callout %} + + +## Advantages and Disadvantages + +| **Advantages** | **Disadvantages** | +|----------------------------------|---------------| +| **Speed:** Dynamic interception offers the best performance. | **Compatibility:** Direct interception does not currently support SUID binaries, Snaps, AppArmor, or Flatpak applications. +| **Set up time:** This is the default mode — just launch an Object Mount shell with the `cuno` CLI command. | **Activation:** In some cases, it is difficult to enable direct interception or keep it enabled (because of environment variable clobbering or lack of privileges to set/use LD_PRELOAD). | +| **Linux native:** For users who are skilled with Linux, using the `cuno` CLI is quick, easy, and non-intrusive. | **Non-technical users:** If Object Mount is being used by team members with only minimal Linux skills, teaching them how to use the Object Mount shell and diagnose if it’s working correctly may be less desirable than setting up a Object Mount on FUSE on their behalf. | + + +## How to Enable + +Direct Interception Mode can be enabled per-session or per-command. + +### Direct Interception: Per-Session + +Enable Object Mount for a **single shell session** by calling `cuno` from a terminal command line: + +```shell +# terminal +cuno +``` + +This will launch a new interactive “wrapped shell” with Object Mount acting in Direct Interception Mode. + +The wrapped shell itself has Object Mount monitoring all its I/O calls — so every command entered or application launched from within the wrapped shell will be monitored and intercepted by Object Mount + +{% callout type="note" %} + **Object Mount CLI `cuno` as a Wrapper** + + When `cuno` is used to launch a new shell, the shell is launched with `LD_PRELOAD` set to point at `cuno.so`. + + - There is no Object Mount shell binary. + - `cuno` only “wraps” existing shells (bash, zsh, etc.) with Object Mount pre-loaded. +{% /callout %} + +### Direct Interception: Per-Command + +To enable direct interception for a **single command**, use: + +```console +cuno run bash -c "" +``` + +{% callout type="note" %} + **Best Practice for Running Single Commands** + + It is recommended to execute one-time commands in this fashion to maintain support for Wildcard expansion (`*`) of filesystem and remote paths. +{% /callout %} + + +## Path Usage + +Inside a `cuno`-wrapped shell session, object storage buckets can be accessed using either **path** or **URI** formats. + +The commands below provide examples of both path and URI usage within a `cuno`-launched shell: + +{% tabs %} + +{% tab label="AWS S3" %} + ```shell + # terminal + cuno + (cuno)$ ls s3:/// + (cuno)$ ls /cuno/s3// + ``` +{% /tab %} + +{% tab label="Storj" %} + ```shell + # terminal + cuno + (cuno)$ ls s3:/// + (cuno)$ ls /cuno/s3// + ``` +{% /tab %} + +{% tab label="Microsoft Azure" %} + ```shell + # terminal + cuno + (cuno)$ ls az://// + (cuno)$ ls /cuno/az/// + ``` +{% /tab %} + +{% tab label="Google Cloud" %} + ```shell + # terminal + cuno + (cuno)$ ls gs:/// + (cuno)$ ls /cuno/gs// + ``` +{% /tab %} + +{% tab label="Other S3 Compatible" %} + ```shell + # terminal + cuno + (cuno)$ ls s3:/// + (cuno)$ ls /cuno/s3// + ``` +{% /tab %} + +{% /tabs %} + +{% callout type="info" %} + **Cloud Paths** + + For additional information on specifying cloud paths using the Object Mount CLI, see the Linux User Guide article: [](docId:jieteeYeyievui9k). +{% /callout %} + + +## Shell Interaction + +`bash` and `zsh` are fully supported when using the Object Mount CLI. + +This includes support for: + - Tab auto-completion of remote paths + - Wildcard expansion (`*`) of filesystem and remote paths + +In either of these fully-supported shells, the prompt will be prefixed with `(cuno)` to indicate that Direct Interception is enabled and that you are using an Object Mount CLI-wrapped shell:: + +```console +(cuno) user@host:~$ +``` + +### Detecting Shells + +To select between the two shells, Object Mount determines if either is present using the following methods (in order): + +1. The shell used to launch Object Mount +2. The user’s preferred login shell +3. All installed shells + +If neither `bash` nor `zsh` are found, then the first shell in this list is used. If no shell is set, then `/bin/sh` is used. + +{% callout type="warning" %} + **Processor Snapshot (`ps`) usage** + + The behavior described here relies on `ps`. + + Depending on your method of installation, you may need to install `ps` manually. (For RedHat/RHEL derivatives, install package `procps`). +{% /callout %} + +### Specifying Shells + +To launch a **specific shell** with Object Mount enabled, use `cuno run`: + +```console +cuno run +``` diff --git a/app/(docs)/object-mount/linux/user-guides/deployment-modes/flexmount/page.md b/app/(docs)/object-mount/linux/user-guides/deployment-modes/flexmount/page.md new file mode 100644 index 000000000..3b2018156 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/deployment-modes/flexmount/page.md @@ -0,0 +1,114 @@ +--- +title: Object Mount FlexMount +hideTitle: false +docId: cFUt9zgCRFFDk5Sq +weight: 3 +metadata: + title: Configuring Object Mount FlexMount + description: + Details on the use and configuration of Object Mount FlexMount for Linux. +hidden: false +--- + +This article dives into the configuration and use of **Object Mount FlexMount** for Linux. + + +## Overview + +**Object Mount FlexMount** combines the wide compatibility and support for Linux Apps via FUSE (including SUID binaries, Snap, AppImage and FlatPak applications) plus all the speed & benefits of Direct Interception Mode whenever possible. + +With Object Mount in FlexMount Mode, Object Mount will recognize that the mount path is an Object Storage mount and use **Direct Interception Mode** whenever possible for the fastest access. Object Mount will fall back to using **Object Mount on FUSE** for anything that cannot be directly intercepted. THis provides both the **best performance** and the **broadest compatibility**. + +Object Mount’s FlexMount benefits are leveraged by: + + 1. Configuring **Object Mount on FUSE**, and + 2. Accessing your cloud storage via the **Object Mount CLI** by specifying your mount path via the **`--flex` parameter**. + +### Advantages + +1. **Speed:** Where possible, Object Mount in FlexMount Mode will operate as fast as possible, using Direct Interception Mode. + +2. **Compatibility:** Support for all POSIX applications: Anything that cannot be directly intercepted will pass through and be handled by Object Mount on FUSE. + +### Disadvantages + +1. **Set up time:** Object Mount on FUSE needs to be configured. + +2. **Shell launch is more complicated:** Each time an Object Mount shell is launched it must be configured to use the mount. This can be worked around by setting up a Object Mount on FUSE [on boot](docId:ZdvWLcm9uFmM5HLk#mount-on-boot), and setting an alias to launch a Object Mount shell with the correct parameters. + + +## How to Enable + +FlexMount is set up as follows: + +1. First, set up a mount using [Object Mount on FUSE](docId:ZdvWLcm9uFmM5HLk): + + ```console + $ mkdir "$HOME/my-object-storage" + $ cuno mount "$HOME/my-object-storage" + ``` + +2. Always use the Object Mount CLI to interact with your object storage. + + Every time the Object Mount CLI is launched, use the `--flex ` parameter. + + For example: + + ```console + $ cuno --flex "$HOME/my-object-storage" + ``` + +3. Then reference your mount using the mount’s path: + + ```console + (cuno) $ ls $HOME/my-object-storage/s3// + ``` + +{% callout type="note" %} + **FlexMount Tilde (~) Use** + + You cannot use a tilde `~` in your `CUNO_OPTIONS` or `cuno -o` CLI options when setting up a FlexMount, as this is something that the shell needs to resolve. + + You may still use it for your `cuno mount` commands, and subsequent FlexMount operations (such as `ls ~/my-object-storage`). +{% /callout %} + +The same FlexMount can be re-used across multiple Object Mount-wrapped shells. + +The `--flex` option can also be used with `cuno run` to run a single command or script with Object Mount enabled. + +For example: + +```shell +# terminal +cuno run --flex "$HOME/my-object-storage" bash -c "touch $HOME/object_storage_mount/s3//newfile" +``` + + +## FlexMount Interception Details + +The `--flex` option is synonymous with the `-o cloudrootover=exact -o cloudroot=""` option, which is used to tell Object Mount to intercept paths that exactly match the `cloudroot` setting, and to handle them using Object Mount on FUSE. + +When using Object Mount Direct Interception in FlexMount Mode, most “local” paths are intercepted but not acted upon as they are handled by the local file system. + +The `cloudrootover` setting tells Object Mount to intercept local paths that match the `cloudroot` setting, and to prioritize itself handling them via the mount. This means that whenever a path is recognized as the `cloudroot`, it will be more efficiently handled in user-space by the Direct Interception/`LD_PRELOAD` library. + +That path recognition can be done in two ways: + + - **Exact String Matching:** (`exact`) This method will match the `cloudroot` setting exactly. This is faster, and will end up relying on the FUSE mount whenever the paths don’t match the `cloudroot` — for example with symbolic links located outside the mount pointing into the mount. + + Exact String Matching is the default method. No additional configuration is needed to enable. + + - **Resolved Path Matching:** (`resolve`) This method will resolve the full path given in a file system call, including symbolic links, to check if the file is ultimately located inside the mount point. + + This requires more calls for every path-based file system call, so is slower when many files are being accessed. It is useful when symbolic links are pointed to the mount, and other cases where the path ultimately is inside the mount. + + This can be more efficient in some special cases (few files, large transfers, complex relationships between files) as `cuno.so` will intercept more calls without requiring them to go through the slower FUSE mount. + + To use Resolved Path Matching Mode, use the parameters `-o cloudrootover=resolve -o cloudroot=""`. + + For example: + + ```console + $ cuno -o cloudrootover=resolve -o cloudroot="$HOME/my-object-storage" + (cuno) $ ls $HOME/object_storage_mount/s3// + ``` diff --git a/app/(docs)/object-mount/linux/user-guides/deployment-modes/fuse/page.md b/app/(docs)/object-mount/linux/user-guides/deployment-modes/fuse/page.md new file mode 100644 index 000000000..5ba106d26 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/deployment-modes/fuse/page.md @@ -0,0 +1,265 @@ +--- +title: Object Mount on FUSE +hideTitle: false +docId: ZdvWLcm9uFmM5HLk +weight: 2 +metadata: + title: Configuring Object Mount on FUSE + description: + Details on the use and configuration of Object Mount on FUSE for Linux. +hidden: false +--- + +This article dives into the configuration and use of **Object Mount on FUSE** for Linux. + +## Overview + +**Object Mount on FUSE** allows you to mount an object storage path as a _directory_ within the local file system hierarchy. + +This allows you (and any other user with access to the mount) to access object storage as if it were just another local directory. + + +## Highlights & Advantages + +Object Mount on FUSE is a FUSE file system. FUSE routes calls through the Object Mount object storage back-end. + + - **Compatibility:** Use Object Mount on FUSE when compatibility is a primary concern (e.g.: when using a variety of applications across multiple different operating systems). + + - **Interoperable:** Object Mount on FUSE can be used when launching the `cuno` binary is not possible (e.g. an automation engine without the required features). + + - **Simple Usage:** After a mount is set up, scripts only need to be changed to point at the new directory. No changes are needed to the workflow and no environment variables need to be set. + + - **Greater Administrative Control:** Object Mount on FUSE can be setup once, by an admin, without giving users any credentials. This removes the need to train users on object storage concepts. + +{% callout type="info" %} + **Performance vs. Compatibility** + + Due to the nature of FUSE file systems, Object Mount on FUSE can be slightly less performant than Object Mount in Direct Interception Mode. + + If speed is your primary objective, consider using Object Mount in [Direct Intercept Mode](docId:UHsd5HnesueQyhnZ) or via the [User-Mode Library](docId:airoogh4Waengi8u#via-user-mode-library-ld-preload). +{% /callout %} + + +## How to Enable + +Create a FUSE-based object storage mount by entering: + +```shell +# terminal +mkdir ~/my-object-storage +# terminal +cuno mount ~/my-object-storage +``` + +Any paired object storage buckets will now be accessible through the mount. + +Choose your storage provider below to see the commands to list files on your S3 mount: + +{% tabs %} +{% tab label="AWS S3" %} +```shell +# terminal +ls ~/my-object-storage/s3// +``` +{% /tab %} + +{% tab label="Storj" %} +```shell +# terminal +ls ~/my-object-storage/s3// +``` +{% /tab %} + +{% tab label="Microsoft Azure" %} +```shell +# terminal +ls ~/my-object-storage/az/// +``` +{% /tab %} +{% tab label="Google Cloud Storage" %} +```shell +# terminal +ls ~/my-object-storage/gs// +``` +{% /tab %} +{% tab label="Other S3 Compatible" %} +```shell +# terminal +ls ~/my-object-storage/s3// +``` +{% /tab %} +{% /tabs %} + +{% callout type="info" %} + **Mount Paths vs URI Paths** + + You cannot use URI paths like “s3://bucket1/foo” when using Object Mount on FUSE. + + To be usable, replace the URI prefix (`s3://`) with the path to your mount. +{% /callout %} + +{% callout type="note" %} + **POSIX Options** + + The `--posix` option requires that a FUSE package be installed on the system. +{% /callout %} + + +## Configuration Options + +There are various options that can be specified from the command line: + +```console +cuno [subsys-options] mount [mount-options] +``` + +These options include: + + - Object Mount --> Mount Options + - Object Mount --> Subsystem Options + - FUSE Options + +### Object Mount: Mount Options + +To _right_ of the `mount` verb, you can specify options that are specific to the mount operation: + +``` +cuno mount [option] ... +``` + +**Noteworthy Mount Options:** + +| **Mount Option** | **Description** | +|-----------------------|-----------------| +| `--root ` | Specifies the root object storage path to be mounted. For example, to mount a single bucket, you would use `--root s3://`. | +| `--posix` | Enables setting and enforcing access permissions, symbolic and hard links, users, groups, etc. Can be used with default permissions. Implicitly sets `-o allow_other`. This option is ideally used with a mount that is only on a single bucket, e.g. `cuno mount --root s3:// --posix `. You should also be using `cuno creds setposix s3://examplebucket true` to enable POSIX handling of the bucket consistently when Direct Interception or a FlexMount is used. | +| `` | Options that are specific to the FUSE mount operation. These options are passed on to `fum` ([FUSE User Mount](https://www.man7.org/linux/man-pages/man8/mount.fuse3.8.html)). Not to be confused with Object Mount subsystem options which use the same `-o` syntax. See below for some of these. See user-guide-cuno-mount-fuse-options. | +| `--no-allow-root` | Do not allow root to access the mount (allowed by default). Disables support for SUID binaries, Snap, AppArmor, and Flatpak applications. | +| `--auto-restart` | Automatically restarts Object Mount on FUSE if problems occur during execution. | +| `--mkdir` | Automatically creates the mount point directory if it does not exist. | +| `--debug` | Enables debug output (same as `cuno mount -o debug`). | +| `--dev-logs` | Enabled debug logging to `/tmp/fuse.logs` | +| `--verbose` | Enables verbose output. | + +### Object Mount: Subsystem Options + +To _left_ of the `mount` verb, you can specify options that are specific to the Object Mount subsystem: + +``` +cuno -o mount ... +``` + +Alternatively, these can be specific using a **system variable**: + +``` +export CUNO_OPTIONS="" +``` + +**Noteworthy Subsystem Options:** + +Some relevant Object Mount Subsystem options are given in the table below. + +Refer to the Linux Advanced Guide article [](docId:phohPoowequie5ji) for additional details. + +| **Subsystem Option** | **Description** | +|----------------------|-----------------| +| `cachehome=` (default: `/cunodb;/dev/shm;/tmp/cache/cuno`) | This option allows you to set a semicolon-delimited list of directories to consider for caching. This is for the internal metadata cache of the Object Mount process behind the mount. Note that the FUSE cache is separate from this cache. Example: `cuno -o cachehome=/mnt/cache;/cunodb;/dev/shm mount ` | +| `uid=` | Define the default user ownership of files and directories within a bucket. These are the UIDs that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. Example: `cuno -o uid=$(id -u ) mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | +| `gid=` | Define the default group ownership of files and directories within a bucket. These are the GIDs that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, prefer to use POSIX file access. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. Example: `cuno -o gid=$(id -g ) mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | +| `filemode=` | Define the default file access permission bits of files within a bucket. Supply the octal (numeric) representation of the permissions you want to apply. These are the permissions that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. For example, to set the default file access permission bits to 0770 (`-rwxrwx---`), use the following command: `cuno -o filemode=0770 mount ` To enforce these settings, use the FUSE mount option `-o default_permissions`. | +| `dirmode=` | Define the default file access permission bits of directories within a bucket. Supply the octal (numeric) representation of the permissions you want to apply. These are the permissions that the Object Mount subsystem will feed to the FUSE mount for non-POSIX (core file access) buckets. Not recommended for most use cases, but may be useful if mounting non-POSIX buckets in the same mount as POSIX buckets. **Ignored** if the bucket has an enabled POSIX tag. For example, to set the default file access permission bits to 0770 (`drwx + +### FUSE Options + +Also on the _right hand-side_ of the `mount` verb, you can specify options that are specific to the FUSE mount operation. + +These options are passed to the FUSE User Mount process: `fum`. (Refer to the 🌐 [FUSE Mount documentation](https://www.man7.org/linux/man-pages/man8/mount.fuse3.8.html). +``` +cuno mount [FUSE option] ... +``` + +{% callout type="note" %} + **Object Mount Options vs. FUSE Options** + + As with the Mount Options, these FUSE options must come _after_ the verb `mount`. + + **Note:** Some of these FUSE options use a similar `-o` syntax as the Object Mount Subsystem options (e.g.: `cuno -o`), but they are _not_ the same. +{% /callout %} + +**Noteworthy FUSE Options:** + +| **Option** | **Description** | +|-----------------|-----------------| +| `-o allow_root` | Allows root access to the mount; required for [SUID](https://www.redhat.com/sysadmin/suid-sgid-sticky-bit) permissions. | +| `-o allow_other` | Allows other users to access the mount; requires `user_allow_other` in `/etc/fuse.conf`. | +| `-o default_permissions` | Enable permission checking by the kernel. To use Enforced POSIX, use `--posix`, which will set this internally. | +| `-o auto_unmount` | Automatically unmounts the mount when the process terminates. | +| `-o ro` | Mounts the file system read-only. To achieve a read-only mount through Object Mount, you can alternatively use `cuno mount --posix` and change the permissions on dirs/files in the mount manually to read-only using `chmod`. See [here](https://www.man7.org/linux/man-pages/man8/mount.8.html) for more on standard mount options like `ro`, `rw`, etc. | +| `-o rw` (default) | Mounts the file system read-write. | +| `-o exec` (default) | Allow execution of binaries on the file system. | +| `-o noexec` | Disallow execution of binaries on the file system. | +| `-o clone_fd` | Uses a separate fuse device fd for each thread (may improve performance). | +| `-o max_idle_threads` | The maximum number of idle worker threads allowed (default: 10). | +| `-s` | Run in single-threaded mode. | +| `-f` | Run in the foreground. | +| `-o debug` | Enable debug output (implies `-f`). | + +**FUSE Kernel Default Attributes:** + +| **Attribute** | **Description** | +|---------------------|-----------------| +| `-o uid=N` | Sets the file owner of all mounted files/dirs to the specified user ID. These are the UIDs that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | +| `-o gid=N` | Sets the file group of all mounted files/dirs to the specified group ID. These are the GIDs that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | +| `-o umask=M` | Sets the file permissions (octal) of all mounted files/dirs. These are the permissions that the FUSE mount will expose to the user. This will ignore any cuonFS POSIX settings including the bucket tag. To enforce these settings, use the FUSE mount option `-o default_permissions`. | + +**FUSE Kernel Cache Configuration:** + +| **Cache Config** | **Description** | +|-------------------------------------|-----------------| +| `-o kernel_cache` | This method caches file data across `open`; i.e. disables flushing of the file data cache on every `open`. Without this option (and neither `direct_io`), data is cached before the next `open` so a `read` syscall may not initiate a `read` operation. | +| `-o auto_cache` | Enables automatic flushing of the data cache on open(). The cache is only flushed if the modification based on modification times (off). | +| `-o entry_timeout=T` | Sets the kernel cache timeout for names (1.0s). | +| `-o negative_timeout=T` | Sets the kernel cache timeout for a negative lookup (file not found) (0.0s). | +| `-o attr_timeout=T` | Sets the kernel cache timeout for attributes (1.0s). | +| `-o ac_attr_timeout=T` | Sets the kernel cache timeout for checking if `-o auto_cache` should flush file data on `open`. | + +### Object Mount on FUSE - Commands + +Once you have configured a mount using Object Mount on FUSE, you can use the following commands to manage it. + +```console +cuno mount [command] +``` + +**Noteworthy Object Mount on FUSE Commands:** + +**List** + - `--list` + - Lists your active Object Mount on FUSE. + - You can also use the Linux utility `mount` directly. + - Example: + ```console + cuno mount --list + ``` + +**Unmount** + - `--unmount` / `-u` + - Unmounts a mount from the specified mount path. + - You can also use the Linux utility `umount` directly. + - Example: + ```console + cuno mount --unmount + ``` + +**Unmount and end the `fum` process** + - `--unmount-kill` / `-U` + - Unmounts a mount from the specified mount path and kills the `fum` process. + - You can also use the Linux utility `umount` directly. + - Example: + ```console + cuno mount --unmount-kill + ``` + +### Mount on Boot + +You can add mount commands to `/etc/fstab` to automatically mount an object storage path on start up. diff --git a/app/(docs)/object-mount/linux/user-guides/deployment-modes/fusion/page.md b/app/(docs)/object-mount/linux/user-guides/deployment-modes/fusion/page.md new file mode 100644 index 000000000..106348eff --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/deployment-modes/fusion/page.md @@ -0,0 +1,610 @@ +--- +title: Object Mount Fusion +hideTitle: false +docId: GVT7eXEaMSZCWESj +weight: 4 +metadata: + title: Configuring Object Mount Fusion + description: + Details on the use and configuration of Object Mount Fusion for Linux. +hidden: false +--- + +{% callout type="warning" %} + **Fusion Beta** + + Object Mount Fusion is currently in **Beta**. + + Your feedback via our 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) will help us improve it! +{% /callout %} + + +## Overview + +**Object Mount Fusion** is designed to enhance local, LAN-based, high-performance, attached storage solutions (like Amazon’s Elastic File System, EFS) with the throughput of object storage. + +It is a **cheaper** and **faster** solution when compared to using EFS alone. + +Object Mount Fusion takes an attached storage filesystem and an (initially) empty object storage bucket/directory and exposes a single interface for both. Object Mount Fusion will migrate files between the object storage location and the local filesystem location depending on their best fit for both performance and cost. + + +## How it Works + +Object Mount Fusion combines both the **local** file storage and the **cloud** object storage into a single, virtual mount (a FUSE mount). + +The files on object storage are represented as hidden links from the host filesystem to the object store. + +Unlike other solutions, Object Mount enables cloud-based object storage to be used as a first-class, high-throughput tier, rather than a traditional slow archival tier. + +Object Mount Fusion automatically and dynamically migrates files between the two locations according to your application behavior and usage. New files may be written into either location depending on predicted and observed access properties. + +Object Mount Fusion supports multiple users, each simultaneously accessing files on multiple nodes, by combining and sharing the attached storage and mount locations. + + +## Setting up Object Mount Fusion + +Object Mount Fusion should be used _within_ the same high-speed LAN as your object storage. + +For example: If you use AWS S3, Object Mount Fusion should _only_ be set up on EC2 nodes within the same +[AWS Region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) as the bucket to to be accessed. + +If you are using an S3-compatible on-premises object storage solution, Object Mount Fusion should be set up on a computer on the same high-speed local area network (LAN). + +_**Note:** If you have already set up and are using an attached storage system (such as EFS), you may skip ahead to [Mounting a Object Mount Fusion filesystem](#mounting-an-object-mount-fusion-filesystem)._ + +Otherwise, follow the steps below to configure Object Mount Fusion: + +### Setup an Empty Bucket on Object Storage + +To use the object storage location when running Object Mount Fusion mount, you will need a location on object storage that is **empty**. + +This will be used as the location that the Fusion filesystem will dynamically migrate data to when it’s more suitably stored on object storage. + +This can either be an **entirely empty bucket**, or an **empty directory** on an existing bucket. This location _must not_ be modified by anything but Object Mount Fusion’s filesystems. + +Review the configuration steps below for your object storage provider: + +{% tabs %} +{% tab label="AWS S3" %} + **AWS S3** + + To create a new AWS empty bucket, follow the instructions for 🌐 [Creating a general purpose bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). +{% /tab %} + +{% tab label="Storj" %} + **Storj** + + To create a new Storj empty bucket, follow the instructions to [Create buckets](docId:pxdnqsVDjCLZgeEXt2S6x). +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + For a new empty container on Azure, follow the instructions to 🌐 [Create a container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container). +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud** + + For a new empty Google Cloud bucket, follow the instructions for 🌐 [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets). +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **Other S3 Compatible** + + For a new empty bucket, follow your storage provider’s instructions for setting up a new bucket. +{% /tab %} +{% /tabs %} + +### Setup a New Compute Node + +Object Mount Fusion needs to be set up on a compute node _within_ the same LAN/region as the object storage bucket to be used. + +You will also need to attach a high-speed file storage solution to the instance. It is recommended to configure this while creating the new instance/image. + +To set up a compute node in the same region as your bucket, follow the relevant steps below for your provider: + +{% tabs %} +{% tab label="AWS S3" %} + **AWS S3** + + Follow the AWS tutorial: 🌐 [Get started with Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html). +{% /tab %} + +{% tab label="Storj" %} + **Storj** + + Typically, you will need a physical or virtualized computer connected to the same LAN as both your object storage location and your file storage device. +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + Follow the Azure Virtual Machine Guide: 🌐 [Quickstart: Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu). +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud** + + Follow the Google Cloud Compute Engine Guide: 🌐 [Instance templates](https://cloud.google.com/compute/docs/instance-templates). +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **Other S3 Compatible** + + Typically, you will need a physical or virtualized computer connected to the same LAN as both your object storage location and your file storage device. +{% /tab %} +{% /tabs %} + +### Create an Empty Directory on Attached Storage + +You will need an empty directory on your attached file storage that is writable by your user. + +If you don’t have this set up yet, follow the relevant steps for your platform to attach a writable storage device to your compute node: + +{% tabs %} +{% tab label="AWS S3" %} + **AWS S3** + + Follow the AWS guide: 🌐 [Getting started with Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/getting-started.html). +{% /tab %} + +{% tab label="Storj" %} + **Storj** + + You can use local storage, or attach an NFS/SMB mount to a file storage device in the same LAN. +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + Follow the Azure documentation to 🌐 [Use the portal to attach a data disk to a Linux VM](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal). + + Additional information on Azure’s fully managed file shares can be found in the 🌐 [What is Azure Files?](https://learn.microsoft.com/en-us/azure/storage/files/storage-files-introduction) article. +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud** + + Choose an appropriate Google Cloud attached disk option be reading through the information in the 🌐 [Choose a disk type](https://cloud.google.com/compute/docs/disks) article. Then, attach your instance to it using the relevant guide. + + Follow the instructions enable to Filestore in the 🌐 [Install and enable components](https://cloud.google.com/filestore/docs/install) article. +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **Other S3 Compatible** + + You can use local storage, or attach an NFS/SMB mount to a file storage device in the same LAN. +{% /tab %} +{% /tabs %} + + +## Mounting an Object Mount Fusion Filesystem + +To set up Object Mount Fusion, you need Object Mount [installed](docId:iethahkeeX5EiJoh) on the compute node, and you need to [Import S3 Credentials](docId:JDK2ED8HGFmyaxk#step-2-import-s3-credentials) so that the bucket/container is accessible. + +Your next steps depend on the filesystem you are using and the options used when mounting/attaching it. + +Refer to the relevant section below: +- Filesystems _without_ extended attribute support (e.g. EFS) +- Filesystems _with_ extended attribute support (e.g. NFSv4 with Linux Kernel 5.9+) + +### Filesystems _Without_ Extended Attribute Support (e.g. EFS) + +Object Mount Fusion is set up and accessed through Object Mount on FUSE. + +To do this, additional options must be passed to `cuno mount` when it is run. + +You will need to: + - Use the `--fusion` option to specify the path to the associated attached storage directory + - Use the `--root` option to specify the path to the object storage + +Use a command similar to the one below to enable both these options: + +```sh +cuno mount \ + --fusion "" \ + --root "" \ + "" +``` + +Example: + +```sh +`cuno mount --fusion /dev/sdf/fusion-store --root s3://bucket/fusion-store $HOME/my-fusion-filesystem` +``` + +By running the above command, the `` becomes the proper way to access the Fusion filesystem. + +All operations, workflows, pipelines, etc. should be pointed to ``. + +### Mounting the Filesystem + +Refer to your environment for specific commands: + +{% tabs %} +{% tab label="AWS S3" %} + **AWS S3** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "/cuno/s3//fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/s3//fusion-store" "~/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Storj" %} + **Storj** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "/cuno/s3//fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/s3//fusion-store" "~/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "/cuno/az///fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/az///fusion-store" "~/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "/cuno/gs//fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/gs//fusion-store" "~/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **Other S3 Compatible** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "/cuno/s3//fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno mount --fusion "/mnt/fast/fusion-store" --root "/cuno/s3//fusion-store" "~/my-fusion-filesystem" + ``` +{% /tab %} +{% /tabs %} + +_**Note:** See tables below for additional `cuno mount` parameters._ + +### Filesystems _With_ Extended Attribute Support (e.g. NFSv4 with Linux Kernel 5.9+) + +When using a filesystem that supports extended attributes you should “bind” it to a cloud location first, as this does additional error checking and makes for a simpler mount/unmount procedure. + +**Bind a Directory to a Cloud Location:** + +To bind a local Fusion directory to a cloud location, use `cuno fusion`. + +This command will configure a local pointer targeted at the cloud mount we will make later. It will also save any relevant options used at this stage as metadata; to be used by default by any mounts that use this fusion binding. + +```shell +#terminal +cuno fusion "" "" +``` + +Example: + +```shell +#terminal +cuno fusion /dev/sdf/fusion-store s3://bucket/fusion-store +``` + +### Mounting the Filesystem (After Binding) + +Object Mount Fusion is set up and accessed through Object Mount on FUSE. + +To do this, additional options must be used when configuring [Object Mount on FUSE](docId:ZdvWLcm9uFmM5HLk). + +If you have already bound a directory to a cloud location, you only need to use the `--root` option to specify the path to the associated attached-storage directory in order to use the mount as a Fusion mount: + +```shell +# terminal +cuno mount \ + --root "" \ + "" +``` + +Refer to your environment for specific commands: + +{% tabs %} +{% tab label="AWS S3" %} + **AWS S3** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + # terminal + cuno run mkdir "s3:///fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Create and mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno fusion "/mnt/fast/fusion-store" "s3:///fusion-store" + #terminal + cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Storj" %} + **Storj** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "s3:///fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Create and mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno fusion "/mnt/fast/fusion-store" "s3:///fusion-store" + #terminal + cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Microsoft Azure" %} + **Microsoft Azure** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "az:////fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Create and mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno fusion "/mnt/fast/fusion-store" "az:////fusion-store" + #terminal + cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Google Cloud" %} + **Google Cloud** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "gs:///fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Create and mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno fusion "/mnt/fast/fusion-store" "gs:///fusion-store" + #terminal + cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" + ``` +{% /tab %} + +{% tab label="Other S3 Compatible" %} + **Other S3 Compatible** + + If the bucket is not empty, create a new empty directory on your bucket: + + ```shell + #terminal + cuno run mkdir "s3:///fusion-store" + ``` + + Create a directory on the attached-storage device (assuming you’ve mounted it at `/mnt/fast`) to use for this purpose: + + ```shell + #terminal + mkdir "/mnt/fast/fusion-store" + ``` + + Create and mount the Object Mount Fusion volume: + + ```shell + #terminal + cuno fusion "/mnt/fast/fusion-store" "s3:///fusion-store" + #terminal + cuno mount --root "/mnt/fast/fusion-store" "$HOME/my-fusion-filesystem" + ``` +{% /tab %} +{% /tabs %} + +_**Note:** See tables below for additional `cuno mount` and `cuno fusion` parameters._ + + +## Using & Testing the Object Mount Fusion filesystem + +All operations, workflows, pipelines, etc. should be pointed to your mount location. + +### Usage Examples + +The following examples assume you have mounted a Fusion filesystem at `$HOME/my-fusion-filesystem`. + +- Download data from the web into the Fusion filesystem: + + ```shell + #terminal + cd $HOME/my-fusion-filesystem + #terminal + wget http://vision.stanford.edu/aditya86/ImageNetDogs/images.tar + ``` + +- List the files in the Fusion filesystem: + + ```shell + #terminal + ls $HOME/my-fusion-filesystem/ + ``` + +- Unpack a tar archive: + + ```shell + #terminal + cd $HOME/my-fusion-filesystem + #terminal + tar -xf images.tar + ``` + +### Testing + +Check that access, modification and migration are working as expected. + +We recommend measuring cost differences, and the overall time taken for equivalent jobs run on attached storage alone. + +**Note:** You can set the `CUNO_DEBUG` environment variable to `trace`, or `debug` at the time of mounting for more details logging information. + +Not seeing what you expect? Contact us at our 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new). + + +## APPENDIX TABLES: Parameters, Arguments & Options + +**TABLE: `cuno fusion` Arguments** + +| **Argument** | **Description** | +|--------------|-----------------| +| `` | Sets the location specified as the place to store file-storage data. This is where files that are migrated off the object store into local storage go. This can happen when access patterns suggest the file would benefit from higher IOPS, or when a file doesn’t meet the minimum size or age thresholds. | +| `` | Text | + +**TABLE: `cuno fusion` Options** + +| **Option** | **Description** | +|------------|-----------------| +| `--fusion-size-threshold ` (optional) | Define the minimum size that a file needs to be for it to be migrated from file storage to object storage. The argument value is of the form `[UNIT]`. If no unit is given, the value is assumed to be in *bytes*. Valid units are `K` (Kilobytes), `M` (Megabytes), `G` (Gigabytes), `T` (Terabytes). | +| `--fusion-age-threshold ` (optional) | Defines the minimum age requirement for files to be considered for migration from file storage to object storage. The age is measured as the time since the most recent of the POSIX creation time, modify time, access time, and change time - in other words, the time since each of these must be greater than the set threshold. The argument value is of the form `[UNIT]`. If no unit is given, the value is assumed to be in seconds. Valid units are `s` (seconds), `m` (minutes), `h` (hours), `d` (days). | + +**TABLE: `cuno fusion` Subcommands** + +| **Subcommand** | **Description** | +|----------------|-----------------| +| `cuno fusion bind ` | Bind a local directory to a cloud directory. | +| `cuno fusion unbind ` | Unbind a local directory from a cloud directory. | +| `cuno fusion rebind ` | Rebind a local directory to a cloud directory. | +| `cuno fusion info ` | Get information about the binding at the location. | + +**TABLE: Relevant `cuno mount` Options for Object Mount Fusion:** + +| **Argument/Option** | **Description** | **Example** | +|---------------------|-----------------|-------------| +| `` | The location from which the Fusion filesystem will be accessed. | `$HOME/my-fusion-filesystem` | +| `--fusion ""` | Enables Fusion for this mount, and sets it to use the location specified to store file-storage data. This is where files that are migrated off the object store into local storage go. This can happen when access patterns suggest the file would benefit from higher IOPS, or when a file doesn’t meet the minimum size or age thresholds. | `/dev/sdf/fusion-store` | +| `--root ""` | Sets the object storage location to be used as the place to store data that is better suited to be stored on object storage rather than on the file storage. | `/cuno/s3/bucket/fusion-store` | +| `--fusion-size-threshold ` (optional) | Defines the minimum size that a file needs to be for it to be migrated from file storage to object storage. The argument value is of the form `[UNIT]`. If no unit is given, the value is assumed to be in *bytes*. Valid units are `K` (Kilobytes), `M` (Megabytes), `G` (Gigabytes), `T` (Terabytes). | | +| `--fusion-age-threshold ` (optional) | Defines the minimum age requirement for files to be considered for migration from file storage to object storage. The age is measured as the time since the most recent of the POSIX creation time, modify time, access time, and change time - in other words, the time since each of these must be greater than the set threshold. The argument value is of the form `[UNIT]`. If no unit is given, the value is assumed to be in seconds. Valid units are `s` (seconds), `m` (minutes), `h` (hours), `d` (days). | | diff --git a/app/(docs)/object-mount/linux/user-guides/deployment-modes/page.md b/app/(docs)/object-mount/linux/user-guides/deployment-modes/page.md new file mode 100644 index 000000000..8e73e6d77 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/deployment-modes/page.md @@ -0,0 +1,30 @@ +--- +title: Deployment Mode Details +hideTitle: false +docId: jqySXsBfj5WGUjgs +weight: 3 +redirects: + - /object-mount/linux/user-guides/basic +metadata: + title: Configuring Deployment Modes + description: + Details on the use and configuration of different Deployment Modes in Object Mount for Linux. +hidden: false +--- + +The articles in this section provide additional details on the different Modes that Object Mount for Linux can be deployed in. + +As was described in the **Getting Started Guide** article [](docId:bRnfbdNE6d5DaZzW), Object Mount for Linux can be used in several different modes: + +1. [Direct Interception Mode](docId:UHsd5HnesueQyhnZ): In Direct Interception Mode, Object Mount inserts itself between your standard Linux applications and the host Operating System. Object Mount detects and intercepts file I/O calls, then translates and redirects them, as needed, to your object storage provider. The Object Mount Command Line Interface, `cuno`, is used to configure Direct Interception Mode. + +2. [Object Mount on FUSE](docId:ZdvWLcm9uFmM5HLk): If Object Mount’s CLI cannot be used, or you require broader support for non-standard Linux applications, Object Mount on FUSE uses Linux FUSE (Filesystem in Userspace) to mount object storage within the native Linux file system environment. This increases compatibility but can reduce performance. + +3. [Object Mount FlexMount](docId:cFUt9zgCRFFDk5Sq) combines the speed and performance of Direct Interception (#1) with the wide application support of Object Mount on FUSE (#2). + +4. [Object Mount Fusion](docId:GVT7eXEaMSZCWESj) can be used to enhance high-performance _attached_ storage solutions (like Amazon’s Elastic File System - EFS) with the throughput of object storage. + + +## Enabling Object Mount in Other Environments + +For information on setting up Docker containers with Object Mount pre-loaded, automatically setting up Object Mount on FUSE at boot, and other containerization and High-Performance Computing (HPC) use-cases, see Linux User Guide article: [](docId:airoogh4Waengi8u). diff --git a/app/(docs)/object-mount/linux/user-guides/enforced-posix-access/page.md b/app/(docs)/object-mount/linux/user-guides/enforced-posix-access/page.md new file mode 100644 index 000000000..0eb415984 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/enforced-posix-access/page.md @@ -0,0 +1,228 @@ +--- +title: Enable Enforced POSIX File Access +hideTitle: false +docId: Eegoo1teiJ8eerae +weight: 4 +redirects: + - /object-mount/linux/getting-started/enforced-posix-access +metadata: + title: Enable Enforced POSIX File Access + description: + This guide represents the general process for setting up Enforced POSIX File Access. The specific steps may vary depending on your object storage provider. +hidden: false +--- + +This guide represents the general process for setting up “Enforced POSIX File Access” mode in Object Mount for Linux. + +**Enforced POSIX File Access** mode will maintain POSIX metadata for your objects, and will enforce POSIX access controls on those objects. Use this when you want to manage what users have access to based on the UID/GID of their UNIX user and the corresponding POSIX metadata (owner, group, mode) on files. + +Users will encounter `access denied` errors if they try to read or write to a file/directory they haven’t been given permission to (by a suitably privileged user doing `chown`, `chgrp` or `chmod`). Note that this is _client-side_ rather than _server-side_ enforcement, and is not enforced using server-side IAM roles or ACL lists. + +The specific steps may vary depending on your object storage provider. Contact the Storj 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) if this is a use-case that you are interested in and these instructions don’t match your expectations or preferences. + +{% callout type="note" %} + **Hybrid Deployment for ACL Support** + + We can also provide a hybrid-approach, which does _both_ client-side access management and server-side access management through ACLs. + + If you need server-side ACL policies to reflect POSIX access controls, contact the Storj 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for assistance. +{% /callout %} + + +## Configuration Overview + +From your object storage provider’s dashboard, generate access credentials with the most complete set of access permissions that any user or admin could need, including permissions to _edit bucket tags_. + + - On a public cloud supporting IAM, you should set up an `admin` IAM user with such credentials. + - The admin credentials are stored privately and are used to set up Object Mount on FUSE in an accessible location. + - Users are only told the path to the mount — they are not given access to `cuno` nor to the admin credentials. + +{% callout type="warning" %} + **Security Considerations** + + The approach given here is **not suitable** if users have the ability to create their own VMs/instances where they can set any POSIX uid/gid. + + In that case, consider using server-side IAM roles or ACL lists to enforce access controls. Contact our 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new) for advice on how this integrates with Object Mount. +{% /callout %} + + +## Key Considerations + +1. If multiple machines or locations are being used, it is important that user and group IDs are controlled and consistent across all machines. + + This is because the POSIX metadata is stored in the object storage, and Object Mount will use the same user and group IDs to present the files on all machines. If you require assistance with managing user and group IDs across multiple machines, contact our 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new). + +2. Independent IAM roles or credentials for each user are not required. The admin credentials are used to set up the mount, and the users are only given access to the mount (not the credentials). + + If the user has access to object storage credentials with server-side privileges beyond this, then the user can potentially access or modify objects outside of these POSIX access controls. + +3. This guide assumes that the _bucket to use is empty_ initially. If you are switching from an ACL-based approach to access management, then you will need to map existing rules to POSIX uids/gids and “apply” those ACLs through POSIX commands like `chown` and `chmod`. + +**Note:** Object Mount does not currently support POSIX ACLs or extended attributes in cloud object storage. Contact 🌐 [Storj Support](https://supportdcs.storj.io/hc/en-us) if you need these features. + + +## How to Enable POSIX Enforced File Access + +Assuming you have already [installed](docId:iethahkeeX5EiJoh) Object Mount for Linux, the following steps will guide you through setting up enforced POSIX access for the users in a generic POSIX-based system. + +### Step 1. Import your Admin Credentials + +Follow the steps in [](docId:JDK2ED8HGFmyaxk#step-2-import-s3-credentials) to obtain, save and import your admin credentials. + +You will need to have run `cuno creds import `. + +After importing, you should ensure that any credentials files (usually in `${XDG_CONFIG_HOME}"/cuno/creds` or `~/.config/cuno/creds`) have been created with, or assigned the, appropriately strict permissions so that non-admin users cannot read them. + +### Step 2. Tag the Bucket + +We check a [bucket tags](docId:phohPoowequie5ji#posix-file-access) when Object Mount tries to access a bucket to assess the POSIX enforcement settings. + +If your bucket or object storage provider does _not support tags_ on buckets, you can skip this step and proceed to [Step 3. Mount the Bucket](#step-3-mount-the-bucket). + +**Note:** You will need to set the `--posix` flag every time you run `cuno mount`. + +Use `cuno creds setposix true` to set the bucket tag which will automatically enable POSIX File Access whenever this bucket is accessed by any Object Mount installation. + +This will affect _everyone_ using the bucket and force all Object Mount users of that bucket into “Enforced POSIX File Access” mode. + +Example: + +```shell + # terminal + cuno creds setposix s3://mybucket true +``` + +By setting this to `true`, Object Mount stores hidden subdirectories inside your object storage directories describing the POSIX metadata (owner/group permissions, change/modify times, etc.) for each file in the directory. This allows Object Mount to present the files in a way that is compatible with POSIX file access semantics. + +{% callout type="note" %} + **File-Based Metadata** + + If you are using S3 buckets, it is possible to set `cuno creds setposix metadata`. This will store POSIX metadata as object metadata _on each individual file_ in the bucket, + + We do not normally recommend this, as it will slow down Object Mount. For more information, see [](docId:phohPoowequie5ji#posix-file-access). +{% /callout %} + + +{% callout type="warning" %} + **Inability to Enforce File Access** + + When a bucket tag enabling POSIX File Access is set, Object Mount Direct Interception, along with any Object Mount on FUSEs/FlexMounts, are compelled to operate in POSIX mode while accessing the bucket. + + However, the mounts will _not be able to enforce POSIX access_ without the additional `--posix` flag at mount time. + + Object Mount Direct Interception Mode necessitates read access to object storage credentials (or some other user-accessible way to authenticate with the object storage (e.g. an IAM role on EC2)]. + + This implies that Direct Interception Mode _is not appropriate for enforcing POSIX_, as users can easily circumvent this by using the accessible credentials or IAM role with a tool other than Object Mount (such as AWS CLI, or directly through an S3 API library). + + Since Object Mount FlexMounts are dependent on Object Mount Direct Interception Mode, they are also unsuitable for use-cases that require POSIX enforcement. +{% /callout %} + +### Step 3. Mount the Bucket + +Mount a bucket in a location that is accessible to the users. + +In this guide, we will use a shared location of `/mnt/cloud/bucket` as the mount point. + +When setting up the mount as an admin user, allow for sudo access using the `-o allow_root` flag. + +To allow other users to access the mount as themselves, use `-o allow_other`. + +```console +$ cuno mount --posix -o allow_root -o allow_other --root s3://mybucket /mnt/cloud/bucket +``` + +Users will now be able to see the files in the bucket at `/mnt/cloud/bucket`, and POSIX permissions persisted by Object Mount for files in that bucket will be respected. + +_You should treat this as any other POSIX file system and set permissions as you need to control access._ + + +## Common Patterns + +Below are some common behaviors and recommendations that you might want to implement when using enforced POSIX access. + +For more examples, see the **Usage Examples** section in the Advanced Guide article [](docId:phohPoowequie5ji#usage-examples). + +### Setting Default Permissions for New Files + +On POSIX systems, you would normally set default permissions using `umask`. + +To determine the umask value you want to set, subtract the value of the permissions you want from `666` (for a file) or `777` (for a directory). + +For example, if `user1` wants the default permissions on new files to be `rw-r--r-- 644`, they would subtract `644` from `666`, giving `022`. + +```console +$ umask 022 +$ touch /mnt/cloud/bucket/user1/newfile +$ ls -l /mnt/cloud/bucket/user1/newfile +-rw-r--r-- 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/user1/newfile +``` + +If you want to set the umask for **all users**, you can set this in the system-wide profile, or in the user’s profile. + +For example, to set it for all users, you could add the umask command to `/etc/profile`. + +See the **Ownership and Permissions** section in the Advanced Guides article [](docId:phohPoowequie5ji#ownership-and-permissions) for additional details on file and directory ownership and permissions. + +### Converting a Bucket to “POSIX Enforced File Access” Mode + +If you have a pre-existing storage location for which you now need to implement POSIX-based access controls, you can simply set the bucket tag as [shown above](#step-2-tag-the-bucket). This will ensure that any _new files_ are stored with POSIX metadata. + +**Note:** This will _not_ affect any _existing files_ until they (or their directories) are accessed. + +To control access to any particular files or directories, proceed with the use of `chown`, `chgrp` and `chmod` as you would normally on a POSIX file system. + +For example, to limit access to `/mnt/cloud/bucket/directory` to only allow `user1`, you could do the following: + +```console +$ chown -R user1 /mnt/cloud/bucket/directory +$ chgrp -R group1 /mnt/cloud/bucket/directory +$ chmod -R 700 /mnt/cloud/bucket/directory +``` + +As a result of the above commands, only `user1` will be able to access the directory and its contents across all systems using Object Mount. + +You can check the POSIX metadata for a file or directory using `ls -l` or `stat`. + +For example: + +```console +$ ls -l /mnt/cloud/bucket/directory +drwx------ 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/directory +$ stat /mnt/cloud/bucket/directory +File: /mnt/cloud/bucket/directory +Size: 0 Blocks: 0 IO Block: 4096 directory +Device: 1ch/28d Inode: 123456 Links: 1 +Access: (0700/drwx------) Uid: ( 1000/ user1) Gid: ( 1000/ group1) +Access: 2022-03-01 12:00:00.000000000 +0000 +Modify: 2022-03-01 12:00:00.000000000 +0000 +Change: 2022-03-01 12:00:00.000000000 +0000 + Birth: - +``` + +### Setting Inherited Permissions on a Directory + +{% callout type="warning" %} + **POSIX ACL Support** + + Work to support POSIX ACLs is under development. + + If this is a feature you need, please contact our 🌐 [Support Team](https://supportdcs.storj.io/hc/en-us/requests/new). +{% /callout %} + +Use the [set file access control lists (setfacl)](https://linux.die.net/man/1/setfacl) command to define additional default permissions to be applied to files in a directory when they are created. + +For example, an admin could create a shared space, `/mnt/cloud/bucket/shared`, for users to collaborate, and set the default permissions to allow all users and groups to read and write files in the directory: + +```console +$ setfacl -m u::rwX,g::rwX,o::- /mnt/cloud/bucket/shared +``` + +This will result in the following behavior: + +```console +$ touch /mnt/cloud/bucket/shared/newfile +$ mkdir /mnt/cloud/bucket/shared/newdir +$ ls -l /mnt/cloud/bucket/shared/ +drwxrwxr-x 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/shared/newdir +-rw-rw-r-- 1 user1 group1 0 Mar 1 12:00 /mnt/cloud/bucket/shared/newfile +``` diff --git a/app/(docs)/object-mount/linux/user-guides/install/page.md b/app/(docs)/object-mount/linux/user-guides/install/page.md deleted file mode 100644 index 6a1f54d89..000000000 --- a/app/(docs)/object-mount/linux/user-guides/install/page.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: Download and Installation -docId: Jae5vooh9iete7wa - -metadata: - title: Download and Installation - description: Download and Installation Guide - -weight: 2 ---- -There are multiple ways to install Object Mount: - -- an installation script, which allows users to install Object Mount without root privileges [user guide](../user-guides/install/scripted-installer). -- standard packages for various Linux distributions [user-guide-package-manager-install](../user-guides/install#package-manager-installation). - -## Install locations - -When Object Mount is installed via the Scripted Installer, it gets installed to different locations depending on whether it was installed by root, or by a non-root user. -Hereafter, root installations are called 'system-wide', and non-root installations are called 'user-local'. - -{% callout type="note" %} -A package manager install of Object Mount is always system-wide, regardless of the privileges of the user installing it. -{% /callout %} - - -| Install type | Location (:code:`CUNO_ROOT`) | Symlinks created in | -| ------------- | ------------- | ------------- | -| User-local | `~/.local/opt/cuno/`| `~/.local/bin/` -|||`~/.local/lib/x86_64-linux-gnu` -|||`~/.local/share/man/`| -| System-wide| `/opt/cuno/` | `/usr/bin/` -|||`/usr/lib/` -|||`/usr/share/man/`| - - -## Scripted installer - -### Distros using glibc (most) - -Most Linux distributions (including Debian/Ubuntu, RedHat and its derivatives) use the GNU Project's C standard library (glibc). This installer/version is compatible with all such Linux distributions. - -See [glibc](../installation/glibc) - -### Distros using musl (Alpine) - -Some Linux distributions (Alpine) use the musl C standard library. This installer/version is compatible with all such Linux distributions. - -See [musl](../installation/musl) - - -## Package manager installation - - -### Debian and derivatives (e.g. Ubuntu) - -See [debian](../installation/debian) - -### Red Hat and derivatives - -See [Red Hat](../installation/redhat) - -### Alpine Linux - -{% callout type="warning" %} -Object Mount APK packages for Alpine will be available soon. -{% /callout %} - -```console -sudo apk add --allow-untrusted ./cuno_{FULL-VERSION}_amd64_musl/cuno_{FULL-VERSION}_amd64_musl.apk -``` - -## Other operating systems - -Object Mount is built for Linux distributions only, and does not {emphasis}`natively` support other operating systems. -However, there are still ways to use Object Mount. - -### macOS using Docker - -{% callout type="note" %} -Alternatives to Docker on Mac are available, some recommended alternatives are: - -- [Rancher Desktop](https://rancherdesktop.io/) (free) - If you're on Apple Silicon, enable Rosetta (Settings > Virtual Machine > VZ: Enable Rosetta support) and VirtioFS (Settings > Virtual Machine > Volumes: virtiofs) -- [OrbStack for Mac](https://orbstack.dev/) (may be faster) - OrbStack works out of the box. - -Unfortunately, [colima](https://github.com/abiosoft/colima) is not currently supported. -{% /callout %} - - -See [](docId:yoopieyewevei1Eo) for more details. - - -### Additional instructions for cuno-mac users - -If you have installed Object Mount directly onto a Mac, you will only have access to Object Mount functionality within Docker containers. We provide the `cuno-mac` utility which you run from Terminal to launch conveniently set-up Linux containers. - -The first time `cuno-mac` is run, a Docker image will be created with Object Mount ready to use, and a user will be set up within the container similar to your local user on the host Mac. - -The Dockerfile found at `~/.local/opt/cuno/share/macos/Dockerfile` can be edited to include any software packages you wish to have available inside the container. However, to update this will require deleting the old image first (by doing `docker rmi cuno-mac`), you can then run `cuno-mac` to rebuild it. - -You must choose between using `cuno-mac` and `cuno` depending on the environment you are currently in. - -To start a new session in which you can use Object Mount, you run `cuno-mac` in Terminal. - -If any arguments are given to `cuno-mac`, it will start a temporary container and pass the arguments on to `cuno`. - -If no arguments are given, `cuno-mac` will start a new interactive container and prefix `(cuno)` to your command-line prompt. This indicated that you are now inside a Docker container, so from here you cannot use `cuno-mac` and should instead use `cuno`. - -To return to macOS, run `exit` until the `(cuno)` prefix is removed. You may only need to do this once, or you may need to do it multiple times if you have started subshells. - -### Windows using WSL2 - -See [Windows](../installation/windows) - -{% callout type="note" %} -Refer to the [Microsoft documentation on installing WSL](https://docs.microsoft.com/en-gb/windows/wsl/install) for more information. -{% /callout %} - -### Exposing mounted object storage to the Host OS - -If you’re running Object Mount in a virtual machine, you can expose any object storage mounted as volumes in it using guest-to-host volume mapping. diff --git a/app/(docs)/object-mount/linux/user-guides/limitations/page.md b/app/(docs)/object-mount/linux/user-guides/limitations/page.md deleted file mode 100644 index 50cfd2f3b..000000000 --- a/app/(docs)/object-mount/linux/user-guides/limitations/page.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: Limitations -docId: Chiec3aS2eizieng - -metadata: - title: Limitations - description: Limitations - -weight: 13 ---- - -{% callout type="note" %} -Object Mount imposes no restrictions on local file accesses. The limitations listed in this section only apply to data stored on object-storage. -{% /callout %} - -{% callout type="note" %} -This document contains technical limitations applying to all Object Mount users. There are additional limitations dictated by your licence tier - set up a [discovery call](https://www.storj.io/landing/get-in-touch) -for more information on licensing and pricing. -{% /callout %} - - -## Direct interception - -Direct interception (using Object Mount CLI or `LD_PRELOAD`) does not currently support SUID binaries, or certain packaged apps like [Snap](https://ubuntu.com/core/services/guide/snaps-intro), [AppImage](https://appimage.org/), or [Flatpak](https://docs.flatpak.org/en/latest/introduction.html) applications. Future updates are planned to address this. -If you need to use such apps, prefer to use [user-guide-object-mount-on-fuse](../user-guides/basic#object-mount-on-fuse) or [user-guide-object-mount-flexmount](../user-guides/basic#object-mount-flex-mount). - -## Maximum object size - -Depending on the solution provider, Object Mount has a limitation on the maximum file size it can store on a remote location. The following table indicates the maximum file sizes per provider. - -| Cloud provider | Maximum file size | -| -------------------- | ----------------- | -| AWS S3 | 5 TB | -| Google Cloud Storage | 5 TB | -| Azure Storage | 4.77 TB | - -## Ownership, permissions and file metadata - -* In the results of the command `ls` the owner of the remote objects is always the current user. Furthermore, remote file permissions are always set to `777`. - -* The creation date of a remote directory is not always available to the system calls. - -In Core File Access mode, the owner of the remote objects is by default always reported as the current user, and remote file permissions are always `777`. Also, the creation time of directories is always displayed as the Unix Epoch (00:00:00 UTC on 1 January 1970). These can be overridden using `CUNO_OPTIONS` ([user-guide-ownership-and-permissions](../user-guides/configuration#ownership-and-permissions)). - -### Directories in Azure - -Creating a directory in Azure Storage (using `mkdir`) will result in a remote blob called \ to be displayed inside the created directory when the user is using the GUI/file explorer that Azure portal provides. However, `ls` and all CLI commands will behave as expected. - -### Auto-completion - -Auto-completion and wildcard characters are fully supported on a Object Mount active shell. This can be created either using the `cuno` command or using `LD_PRELOAD` (e.g. `LD_PRELOAD=/usr/lib/cuno.so bash`). In the latter, paths containing colons such as `s3://bucket` on cloud paths will only succeed if `:` is removed from the separator list variable `COMP_WORDBREAKS`. For example: - -```bash -[[ "$LD_PRELOAD" =~ cuno ]] && export COMP_WORDBREAKS=${COMP_WORDBREAKS/:/} -``` - -### Memory-mapping - -Currently, only read-only private file memory mapping is supported. - -### Applications - -You may want to check the secton [user-guide-tips-for-apps](../user-guides/tips). diff --git a/app/(docs)/object-mount/linux/user-guides/ml/page.md b/app/(docs)/object-mount/linux/user-guides/ml/page.md deleted file mode 100644 index 4099205da..000000000 --- a/app/(docs)/object-mount/linux/user-guides/ml/page.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Machine Learning Accelerator -docId: aiwoo2Gohshah8Zo - -metadata: - title: Machine Learning Accelerator - description: Machine Learning Accelerator Guide - -weight: 11 ---- - -The Object Mount Machine Learning Accelerator (Object Mount MLA) is a software library that accelerates your existing machine learning models by advanced leveraging of our data retrieval and storage technology. - -It is available for our Professional and Enterprise customers. - -## Requirements - -Python 3.7+ is required to use the Object Mount Machine Learning Accelerator (Object Mount MLA). If you require lower versions of Python 3, please contact us at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). - -### Supported functions - -The Object Mount MLA has been tested with the following data loading functions: - -- `open` -- `numpy.loadfile` -- `numpy.load` -- `numpy.loadtxt` -- `PIL.Image.open` -- `torchaudio.load` - -It may also work with additional data loading functions, and support is available at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) if you need assistance to enable additional functions. - -## Usage - -First, you must install the MLA dependencies in your environment. You can use `pip` to do so: - -```console -pip install -r "/requirements.txt" -``` - -You must also set the following environment variables to enable the Object Mount Machine Learning capability: - -- `CUNO_SPEEDUP_PYTHON=1` -- `PYTHONPATH="${PYTHONPATH+:$PYTHONPATH}"` - -In this example, Python will load the Object Mount MLA's `sitecustomize.py` first. The `CUNO_SPEEDUP_PYTHON` environment variable will signal the Object Mount MLA to enable our ML optimizations. - -Next, you can use any of the standard ways to enable Object Mount, including Direct Interception and Object Mount on FUSE. - -### Using Object Mount Direct Interception - -To enable the Object Mount Direct Interception you can launch a new shell with Object Mount enabled as usual with `cuno`. - -If this is not an option, you can use `cuno run` to enable Object Mount for a single command. You can also set the `LD_PRELOAD` environment variable directly, like `LD_PRELOAD=/lib/cuno.so`. - -#### Example usage - -Activating a shell with Object Mount as an example: - -```console -$ export CUNO_SPEEDUP_PYTHON=1 -$ export PYTHONPATH="${PYTHONPATH+:$PYTHONPATH}" -$ cuno -(cuno) $ python3 demo/verify.py -``` - -### Using Object Mount on FUSE - -To use Object Mount on FUSE with your object storage data, you will need to configure your Python scripts to look for data inside the mount. - -#### Example usage - -Mounting an S3 bucket named `dataset_bucket` at the location `$HOME/cloudmount` as an example: - -```console -export CUNO_SPEEDUP_PYTHON=1 -export PYTHONPATH="/dataset_preload2${PYTHONPATH+:$PYTHONPATH}" -cuno mount --root s3://dataset_bucket "$HOME/cloudmount" -python3 dataset_preload2/demo/verify.py --location "$HOME/cloudmount/training_set_1" -``` diff --git a/app/(docs)/object-mount/linux/user-guides/page.md b/app/(docs)/object-mount/linux/user-guides/page.md index 07c8ade35..6f664f30e 100644 --- a/app/(docs)/object-mount/linux/user-guides/page.md +++ b/app/(docs)/object-mount/linux/user-guides/page.md @@ -1,51 +1,24 @@ --- -title: User Guides +title: User Guides for Linux +hideTitle: false docId: ohs0ailohSh0Vie3 - +weight: 3 metadata: title: User Guides description: User Guides Overview -weight: 4 +hidden: false --- -Object Mount is a scalable, high-performance POSIX compatibility layer that lets you interact with files stored on object storage such as Amazon S3, Azure Blob Storage, Google Cloud Storage, or any S3-compatible object store hosted in the cloud or locally. - -## The package - -Object Mount is Linux software: there's a Object Mount Command Line Interface (Object Mount CLI), `cuno`, providing the highest performance and most straightforward way to interact with object storage. This works through a user-mode library, `cuno.so`, which intercepts (both dynamic and static) applications using [LD_PRELOAD] functionality and fast dynamic binary instrumentation. - -Object Mount can also be used with our modified [FUSE] mount solution, [Object Mount on FUSE](./user-guides/basic#object-mount-on-fuse), providing wider support for applications where the [Object Mount CLI cannot be used](./user-guides/limitations#direct-interception). - -To match the best performance with the widest support, consider the hybrid solution: [Object Mount FlexMount](./user-guides/basic#object-mount-flex-mount). - -Access credentials can also be optionally managed by Object Mount. - -## Wide support for object storage providers - -Object Mount has native support for: - -- [Amazon Web Services S3](https://aws.amazon.com/s3/) -- [Microsoft Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) -- [Google Cloud Storage](https://cloud.google.com/storage/) (however for best performance we currently recommend using S3 with their [S3 accesspoint](https://cloud.google.com/storage/docs/interoperability)) - -In theory, Object Mount supports any S3-compatible object storage provider. In practice, the "S3 API" implementation can have differences in behaviour between providers, so some additional configuration is sometimes necessary. Object Mount has been tested on: - -- [Oracle Cloud Infrastructure Object Storage](https://www.oracle.com/cloud/storage/object-storage.html) -- [Storj](https://storj.io/) -- [Wasabi](https://wasabi.com/) -- [MinIO](https://min.io/) -- [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/) -- [Dell ECS Object Storage](https://www.delltechnologies.com/en-gb/storage/ecs/index.htm) - -The following providers have not yet been validated; however, users have reported success with: +Once Object Mount for Linux is installed and running on your computer, browse through these Linux **User Guides** articles for additional usage and configuration settings. -- [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage) -- [Backblaze B2 Cloud Storage](https://www.backblaze.com/cloud-storage) -- [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/) -- [Cloudflare R2](https://www.cloudflare.com/en-gb/developer-platform/r2/) -- [Scality](https://www.scality.com/) -- [DataDirect Networks (DDN) Storage](https://www.ddn.com) +**Articles Include:** -[fuse]: https://www.kernel.org/doc/html/next/filesystems/fuse.html -[ld_preload]: https://man7.org/linux/man-pages/man8/ld.so.8.html +- Validate Functionality & Access Object Storage +- Activate a License Key +- Understand Deployment Modes +- Enable Enforced POSIX File Access +- Cloud Path Details +- Application Tips +- Media & Entertainment Mode +- High-Performance Computing Mode diff --git a/app/(docs)/object-mount/linux/user-guides/profile-mode/page.md b/app/(docs)/object-mount/linux/user-guides/profile-mode/page.md new file mode 100644 index 000000000..547d0fae9 --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/profile-mode/page.md @@ -0,0 +1,111 @@ +--- +title: Media & HPC Modes +hideTitle: true +docId: linux-profile-mode +weight: 9 +metadata: + title: Linux Profile Mode (CUNO_PROFILE) + description: + Explains the CUNO_PROFILE environment variable for Linux builds and how to choose the appropriate mode (M&E vs HPC). +--- + +# Media & Entertainment and HPC Modes + +Object Mount for Linux includes an environment variable called `CUNO_PROFILE` that modifies internal application behavior for different types of workloads. + +This setting only applies to Object Mount on Linux and helps Object Mount optimize itself for Media & Entertainment (M&E) and High-Performance Computing (HPC) workflows: + +| **CUNO Profile Setting** | **Workflow** | +|---------------------------|--------------| +| `CUNO_PROFILE=M&E` | Media & Entertainment workflows +| `CUNO_PROFILE=hpc` | High-Performance Computing environments + +Understanding the behavior of each mode is important. + +Setting the wrong mode can negatively impact expected performance, especially in large-scale automated environments. + + +## Purpose of CUNO_PROFILE + +The `CUNO_PROFILE` variable adjusts how Object Mount behaves internally, including: + + - How aggressively metadata is cached + - Which filesystem features are exposed + - Logging, debugging, and memory handling profiles + - Compatibility expectations with the Linux **UserLAnd** and **toolchain** tools + +{% callout type="info" %} + **Default Profile Mode** + + If unset, the default is for Media & Entertainment workflows: `CUNO_PROFILE=M&E`. +{% /callout %} + +Media & Entertainment Mode is well-suited to desktop users, editors, and creative professionals using Object Mount interactively, but less appropriate for headless systems or scripted automations. + + +## Choosing the Right Profile + +Before installing or configuring Object Mount for Linux, consider the following questions: + +1. **Is the user working interactively? Or via headless automation?** + + - `M&E` Mode is tuned for interactive, graphical workflows + - `hpc` Mode is preferred for background tasks, CLI tools, and scripts + +2. **Will the mount be used by creative software applications?** + + - If using Apps such as Avid Media Composer, DaVinci Resolve, Apple Final Cut Pro, Adobe Premiere Pro, etc.: + - Set: `CUNO_PROFILE=M&E` + - If running scripted automation tools, media transformation pipelines, background archive/backup jobs, etc.: + - Set: `CUNO_PROFILE=hpc` + +3. **Is the system a personal workstation? Or shared server node?** + + - Personal Workstation — Set: `CUNO_PROFILE=M&E` + - Server node / render farm — Set: `CUNO_PROFILE=hpc` + +4. **Is the mount expected to be long-lived and static? Or frequently toggled?** + + - Session-based, user-initiated mounts — Set: `CUNO_PROFILE=M&E` + - Long-lived, persistent mounts — Set: `CUNO_PROFILE=hpc` + +5. **Will users be working via GUI, or purely via CLI/API?** + + - GUI or desktop usage — Set: `CUNO_PROFILE=M&E` + - CLI tools, automation, batch jobs — Set: `CUNO_PROFILE=hpc` + +6. **Which is more important: latency or throughput?** + + - Low-latency I/O, fast response time “desktop feel” — Set: `CUNO_PROFILE=M&E` + - Bulk, high-throughput, background tasks (e.g.: rendering, transcoding) — Set: `CUNO_PROFILE=hpc` + + +## Setting the Profile + +To explicitly set the profile, define the environment variable _before_ launching Object Mount. + +For example: + +``` +bash +export CUNO_PROFILE=hpc +``` + +For permanent use, add the environment variable setting to: + - Your shell profile (`~/.bashrc`, `~/.zshrc`) + - A systemd service or login script + - Docker or container entrypoints (if applicable) + + +## Table: Use Case Quick Reference + +| **Use Case** | **Recommended Profile** | +|--------------------------------------|-------------------------| +| **Media & Entertainment Workflows:** | +| • Desktop video editing | `M&E` | +| • Interactive audio work | `M&E` | +| • Local workstation media previews | `M&E` | +| **High-Performance Computing Environments:** | +| • Automated ingest pipelines | `hpc` | +| • Render farm nodes | `hpc` | +| • Remote headless access | `hpc` | diff --git a/app/(docs)/object-mount/linux/user-guides/tips/page.md b/app/(docs)/object-mount/linux/user-guides/tips/page.md deleted file mode 100644 index 6f64c2ed6..000000000 --- a/app/(docs)/object-mount/linux/user-guides/tips/page.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: Tips for Apps -docId: lawaDooteicha9li - -metadata: - title: Tips for Apps - description: Usage tips for some applications - -weight: 12 ---- - -There are some general ideas presented in [user-guide-core-concepts](../user-guides/core-concepts) that should help inform usage of object storage with your software. For example, object storage is not very good for high IOPS work loads, so prefer to use [Object Mount Fusion](Object Mount-fusion-guide) for that use-case. - -The following are some applications for which we have additional guidance when using then in combination with Object Mount. If you have more to add to the list, let us and the community know at our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new). - -## Applications - -### Python - -Python's `os.path.realpath(path)` is not supported for URI-based access like `xx://`. Use path-format instead (e.g. `/cuno/s3/bucket/path`). - -### Rsync - -We strongly recommend running `rsync` with the options `--inplace -W`. This makes rsync work more efficiently with object storage. - -To use rsync options that preserve permissions (`-p`) and modification times (`-t`), such as when you want to update files only when the source has been updated, you must enable [POSIX File Access](../getting-started/configuration-modes#posix-file-access). - -### Fpsync - -When using `fpysync`, use the `-o` option to pass the options recommended for rsync down to the worker processes, e.g. `-o "--inplace -W"`. Further, because Object Mount is already parallelised, we recommend limiting the number of Fpsync worker processes using the `-n` option. - -To use rsync options that preserve permissions (`-p`) and modification times (`-t`), such as when you want to update files only when the source has been updated, you must enable [POSIX File Access](../getting-started/configuration-modes#posix-file-access). - -### FFmpeg - -While ``ffmpeg`` works, it may be slow for the following use-cases: - -- Writing to object storage, if the task requires many randomly-placed writes during an upload. For example, for a large mp4 file being written with the flags ``-movflags +faststart``. - -- Reading from object storage, a complex filterchain where multiple subtitle streams are being read from the same input file. (This will be improved soon! Be the first to find out by contacting our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new)) - -- Reading from object storage, files that contain title screen and extra credit scenes can be slow to start. - -### sudo with Direct Interception - -Using [Direct Interception](../user-guides/basic#direct-interception-with-object-mount-cli) (including when using the Object Mount CLI) requires the `LD_PRELOAD` environment variable to be set and maintained for all executed child processes. Since `sudo` usage by default does not preserve the environment variables set, the following requirements apply: - -- `sudo` needs to be run with `--preserve-env` to preserve `CUNO_OPTIONS` -- `sudo` needs to launch a child shell that will then run the command, so that the `LD_PRELOAD` environment variable can be set before running the command to be intercepted. -- `LD_PRELOAD` needs to be manually set inside the child shell launched - -To use `sudo` with Direct Interception, please do the following: - -1. Start a wrapped-shell using Object Mount CLI: `cuno` - -2. Run sudo in the following way: - - ```console - sudo --preserve-env /bin/bash -c "export LD_PRELOAD=$LD_PRELOAD && " - ``` - -### Locate - -The `locate` application requires some heightened privileges to create the database, and also has some incompatibilities with Object Mount [Direct Interception](../user-guides/basic#direct-interception-with-object-mount-cli) . - -#### Issues with Direct Interception - -When using [Direct Interception](../user-guides/basic#direct-interception-with-object-mount-cli), note that `locate` and `updatedb` do not work with URI-style paths. Please use directory-style paths of the form `/cuno/xx/`. - -Furthermore, Direct Interception (even when using the Object Mount CLI) requires the `LD_PRELOAD` environment variable to be set and maintained for all executed child processes. Since `updatedb` typically needs to be run with `sudo`, the limitations specifed in [user-guide-limitations-sudo](#sudo-with-direct-interception) apply here. - -#### Instructions for using Locate - -To help work around these limitations, we provide steps below on how to use `locate`. - -1. Create a new database, which we call `cunoloc.db`: -{% tabs %} -{% tab label="Using Object Mount on FUSE" %} -Assuming you have a Object Mount on FUSE set up at ``~/my-object-storage``, you can use ``updatedb`` directly to crawl all paired buckets from all your object storage providers: -```console -sudo updatedb -U ~/my-object-storage -o cunoloc.db -``` -{% callout type="warning" %} -The mount location should not change (``~/my-object-storage`` in the example above), because it will be written into the database created. -{% /callout %} -{% /tab %} -{% tab label="Using Direct Interception" %} - Run ``updatedb`` with the workarounds for sudo: - ```shell - # terminal - cuno - ``` - - Run ``updatedb`` inside the shell, using a cloud path in the directory format: -```console -(cuno) $ sudo --preserve-env /bin/bash -c "export LD_PRELOAD=$LD_PRELOAD && updatedb -U /cuno// -o cunoloc.db" -``` -{% /tab %} -{% /tabs %} - -2. Change the database ownership back to your current user: - - ```console - sudo chown $(whoami):$(whoami) cunoloc.db - ``` - -3. Add the database to your `LOCATE_PATH` environment variable and use locate normally; or search within the database, as follows: - - ```console - locate -d cunoloc.db myfile - ``` - -#### Setting up the Locate cron job - -By default, the global locate database is periodically updated by a cron job. To setup the cron job for Object Mount, you need to edit the file `/etc/cron.daily/mlocate`. The last line updates the global database: - -```console -flock --nonblock /run/mlocate.daily.lock $NOCACHE $IONICE nice /usr/bin/updatedb.mlocate -``` - -This should be replaced with something like: - -```console -LD_PRELOAD='/usr/lib/cuno.so' CUNO_OPTIONS='' CUNO_CREDENTIALS='' flock --nonblock /run/mlocate.daily.lock $NOCACHE $IONICE nice /usr/bin/updatedb.mlocate -``` - -If you don't want to index all of your object storage, you can specify where `updatedb` does **not** look for files by adding paths to `PRUNEPATHS` in the file `/etc/updatedb.conf`. diff --git a/app/(docs)/object-mount/linux/user-guides/uninstall/page.md b/app/(docs)/object-mount/linux/user-guides/uninstall/page.md deleted file mode 100644 index 3b799ea9c..000000000 --- a/app/(docs)/object-mount/linux/user-guides/uninstall/page.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: Uninstallation -docId: Oochi9lee7ookaih - -metadata: - title: Uninstallation - description: Uninstallation Guide - -weight: 13 ---- - -Remove the directory pointed to by `CUNO_ROOT`. -Refer to [user-guide-install-locations](../user-guides/install#install-locations) for more information. - -## User-local uninstallation - -Remove the following additional files: - -``` -~/.local/lib/x86_64-linux-gnu/cuno.so -~/.local/bin/cuno -~/.local/share/man/man1/cuno.1 -~/.local/share/man/man1/cuno-creds.1 -~/.local/share/man/man1/cuno-mount.1 -~/.local/share/man/man8/cuno.so.8 -``` - -## System-wide uninstallation - -Remove the following additional files: - -``` -/usr/lib/cuno.so -/usr/bin/cuno -/usr/share/man/man1/cuno.1 -/usr/share/man/man1/cuno-creds.1 -/usr/share/man/man1/cuno-mount.1 -/usr/share/man/man8/cuno.so.8 -``` - -## User-specific settings - -User-specific configuration is stored in `~/.config/cuno`. -Remove this directory to remove all user-specific settings. - -{% callout type="note" %} -Remove any customisations made to shell initialisation scripts such as `~/.bashrc`, `~/.profile`, `.zshrc`, `/etc/profile.d`, etc. -{% /callout %} - -Please also remember to remove any customizations you might have made to your shell's runtime initialization command scripts, `.bashrc`, etc. - -## Package manager uninstallation - -Use the commands in the following sections to uninstall Object Mount for specific Linux distributions. - -### Debian derivatives (e.g. Ubuntu) - -```console -sudo apt-get remove cuno\* -``` - -### RedHat derivatives (e.g. CentOS) - -```console -sudo yum remove cuno\* -``` - -### Alpine Linux derivatives - -```console -sudo apk del cuno -``` diff --git a/app/(docs)/object-mount/linux/user-guides/validate/page.md b/app/(docs)/object-mount/linux/user-guides/validate/page.md new file mode 100644 index 000000000..ab9a946db --- /dev/null +++ b/app/(docs)/object-mount/linux/user-guides/validate/page.md @@ -0,0 +1,335 @@ +--- +title: Validate Functionality & Access Object Storage +hideTitle: false +docId: JDK2ED8HGFmyaxk +weight: 1 +metadata: + title: Validate Object Mount & Access Object Storage + description: + After installation, steps to run and validate Object Mount for Linux. +hidden: false +--- + +After [installing Object Mount](docId:iethahkeeX5EiJoh) for Linux, you should: + + 1. Run and Validate global functionality by accessing public buckets + 2. Import your private S3 credentials into Object Mount + 3. Connect to your private Object Storage buckets + +Details for each step is outlined below. + + +## Step 1. Validate Object Mount’s Public Connectivity + +The following steps will help you verify that Object Mount is running correctly and can connect to publicly accessible S3 storage buckets. + +**1a. Enter Object Mount’s CLI Console (aka “cunoFS”)** + +- From a Linux shell prompt enter the `cuno` command: + ``` + user:~$ cuno + ``` + +- You should see the `(cuno)` CLI prefix to the left of your user prompt. This indicates that a new shell has been launched, wrapped in Object Mount’s `cuno` wrapper: + ``` + user:~$ cuno + (cuno) user:~$ + ``` + + {% callout type="info" %} + **Adding `cuno` to the PATH** + + If the Object Mount `cuno` application cannot be found (`cuno: command not found`), add the [install location](docId:ahWohd5eegh6eizi#installation-locations-scripted-installers) to your `$PATH` variable. + {% /callout %} + +- Enter `exit` at any time to close the Object Mount CLI console and return to the unwrapped shell prompt: + ``` + (cuno) user:~$ exit + user:~$ + ``` + +**1b. Explore Public Datasets** + +Validate that Object Mount can reach Internet-based object storage buckets by listing the files in several public S3 repositories: + +- View image file listings from the James Webb Space Telescope on AWS S3: + + ``` + (cuno) $ ls s3://stpubdata/jwst/public/ + ``` + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-list-aws-bucket.jpg) + +- Browse satellite images on Google Cloud Storage: + ``` + (cuno) $ ls gs://gcp-public-data-landsat/ + ``` + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-linux-cuno-list-gc-bucket.jpg) + + +## Step 2. Import S3 Credentials + +In order to access and mount your private Object Storage buckets you will need to save your S3 credentials to a file. + +Instructions on obtaining your credentials and saving them to a file can be found in the Appendix article: [](docId:E4NhE5kPdjURRajJ). + +Once your credentials have been saved to a file on your Linux drive, proceed with the following steps: + +**2a. Import Your Credential File into Object Mount** + +Assuming you have saved your credentials to a file named `credentials.txt`, run the following command to add these credentials to the local set of Object Mount managed credentials: + +```sh +cuno creds import credentials.txt +``` + +This command will attempt to discover all the buckets that your credentials have access to, as well as the settings, limitations, and compatibility of these buckets. This may take a while if you have _many_ buckets associated with the credentials you are importing. + +```sh +cuno creds import credentials.txt +- Examining credential file.. +- Importing credential file into CUNO's credentials store.. +- Detecting available buckets/containers.. +Detected buckets: 2 +- Attempting to pair 2 buckets/containers.. +Paired 2/2 buckets: + [+] s3://bucket01 + [+] s3://bucket02 +``` + +You can display your imported credentials with the `cuno creds list` command: + +``` ++-------------+ +| credentials | ++-------------+ + [*] credentials.s3c + ++----------+ +| pairings | ++----------+ + [*] s3://bucket01 -> credentials.s3c + [*] s3://bucket02 -> credentials.s3c +``` + +Notice that a new credential file has been created with the `.s3c` extension, and that two buckets were found using the credentials. + +{% callout type="note" %} + **Generated `.s3c` Credential File Location** + + The `cuno creds import` command creates an `.s3c` file with the corresponding bucket entries and adds appropriate configuration settings such as: region, URL path style, etc. + + The default location for storing these imported credentials is the directory `$XDG_CONFIG_HOME/cuno/creds` (if unset, `$XDG_CONFIG_HOME` defaults to `~/.config`). + + To use an alternative location, set the `CUNO_CREDENTIALS` environment variable to point to your preferred path. + + For example: `export CUNO_CREDENTIALS=/home/user/my-cloud-credentials`. + + **Note:** You should _not_ manually copy your credentials directly into these locations. +{% /callout %} + +**2b. Troubleshooting Credential Import** + +**Purge and Re-Import:** + +If you encounter an issue and need to re-import your credential file, you must first remove the previously generated (and unwanted) `.s3c` credential file using the `purge` command: + +```sh +cuno creds purge credentials.s3c +``` + +See the [](docId:aish4shuiheeZaig) article for additional `cuno creds` details. + +**S3 Compatibility Check:** + +If you are using an S3-compatible service and are having problems, you can run a compatibility check using the `detectfeatures` command: + +```sh +cuno creds detectfeatures s3://bucket-to-test credentials.txt +``` + +This command will iterate through multiple S3-compatibility tests, settings, and limitations, and then reconfigure and adjust the generated credentials `.s3c` file based on its findings. + +**Note:** The bucket specified must permit **write access** to allow for the creation of temporary files for testing purposes. + +{% callout type="warning" %} + **Feature Detection: Bandwidth & Time Usage** + + Running feature detection may use up to a few gigabytes of bandwidth and may take a several minutes to complete depending on the machine’s connection speed and the S3 storage provider. +{% /callout %} + + +## Step 3. Test Connectivity to Your Private Object Storage Bucket(s) + +Once your credentials have been successfully imported you can test access to your private buckets. + +**3a. Enter Object Mount’s CLI Console (aka “cunoFS”)** + +- From a Linux shell prompt enter the `cuno` command: + ``` + user:~$ cuno + ``` + +- You should see the `(cuno)` CLI prefix to the left of your user prompt. This indicates that a new shell has been launched, wrapped in Object Mount’s `cuno` wrapper: + ``` + user:~$ cuno + (cuno) user:~$ + ``` + +**3b. Access Your Bucket’s Contents** + +Select your Object Storage Provider below for the appropriate commands to: + + - View buckets + - View files + - Create files + - Display files + - Delete files + +{% tabs %} + + {% tab label="AWS S3" %} + List your accessible buckets: + ``` + (cuno) $ ls s3:// + ``` + + List files and folders in a bucket: + ``` + (cuno) $ ls s3://bucket01/ + ``` + + Create a new object in your bucket + ``` + (cuno) $ echo "hello world" > s3://bucket01/helloworld.txt + ``` + + Display that new file back to the console: + ``` + (cuno) $ cat s3://bucket01/helloworld.txt + ``` + + Delete that new file: + ``` + (cuno) $ rm s3://bucket01/helloworld.txt + ``` + {% /tab %} + + {% tab label="Storj" %} + List your accessible buckets: + ``` + (cuno) $ ls s3:// + ``` + + List files and folders in a bucket: + ``` + (cuno) $ ls s3://bucket01/ + ``` + + Create a new object in your bucket + ``` + (cuno) $ echo "hello world" > s3://bucket01/helloworld.txt + ``` + + Display that new file back to the console: + ``` + (cuno) $ cat s3://bucket01/helloworld.txt + ``` + + Delete that new file: + ``` + (cuno) $ rm s3://bucket01/helloworld.txt + ``` + {% /tab %} + + {% tab label="Microsoft Azure" %} + List your accessible buckets: + ``` + (cuno) $ ls az:// + ``` + + List files and folders in a bucket: + ``` + (cuno) $ ls az://your-azure-storage-account/bucket01/ + ``` + + Create a new object in your bucket + ``` + (cuno) $ echo "hello world" > az://your-azure-storage-account/bucket01/helloworld.txt + ``` + + Display that new file back to the console: + ``` + (cuno) $ cat az://your-azure-storage-account/bucket01/helloworld.txt + ``` + + Delete that new file: + ``` + (cuno) $ rm az://your-azure-storage-account/bucket01/helloworld.txt + ``` + {% /tab %} + + {% tab label="Google Cloud" %} + List your accessible buckets: + ``` + (cuno) $ ls gs:// + ``` + + List files and folders in a bucket: + ``` + (cuno) $ ls gs://bucket01/ + ``` + + Create a new object in your bucket + ``` + (cuno) $ echo "hello world" > gs://bucket01/helloworld.txt + ``` + + Display that new file back to the console: + ``` + (cuno) $ cat gs://bucket01/helloworld.txt + ``` + + Delete that new file: + ``` + (cuno) $ rm gs://bucket01/helloworld.txt + ``` + {% /tab %} + + {% tab label="Other S3 Compatible" %} + List your accessible buckets: + ``` + (cuno) $ ls s3:// + ``` + + List files and folders in a bucket: + ``` + (cuno) $ ls s3://bucket01/ + ``` + + Create a new object in your bucket + ``` + (cuno) $ echo "hello world" > s3://bucket01/helloworld.txt + ``` + + Display that new file back to the console: + ``` + (cuno) $ cat s3://bucket01/helloworld.txt + ``` + + Delete that new file: + ``` + (cuno) $ rm s3://bucket01/helloworld.txt + ``` + {% /tab %} +{% /tabs %} + +**3c. Exit Object Mount’s CLI** + +Enter `exit` at any time to close the Object Mount CLI console and return to the unwrapped shell prompt: + +``` +(cuno) user:~$ exit +user:~$ +``` diff --git a/app/(docs)/object-mount/macos/_meta.json b/app/(docs)/object-mount/macos/_meta.json deleted file mode 100644 index f7b6d5aca..000000000 --- a/app/(docs)/object-mount/macos/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "macOS", "weight": 3} diff --git a/app/(docs)/object-mount/macos/fast-copy-paste/_meta.json b/app/(docs)/object-mount/macos/fast-copy-paste/_meta.json deleted file mode 100644 index 16da49cba..000000000 --- a/app/(docs)/object-mount/macos/fast-copy-paste/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Fast Copy Paste", "weight": 4} diff --git a/app/(docs)/object-mount/macos/fast-copy-paste/page.md b/app/(docs)/object-mount/macos/fast-copy-paste/page.md deleted file mode 100644 index 39521bb41..000000000 --- a/app/(docs)/object-mount/macos/fast-copy-paste/page.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Fast Copy and Move on macOS -docId: fast-copy-paste -hideTitle: true -weight: 7 -metadata: - title: Fast Copy and Move on macOS - description: - A semi-technical explanation of how Object Mount’s Fast Copy and Move Finder integration works on macOS. ---- - -# Fast Copy and Move on macOS - -As of Object Mount **v1.0.4-beta**, macOS now supports **Fast Copy** and **Fast Move** options directly from Finder. This provides a much faster way to send files to your mounted S3 or Storj storage which ideal for working with large media assets or entire folders. - -You'll see new right-click options called: - -- **Fast Copy To Object Mount** -- **Fast Move To Object Mount** - -These options appear in Finder when you right-click a file or folder. - ---- - -## How to Enable Fast Copy / Move - -Fast Copy and Fast Move are powered by a Finder Extension and must be enabled manually the first time. - -### Steps: - -1. Open **System Settings** -2. Navigate to **General → Login Items** -3. Scroll to the bottom to find **Extensions** -4. Click **File Provider Extensions** -5. Enable the checkbox for **Object Mount** - -Once enabled, you’ll see the new right-click options in Finder. - -{% callout type="info" %} -You may need to right-click again or relaunch Finder (`Cmd + Option + Esc`) to see the new menu items. -{% /callout %} - ---- - -## When to Use Fast Copy or Move - -These options are ideal when: - -- Moving footage, projects, or renders from a local disk into the cloud -- Copying large files or folders from one Object Mount volume to another -- Working with high-latency networks or remote S3 endpoints -- Avoiding bottlenecks from Finder’s standard file operations - ---- - -## Why It’s Faster - -Standard Finder copy/move uses macOS file APIs that copy data one block at a time, without any awareness of how cloud storage works. This can lead to slower performance, especially for large files or nested folder structures. - -Fast Copy and Fast Move bypass these routines and instead: - -- Use Object Mount’s **optimised S3 transfer logic** -- Support **multipart, parallel uploads** -- Automatically manage metadata and cloud layout -- Integrate directly with your mount’s caching settings - ---- - -## Behind the Scenes - -Fast Copy / Move: - -- Leverages the Finder Extension and Object Mount’s core engine -- Uses **direct S3 transfer** logic, not shell scripting or AppleScript -- Respects your mount settings (e.g. POSIX mode, cache location) -- Logs operations in your standard Object Mount log file - ---- - -## Best Practices - -- Store original media locally, then **Fast Move** it to cloud when ready -- For large archive folders, use **Fast Copy** to quickly duplicate into your S3 bucket -- Use Fast Move to clear local storage while archiving to the cloud in one step - -{% callout type="info" %} -Unlike drag-and-drop or standard Copy + Paste, **Fast Copy / Move** ensures optimal use of bandwidth and avoids unnecessary local I/O. -{% /callout %} - -{% callout type="info" %} -**Log File Location** --`~/Library/Application Support/Object Mount/cunoFS.log` -{% /callout %} - ---- - -## Related Settings - -For further tuning, see: -- [Performance Tuning](../faq/tuning/) - ---- - -Need help getting Fast Copy or Move working? [Contact support](https://supportdcs.storj.io/hc/en-us/requests/new) — we’re happy to help. - ---- diff --git a/app/(docs)/object-mount/macos/features/page.md b/app/(docs)/object-mount/macos/features/page.md new file mode 100644 index 000000000..517b6e571 --- /dev/null +++ b/app/(docs)/object-mount/macos/features/page.md @@ -0,0 +1,110 @@ +--- +title: Feature Guide +hideTitle: true +docId: ehHbpq6KFndcVCgc +weight: 3 +metadata: + title: Feature Guide for macOS + description: + Details on unique features of the macOS native app. +hidden: false +--- + +# Feature Guide: macOS Native App + +This guide provides details on unique and noteworthy features that are specific to the **macOS Native App** of Object Mount. + +Features that are global and apply to **all OS versions** of Object Mount (macOS, Windows, Linux) can be found in the [Global Features](#global-features-and-concepts) section below. + + +## Fast Copy and Fast Move + +Starting with **v1.0.4**, Object Mount for macOS now supports **Fast Copy** and **Fast Move** options directly from the Mac Finder. + +- **Note:** macOS 13 Ventura (or later) is required to use Fast Copy and Fast Move. + +**Fast Copy** and **Fast Move** provide a much faster way to upload files to your object storage — ideal when working with large media assets or entire folder trees of content. + +Once enabled, two new context-sensitive menu options will be displayed when you right-click on a file or folder on the Mac’s Desktop or within the Finder: + +- **Object Mount > Fast Copy To...** +- **Object Mount > Fast Move To...** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-fast-copy.jpg) + +### Enabling Fast Copy and Fast Move + +Fast Copy and Fast Move are enabled through a Finder Extension. The extension must be manually activated. + +Follow the steps below to enable Fast Copy and Fast Move in **two** places within the Mac’s System Settings: + +1. Open **System Settings**. + +2. Navigate to **General → Login Items & Extensions**. + +3. Scroll down to the **Extensions** section. + +4. Click the **Info Circle** `(i)` next to **File Providers**. + - Activate the slider for **Object Mount** and click **Done**. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-fast-copy-enable.jpg) + +5. Click the **Info Circle** `(i)`next to **File System Extensions**. + - Activate the slider for **macFuse** and click **Done**. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-enable-macfuse-sys-ext.jpg) + +Once enabled, you will see the new “right-click” options in the Finder. + +{% callout type="info" %} + **“Right-Click” Options** + + You may need to right-click more than once to activate the extension for the first time. If necessary, reboot your Mac to see the new menu items. +{% /callout %} + +### When to Use Fast Copy or Fast Move + +Using **Fast Copy** or **Fast Move** is ideal when you want to: + +- Move large files (media footage, projects, renders, etc.) or large folder trees from a local disk to a mounted S3 bucket +- Copy large files or folders from one Object Mount volume to another Object Mount volume +- Improve performance on high-latency networks (or slow remote S3 endpoints) +- Avoid the bottlenecks of the Finder’s standard file operations + +### How Fast Copy and Fast Move Work + +Standard **copy and move** actions performed by the Finder use the native macOS file system’s I/O calls. These standard file operations copy data **one block** at a time, without any awareness of how cloud storage works. This can lead to slow performance — especially for large files or nested folder structures. + +**Fast Copy** and **Fast Move** bypass these standard OS-based read/write calls and instead: + +- Use Object Mount’s **optimized S3 transfer logic** +- Support **multipart and parallel uploads** +- Automatically manage metadata and cloud layout +- Integrate directly with your mount’s caching settings + +Behind the scenes, Fast Copy and Fast Move: + +- Leverage both the Finder Extension and Object Mount’s core engine +- Use **direct S3 transfer** logic, not shell scripting or AppleScript +- Respect your mount settings (e.g.: POSIX mode, cache location) +- Log operations to the standard Object Mount log file + +### Use Cases for Using Fast Copy and Fast Move + +- Store original media locally, then **Fast Move** it to the cloud when ready. +- For large archive folders, use **Fast Copy** to quickly duplicate them into your S3 bucket. +- Use **Fast Move** to free up local storage when archiving to the cloud. + + +## Global Features & Concepts + +See the many articles in the [Appendix](docId:xzAqS9Um7xjQSbmD) for additional details on features that apply to Object Mount on _all_ operating systems (macOS, Windows, and Linux): + +- Object Storage Core Concepts +- POSIX explained +- S3 Credential Management +- Supported S3 Providers +- Performance Tuning +- Troubleshooting Guide +- FAQ +- Release Notes diff --git a/app/(docs)/object-mount/macos/installation/page.md b/app/(docs)/object-mount/macos/installation/page.md index 588f43fe3..5d9d8b355 100644 --- a/app/(docs)/object-mount/macos/installation/page.md +++ b/app/(docs)/object-mount/macos/installation/page.md @@ -1,72 +1,166 @@ --- -title: Installation -weight: 1 +title: Installation Guide hideTitle: true +docId: MyV4Bv3fKJEZucAb +weight: 1 metadata: - title: macOS Native Installation + title: Installation Guide for Native macOS App description: Details about installing Object Mount on macOS +hidden: false --- -# macOS Native Installation Guide -This section covers how to install the app on macOS using the standard `.dmg` installer. The process is quick and straightforward, with just a couple of setup steps to ensure everything works smoothly on both Intel and Apple Silicon (M1/M2/M3/M4) machines. +# Installation Guide: macOS Native App + +This section covers how to install the Object Mount native application for macOS using a standard `.dmg` image. + +The steps below work for Macs with either Intel processors or Apple Silicon (M1-M5) chips. ---- ## System Requirements -Before you begin, make sure your system meets the following: +Before you begin, make sure your Mac meets the following system requirements: -- macOS 12.0 Monterey or later -- At least 300MB of free disk space -- Internet connection for mounting cloud-hosted buckets, or LAN connection for mounting local buckets -- **macFUSE 4.8.2 or later** must be installed (we’ll walk you through it) +- macOS 12 Monterey or later ---- + (Note: macOS 13 Ventura or later is required for advanced features such as Fast Copy & Fast Move.) -## Step 1: Download the Installer +- At least 300MB of free disk space +- Internet connection for mounting cloud-hosted buckets, or LAN connection for mounting local S3 buckets +- macFUSE 4.8.2 or later must be installed (see instructions below) -Download the correct `.dmg` file for your system. +{% callout type="note" %} + **S3 Provider Support:** -{% callout type="info" %} -Choose the version that matches your chip architecture: x86 for Intel or arm64 for Apple Silicon). + Object Mount for Mac fully supports S3 container services from Amazon Web Services, Storj, and other fully S3-compatible Object Storage providers. + + - See the [Supported S3 Providers](docId:xCjeasn8SLQn2vfd) article in the Appendix for a complete S3 provider list. + + **Azure & GCS Support:** + + - Support for _Native_ Microsoft Azure Storage and _Native_ Google Cloud Storage services are still under development in Object Mount for Mac. + - For Google Cloud access: Using GCS _S3_ is recommended instead of using GCS _S3 Gateways_. + - Native Azure and Native GCS (S3 Gateways) should be considered “experimental” at this time. {% /callout %} -If you're unsure which chip you have: -- Click the Apple  menu in the top left -- Select **About This Mac** -- Look for **Processor** (Intel) or **Chip** (Apple M1, M2, etc.) ---- +## Step 1: Download the latest Installation Image + +Download the correct macOS `.dmg` disk image file for your system. + +**Note:** Contact your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) if you need access to the most up-to-date disk image. + +{% callout type="info" %} + **Mac Chip Types** -## Step 2: Install the App + Be sure to choose the disk image that matches your chip architecture: + - **x86** for Intel processors + - **arm64** for Apple Silicon/chips (M1-M5). + + To determine which chip you have: + - Click the Apple  menu in the top left + - Select **About This Mac** + - Look for **Processor** (Intel) or **Chip** (Apple M1, M2, etc.) +{% /callout %} + + +## Step 2: Install the Object Mount App 1. Double-click the `.dmg` file to mount it -2. Drag the app into your **Applications** folder +2. Drag the Object Mount app into your **Applications** folder + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/on-mac-drag-to-install.jpg) ---- ## Step 3: Install macFUSE (Required) -The app uses **macFUSE** to handle filesystem mounting. If you don’t have it already: +The native macOS Object Mount app requires the **macFUSE** system extension. -1. Download the latest version from the [macFUSE releases page](https://osxfuse.github.io/) -2. Open the `.dmg` and run the installer -3. You’ll likely be prompted to allow a System Extension in **System Settings → Privacy & Security** -4. Click “Allow” and restart your Mac into Recovery Mode if prompted +macFuse, on both Intel and Apple silicon Macs, provides system-level volume mounting extensions that Object Mount leverages. -{% callout type="warning" %} -You must allow macFUSE in System Settings, or the app won't be able to mount. -{% /callout %} +Download and install macFUSE following the steps below for your Mac hardware type: ---- +{% tabs %} -## Step 4: Launch the App +{% tab label="Intel Processor" %} + 1. Download the **latest** version (not pre-release) `.dmg` image from the [macFUSE releases page](https://macfuse.github.io/). -Once installed: + 2. Open the macFUSE `.dmg` and run the **Install macFUSE.pkg**. + - As prompted, click **Allow**, **Continue** and/or **Agree** to install the package -- Open the app from your **Applications** folder -- You will be prompted to enter your **Licence Key** automatically -- You may be prompted to grant access to removable volumes or network locations — click **Allow** + 3. As a new System Extension, the package will generate a “System Extension Blocked” message. + + - Open **System Settings** + - Open **Security & Privacy** + - Click the **lock** 🔒 icon to allow changes + - Click the **Allow** button to permit the installation ---- + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-macfuse-01.jpg) + + 4. Restart your Mac when prompted. +{% /tab %} + +{% tab label="Apple Silicon (M Chip)" %} + 1. Download the **latest** version (not pre-release) `.dmg` image from the [macFUSE releases page](https://macfuse.github.io/). + + 2. Open the macFUSE `.dmg` and run the **Install macFUSE.pkg**. + - As prompted, click **Allow**, **Continue** and/or **Agree** to install the package + + 3. As a new System Extension, the package will generate a “System Extension Blocked” message. + - Open **System Settings** + - Open **Privacy & Security** + - Scroll down to the **Security** section + - Find and click the **Enable System Extensions...** button + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-fuse-install-01-enable.jpg) + + 4. You will then be prompted to **Shut Down** and restart your Mac into its **Recovery Mode** + -Save any work you have in progress + - Click **Shut Down** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-fuse-install-02-shut-down.jpg) + + 5. With your Mac powered off, **PRESS AND HOLD** your power button (or Touch ID button) for 3-5+ seconds. + - You should see an Apple message that says **Continue holding for startup options...** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-recovery-mode-01-hold.jpg) + + - Continue holding the power button until the message changes to **Loading startup options...** + + 6. In the modified boot screen, click the gear icon labeled **Options** then click **Continue** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-recovery-mode-02-gear.jpg) + + - Select your admin user, click **Next** and enter your password to view the **Recovery Mode** screen and its menu options + + 7. From the **Recovery** menu, select the **Utilities** drop-down menu and choose **Startup Security Utility** + + 8. Select your primary hard drive and click **Unlock...** + + - Enter your Admin password and click **Unlock** + + 9. Once unlocked, click the **Security Policy...** button + + 10. Click the **Reduced Security** radio button + + - Then, tick the first box for **Allow user management of kernel extensions from identified developers** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-recovery-mode-03-reduced-security.jpg) + + - Click **OK** and enter your admin password one last time + + 11. Lastly, from the **Startup Disk** Menu, choose **Shut Down** + + Restart your Mac as normal. +{% /tab %} + +{% /tabs %} + + +## Next Steps + +Proceed to the macOS [User Guide](docId:QpBba8p4bMTXAkBK) to **activate** your Object Mount installation, **configure** your object storage credentials, and **create** your first mount. + +Then review the macOS [Feature Guide](docId:ehHbpq6KFndcVCgc). + +Finally, read through the usage tips for your specific [Media Workflows](docId:JXF4_NLsV@nh9QcT) and Applications. diff --git a/app/(docs)/object-mount/macos/page.md b/app/(docs)/object-mount/macos/page.md index a8809c796..69fc96842 100644 --- a/app/(docs)/object-mount/macos/page.md +++ b/app/(docs)/object-mount/macos/page.md @@ -6,31 +6,37 @@ weight: 3 metadata: title: macOS Installation and Usage description: - Step-by-step guidance for installing and using the native Object Mount app on macOS. + Home Page for guidance on installing and using the native Object Mount app on macOS. +hidden: false --- # macOS Installation and Usage -This section contains everything you need to install and use the native **Object Mount** app on macOS. +This section contains everything you need to install and use the native **Object Mount** app on an Apple Mac. -Whether you're setting up for creative workflows, remote storage access, or team-based collaboration, these pages will walk you through installation, setup, and day-to-day use. +Whether you’re setting up for creative workflows, remote storage access, or team-based collaboration, these pages will walk you through installation, configuration and day-to-day use. ---- ## What’s Covered -- Native installation using the `.dmg` package +- Native installation using a `.dmg` package - System requirements and FUSE dependencies -- Setup guidance for Apple Silicon and Intel Macs -- User guide for mounting, configuring, and managing storage +- Setup guidance for Macs using either Apple Silicon and Intel chips +- User guides for mounting, configuring, and managing your object storage +- Tips and recommendations for unique macOS features, media applications, and performance improvements ---- -## Related Pages +## Next Steps -- [macOS Installation Guide](macos/installation/) -- [macOS User Guide](macos/user-guides/) +Start with the **Installation Guide**: -We recommend starting with the installation guide, then moving on to the user guide for tips on mounting and working with storage in your daily workflows. +- macOS [Installation Guide](docId:MyV4Bv3fKJEZucAb) ---- +Move on to the **User Guide** for instructions on creating mounts and working with object storage in your daily workflows: + +- macOS [User Guide](docId:QpBba8p4bMTXAkBK) + +Lastly, review the macOS OS-specific **Feature Guide** and the appropriate **Media Workflow Guide**: + +- macOS [Feature Guide](docId:ehHbpq6KFndcVCgc) +- [Media Workflow Guides](docId:JXF4_NLsV@nh9QcT) diff --git a/app/(docs)/object-mount/macos/user-guide/page.md b/app/(docs)/object-mount/macos/user-guide/page.md new file mode 100644 index 000000000..32ed9de2b --- /dev/null +++ b/app/(docs)/object-mount/macos/user-guide/page.md @@ -0,0 +1,217 @@ +--- +title: User Guide +hideTitle: true +docId: QpBba8p4bMTXAkBK +redirects: + - /object-mount/credentials/macos/import +weight: 2 +metadata: + title: Object Mount User Guide for macOS + description: + A user guide for getting started with Object Mount on macOS. +hidden: false +--- + +# User Guide: macOS Native App + +This guide walks you through the core steps to get started with **Object Mount on macOS**, including: activation, adding credentials, and mounting your cloud-based object storage as a local volume. + + +## Step 1. Launch & Activate Object Mount + +Once Object Mount and macFUSE have been installed on your Mac (see the macOS [](docId:MyV4Bv3fKJEZucAb)), follow the steps below to launch and activate Object Mount: + +- Run Object Mount from your **Applications** folder. + +- You will be prompted to enter your Object Mount **License Key**. + - Contact your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) if you need to obtain an Object Mount License Key. + +- Click **Install New License** and paste your Object Mount license key into the text box. + +- Click **Activate**. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-license-key-valid.jpg) + +- If prompted, click **OK** to allow Object Mount to access files on network volumes. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-permission.jpg) + +Once activated, you can view your license key details in the **About** tab within Object Mount. + + +## Step 2. Configure S3 Credentials & Create a Mount + +Object Mount can connect to object storage from Storj as well as other major cloud storage providers. Standard S3-compatible credentials from your cloud storage provider are required. + +### Obtain Your Object Storage Credentials + +- See the Appendix article [](docId:E4NhE5kPdjURRajJ#obtaining-your-s3-credentials) for instructions. + +### Enter Your Credentials and Create a Mount + +There are two easy paths to create a mount — both paths achieve identical results and both require your object storage credentials: + +- Click **Create New Mount** from the **Mounts** tab, or +- Click **Add New Credentials** from the **Credentials** tab +- Then select and configure one or more buckets to mount as a local drive + +Follow the steps below to create a new mount using the **Add New Credentials** path: + +1. Select the **Credentials** tab in Object Mount. + +2. Click the **Add New Credentials** button. + - Give the credential set a clear name. + - Choose your provider tab (e.g.: AWS, Storj, etc.). + - Enter the **Access key ID**, **Secret key**, and **Endpoint** (or Region, if applicable) that you obtained earlier. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-new-credentials.jpg) + + {% callout type="info" %} + **S3 Compatible Providers** + + If your provider is not listed, select **Other S3** and choose the provider from the dropdown. + + - Ensure the endpoint is accessible from the Internet. + - Some providers may require extra compatibility settings like a defined region. + + Below is an example of providing an additional S3 Compatibility Setting for region: + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-custom-s3-region.png) + {% /callout %} + +3. Click **Next**. + - Object Mount will authenticate to your cloud storage provider and attempt to discover any buckets your credentials have access to. + +4. Select **Mount all buckets** or choose one or more specific buckets to create a mounted drive for, then click **Next**. + +5. Enter a **Mount name** to use and, if necessary, alter the path location for the local mount drive. + +6. Tick the **Mount as read-only** box if all you need is to browse and copy content. + - Uncheck the box to enable **full read/write access**. + +7. Tick the box to **Enable POSIX metadata** (recommended). + - For more details, see the Appendix article: [POSIX Explained](docId:ySneAEd79CVewSSr). + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-configure-your-mount.jpg) + +8. Click **Next** to create the mount. Then enable the mount from the **Mounts** tab. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-enable-your-mount.jpg) + + {% callout type="info" %} + **“Extension Blocked” Message** + + Depending on your Mac hardware and OS version, an **“Extension Blocked”** message may be presented when Object Mount attempts to mount your new drive. + + If so, perform the following steps: + - Open **System Settings** + - Navigate to **Privacy & Security** + - Scroll down to the **Security** section + - Click **Allow** to permit the macFUSE extension (from developer “Benjamin Fleischer”) to be loaded + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-allow-macfuse.jpg) + {% /callout %} + +**Note:** First-time access to a new mount may take a few moments if the bucket contains many files — performance improves after the initial mount. + + +## Newly Mounted Drive & Buckets + +Your newly mounted drive will appear on your Mac using a **FUSE** drive icon: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-fuse-drive-icon.jpg) + +If there are multiple buckets accessible using the same credential set, those buckets will each appear as separate **folders** within the single mounted drive icon: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-multiple-buckets.jpg) + + +## Preferences & Advanced Settings + +Object Mount for macOS includes a flexible **Settings** panel where you can fine-tune caching, logging, and performance settings. + +{% callout type="info" %} + **Settings** + + - These settings are global and apply to all mounts. + - These settings cannot be edited while a mount is active. Disable all mounts before making changes. +{% /callout %} + +Refer to the following sections for details on each setting: + +### Data Cache + +The local **Data Cache** temporarily stores small portions of accessed cloud files on your local disk — useful for repeated or random access operations. + +- **Location:** The local path where data cache files are stored +- **Maximum cache size:** The upper limit (in MB) for total cached data + - **Note:** This feature does *not* delay cloud writes — writing to the cloud happens immediately. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-settings-01.jpg) + +{% callout type="info" %} + **Data Cache Recommendations** + + - Use a fast local drive (SSD or NVMe) for best caching performance. + - If the connection to your object storage is on a high-speed, low-latency network, you may see improved performance by turning caching _off_. +{% /callout %} + +### Metadata Cache Settings + +Metadata caching helps Object Mount load and retain file structure, timestamps, and permissions more efficiently. + +- **Location:** The local folder for storing metadata information +- **Cache timeout:** The duration, in seconds, that metadata remains valid before being refreshing from the cloud (default: 60 seconds) + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-settings-02.jpg) + +### Advanced Settings + +These options let you further customize Object Mount’s performance and diagnostic behavior. + +| **Setting** | **Description** | +|--------------------------|-----------------| +| **S3 connections limit** | Controls the number of concurrent S3 operations (default: 100). +| **Log level** | Adjusts logging detail. Options: `error`, `warning`, `debug`, or `trace`. +| **Single threaded mode** | Runs the filesystem in single-threaded mode (rarely needed; leave `off` unless instructed by Storj Support). +| **Debug logging** | Enables extensive debug output to assist with support issues. +| **Custom Environment Variables** | Custom variables to modify Object Mount’s behavior, add variables when recommended by Storj Support. + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-settings-03.jpg) + +{% callout type="info" %} + **Log File Location** + + On macOS the default location for storing the log file is: + + `~/Library/Application Support/Object Mount/cunoFS.log`. +{% /callout %} + +Click **Apply** to save any changes. Then re-enable your mount. + +Click **Reset** to revert all settings back to their default values. + + +## Updating macFUSE + +You can update and/or uninstall macFUSE within the Mac’s System Preference. + +{% tabs %} + +{% tab label="Monterey (12) & Older" %} +For macOS versions Monterey and prior, you will find the macFUSE extension at the bottom of the **System Preferences** window: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-macfuse-preferences-01.jpg) + +Click the macFUSE icon to see options to **update** and/or **uninstall** macFUSE: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-macfuse-preferences-02.jpg) +{% /tab %} + +{% tab label="Ventura (13) & Newer" %} +For **newer macOS versions**, you will find the macFUSE extension listed at the very bottom within **System Settings**: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-update-macfuse-m-chip.jpg) +{% /tab %} + +{% /tabs %} diff --git a/app/(docs)/object-mount/macos/user-guides/page.md b/app/(docs)/object-mount/macos/user-guides/page.md deleted file mode 100644 index 8f2f633ce..000000000 --- a/app/(docs)/object-mount/macos/user-guides/page.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -title: macOS User Guide -docId: macos-user-guide -redirects: - - /object-mount/credentials/macos/import -hideTitle: true -weight: 1 -metadata: - title: Object Mount macOS User Guide - description: - A simple one-page user guide for getting started with Object Mount on macOS. ---- - -# Object Mount macOS User Guide - -This guide walks you through the core steps to get started with **Object Mount** on macOS — including activation, importing credentials, and mounting your object storage as a local volume. - ---- - -## 1. Initial Setup & Licence Activation - -When you first launch Object Mount, you'll be prompted to enter your **licence key**. This should have been provided via email. - -- Enter your activation key when prompted -- To verify your licence period, go to the **About** tab in the top navigation — this displays your current licence status and expiry - ---- - -## 2. Importing Cloud Credentials - -Object Mount connects to major cloud storage providers using standard S3-compatible credentials. - -### Steps: - -1. Go to the **Credentials** tab -2. Click the green **Import new credentials** button -3. In the modal: - - Give the credential set a clear name - - Choose your provider tab (e.g., AWS, Storj, etc.) - - Fill in the access key, secret, and (if applicable) region or endpoint - -{% callout type="info" %} -If you're using a different provider, select **S3-compatible**, choose the provider from the dropdown, and ensure the endpoint is accessible. Some providers may require extra compatibility settings like a defined region. -{% /callout %} - - -For **Storj users**, we recommend using **lexicographically ordered buckets** for the best listing performance. - -4. Click **Import** - - Object Mount will attempt to discover any buckets your credentials can access -5. Select one or more buckets to create mount configurations - ---- - -## 3. Creating and Managing Mounts - -Once your credentials are active and buckets have been detected: - -1. Go to the **Mounts** tab -2. Click **Create new mount** -3. Configure the mount: - - Choose bucket and local mount path - - Decide whether to enable **POSIX mode** (adds permission emulation) - - Configure caching or advanced options if needed - -4. Click **Add Mount** to save it -5. On the Mounts page, toggle the mount to **activate** it -6. Click the mount name to open it in **Finder** - -{% callout type="info" %} -First-time access may take a few seconds if the bucket contains many files — performance improves after initial load. -{% /callout %} - - - ---- - -## 4. Preferences & Advanced Settings - -Object Mount includes a flexible **Preferences** panel where you can fine-tune caching, logging, and performance settings across all mounts. These settings are global and apply to every active mount. - ---- - -### Data Cache - -The **data cache** temporarily stores small portions of accessed files on your local disk — useful for repeated or random access operations. - -- **Location**: The local path where data cache files are stored -- **Maximum cache size**: The upper limit (in MB) for total cached data -- **Note**: This does *not* delay cloud writes — writing to the cloud happens immediately - -{% callout type="info" %} -Use a fast local drive (SSD or NVMe) for best caching performance. -{% /callout %} - -{% callout type="info" %} -If you are on a high-speed, low-latency network connected to your object storage, you may see improved performance by turning caching off. -{% /callout %} - ---- - -### Metadata Cache - -Metadata caching helps Object Mount remember file structure, timestamps, and permissions more efficiently. - -- **Location**: The local directory for storing metadata information -- **Cache timeout (s)**: The duration metadata remains valid before refreshing from the cloud (default: 60 seconds) - ---- - -### Advanced Settings - -These options let you further customise Object Mount's performance and diagnostic behaviour: - -| Setting | Description | -|-------------------------------------|-------------| -| **S3 Connections Limit** | Controls the number of concurrent S3 operations (default: 100) | -| **Log Level** | Adjusts logging detail. Options: `error`, `warning`, `debug`, or `trace` | -| **Single Threaded Mode** | Runs the filesystem in single-threaded mode (rarely needed; leave off unless instructed by Support) | -| **Debug Logging** | Enables detailed debug output to assist with support issues | -| **Environment Variables** | Custom variables to modify Object Mount behaviour, adding variables may be recommended by Support | - -{% callout type="info" %} -**Log File Location** --`~/Library/Application Support/Object Mount/cunoFS.log` -{% /callout %} - -Click **Reset** to revert changes to default settings. - ---- - -## More Info - -- [Performance Tuning](../faq/tuning/) -- [POSIX mode explained](../faq/posix-mode/) - ---- diff --git a/app/(docs)/object-mount/media-workflows/_meta.json b/app/(docs)/object-mount/media-workflows/_meta.json deleted file mode 100644 index 359dce3e6..000000000 --- a/app/(docs)/object-mount/media-workflows/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Object Mount in Media Workflows", "weight": 1} diff --git a/app/(docs)/object-mount/media-workflows/davinci-resolve/page.md b/app/(docs)/object-mount/media-workflows/davinci-resolve/page.md index 0ec048ccf..001d0e689 100644 --- a/app/(docs)/object-mount/media-workflows/davinci-resolve/page.md +++ b/app/(docs)/object-mount/media-workflows/davinci-resolve/page.md @@ -1,75 +1,77 @@ --- -title: Object Mount with DaVinci Resolve +title: DaVinci Resolve hideTitle: true +docId: xZ5tdyuyYGM8Lrfp +weight: 3 metadata: title: Object Mount with DaVinci Resolve description: - Recommended settings and tips for using Object Mount with DaVinci Resolve + Recommended settings and tips for using Object Mount with DaVinci Resolve. +hidden: false --- # Using Object Mount with DaVinci Resolve This guide walks you through the best setup for using Object Mount with **DaVinci Resolve**, along with a few optional tweaks and performance tips that may help smooth things out depending on your system and workflow. ---- ## Recommended Settings (Per Project) After opening or importing your Resolve project, please check the following: -1. From the top menu bar, go to: +1. From the top menu bar, go to: **Preferences > System > Media Storage** -2. Add your **Object Mount** volume as a **Media Storage Location** - - The Object Mount volume **should not** be first in the list - - Resolve uses the top location for gallery stills and cache files, so keep this separate +2. Add your **Object Mount** volume as a **Media Storage Location**: + - The Object Mount volume **should not** be first in the list. + - Resolve uses the _top location_ for gallery stills and cache files, so keep this separate. -3. Disable **Direct I/O** for the Object Mount path - - This helps ensure better compatibility and smoother read/write behaviour +3. Disable **Direct I/O** for the Object Mount path: + - This helps ensure better compatibility and smoother read/write behavior. ---- ## Optional Performance Tweaks -These settings aren’t required, but can improve responsiveness or playback performance — especially on larger projects or when working over cloud storage. +These settings aren’t required, but can improve responsiveness or playback performance — especially on larger projects or when working over cloud storage. -### Playback and Caching +**Playback and Caching:** -- **Playback > Render Cache** → Set to **Smart** -- **Playback > Timeline Proxy Resolution** → Set to **Half** or **Quarter** -- **Playback > Proxy Handling** → Set to **Prefer Proxies** -- **Playback > Use Optimized Media if Available** → Enable + - **Playback > Render Cache** → Set to **Smart** + - **Playback > Timeline Proxy Resolution** → Set to **Half** or **Quarter** + - **Playback > Proxy Handling** → Set to **Prefer Proxies** + - **Playback > Use Optimized Media if Available** → Enable -### Preview Panels and Navigation +**Preview Panels and Navigation:** -- In the **Edit** tab: - - Use the **Timeline View Options** (top-left of the timeline) - - Set **Thumbnail View** to **Thumbnail** or **None** - - Disable **Display Audio Waveform** + - In the **Edit** tab: + - Use the **Timeline View Options** (top-left of the timeline) + - Set **Thumbnail View** to **Thumbnail** or **None** + - Disable **Display Audio Waveform** -- In the **Media** tab: - - Switch **Timelines panel** to **List View** (top right) - - Open the overflow menu: - - Disable **Film Strip** - - Disable **Audio Waveforms** + - In the **Media** tab: + - Switch **Timelines panel** to **List View** (top right) + - Open the overflow menu: + - Disable **Film Strip** + - Disable **Audio Waveforms** -- In **Media Storage**, switching from **Thumbnail View** to **List View** can also help when browsing through folders over Object Mount. + - In **Media Storage**, switching from **Thumbnail View** to **List View** can also help when browsing through folders on Object Mount volumes. ---- ## Monitoring Performance -Because of how Object Mount optimises transfers, the built-in file copy progress indicators (in Finder or File Explorer) may not accurately reflect what's happening behind the scenes. +Because of how Object Mount optimizes transfers, the built-in file copy progress indicators (in Finder or File Explorer) may not accurately reflect what’s happening behind the scenes. For a better view of performance: -- On **macOS**: Open **Activity Monitor > Network tab** -- On **Windows**: Go to **Task Manager > Performance > Network** + - On **macOS**: Open **Activity Monitor > Network tab** + - On **Windows**: Go to **Task Manager > Performance > Network** -If Object Mount is saturating the network connection, that’s a good sign — it’s doing what it’s designed to do. Transfers may still appear idle in the UI, but they are active under the hood. +If Object Mount is saturating the network connection — that’s a good sign! It’s doing what it’s designed to do. + +Transfers may still appear idle in the UI, but they are active “under the hood”. {% callout type="info" %} -Don’t cancel a transfer just because the Finder or Explorer bar seems stuck — always check your system’s network usage first. + **Use Patience with Slow Transfers** + + Don’t cancel a transfer just because the Finder or Explorer bar seems stuck — always check your system’s network usage first. {% /callout %} - ---- diff --git a/app/(docs)/object-mount/media-workflows/daw/page.md b/app/(docs)/object-mount/media-workflows/daw/page.md deleted file mode 100644 index f1dbf5dee..000000000 --- a/app/(docs)/object-mount/media-workflows/daw/page.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Object Mount with Digital Audio Workstations -hideTitle: true -metadata: - title: Object Mount with Digital Audio Workstations (DAWs) - description: - Best practices for using Object Mount with audio software including Pro Tools, Logic Pro X, Ableton Live, and more. ---- - -# Using Object Mount with Digital Audio Workstations (DAWs) - -Object Mount works well alongside many industry-standard audio tools by offering high-performance access to cloud-based sample libraries, project archives, and media assets. This guide offers high-level advice for working with **Pro Tools**, **Logic Pro X**, **Ableton Live**, **Cubase**, **FL Studio**, **Reaper**, **Studio One**, and **Adobe Audition**. - ---- - -## Recommended Workflow - -While Object Mount behaves like a local drive, most DAWs perform best when: - -- **Project/session files** are saved to a **local SSD** -- **Media and asset libraries** (e.g. sample packs, stems, field recordings) are stored on Object Mount -- **Render/export targets** are local, then archived to cloud storage after completion - -{% callout type="info" %} -In all cases, placing scratch/cache folders on fast internal drives is strongly recommended. -{% /callout %} - ---- - -## Tool-Specific Tips - -{% tabs %} -{% tab label="Avid Pro Tools" %} -- Store `.ptx` session files and audio files on **local volumes** -- Use Object Mount for **archived session material**, bounced stems, or loop/sample libraries -- Avoid using cloud storage for active playback/editing unless the session is fully loaded into RAM -{% /tab %} - -{% tab label="Logic Pro X" %} -- Store your `.logicx` project files locally -- Link external media (samples, bounces, multis) from Object Mount -- Enable **Project > Consolidate** to pull linked cloud assets into local sessions if needed -{% /tab %} - -{% tab label="Reaper" %} -- Reaper is flexible and lightweight and can often read directly from mounted cloud volumes -- Recommended: use **Save project with media** to consolidate sessions if moving between machines -{% /tab %} - -{% tab label="Adobe Audition" %} -- Keep `.sesx` project files local -- Use Object Mount for access to broadcast archives, interview audio, and backing tracks -- When exporting long sessions, render to a local path before uploading to cloud -{% /tab %} - -{% tab label="Ableton Live / FL Studio / Cubase / Studio One" %} -- Save `.als`, `.flp`, `.cpr`, and `.song` project files locally -- Use Object Mount to organise reusable assets like: - - Drum kits - - Sound packs - - Rendered stems - - Collab folders -- Consider using proxies or pre-rendered versions of multi-track material stored on the cloud. -{% /tab %} -{% /tabs %} - ---- - -## Summary - -DAWs are highly sensitive to disk latency. For best results: - -- Store active sessions and renders locally -- Use Object Mount for assets, libraries, and archives -- Enable caching in Preferences for faster access to large sample sets -- Avoid real-time recording or playback **to** Object Mount volumes - -Looking to integrate Object Mount into a studio or broadcast pipeline? [Contact support](https://supportdcs.storj.io/hc/en-us/requests/new) for advice on multi-user and high-throughput setups. - ---- diff --git a/app/(docs)/object-mount/media-workflows/final-cut-pro/page.md b/app/(docs)/object-mount/media-workflows/final-cut-pro/page.md deleted file mode 100644 index cd0b197ff..000000000 --- a/app/(docs)/object-mount/media-workflows/final-cut-pro/page.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: Object Mount with Final Cut Pro -hideTitle: true -metadata: - title: Object Mount with Final Cut Pro - description: - Recommended workflow and tips for using Object Mount with Final Cut Pro on macOS. ---- - -# Using Object Mount with Final Cut Pro - -This guide outlines how to use **Object Mount** with **Final Cut Pro (FCP)** on macOS. While FCP doesn’t offer the same granular media storage configuration as other NLE software, Object Mount integrates cleanly into your workflow with a few key recommendations. - ---- - -## Best Practices for Cloud-Mounted Media - -Final Cut Pro expects media to be available on fast, local volumes. When using cloud-mounted storage (like Object Mount), you should consider: - -1. **Always import media “without copying”** into the library - - When prompted during import, select: - **Leave files in place** (not “Copy to library”) - - This ensures media is read directly from your Object Mount volume without unnecessary duplication - -2. **Store Libraries locally** - - Final Cut Pro libraries perform best when saved to a high-speed local disk - - We recommend saving your `.fcpbundle` file outside the Object Mount volume - -3. **Use Proxy or Optimized Media for playback** - - FCP supports background generation of proxy media and optimised formats - - Go to **Preferences > Playback**, and set: - - **Playback Media** → **Use Proxy Media (if available)** - - Generate proxies for high-res footage when working remotely - ---- - -## Importing from Object Mount - -1. Mount your cloud bucket using Object Mount -2. Open Final Cut Pro and create or open a library (stored locally) -3. In the **Import Media** window, navigate to your Object Mount volume -4. Select the footage you want, and ensure: - - **Leave files in place** is checked - - (Optional) Enable **Create proxy media** on import for large or high-bitrate files - -{% callout type="note" %} -Object Mount behaves like a local drive in Finder, so you can browse and import as usual — but behind the scenes, data is streamed directly from the cloud. -{% /callout %} - ---- - -## Recommended System Preferences - -- Object Mount’s **data cache** should be enabled (default) -- Place metadata and data caches on fast local storage (e.g. internal SSD) -- For large projects or remote assets, enable `CUNO_OPTIONS = -filePrefetch` via the Preferences tab to accelerate first access - ---- - -## Performance Tips - -- Avoid saving cache or render files inside the Object Mount volume — use local scratch disks -- Enable Final Cut’s **background rendering** for smoother editing -- Monitor network activity via **Activity Monitor > Network tab** to confirm streaming performance - ---- - -## Known Nuances - -- Final Cut Pro doesn’t show granular file I/O errors in the interface — if playback stalls or clips don't link, double-check: - - Network status - - Object Mount activity (log file or mount state) - - File permissions if POSIX mode is enabled -- Finder may take longer to open cloud-mounted folders with thousands of files. Consider using subfolders or proxy workflows for better navigation. - ---- - -## Summary - -Using Object Mount with Final Cut Pro is straightforward when media is imported correctly and libraries are kept local. Proxy workflows and caching can significantly improve responsiveness, especially when working with large projects or slower connections. - -If you're unsure whether a workflow is well suited to cloud-mounted editing, feel free to [reach out to support](https://supportdcs.storj.io/hc/en-us/requests/new). - diff --git a/app/(docs)/object-mount/media-workflows/graphics/page.md b/app/(docs)/object-mount/media-workflows/graphics/page.md deleted file mode 100644 index d2d5ea412..000000000 --- a/app/(docs)/object-mount/media-workflows/graphics/page.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: Object Mount with Graphics Applications -hideTitle: true -metadata: - title: Object Mount with Graphics Applications - description: - Tips and best practices for using Object Mount with Blender, Unity, and Adobe After Effects. ---- - -# Using Object Mount with Graphics Applications - -This guide outlines best practices for using **Object Mount** with common graphics and animation tools such as **Blender**, **Unity**, and **Adobe After Effects**. These applications all expect fast, local file access and Object Mount helps you work directly from the cloud while maintaining a smooth creative workflow. - ---- - -## Blender - -Blender works well with mounted cloud volumes, as long as you’re aware of how it handles linked assets: - -- Save `.blend` files locally for best stability -- Use Object Mount for linked assets such as textures, models, and simulations -- Avoid opening or rendering directly to Object Mount volumes unless using proxy caching -- Use **File > External Data > Pack Resources** before sharing a project across machines - -{% callout type="info" %} -Enable data and metadata caching in Object Mount Preferences to speed up texture and geometry access. -{% /callout %} - ---- - -## Unity - -Unity projects can technically be stored on mounted cloud storage, but this is not recommended for live editing. - -- Keep your Unity project folder on a **local drive** -- Use Object Mount for: - - Asset archives (e.g. large textures, sound libraries, HDRIs) - - Downloadable content folders - - Scripting or automation tools accessing external datasets - -{% callout type="info" %} -If using Unity Cloud Build or version control (e.g. Git + LFS), Object Mount is a great place to store large binary files for later retrieval. -{% /callout %} - ---- - -## Adobe After Effects - -After Effects can read media files directly from Object Mount, but you’ll get the best experience when: - -- Project files (`.aep`) are stored locally -- Cloud storage is used for: - - Footage libraries (video, audio, stills) - - Linked compositions or archive material -- Enable **“Cache frames to disk”** on a fast local drive to avoid render slowdowns - -{% callout type="info" %} -For larger video files, consider using proxies or transcoded formats for previewing from the cloud. -{% /callout %} - ---- - -## Summary - -Object Mount makes it easy to integrate cloud storage into graphics workflows, especially for storing and retrieving large, reusable assets. For best results: - -- Keep project files and cache folders local -- Use cloud-mounted volumes for footage, libraries, textures, and external content -- Enable prefetch and caching settings in Object Mount for faster first-time access - -Need help tuning for your specific creative app? [Reach out to support](https://supportdcs.storj.io/hc/en-us/requests/new) — we’re happy to assist. - ---- diff --git a/app/(docs)/object-mount/media-workflows/media-composer/page.md b/app/(docs)/object-mount/media-workflows/media-composer/page.md index bb4215a8e..0d1edcffd 100644 --- a/app/(docs)/object-mount/media-workflows/media-composer/page.md +++ b/app/(docs)/object-mount/media-workflows/media-composer/page.md @@ -1,89 +1,92 @@ --- -title: Object Mount with Avid Media Composer +title: Avid Media Composer hideTitle: true +docId: eSTEd9V5JbexxYDQ +weight: 2 metadata: title: Object Mount with Avid Media Composer description: Recommended workflow and setup tips for using Object Mount with Avid Media Composer on macOS and Windows. +hidden: false --- # Using Object Mount with Avid Media Composer -This guide walks through how to use **Object Mount** with **Avid Media Composer**, including key setup considerations and workflow tips to ensure smooth media management when working with cloud-mounted storage. +This guide walks you through how to use **Object Mount** with **Avid Media Composer**, including key setup considerations and workflow tips to ensure smooth media management when working with cloud-mounted storage. ---- ## Recommended Setup -Avid Media Composer expects all media to reside on high-performance, locally mounted storage paths. Object Mount allows you to treat object storage as a native volume, but there are a few important things to keep in mind: +Avid Media Composer expects all media to reside on high-performance, locally-mounted storage paths. -1. **Mount to a simple, short file path** - - Avid prefers media directories without long paths or special characters - - Example: `D:\Avid_Mount` or `/Volumes/AvidMedia` +Object Mount allows you to treat object storage as a native volume, but there are a few important things to keep in mind: -2. **Ensure visibility using the `alldrive` Console command** - - From within Avid Media Composer, open the Console from the Tools menu or use the keyboard shortcut of Cntl.(cmd.)+6 - - Type `AllDrives` and press Enter. This makes all mounted volumes visible to Avid +1. **Mount to a simple, short file path** + - Avid prefers media directories without long paths or special characters. + - Example: `D:\Avid_Mount` or `/Volumes/AvidMedia`. -3. **Save Avid project and bin files locally** - - Save your `.avb`, `.avp`, and `.avf` files to a local disk - - Use Object Mount for read access to media assets (e.g., linked clips, raw footage) +2. **Ensure visibility using the `alldrive` Console command** + - From within Avid Media Composer, open the Console from the Tools menu (CTRL+6 / CMD+6). + - Type `AllDrives` and press Enter. This makes all mounted volumes visible to Avid. +3. **Save Avid project and bin files locally** + - Save your `.avb`, `.avp`, and `.avf` files to a local volume. + - Use Object Mount for read access to media assets (e.g., linked clips, raw footage). ---- ## Media and Performance Tips To ensure a smooth editing experience: -- Enable **Object Mount caching** in the Preferences tab -- Store **data and metadata cache** on a fast internal SSD -- If you’re working with large high-resolution files (4K, RAW), enable the following: - - - Set `CUNO_OPTIONS = -filePrefetch` as a custom environment variable - - Increase **S3 connection pool size** in Advanced Settings (e.g., 150–200) - - Use Avid’s **Background Bin Saving** to reduce interruptions + - Enable **Object Mount caching** in the Preferences tab. + - Store **data and metadata cache** on a fast, local volume. + - If you’re working with large, high-resolution files (4K, RAW), enable the following: + - Set `CUNO_OPTIONS = -filePrefetch` as a custom environment variable. + - Increase **S3 connection pool size** in Advanced Settings (e.g., 150-200). + - Use Avid’s **Background Bin Saving** to reduce interruptions. + - **Avoid background virus scanning** or indexing on the mounted path (especially on Windows) -- **Avoid background virus scanning** or indexing on the mounted path (especially on Windows) - ---- ## Proxy & Optimized Workflows -Avid’s DNxHR LB and DNxHD LB proxy workflows are recommended when working with high resolution media on cloud storage. You may want to consider: +Avid’s DNxHR LB and DNxHD LB proxy workflows are recommended when working with high resolution media on cloud storage. -- Using the **Create Proxies** command to generate proxy media and toggle between proxy and high-resolution clips via the **Play button drop menu** -- Or, transcode media to proxy formats such as **DNxHD LB** or **DNxHR LB** and later use **Relink** to conform back to high resolution for finishing +You may want to consider: -In either case, **generating proxies to local storage** is recommended for optimal playback performance, especially when working over slower or high-latency connections. + - Using the **Create Proxies** command to generate proxy media and toggle between proxy and high-resolution clips via the **Play button drop menu**. + - Or, transcode media to proxy formats such as **DNxHD LB** or **DNxHR LB** and later use **Relink** to conform back to high resolution for finishing. +In either case, **generating proxies to local storage** is recommended for optimal playback performance, especially when working over slower or high-latency connections. ---- ## Monitoring Performance Transfers via Object Mount may not show progress in Avid’s interface, so we suggest: -- On **macOS**: Open **Activity Monitor > Network** -- On **Windows**: Use **Task Manager > Performance > Network** -- For detailed diagnostics, set the Object Mount log level to `debug` or `trace` in Preferences + - On **macOS**: Open **Activity Monitor > Network**. + - On **Windows**: Use **Task Manager > Performance > Network**. + - For detailed diagnostics, set the Object Mount log level to `debug` or `trace` in Preferences. {% callout type="info" %} -If Object Mount appears slow when linking large files, it's often due to first-time access from cloud storage. This improves after initial reads thanks to caching and file prefetch. + **Use Patience with Slow Transfers** + + If Object Mount appears slow when linking large files, it can be due to first-time access from cloud storage. + + Performance improves after initial reads, thanks to caching and file prefetch. {% /callout %} ---- ## Known Considerations -- Avid doesn’t natively understand cloud latency so playback may stutter if accessing untranscoded high-resolution files over slower connections -- Ensure the mount path is stable and doesn’t unmount or disconnect mid-session -- POSIX mode can help with permission-sensitive environments, but is not required for typical read workflows + - Avid doesn’t natively understand cloud latency so playback may stutter if accessing un-transcoded high-resolution files over slower connections. + - Ensure the mount path is stable and doesn’t unmount or disconnect mid-session. + - POSIX mode can help with permission-sensitive environments, but is not required for typical read-only workflows. ---- ## Summary -Avid Media Composer can work effectively with Object Mount when media is properly organized and cache settings are optimized. Local project files, proxy workflows, and sensible mount paths all contribute to a faster, more reliable experience. +Avid Media Composer works effectively with Object Mount when media is properly organized and cache settings are optimized. + +Local project files, proxy workflows, and sensible mount paths all contribute to a faster, more reliable experience. -For help tuning your Avid workflow with Object Mount, [contact our support team](https://supportdcs.storj.io/hc/en-us/requests/new) — we’re happy to assist. +For additional help tuning your Avid workflow with Object Mount, [contact our support team](https://supportdcs.storj.io/hc/en-us/requests/new) — we’re happy to assist. diff --git a/app/(docs)/object-mount/media-workflows/page.md b/app/(docs)/object-mount/media-workflows/page.md index 0b83f439b..c9ce567dc 100644 --- a/app/(docs)/object-mount/media-workflows/page.md +++ b/app/(docs)/object-mount/media-workflows/page.md @@ -1,91 +1,73 @@ --- -title: Object Mount in Media Workflows +title: Media Workflow Guides hideTitle: true -weight: 1 +docId: JXF4_NLsV@nh9QcT +weight: 6 metadata: - title: Object Mount in Media Workflows + title: Media Workflow Guides description: - Overview of using Object Mount with creative tools like DaVinci Resolve and Adobe Premiere Pro + Overview of using Object Mount with creative tools like DaVinci Resolve and Adobe Premiere Pro. +hidden: false --- # Object Mount in Media Workflows -Object Mount is designed to make working with cloud storage as seamless as possible — especially in media production environments where performance, compatibility, and simplicity matter. +Object Mount is designed to make working with cloud storage as seamless as possible — especially in media production environments where performance, compatibility, and simplicity matter. -This page gives a quick overview of how Object Mount fits into post-production workflows, and where it works well with tools like **DaVinci Resolve** and **Adobe Premiere Pro**. - ---- +{% youtube-embed videoId="Si__mi8HXPE" /%} -## What is Object Mount? +The articles in this **Media Workflow Guides** section will provide an overview of how Object Mount fits into post-production workflows, as well as configuration tips for tools like **Adobe Premiere Pro**, **Avid Media Composer** and **DaVinci Resolve**. -Object Mount is a native S3 mount tool that lets you treat remote object storage like a local drive. It works with **Storj**, **AWS S3**, and other compatible providers, and is available on **macOS** and **Windows**. -Once mounted, your object storage behaves like any other drive — you can browse, preview, and edit media directly from it using your preferred creative software. +## Adobe Premiere Pro ---- +Premiere Pro users benefit from Object Mount: -## Object Mount Demo + - Projects and source footage can be stored remotely and edited directly. + - Media can be linked and relinked to Object Mount volumes just like local storage. + - Assign proxies and cache files to local disks for the best performance. + - Object Mount volumes appear like regular drives in Finder or File Explorer, making them easy to integrate with Premiere’s media browser and import panels. -{% youtube-embed videoId="Si__mi8HXPE" /%} +See [Using Object Mount with Premiere Pro](docId:fdz5QyWwRqaDFYS9) for complete configuration tips. ---- -## DaVinci Resolve +## Avid Media Composer -Object Mount integrates cleanly into DaVinci Resolve workflows: +Object Mount integrates cleanly into Avid Media Composer workflows: -- Media can be accessed directly from your Object Mount volume -- Stills, cache, and preview files are best kept on local SSD/NVMe to avoid bottlenecks -- Direct I/O should be disabled for the Object Mount path -- Optional settings like proxy mode and smart caching can improve playback on cloud-mounted media + - Preview and scrub footage before importing. + - Store Scratch Disks and Proxies on local, fast volumes. + - Adjust Proxy file formats to improve performance. -See [Using Object Mount with DaVinci Resolve](./media-workflows/davinci-resolve) for the full setup guide. +See [Using Object Mount with Avid Media Composer](docId:eSTEd9V5JbexxYDQ) for full setup and setting guidelines. ---- -## Adobe Premiere Pro +## DaVinci Resolve -Premiere Pro users can also benefit from Object Mount in similar ways: +Object Mount interacts seamlessly with DaVinci Resolve: -- Projects and source footage can be stored remotely and edited directly -- Media can be linked and relinked to Object Mount volumes just like local storage -- Proxies and cache files are recommended to remain on local disks for speed + - Media can be accessed directly from your Object Mount volume. + - Stills, cache, and preview files are fastest when stored on local SSD/NVMe volumes. + - Direct I/O should be disabled for the Object Mount path. + - Optional settings like proxy mode and smart caching can improve playback on cloud-mounted media. -{% callout type="info" %} -Object Mount volumes appear like regular drives in Finder or File Explorer, making them easy to integrate with Premiere’s media browser and import panels. -{% /callout %} +See [Using Object Mount with DaVinci Resolve](docId:xZ5tdyuyYGM8Lrfp) for more details. ---- ## When to Use Object Mount Object Mount is ideal for: -- Remote editing workflows -- Team environments where storage is centralised -- Archival footage access over cloud -- High-speed read access (e.g. proxy editing or frame review) + - Remote editing workflows + - Team environments where storage is centralized + - Archival footage access directly from the cloud + - High-speed read access (e.g.: proxy editing or frame review) -It’s particularly well-suited for **hybrid workflows** where some assets live on fast local disks and others come in from the cloud — without needing clunky manual sync tools or complex workflows. +Object Mount is particularly well-suited for **hybrid workflows** where some assets live on fast local disks, while other content comes in from the cloud — with no need for manual sync tools or complex workflows. ---- ## Final Notes -Object Mount is not a replacement for all local workflows — cache files, renders, and scratch disks are still best kept local. But for accessing and working with cloud-hosted media, it bridges the gap between storage and usability beautifully. - -For app-specific setup steps, visit: - -- [Object Mount with DaVinci Resolve](./media-workflows/davinci-resolve/) -- [Object Mount with Adobe Premiere Pro](./media-workflows/premiere-pro/) - -Others: - -- [Object Mount with Final Cut Pro](./media-workflows/final-cut-pro/) -- [Object Mount with Avid Media Composer](./media-workflows/media-composer/) -- [Object Mount with Graphics Applications](./media-workflows/graphics/) -- [Object Mount with Digital Audio Workstations](./media-workflows/daw/) - ---- - +Object Mount is not a replacement for all local workflows — cache files, renders, and scratch disks are still best kept on local volumes. +But for accessing and working with cloud-hosted media, Object Mount bridges the gap between cloud storage and simplified usability beautifully. diff --git a/app/(docs)/object-mount/media-workflows/premiere-pro/page.md b/app/(docs)/object-mount/media-workflows/premiere-pro/page.md index 4d07600af..55fecf336 100644 --- a/app/(docs)/object-mount/media-workflows/premiere-pro/page.md +++ b/app/(docs)/object-mount/media-workflows/premiere-pro/page.md @@ -1,91 +1,97 @@ --- -title: Object Mount with Premiere Pro +title: Adobe Premiere Pro hideTitle: true +docId: fdz5QyWwRqaDFYS9 +weight: 1 metadata: - title: Object Mount with Premiere pro + title: Object Mount with Adobe Premiere Pro description: - Recommended workflow and setup for editing with Adobe Premiere Pro using Object Mount volumes + Recommended workflow and setup for editing with Adobe Premiere Pro using Object Mount volumes. +hidden: false --- # Using Object Mount with Adobe Premiere Pro -Object Mount makes it easy to browse, import, and edit footage directly from S3-compatible storage in Adobe Premiere Pro. Whether you're working with local proxies, editing remotely, or centralising project media, Object Mount integrates smoothly into your editing pipeline. +Object Mount makes it easy to browse, import, and edit footage directly from S3-compatible storage in Adobe Premiere Pro. + +Whether you’re working with local proxies, editing remotely, or centralizing project media, Object Mount integrates smoothly into your editing pipeline. ---- ## Recommended Setup -Once you've mounted your object storage with Object Mount, your volume will appear like a standard drive in **File Explorer (Windows)** or **Finder (macOS)**. From there, you can work as usual in Premiere Pro with just a few things to keep in mind. +Once you’ve mounted your object storage with Object Mount, your volume will appear like a standard drive in **File Explorer (Windows)** or **Finder (macOS)**. -### 1. Importing Footage +From there, you can work as usual in Premiere Pro with just a few things to keep in mind. -- Go to **File > Import**, or use the **Media Browser** -- Navigate to your **Object Mount** drive and select your files -- Footage can be previewed and scrubbed before importing +**Importing Footage** + - Select **File > Import** (or use the **Media Browser**). + - Navigate to your **Object Mount** drive and select your files. + - Footage can be previewed and scrubbed before importing. {% callout type="info" %} -If you notice slower performance with large files or over remote networks, consider using proxy workflows. + **Large File Transfer Performance** + + If you notice slower performance with large files or over remote networks, consider using proxy workflows (see below). {% /callout %} ---- -## Proxy and Cache Settings (Recommended) +## Recommended Proxy and Cache Settings While Object Mount enables direct-from-cloud editing, Premiere Pro still benefits from local cache for smooth playback and faster timeline scrubbing. -### Suggested settings: +**Suggested Settings:** -- **File > Project Settings > Scratch Disks** - - Set **Media Cache** and **Previews** to a **fast local SSD** +1. **File > Project Settings > Scratch Disks**: + - Set **Media Cache** and **Previews** to a fast, local SSD. -- **Preferences > Media Cache** - - Enable **Auto delete cache files older than X days** to keep things tidy +2. **Preferences > Media Cache**: + - Enable **Auto delete cache files older than X days** to keep things tidy. -- **Create Proxies** via **Ingest Settings** or manually using the **Project panel** - - Store proxies on a local drive - - Set **Proxy File Format** to ProRes Proxy or H.264 Low Res for faster performance +3. **Create Proxies** via **Ingest Settings** or manually using the **Project panel**: + - Store proxies on a local drive. + - Set **Proxy File Format** to “ProRes Proxy” or “H.264 Low Res” for faster performance. -- Enable **Toggle Proxies** in the Program Monitor to switch views on the fly +4. Enable **Toggle Proxies**: + - Navigate to the Program Monitor. + - Set **Toggle Proxies** to switch views on the fly. ---- ## Relinking Media + If a project was opened on a different machine or the mount point has changed: -- Go to **File > Link Media** -- Browse to your Object Mount volume -- Premiere will relink to all matching file paths automatically + - Go to **File > Link Media**. + - Browse to your Object Mount volume. + - Premiere will relink to all matching file paths automatically. + +{% callout type="info" %} + **Ease of File Management** -{% callout type="info"} -Object Mount maintains consistent folder structures across platforms, which makes relinking straightforward. -{% /callout } + Object Mount maintains consistent folder structures across platforms, which makes relinking straightforward. +{% /callout %} ---- ## Performance Tips -- Stick to **List View** in Media Browser when browsing large folders -- Use proxies for multicam or high-resolution editing (e.g. 4K, RAW) -- Avoid saving the Premiere project file directly to the Object Mount drive — store locally and backup to cloud separately if needed +- Stick to **List View** in Media Browser when browsing large folders. +- Use proxies for multicam or high-resolution editing (e.g. 4K, RAW). +- Avoid saving the Premiere project file directly to the Object Mount drive. + - Store locally, and then backup to the cloud separately, if needed. ---- ## Storage Visibility -Your Object Mount volume will show up like this: -- On **macOS**: Under **Locations** in Finder -- On **Windows**: As a mapped drive under **This PC** +Your Object Mount volume will show up: + - On **macOS**: Under **Locations** in Finder. + - On **Windows**: As a mapped drive under **This PC**. -Once mounted, there’s no need to sync or copy files — you’re working live from the source. +Once mounted, there’s no need to sync or copy files — you’re working live from the source in the cloud. ---- ## Reminder on Write Operations Premiere Pro reads from cloud storage efficiently, but for best results: -- Keep render outputs, scratch files, and auto-saves on a local drive -- Use Object Mount for **read access to source footage** -- Export final renders locally, then upload back to your storage if needed - ---- + - Keep render outputs, scratch files, and auto-saves on a local drive. + - Use Object Mount for **read access to source footage**. + - Export final renders locally, then upload back to your storage if needed. diff --git a/app/(docs)/object-mount/page.md b/app/(docs)/object-mount/page.md index 437bc22ef..71a30d316 100644 --- a/app/(docs)/object-mount/page.md +++ b/app/(docs)/object-mount/page.md @@ -1,42 +1,52 @@ --- -title: Object Mount Overview +title: Object Mount +hideTitle: true docId: okai0aiJei9No1Sh -weight: 0 +weight: 1 metadata: title: Object Mount Overview description: Object Mount makes working with object storage feel like using a fast, native file system. It’s POSIX-compatible and built to support new and existing applications without change. +hidden: false --- -**Object Mount** makes working with object storage as smooth and seamless as using a local disk. +Welcome to the Storj Object Mount Documentation! -Whether you're working in media, research, development, or backup workflows, Object Mount provides the speed and flexibility of local storage with the scale of the cloud. +Storj’s Object Mount makes working with object storage as fast and seamless as using a local drive. -With full POSIX compatibility, it lets existing applications interact with data on object storage, without needing to rewrite a thing — just mount and go. +Whether you’re working in media production, research & development, or backup workflows, Object Mount combines the convenience of local storage with the scalability of the cloud. ---- -## Learn More +## What is Object Mount? -- Product features, benchmarks, and case studies: - Visit the [Object Mount product page](https://www.storj.io/fuse-filesystem) +Object Mount is a high-performance mount client that provides seamless, native access to object storage as if it were a fast, local file system. It allows existing applications to interact with object storage content without needing to recode apps or change your workflow — just mount and go. -- Talk to us about licensing, free trials, or whether Object Mount is right for you: - [Book a discovery call](https://www.storj.io/landing/get-in-touch) +| **Feature** | **Benefit** | +| -- | -- | +| **Native Clients** | Object Mount offers native clients for Linux, Windows and macOS. Installs in minutes and can run inside unmodified containers.| +| **Compatible and Seamless** | Run your same, unmodified tools and applications — now with fast access to object storage data. Files and directories on object storage behave the same as they do on a local disk. Fast, transparent, user-friendly.| +| **POSIX Compliant** | Run your Linux tools, applications and workloads as expected — with all the same user and group permissions, symbolic & hard links, etc. | +| **High Performance** | Intelligent caching and high-throughput make editing large files in real-time a breeze. Adobe Premiere Pro, DaVinci Resolve and any other industry standard toolkit all work “out-of-the-box.” Cloud storage is now just as responsive as local storage. | +| **No Proprietary Formatting** | Object Mount keeps your files in their original format without any proprietary conversions. Your data is accessible across workflows, teams, tools, apps and dashboards. No risk of vendor lock-in. | +| **Cost-efficient and Scalable** | By only accessing the bytes you need, Object Mount minimizes download bandwidth and reduces the need for expansive (and expensive) local storage arrays, lowering egress costs and saving you money on every project.| +| **Eco-friendly Collaboration** | Reduce the energy and resource footprint of your Media and Entertainment workflows. Global teams access the same ‘single source of truth’ content without moving, copying or replicating media across multiple, disparate platforms. | -- Need help? - Reach out directly via our [help desk](https://supportdcs.storj.io/hc/en-us/requests/new) ---- +## Where to Start -## Installation Guides +**Learn more** about how Object Mount works in the many **Appendix** articles: -To get started, head over to the relevant setup guide for your platform: +- Object Storage [Core Concepts](docId:xzAqS9Um7xjQSbmD) +- [POSIX Explained](docId:ySneAEd79CVewSSr) -- [Linux Installation](./object-mount/linux/) -- [macOS Installation](./object-mount/macos/) -- [Windows Installation](./object-mount/windows/) +Follow the **Installation Procedure** and **User Guides** for your operating system: -Each guide includes system requirements, setup steps, and optional performance tips to help you get the most out of Object Mount. +- [macOS](docId:apog2ij9jk6f) +- [Windows](docId:TSMB5yXSc4FcqwXj) +- [Linux](docId:wxtofwqcb5f2) ---- +**Optimize your workflows** with Media and Application tips: + +- [Premiere Pro](docId:fdz5QyWwRqaDFYS9) from Adobe +- [Media Composer](docId:eSTEd9V5JbexxYDQ) from Avid +- [DaVinci Resolve](docId:xZ5tdyuyYGM8Lrfp) from Blackmagic Productions diff --git a/app/(docs)/object-mount/release-notes/_meta.json b/app/(docs)/object-mount/release-notes/_meta.json deleted file mode 100644 index df04eac8f..000000000 --- a/app/(docs)/object-mount/release-notes/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Release Notes", "weight": 6} diff --git a/app/(docs)/object-mount/release-notes/page.md b/app/(docs)/object-mount/release-notes/page.md deleted file mode 100644 index 9ab9a4994..000000000 --- a/app/(docs)/object-mount/release-notes/page.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Release Notes -metadata: - title: Object Mount Release Notes - description: - Object Mount Release Notes ---- - -Release notes for Object Mount can be found over on our [GitHub page](https://github.com/storj/object-mount/releases) diff --git a/app/(docs)/object-mount/windows/_meta.json b/app/(docs)/object-mount/windows/_meta.json deleted file mode 100644 index e3b1315db..000000000 --- a/app/(docs)/object-mount/windows/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Windows", "weight": 4} diff --git a/app/(docs)/object-mount/windows/fast-copy-paste/_meta.json b/app/(docs)/object-mount/windows/fast-copy-paste/_meta.json deleted file mode 100644 index 16da49cba..000000000 --- a/app/(docs)/object-mount/windows/fast-copy-paste/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Fast Copy Paste", "weight": 4} diff --git a/app/(docs)/object-mount/windows/fast-copy-paste/page.md b/app/(docs)/object-mount/windows/fast-copy-paste/page.md deleted file mode 100644 index d2b3a1ec1..000000000 --- a/app/(docs)/object-mount/windows/fast-copy-paste/page.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Fast Paste on Windows -docId: fast-copy-paste -hideTitle: true -weight: 7 -metadata: - title: Fast Paste on Windows - description: - A semi-technical explanation of how Object Mount’s “Fast Paste” feature works and why it’s faster than standard Windows file operations. ---- - -# Fast Copy + Paste on Windows - -**Fast Copy + Paste** is feature in Object Mount that significantly improves file copy performance when pasting data into an Object Mount mounted drive. This page walks through how to leverage this feature on Windows. - -Once you have copied a file, you’ll see this option when you right-click on a folder within an Object Mount volume, look for the menu item **Fast Paste here** with a white and blue cloud icon. - -{% callout type="info" %} -You may need to press **Show more options** on the right-click menu to see this option! -{% /callout %} - ---- - -## How It Works - -When you use standard Windows copy/paste (Ctrl+C then Ctrl+V, or right-click Copy + right-click Paste), Windows uses its own built-in file I/O logic. It handles cloud storage just like any other drive, moving files one block at a time, often with less awareness of cloud-specific behaviour like multipart uploads, data layout, or server-side copy capabilities. - -**Fast Paste**, by contrast, uses a custom, optimised transfer path built into Object Mount. It directly: - -- Reads the files from the Windows clipboard using native Windows APIs -- Bypasses generic shell copy routines -- Streams files into the mount using **multipart-aware**, **concurrent** upload operations tuned for S3-compatible storage - ---- - -## Why It’s Faster - -| Feature | Standard Paste | Fast Paste in Object Mount | -|-----------------------------|------------------------|----------------------------------| -| Uses Windows shell I/O | ✅ Yes | ❌ No | -| Streams via S3 multipart | ❌ No | ✅ Yes | -| Uploads in parallel | ❌ Single-threaded | ✅ Parallel with tuning options | -| Respects Object Mount cache | ❌ Not aware | ✅ Integrated | -| POSIX metadata support | ❌ No | ✅ Optional | - ---- - -## When to Use It - -Fast Paste is ideal when: - -- Copying large files (e.g. media assets, backups) -- Copying folder(s) with lots of files -- Copying content already inside a Object Mount directory -- Pasting **from** or **into** a mounted S3 bucket or Storj bucket -- Using slow or high-latency networks where traditional paste stalls - -{% callout type="info" %} -If you’ve copied files locally and are pasting them into an Object Mount drive, **Fast Paste** ensures they reach the cloud using the most efficient method available. -{% /callout %} - -{% callout type="info" %} -If you want to copy a whole file from your Object Mount drive, **Fast Paste** ensures they pull from the cloud using the most efficient method available. -{% /callout %} - ---- - -## Behind the Scenes - -Fast Paste integrates tightly with: - -- Object Mount’s data cache (if enabled) -- Multipart thresholds defined by the mount configuration -- Active logging (you’ll see Fast Paste operations noted in **log files**) - -It avoids unnecessary disk I/O and background indexing that can affect standard clipboard operations on cloud-mapped drives. - ---- - -## Related Settings - -To further tune Fast Paste performance, adjust: - -- [Performance Tuning](../faq/tuning/) - ---- diff --git a/app/(docs)/object-mount/windows/features/page.md b/app/(docs)/object-mount/windows/features/page.md new file mode 100644 index 000000000..a0f3a5f4e --- /dev/null +++ b/app/(docs)/object-mount/windows/features/page.md @@ -0,0 +1,150 @@ +--- +title: Feature Guide +hideTitle: true +docId: xMKcaWrTVK6QMD8h +redirects: + - /object-mount/windows/pinning +weight: 3 +metadata: + title: Feature Guide for Windows + description: + Details on unique features of the Windows native app. +hidden: false +--- + +# Feature Guide: Windows Native App + +This guide provides details on unique and noteworthy features that are specific to the **Windows Native App** of Object Mount. + +Features that are global and apply to **all OS versions** of Object Mount (macOS, Windows, Linux) can be found in the [Global Features](#global-features-and-concepts) section below. + + +## Fast Paste on Windows + +**Fast Paste** provide a much faster way to upload files to your object storage — ideal when working with large media assets or entire folder trees of content. + +To use **Fast Paste**: + +1. Copy a file or folder from your local drive as normal: using **CTRL+C** or **right-click + Copy**. + +2. Navigate to an Object Mount drive and directory. + +3. Right click and choose the **Fast Paste here** menu option (shown with a white and blue Object Mount icon). + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-fast-paste.jpg) + +### When to Use Fast Paste + +Using Fast Paste is ideal when you want to: + +- Move large files (media footage, projects, renders, etc.) or large folder trees from a local disk to a mounted S3 bucket +- Copy large files or folders from one Object Mount volume to another Object Mount volume +- Improve performance on high-latency networks (or slow remote S3 endpoints) +- Avoid the bottlenecks of Windows’ standard file operations + +### How Fast Paste Works + +Standard **copy and paste** actions performed by Windows (Ctrl+C/Ctrl+V, or right-click + Copy/Paste) invokes Windows’ built-in file I/O calls. These standard file operations copy data **one block** at a time, without any awareness of how cloud storage works. This can lead to slow performance — especially for large files or nested folder structures. + +**Fast Paste** bypasses the standard OS-based write calls and instead: + +- Reads the files from the Windows clipboard using native Windows I/O calls +- Streams files up to the mount using **multipart-aware** and **concurrent** upload operations tuned for S3-compatible storage + +A live **progress indicator** will be displayed when you use Fast Paste: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-fast-paste-progress.jpg) + +Additionally, **Fast Paste**: + +- Respects your mount settings (e.g.: POSIX mode, cache location) +- Applies performance and tuning options (cache settings & location, multipart thresholds, etc.) +- Logs operations to the standard Object Mount log file + +### Use Cases for using Fast Paste + +- Store original media locally, then copy and **Fast Paste** it to cloud when ready. +- For large archive folders, use copy then **Fast Paste** to quickly duplicate them into your S3 bucket. +- Use **Fast Paste** to free up local storage when archiving to the cloud. + + +## File Pinning on Windows + +Starting with **Object Mount v1.0.5**, you can now **pin files** in Windows to ensure they remain in the **local Object Mount cache**. + +This improves performance and is especially useful for: +- Preventing delays when reopening large media assets +- Ensuring critical project files stay cached locally + +### What Does File Pinning Do? + +When you **pin a file**, Object Mount: + - Downloads and stores the file in Object Mount’s local data cache + - Prevents the file from being _evicted_ due to cache size limitation settings + - Flags the file for _persistent caching_, even across restarts + +{% callout type="info" %} + **Windows File Pinning** + + Pinning does *not* duplicate or move the file elsewhere. It simply marks the file as _persistent_ in the Object Mount local cache. + + **Important:** Pinned files _do not remain accessible_ when Object Mount is off-line, or when your mount is disabled or disconnected. An active, live connection is still required for access, even when a file is pinned. +{% /callout %} + +### Pinning Files + +To Pin a file: + +1. Right-click any file _stored within_ an **Object Mount volume**. + +2. Select the **Pin selected item** menu option (shown with a white and blue Object Mount icon). + +3. A small icon overlay in File Explorer will indicate the file is pinned: 🟢 + +To Unpin a file: + +1. Right-click the file and choose **Unpin selected item**. + +**Note:** Currently you can only pin individual _files_, not entire _folders_. + +Pinned files can be identified by: + - A visual “green dot” 🟢 overlay indicator when viewed in **File Explorer** + - Their status in the **Object Mount logs** + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-pinned-file-marker.jpg) + + **Note:** A restart may be necessary before you will see the “pinned file” indicator. + +### Pinned Files & Cache Space + +- Pinned files do _not_ count toward the maximum data cache size set in the Settings tab of Object Mount. +- However, pinning requires that the **Data Cache** be enabled. + - If the Data Cache is disabled, the pinning menu items will no longer be displayed. + - When the Data Cache is re-enabled, any previously pinned files will retain their pinned status. + +{% callout type="warning" %} + **Clearing the Cache** + + If you click **Clear Caches** in the Settings tab of Object Mount, all pinned files will be removed from the cache and will revert to an “unpinned” status. +{% /callout %} + +### Best Practices for Using Pinning + +Use Pinning to: + - Keep critical assets readily available without re-downloading + - Avoid cache eviction for large files you frequently revisit + - Preload media for use in slow-bandwidth environments + + +## Global Features & Concepts + +See the many articles in the [Appendix](docId:xzAqS9Um7xjQSbmD) for additional details on features that apply to Object Mount on _all_ operating systems (macOS, Windows, and Linux): + +- Object Storage Core Concepts +- POSIX explained +- S3 Credential Management +- Supported S3 Providers +- Performance Tuning +- Troubleshooting Guide +- FAQ +- Release Notes diff --git a/app/(docs)/object-mount/windows/installation/page.md b/app/(docs)/object-mount/windows/installation/page.md index b7afc0821..2a68cde05 100644 --- a/app/(docs)/object-mount/windows/installation/page.md +++ b/app/(docs)/object-mount/windows/installation/page.md @@ -1,56 +1,64 @@ --- -title: Installation -weight: 1 +title: Installation Guide hideTitle: true - +docId: VFmmZcEdcxpZZCx5 +weight: 1 metadata: - title: Windows Native Installation + title: Windows Native Installation Guide description: Details about installing Object Mount on Windows +hidden: false --- -# Windows Native Installation Guide +# Installation Guide: Windows Native App -This section covers how to install the app on Windows using the standard `.exe` installer. The process is quick and straightforward, with just a couple of setup steps to ensure everything runs smoothly on your Windows machines. +This section covers how to install the the Object Mount native application for Windows using a standard `.exe` installer. ---- ## System Requirements -Before you begin, make sure your system meets the following: +Before you begin, make sure your PC meets the following system requirements: -- Windows 10 or Windows Server 2016, and newer +- Windows 10 or later (or Windows Server 2016 or newer) +- Intel x86_64 or AMD processor (no ARM CPU support at this time) - At least 300MB of free disk space - Internet connection for mounting cloud-hosted buckets, or LAN connection for mounting local buckets ---- - -## Step 1: Download the Installer +{% callout type="note" %} + **S3 Provider Support:** -Download the latest `.exe` file. + Object Mount for Windows fully supports S3 container services from Amazon Web Services, Storj, and other fully S3-compatible Object Storage providers. + + - See the [Supported S3 Providers](docId:xCjeasn8SLQn2vfd) article in the Appendix for a complete S3 provider list. -{% callout type="info" %} -If your organisation restricts installer downloads, you may need to right-click the file and choose **Unblock** before running. + **Azure & GCS Support:** + + - Support for _Native_ Microsoft Azure Storage and _Native_ Google Cloud Storage services are still under development in Object Mount for Windows. + - For Google Cloud access: Using GCS _S3_ is recommended instead of using GCS _S3 Gateways_. + - Native Azure and Native GCS (S3 Gateways) should be considered “experimental” at this time. {% /callout %} ---- -## Step 2: Install the App +## Step 1: Download the Latest Installer -Adminstrative permissions are required to install some of the necessary components. +Download the latest Object Mount `.exe` Windows installer program. -1. Double-click the `.exe` file to start the installation -2. Follow the prompts in the installer -3. When prompted, allow the app to make changes (click **Yes**) +**Note:** Contact your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) if you need access to the most up-to-date installer. ---- -## Step 3: Launch the App +## Step 2: Install the Object Mount App -Once installed: +**Note:** Administrative permissions are required to install some of the Object Mount components. -- Open **Object Mount** from your Start Menu or Desktop shortcut -- You will be prompted to enter your **Licence Key** automatically -- Windows may prompt you to allow firewall access — click **Allow Access** +1. Double-click the setup `.exe` program to start the installation. ---- +2. Follow the prompts to complete the installation process. + + +## Next Steps + +Proceed to the Windows [User Guide](docId:khHGfZsyY9NJ2uGK) to **activate** your Object Mount installation, **configure** your object storage credentials, and **create** your first mount. + +Then review the Windows OS-specific [Feature Guide](docId:xMKcaWrTVK6QMD8h). + +Finally, read through usage tips for your specific [Media Workflows](docId:JXF4_NLsV@nh9QcT) and Applications. diff --git a/app/(docs)/object-mount/windows/page.md b/app/(docs)/object-mount/windows/page.md index e5acde0ea..343b84c0d 100644 --- a/app/(docs)/object-mount/windows/page.md +++ b/app/(docs)/object-mount/windows/page.md @@ -1,35 +1,40 @@ --- title: Windows hideTitle: true -docID: veoobbmaya4z +docId: TSMB5yXSc4FcqwXj weight: 4 metadata: title: Windows Installation and Usage description: - Step-by-step guidance for installing and using the native Object Mount app on Windows. + Home Page for guidance on installing and using the native Object Mount app on Windows. +hidden: false --- # Windows Installation and Usage -This section covers how to install and use the native **Object Mount** app on Windows. +This section contains everything you need to install and use the native **Object Mount** app on Microsoft Windows. -Whether you’re setting up a single workstation or deploying across a team, these pages will walk you through the key steps for installation, configuration, and daily use with cloud storage. +Whether you’re setting up a single workstation or deploying across a team, these pages will walk you through installation, configuration and day-to-day use. ---- ## What’s Covered - Native installer and system requirements -- Guidance for mounting, logging, and configuration -- Tips for media workflows and storage performance +- User guides for mounting, configuring, and managing your object storage +- Tips and recommendations for unique Windows features, media applications, and performance improvements ---- -## Related Pages +## Next Steps -- [Windows Installation Guide](./windows/installation) -- [Windows User Guide](./windows/user-guides) +Start with the **Installation Guide**: -Start with the installation guide to get Object Mount up and running, then explore the user guide for tips on storage access, logging, and daily use. +- Windows [Installation Guide](docId:VFmmZcEdcxpZZCx5) ---- +Move on to the **User Guide** for instructions on creating mounts and working with object storage in your daily workflows: + +- Windows [User Guide](docId:khHGfZsyY9NJ2uGK) + +Lastly, review the Windows OS-specific **Feature Guide** and the appropriate **Media Workflow Guide**: + +- Windows [Feature Guide](docId:xMKcaWrTVK6QMD8h) +- [Media Workflow Guides](docId:JXF4_NLsV@nh9QcT) diff --git a/app/(docs)/object-mount/windows/pinning/_meta.json b/app/(docs)/object-mount/windows/pinning/_meta.json deleted file mode 100644 index 75975392d..000000000 --- a/app/(docs)/object-mount/windows/pinning/_meta.json +++ /dev/null @@ -1 +0,0 @@ -{"title": "Windows Pinning", "weight": 8} diff --git a/app/(docs)/object-mount/windows/pinning/page.md b/app/(docs)/object-mount/windows/pinning/page.md deleted file mode 100644 index 37ca111be..000000000 --- a/app/(docs)/object-mount/windows/pinning/page.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Pinning on Windows -docId: pinning-windows -hideTitle: true -weight: 8 -metadata: - title: Pinning on Windows - description: - Explains how to use the file pinning feature in Object Mount for Windows to keep files cached and accessible offline. ---- -# Pinning on Windows - -As of **Object Mount v1.0.5**, you can **pin files** on Windows to ensure they remain in your local cache. - -This is especially useful for: - -- Preventing delays when reopening large media assets -- Ensuring critical project files stay cached locally - ---- - -## What Does Pinning Do? - -When you **pin a file**, Object Mount: - -- Downloads and stores the file in the local data cache -- Prevents it from being evicted due to cache size limits -- Flags it for persistent caching even across restarts - -{% callout type="info" %} -Pinning does *not* duplicate or move the file elsewhere. It simply marks the file as persistent in cache. -{% /callout %} - -{% callout type="warning" %} -**Important:** Pinned files **do not remain accessible** when Object Mount is offline or disconnected. A live connection is still required for access, even if the file is cached. -{% /callout %} - ---- - -## How to Pin a File - -1. Right-click any file inside an **Object Mount volume** -2. Select **"Pin to Object Mount"** from the context menu -3. A small icon overlay in File Explorer will indicate the file is pinned (✓) - -To **unpin**, right-click the file again and choose **"Unpin from Object Mount"**. - -> You can currently pin individual files, not entire folders. - ---- - -## How to Tell if a File Is Pinned - -Pinned files can be identified by: - -- A visual overlay icon in **File Explorer** -- Their status in the **Object Mount logs** and future GUI updates - ---- - -## Managing Cache Space - -- **Pinned files do *not* count toward the cache size limit** set in Preferences -- However, pinning requires the **Data Cache** to be enabled -- If you **clear the Data Cache**, all pinned files will be removed from disk and lose their pinned status - -To retain pinned files: - -- Keep the **Data Cache** enabled under - **Preferences → Data Cache** - ---- - -## Best Use Cases - -- Keeping critical assets readily available without re-download -- Avoiding cache eviction for large files you frequently revisit -- Preloading media for use in bandwidth-limited environments (while online) - ---- - -## Related Settings - -See also: - -- [Performance Tuning](../faq/tuning/) diff --git a/app/(docs)/object-mount/windows/user-guide/page.md b/app/(docs)/object-mount/windows/user-guide/page.md new file mode 100644 index 000000000..20aa98ba7 --- /dev/null +++ b/app/(docs)/object-mount/windows/user-guide/page.md @@ -0,0 +1,187 @@ +--- +title: User Guide +hideTitle: true +docId: khHGfZsyY9NJ2uGK +redirects: + - /object-mount/credentials/windows/import +weight: 2 +metadata: + title: Object Mount User Guide for Windows + description: + A user guide for getting started with Object Mount on Windows. +hidden: false +--- + +# User Guide: Windows Native App + +This guide walks you through the core steps to get started with **Object Mount** on Windows, including: activation, adding credentials, and mounting your cloud-based object storage as a local drive. + + +## Step 1. Launch & Activate Object Mount + +- Run Object Mount from the Start Menu or Desktop shortcut. + +- You will be prompted to enter your Object Mount **License Key**. + - Contact your 🌐 [Storj Representative](https://www.storj.io/landing/get-in-touch) if you need to obtain an Object Mount License Key. + +- Click **Install New License** and paste your Object Mount license key into the text box. + +- Click **Activate**. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-license-key-valid.jpg) + +- If prompted, click **Allow Access** to permit firewall changes. + +Once activated, you can view your license key details in the **About** tab within Object Mount. + + +## Step 2. Configure S3 Credentials & Create a Mount + +Object Mount can connect to object storage from Storj as well as other major cloud storage providers. Standard S3-compatible credentials from your cloud storage provider are required. + +### Obtain your Object Storage Credentials + +- See the Appendix article [](docId:E4NhE5kPdjURRajJ#obtaining-your-s3-credentials) for instructions. + +### Enter Your Credentials and Create a Mount + +There are two easy paths to create a mount — both paths achieve identical results and both require your object storage credentials: + +- Click **Create New Mount** from the **Mounts** tab, or +- Click **Add New Credentials** from the **Credentials** tab +- Then select and configure one or more buckets to mount as a local drive + +Follow the steps below to create a new mount using the **Add New Credentials** path: + +1. Select the **Credentials** tab in Object Mount. + +2. Click the **Add New Credentials** button. + - Give the credential set a clear name + - Choose your provider tab (e.g.: Storj, AWS, etc.). + - Enter the **Access key ID**, **Secret key**, and **Endpoint** (or Region, if applicable) that you obtained earlier. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-new-credentials.jpg) + + {% callout type="info" %} + **S3 Compatible Providers** + + If your provider is not listed, select **Other S3** and choose the provider from the dropdown. + + - Ensure the endpoint is accessible from the Internet. + - Some providers may require extra compatibility settings like a defined region. + + Below is an example of providing an additional S3 Compatibility Setting for region: + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-mac-custom-s3-region.png) + {% /callout %} + +3. Click **Next**. + - Object Mount will authenticate to your cloud storage provider and attempt to discover any buckets your credentials have access to. + +4. Select **Mount all buckets** or choose one or more specific buckets to create a mounted drive for. Click **Next**. + +5. Enter a **Display name** for your mount. + - Make note of the drive letter that Windows initially assigns to the new mount. + - If this drive letter is in use when you enable your mount, you will be prompted to select a new drive letter (see step 8, below). + - **Note:** You can also set a **Preferred drive letter** to use under the **Settings** tab. + +6. Tick the **Mount as read-only** box if all you need is to browse and copy content. + - Uncheck the box to enable **full read/write access**. + +7. Tick the box to **Enable POSIX metadata** (recommended). + - For more details, see the Appendix article: [POSIX Explained](docId:ySneAEd79CVewSSr). + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-configure-your-mount.jpg) + +8. Click **Next** to create the mount. Then enable the mount from the **Mounts** tab. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-enable-your-mount.jpg) + + - If the drive letter assigned by Windows is already in use, you will be prompted to select a new drive letter: + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-drive-assignment.jpg) + +**Note:** First-time access to a new mount may take a few moments if the bucket contains many files — performance improves after the initial mount. + + +## Newly Mounted Drive & Buckets + +Your newly mounted drive will appear on your PC using the assigned drive letter: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-new-drive-01.jpg) + +If there are multiple buckets accessible using the same credential set, those buckets will each appear as separate **folders** within the single mounted drive letter: + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-new-drive-02.jpg) + + +## Preferences & Advanced Settings + +Object Mount includes a flexible **Settings** panel where you can fine-tune caching, logging, and performance settings across all mounts. + +{% callout type="info" %} + **Settings** + + - These settings are global and apply to all mounts. + - These settings cannot be edited while a mount is active. Disable all mounts before making changes. +{% /callout %} + +Refer to the following sections for details on each setting: + +### General + +Select a **Preferred drive letter** for Object Mount to use. + +- **Note:** Object Mount only allows for a single mount to be active at any time. This preferred drive letter will be used for that one, active mount. + +### Data Cache + +The local **Data Cache** temporarily stores small portions of accessed cloud files on your local disk — useful for repeated or random access operations. + +- **Location:** The local path where data cache files are stored +- **Maximum cache size:** The upper limit (in MB) for total cached data + - **Note:** This feature does *not* delay cloud writes — writing to the cloud happens immediately. + + ![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-settings-01.jpg) + +{% callout type="info" %} + **Data Cache Recommendations** + + - Use a fast local drive (SSD or NVMe) for best caching performance. + - If the connection to your object storage is on a high-speed, low-latency network, you may see improved performance by turning caching _off_. +{% /callout %} + +### Metadata Cache Settings + +Metadata caching helps Object Mount load and retain file structure, timestamps, and permissions more efficiently. + +- **Location:** The local directory for storing metadata information +- **Cache timeout:** The duration, in seconds, that metadata remains valid before being refreshing from the cloud (default: 60 seconds) + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-settings-02.jpg) + +### Advanced Settings + +These options let you further customize Object Mount’s performance and diagnostic behavior. + +| **Setting** | **Description** | +|--------------------------|-----------------| +| **S3 connections limit** | Controls the number of concurrent S3 operations (default: 100). +| **Log level** | Adjusts logging detail. Options: `error`, `warning`, `debug`, or `trace`. +| **Single threaded mode** | Runs the filesystem in single-threaded mode (rarely needed; leave `off` unless instructed by Storj Support). +| **Debug logging** | Enables extensive debug output to assist with support issues. +| **Custom Environment Variables** | Custom variables to modify Object Mount’s behavior, add variables when recommended by Storj Support + +![](https://link.us1.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/om-docs/om-win-settings-03.jpg) + +{% callout type="info" %} + **Log File Location** + + On Windows the default location for storing the log file is: + + `C:\Users\%username%\AppData\Local\Object Mount\cunofs.log`. +{% /callout %} + +Click **Apply** to save any changes. Then re-enable your mount. + +Click **Reset** to revert all settings back to their default values. diff --git a/app/(docs)/object-mount/windows/user-guides/page.md b/app/(docs)/object-mount/windows/user-guides/page.md deleted file mode 100644 index 39f8102c2..000000000 --- a/app/(docs)/object-mount/windows/user-guides/page.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Windows User Guide -hideTitle: true -docId: windows-user-guide -redirects: - - /object-mount/credentials/windows/import -weight: 1 -metadata: - title: Object Mount Windows User Guide - description: - A simple one-page user guide for getting started with Object Mount (formerly cunoFS) on Windows. ---- - -# Object Mount Windows User Guide - -This guide walks you through the essential steps to get started with **Object Mount** on Windows — including activation, adding credentials, and mounting object storage as a local drive. - ---- - -## 1. Initial Setup & Licence Activation - -The first time you launch Object Mount, you’ll be prompted to enter your **licence key**. This should have been emailed to you at the time of setup. - -- Enter the activation key when prompted -- You can verify your licence at any time via the **About** tab, which shows your current licence details - ---- - -## 2. Importing Cloud Credentials - -Object Mount connects to most cloud providers using standard **S3-compatible credentials**. - -### Steps: - -1. Navigate to the **Credentials** tab -2. Click the green **Import new credentials** button -3. In the popup window: - - Give the credential set a clear, recognisable name - - Select the appropriate tab for your cloud provider (e.g., AWS, Storj, Backblaze) - - Fill in your access key, secret, and any required region or endpoint - -{% callout type="info" %} -If you're using an S3-compatible provider like Wasabi, choose the **S3-compatible** tab and select the provider from the dropdown. Make sure the endpoint is accessible from your system. -{% /callout %} - -Storj users: Use **lexicographically ordered buckets** for best listing performance. - -4. Click **Import** to verify the credentials and discover available buckets -5. Select one or more buckets to begin creating mount configurations - ---- - -## 3. Creating and Managing Mounts - -After credentials have been successfully added: - -1. Go to the **Mounts** tab -2. Click **Create new mount** -3. Choose: - - The bucket to mount - - The local drive letter to assign - - Whether to enable **POSIX mode** (for permission support) - - Any optional caching or mount parameters - -4. Click **Add Mount** to save -5. Back in the Mounts view, toggle the switch to **enable** the mount -6. Click the mount name to open it in **File Explorer** - -{% callout type="info" %} -On first access, loading may take a moment — especially for large buckets. Subsequent access will be faster thanks to internal caching. -{% /callout %} - ---- - -## 4. Preferences & Advanced Settings - -Object Mount includes a flexible **Preferences** panel where you can fine-tune caching, logging, and performance settings across all mounts. These settings are global and apply to every active mount. - -### Data Cache - -The **data cache** temporarily stores small portions of accessed files on your local disk — useful for repeated or random access operations. - -- **Location**: The local path where data cache files are stored -- **Maximum cache size**: The upper limit (in MB) for total cached data -- **Note**: This does *not* delay cloud writes — writing to the cloud happens immediately - -{% callout type="info" %} -Use a fast local drive (SSD or NVMe) for best caching performance. -{% /callout %} - -{% callout type="info" %} -If you are on a high-speed, low-latency network connected to your object storage, you may see improved performance by turning caching off. -{% /callout %} - -### Metadata Cache - -Metadata caching helps Object Mount remember file structure, timestamps, and permissions more efficiently. - -- **Location**: The local directory for storing metadata information -- **Cache timeout (s)**: The duration metadata remains valid before refreshing from the cloud (default: 60 seconds) - -### Advanced Settings - -These options let you further customise Object Mount's performance and diagnostic behaviour: - -| Setting | Description | -|-------------------------------------|-------------| -| **S3 Connections Limit** | Controls the number of concurrent S3 operations (default: 100) | -| **Log Level** | Adjusts logging detail. Options: `error`, `warning`, `debug`, or `trace` | -| **Single Threaded Mode** | Runs the filesystem in single-threaded mode (rarely needed; leave off unless instructed by Support) | -| **Debug Logging** | Enables detailed debug output to assist with support issues | -| **Environment Variables** | Custom variables to modify Object Mount behaviour, adding variables may be recommended by Support | - -{% callout type="info" %} -**Log File Location** -`C:\Users\%username%\AppData\Local\Object Mount\cunofs.log` -{% /callout %} - -Click **Reset** to revert changes to default settings. - ---- - -## More Info - -- [Performance Tuning](../faq/tuning/) -- [POSIX mode explained](../faq/posix-mode/) - ----