Usage
dvc supports different remote storage site configurations that work with the most common providers.
Following the dvc http remote specification, the Imaging Platform implements a custom remote that can be used just like any other remote configuration. The special thing is that the platform is not actually storing anything, but acting as a proxy that forwards the requests to the desired remote, providing the necessary authorization for the user.
The advantage of this approach is that users can easily provide authorization based on GitLab tokens and administrators can control access and rotate keys by simply updating GitLab repositories.
Note that the platform cannot display data outside the proxy storage. For this reason is mandatory to add this configuration for the platform to work properly.
Register real storage sites
Note: Azure is the only configuration support at the time of writing.
From the storage site view you can register a storage site to be proxied by the platform. Depending on the provider the platform will require a different set of parameters following the dvc remote documentation. The process will create a git repository on GitLab that holds the provided configuration. If you need to modify it, you can do it directly on this repository.
It is important to note that the registered storage must exist. For example, if you register an Azure container, the container must exist.
This example registers a container named jisap-test under the jisap account.
The connection string must be an access key with account name. It can be easily obtained from “Azure > Settings > Access Keys”. Here are the mandatory fields.
DefaultEndpointsProtocol=https;
AccountName=jisap;
AccountKey=accountkey;
EndpointSuffix=core.windows.net
SAS keys are not support at the moment
Use storage site for a dataset
Once the storage site is registered, you will be able to find it in the storage site list. If you go to a dataset without storage site configuration, you will see a section that prompts you to select the desired storage site. You can click on the desired one and this operation will create a new commit on your repository adding the proxy configuration.
Access control
The platform handles access to the storage site based on the GitLab permissions. This means that an user must have access to the storage site repository in order to upload/download files from it.
Also, you can use a project access token to access to the storage site from CI/CD pipelines. This can be obtained from project settings on GitLab. This token will allow to upload/download data from the storage site. It requires API permission.