This is the command line tool for execution on the Linux or Windows terminal – all parameters and functions are available for execution on-demand or in automated scripting environments, e.g. CI/CD pipelines.
This is the online version, which makes standard functions directly available within the web.
Go to releases. The executable binary file jisap-cli (for Linux) is linked under the section “Assets” below the heading “Other”. Alternatively, use curl to download the binary, then move it to /bin, and mark it as executable.
$ repo_token="xyz"
$ sudo curl -ks -H "cookie: tokenGitLab=$repo_token" "https://jisap.basf.net/api/v1/datasets/9036/files/build/jisap-cli" --output /bin/jisap-cli
$ sudo chmod +x /bin/jisap-cli
$ jisap-cli
A cli tool to ease integration with the JISAP AI Workbench.
Usage:
jisap-cli [command]
...
Go programs can be compiled and build for all common operating systems and architectures. Create an Gitlab issue in this project, and give some details on the intended operating system and architecture. The build scripts can then easily be extended.
$ jisap-cli
A cli tool to ease integration with the JISAP AI Workbench.
Usage:
jisap-cli [command]
Available Commands:
comment Publish a comment to the current commit
create-tar Downloads project files and creates tar (at stdout)
csv2md Convert CSV content to markdown table format
dvc-metrics-log List commit info and related dvc metrics
help Help about any command
list-datasets Print list of JISAP datasets
list-deployments List project deployments
list-files Print list of (DVC-managed) files in a JISAP project
list-projects Print list of JISAP projects
list-releases List project releases
list-templates Print list of JISAP templates
md2html Converts markdown to html
md2md Corrects spacing of markdown table cells
open-port (not yet implemented) Create port-mapping, and kubernetes/docker namespaces at App Store
process-template injects environment variables and/or Gitlab CI/CD variables
publish Publish an image or file to the platform
release Creates a new release in GitLab
smb-dvc-sync Syncs files in a Samba share to a dvc tracked folder
version Print version info
Flags:
-h, --help help for jisap-cli
Use "jisap-cli [command] --help" for more information about a command.
If a command takes Gitlab project ID(s) as input, or otherwise needs to access the Gitlab API, the environment variables CI_SERVER_URL (e.g. https://gitlab.roqs.basf.net) and repo_token (a personal Gitlab token) need to be defined. An hopefully error message is printed, if an requirement for execution is not met.
Use jisap-cli comment in your .gitlab-ci.yml
Use jisap-cli release in your .gitlab-ci.yml
The command jisap-cli process-template is used to inject Gitlab CI/CD variables and/or environment variables into a template.
If a Gitlab project ID is provided, the environment variables CI_SERVER_URL (e.g. https://gitlab.roqs.basf.net) and repo_token (a personal Gitlab token) need to be set. This is true for all JISAP-CLI commands that utilize the Gitlab API. An hopefully error message is printed, if an requirement for execution is not met.
The following examples illustrate, how it works:
$ jisap-cli process-template --environment
My home directory is: {{ .HOME}} # Enter this text
My user name is: {{ .USER }} #
The current directory is: {{ .PWD }} #
# Ctrl+D
My home directory is: /home/klukasc
My user name is: klukasc
The current directory is: /home/klukasc/git/jisap-cli
$ jisap-cli process-template --project-id 9036
App name: {{ .CI_APP_NAME }} # Enter this text
Gateway port: {{ .CI_APIGATEWAY_PORT }} #
# Ctrl+D
App name: JISAP-CLI
Gateway port: 32583
Inject CI/CD variables into a given template file:
$ cat template.txt
App name: {{ .CI_APP_NAME }}
Gateway port: {{ .CI_APIGATEWAY_PORT }}
$ cat template.txt | jisap-cli process-template --project-id [gitlab project ID]
App name: JISAP-CLI
Gateway port: 32583
$ jisap-cli process-template --debug --project-id 9036 # debug prints project variables to StdErr
CI_APIGATEWAY_PORT=32583
CI_APP_NAME=JISAP-CLI
CI_HAPPYPOTTER_TEMPLATES=flask
CI_KUBERNETES_NAME=jisap-cli
CI_KUBERNETES_SERVICE_ACCOUNT_DEV=jisap-cli-dev-ci
CI_KUBERNETES_SERVICE_ACCOUNT_PROD=jisap-cli-prod-ci
CI_KUBERNETES_SERVICE_ACCOUNT_QUAL=jisap-cli-qual-ci
CI_KUBERNETES_TOKEN_DEV=*** masked ***
CI_KUBERNETES_TOKEN_PROD=*** masked ***
CI_KUBERNETES_TOKEN_QUAL=*** masked ***
CI_REGISTRY_NAMESPACE=jisap_cli
CI_REGISTRY_TOKEN=*** masked ***
CI_REGISTRY_USER=jisap-cli
repo_token=*** masked ***
# Ctrl+D - no input is provided, thus no output will be printed to StdOut
This command is used in templates .gitlab-ci.yml files to inject CI/CD variables into the files deployment.yml and Docker. If the parameter --debug is added, the complete list of environment variables and/or CI/CD Gitlab project variables is printed to stderr. Keep in mind that secret tokens may be included in the output. Press Ctrl+D to end data input if you try out the command without passing file content via Pipe mechanism as input. On Microsoft Windows press Ctrl+Z and then press Enter to end the interactive input into the command.
Datasets may be listed either in markdown or in CSV format. Markdown can be converted to HTML by piping the output to jisap-cli md2html.
$ jisap-cli list-datasets
| ID | Namespace | Name | LastActivityAt | ForkedFrom | Visibility | Tags |
|------|-----------------|-----------------------------------|--------------------------|------------|------------|---------------------------------------------------------|
| 9253 | jisap/datasets | Fruit-Images-Dataset | 2021-01-14T09:12:40.479Z | | internal | Classification/JISAP-Example/Multi-Label Classification |
| 9265 | jisap/datasets | The Oxford-IIIT Pet Dataset | 2021-01-08T08:13:40.088Z | | internal | Classification/JISAP-Example |
| 9256 | jisap/datasets | H-Leadfinder Hit-No Hit Data | 2021-01-07T14:06:54.873Z | | internal | Explainability/JISAP-Example |
| 9126 | jisap/datasets | CIFAR-10-png | 2020-12-18T15:38:44.856Z | | internal | JISAP-Example |
| 9120 | jisap/datasets | MNIST-png | 2020-12-18T09:43:49.771Z | | internal | JISAP-Example |
...
$ jisap-cli list-datasets --members
| ID | Name | Owner | Maintainers | Developers | Reporters | Guests |
|------|-------------------------------------------|---------------------|-------------------------------------------|------------------------|-----------|--------|
| 9253 | Fruit-Images-Dataset | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
| 8934 | stargate training data | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM/Tobias Mentzel | | |
| 9713 | IMDB Dataset of 50K Movie Reviews | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
| 9645 | ovaries sample data | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
| 9709 | Titanic Dataset | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
| 9701 | Hymenoptera - Ants vs Bees Classification | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
| 9662 | CropDesign Image-Archive | Christian Klukas | Tobias Reinhard Umbach/Adrian Tobias Noll | FAROMBM | | |
...
Projects may be listed either in markdown or in CSV format. Markdown can be converted to HTML by piping the output to jisap-cli md2html. A list of sub-directories may be specified as additional arguments, otherwise the whole project is processed.
$ jisap-cli list-projects
| ID | Namespace | Name | LastActivityAt | ForkedFrom | Visibility | Tags |
|------|-----------------|-----------------------------------------|--------------------------|------------|------------|-------------------------------------|
| 9254 | jisap/projects | Fruit-Images-Training | 2021-01-14T06:41:05.186Z | 9253 | internal | AppStore/JISAP-Example/Keras/R-Stat |
| 9257 | jisap/projects | H-Leadfinder Hit-No Hit Explainability | 2021-01-13T18:00:13.282Z | 9256 | internal | Classification/Explainability |
| 9036 | jisap | JISAP-CLI | 2021-01-13T17:39:23.038Z | | public | Go/JISAP |
| 8906 | jisap/projects | LeafClassification | 2021-01-13T13:25:58.496Z | 8905 | internal | Classification/JISAP |
| 9296 | jisap/projects | SebTraining1 | 2021-01-11T09:53:23.738Z | | internal | |
| 9266 | jisap/projects | Cats vs Dogs Classification with FastAI | 2021-01-08T18:09:02.569Z | 9265 | internal | FastAI/JISAP-Example |
...
The list-files command creates by default all hierarchical view of the directories and files of a project. For options on how to hide files, .files and directories, or how to create a flat list of file names, consult the help for the command (“jisap-cli list-files –help”).
Either releases for a given project, or all releases may be printed in markdown format (default) or as a csv table. If you pass --all-versions not only the latest release for a project but all are listed.
$ go run . list-releases --all |
Rendered output:
| PID | Project | Release Created | Tag | Name | User | Assets |
|---|---|---|---|---|---|---|
| 9036 | JISAP-CLI | 2021-01-13 18:47:29.001 +0100 CET | v1.82.956716 | jisap-cli version 1.82.956716 | klukas | jisap-cli.exe jisap-cli |
| 8906 | LeafClassification | 2020-12-04 11:02:59.805 +0100 CET | dev_model_v1.0.0 | First model | klukas | TensorFlow Model H5 V1.0.0 |
| 8962 | stargate_training_process | 2020-12-15 21:31:33.252 +0100 CET | test_training_bs_16_v1 | Test training results with larger batch size | klukas |
Lists deployments (Name, URL, …) for a given project, or all projects. Output format is as for the other list-commands either markdown (default) or CSV.
$ jisap-cli list-deployments --all
| PID | Project | ID | Name | Slug | State |
|------|-----------------------------------------|-----|-----------------------------------------------------------------------------|--------------------------|-----------|
| 9254 | Fruit-Images-Training | 114 | [Development API](https://app-dev.roqs.basf.net/fruit_images_training_api/) | development-api-od9002 | available |
| 9254 | Fruit-Images-Training | 115 | [QA API](https://app-qa.roqs.basf.net/fruit_images_training_api/) | qa-api-3lqa9i | available |
| 9254 | Fruit-Images-Training | 116 | [Production API](https://app.roqs.basf.net/fruit_images_training_api/) | production-api-hobmb3 | available |
| 8962 | Stargate Model Training | 109 | [swagger API for model v1](http://jisap.basf.net:5400) | swagger-api-for-m-hahesg | available |
| 8962 | Stargate Model Training | 117 | [App Store Dev Deployment](https://app-dev.roqs.basf.net/stargate_api) | app-store-dev-dep-9nfgm4 | available |
| 9257 | H-Leadfinder Hit-No Hit Explainability | 113 | [swagger API dev](https://app.roqs.basf.net/fruit_training/) | swagger-api-dev-hzvbdk | stopped |
| 9266 | Cats vs Dogs Classification with FastAI | 112 | [model_v1_dev](https://jisap.basf.net/api/v1) | model-v1-dev-a9xad1 | available |
Rendered output:
| PID | Project | ID | Name | Slug | State |
|---|---|---|---|---|---|
| 9254 | Fruit-Images-Training | 114 | Development API | development-api-od9002 | available |
| 9254 | Fruit-Images-Training | 115 | QA API | qa-api-3lqa9i | available |
| 9254 | Fruit-Images-Training | 116 | Production API | production-api-hobmb3 | available |
| 8962 | Stargate Model Training | 109 | swagger API for model v1 | swagger-api-for-m-hahesg | available |
| 8962 | Stargate Model Training | 117 | App Store Dev Deployment | app-store-dev-dep-9nfgm4 | available |
| 9257 | H-Leadfinder Hit-No Hit Explainability | 113 | swagger API dev | swagger-api-dev-hzvbdk | stopped |
| 9266 | Cats vs Dogs Classification with FastAI | 112 | model_v1_dev | model-v1-dev-a9xad1 | available |
The dvc-metrics-log command queries connects with Gitlab, to retrieve the list of branches and their commits. Within these commits changes to the dvc.lock file are searched for. Within dvc.lock the metric output is listed as a JSON file artifact. This command loads the according information and processes it into a markdown table or as a csv list.
By default only the latest commit to each branch is included in the output. Use --pageLen X to specify the desired maximum number of commits to the dvc.lock-file.
Use --mdIgnore A,B,C to ignore metrics for A, B, C. The named columns are then not included in the output.
By default the command will list the metric results for all stages and metric files individually as separated markdown tables.
List commits and related dvc metrics of commits that changed metrics results
Usage:
jisap-cli dvc-metrics-log [flags]
Flags:
--header enable to print CSV header row
-h, --help help for dvc-metrics-log
--list print as csv list instead of markdown
--mdAddOutdatedColumns add metric columns not included in latest commit
--mdDateFormat string format string of commit time date format (default "02 Jan 06 15:04 MST")
--mdFloatFormat string format string for float values in output (default "%.3f")
--mdFullAuthor don't shorten author name
--mdHeadingSuffix string suffix for stage and metrics headings (default "\n\n")
--mdIgnore string comma-separated list of metrics keys that shouldn't be included in output
--mdMetricsFilePrefix string prefix for metrics file section, use / to disable output of metrics file heading (default "## ")
--mdNoAuthor don't include commit author in output
--mdNoRef don't include ref in output
--mdNoReplaceUnderScore disable replacing _ by space for ref column and metric key table column headers
--mdNoTitle don't include commit message title in output
--mdStagePrefix string prefix for stage header, use / to disable output of stage heading (default "# ")
--out string comma-separated list of metrics out file names
--outprefix string comma-separated list of metric-prefix for outs
--pageLen int maximum count of commits per branch to be listed (max 100) (default 1)
--project-ids string comma-separated list of IDs of Gitlab projects
--ref string ref, e.g. master, if empty, list all branches
$ jisap-cli dvc-metrics-log --project-ids 8962 --mdNoTitle --mdIgnore n_images,species,train_time_s
# train
## metrics/BOTRCI/summary.json
| Short ID | Authored | Ref | train R2 | train loss | train mae | train mse | val R2 | val loss | val mae | val mse |
|----------|--------------------------------------------------------------|-----------------|----------|------------|-----------|-----------|--------|----------|---------|---------|
| a4a05aa1 | 28 Jan 21 19:33 CET ([CK](mailto:christian.klukas@basf.com)) | master | 0.957 | 0.229 | 0.048 | 0.006 | 0.833 | 0.207 | 0.064 | 0.018 |
| 9d90fb3e | 28 Jan 21 19:31 CET ([CK](mailto:christian.klukas@basf.com)) | higher lr | 0.990 | 0.209 | 0.024 | 0.001 | 0.864 | 0.195 | 0.057 | 0.015 |
| 7c7d3251 | 28 Jan 21 18:03 CET ([CK](mailto:christian.klukas@basf.com)) | try bs 16 | 0.928 | 0.241 | 0.058 | 0.010 | 0.816 | 0.213 | 0.071 | 0.020 |
| cb094b8e | 26 Jan 21 18:02 CET ([CK](mailto:christian.klukas@basf.com)) | v2 | 0.957 | 0.229 | 0.048 | 0.006 | 0.833 | 0.207 | 0.064 | 0.018 |
| 65463012 | 06 Jan 21 12:27 CET ([CK](mailto:christian.klukas@basf.com)) | large lr | 0.967 | 0.223 | 0.041 | 0.004 | 0.835 | 0.207 | 0.063 | 0.018 |
| 565bdc4a | 16 Dec 20 06:17 CET ([CK](mailto:christian.klukas@basf.com)) | bs 16 | | 0.219 | 0.037 | 0.003 | 0.834 | 0.207 | 0.062 | 0.018 |
...
The following commands do not need a mounted network drive, they operate stand-alone.
This mode of operation can be used to update datasets, without the need for large amounts of free local space. During copy operation each file is downloaded and then uploaded individually. Already copied files (determined by name and file size) are not downloaded or uploaded again.
Use the following command to sync B to the JISAP storage:
jisap-cli smb-dvc-sync --help
This mode of operation can be used to download data from network drive B. Already downloaded files (determined by name and file size) are not transfered again.
Use the following command to sync B to a local directory:
jisap-cli smb-local-sync --help
This mode of operation can be used to upload data from a local directory to network drive B. Already uploaded files (determined by name and file size) are not transfered again.
Use the following command to the local directory with a directory on network drive B:
jisap-cli local-smb-sync --help