cloud-function (cf)¶
For information about concepts and other user interfaces, see the Overview from the NVCF Documentation.
Cloud Function Commands
usage: ngc cloud-function [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
{available-gpus,cluster,function,gpu,registry-credential,task,telemetry-endpoint}
...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
cloud-function¶
- {available-gpus,cluster,function,gpu,registry-credential,task,telemetry-endpoint}
Possible choices: available-gpus, cluster, fn, function, gpu, instance-type, registry-credential, task, telemetry-endpoint
Sub-commands¶
function (fn)¶
Cloud function function commands.
ngc cloud-function function [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
{authorization,create,deploy,info,instance,list,remove,remove-rate-limit,update,update-rate-limit,update-secret}
...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
function¶
- {authorization,create,deploy,info,instance,list,remove,remove-rate-limit,update,update-rate-limit,update-secret}
Possible choices: auth, authorization, create, delete, deploy, info, instance, list, ls, remove, remove-rate-limit, rm, update, update-rate-limit, update-secret
Sub-commands¶
deploy¶
Function deployment commands
ngc cloud-function function deploy [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
{create,info,list,log,remove,restart,update}
...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
deploy¶
- {create,info,list,log,remove,restart,update}
Possible choices: create, delete, info, list, log, ls, redeploy, remove, restart, rm, update
Sub-commands¶
list (ls)¶
List deployments
ngc cloud-function function deploy list [--debug] [--format_type <fmt>]
[--org <name>] [--status <status>]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --status
Possible choices: ACTIVE, ERROR
Filter deployments by status type
info¶
Info about a function deployment
ngc cloud-function function deploy info [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
remove (rm, delete)¶
Undeploy a function
ngc cloud-function function deploy remove [--debug] [--format_type <fmt>]
[--graceful] [--org <name>]
[--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --graceful
Allow current tasks to complete before deleting deployment.
restart (redeploy)¶
Restart a currently deployed function
ngc cloud-function function deploy restart [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
update¶
Update an existing deployment
ngc cloud-function function deploy update [--configuration CONFIGURATION]
[--configuration-file <file.yaml>]
[--debug]
[--deployment-specification <dep-spec>]
[--format_type <fmt>] [--org <name>]
[--set <key1.key2=value>]
[--targeted-deployment-specification <targeted-dep-spec>]
[--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --configuration
Helm chart overrides json will apply cross the functions, only for Helm chart deployment
- --configuration-file, -f
Helm chart overrides yaml. Can apply multiple times.The priority will be given to the last (right-most) file specified.
- --set
Set configuration values (e.g., --set image.tag=latest). Can apply multiple times.
- --deployment-specification, --dep-spec
Deployment specs with GPU and Backend details, can specify multiple times. (Format: backend:gpu:instance_type:min_instances:max_instances[:max_request_concurrency])
- --targeted-deployment-specification, --targeted-dep-spec
Deployment specs with GPU and instance type, can specify multiple times.Clusters are mandatory for non GFN backends.(Format: <gpu:instance_type:min_instances:max_instances[:max_request_concurrency][:clusters(cluster_1,cluster_2)][:regions(region_1,region_2)][:attributes(attribute_1,attribute_2)][:preferredOrder])
create¶
Create a deployment
ngc cloud-function function deploy create [--configuration CONFIGURATION]
[--configuration-file <file.yaml>]
[--debug]
[--deployment-specification <backend:gpu:instance_type:min_instances:max_instances[:maxRequestConcurrency]>]
[--format_type <fmt>] [--org <name>]
[--set key1.key2=value]
[--targeted-deployment-specification <gpu:instance_type:min_instances:max_instances[:max_request_concurrency][:clusters(cluster_1,cluster_2)][:regions(region_1,region_2)][:attributes(attribute_1,attribute_2)][:preferredOrder]>]
[--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --configuration
Helm chart overrides json will apply cross the functions, only for Helm chart deployment
- --configuration-file, -f
Helm chart overrides yaml. Can apply multiple times. The priority will be given to the last (right-most) file specified.
- --set
Set configuration values (e.g., --set image.tag=latest). Can apply multiple times.
- --deployment-specification, --dep-spec
Deployment specs with GPU and Backend details, can specify multiple times
- --targeted-deployment-specification, --targeted-dep-spec
Deployment specs with GPU and instance type, can specify multiple times.Clusters are mandatory for non GFN backends.
log¶
Query Logs for Deployment
ngc cloud-function function deploy log [--debug] [--duration <t>]
[--end-time <t>] [--format_type <fmt>]
[--org <name>] [--start-time <t>]
[--team <name>] [-h]
function-id:[<function-id>]
Positional Arguments¶
- function-id:[<function-id>]
Function. Format: function-id:function-version
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --start-time
Specifies the start time for querying logs. Format: [yyyy-MM-dd::HH:mm:ss].
- --end-time
Specifies the end time for querying logs. Format: [yyyy-MM-dd::HH:mm:ss]. Default: now
- --duration
Specifies the duration of time, either after begin-time or before end-timelogs. Format: [nD][nH][nM][nS]. Default 1 day, doesn't respect decimal measurements
instance¶
Function instance commands
ngc cloud-function function instance [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
{execute,list,logs} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
instance¶
- {execute,list,logs}
Possible choices: exec, execute, list, logs, ls
Sub-commands¶
list (ls)¶
List specific function version instances
ngc cloud-function function instance list [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
execute (exec)¶
Execute command on target container
ngc cloud-function function instance execute --command <command>
--container-name <container-name>
[--debug] [--format_type <fmt>]
--instance-id <instance-id>
[--org <name>] --pod-name
<pod-name> [--team <name>]
[--timeout <timeout>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --timeout
Timeout for the command execution
Required named arguments¶
- --container-name
Target container to execute container
- --pod-name
Target pod which including the target container
- --instance-id
Target instance id.
- --command
Command to execute
logs¶
Read the target instance logs
ngc cloud-function function instance logs [--container-name <container-name>]
[--debug] [--format_type <fmt>]
--instance-id <instance-id>
[--org <name>]
[--pod-name <pod-name>]
[--team <name>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --container-name
Target container to query logs. Must be used with --pod-name
- --pod-name
Target pod which including the target container. Must be used with --container-name
Required named arguments¶
- --instance-id
Target instance id.
list (ls)¶
List functions
ngc cloud-function function list [--access-filter <filter>]
[--column <column>] [--debug]
[--format_type <fmt>] [--name-pattern <name>]
[--org <name>] [--team <name>] [-h]
[[<function-id>]]
Positional Arguments¶
- [<function-id>]
Function ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --access-filter
Possible choices: authorized, private, public
Filter functions by access, choices are: [private,public,authorized]
- --name-pattern
Filter functions by names, supports globs.
- --column
Specify output column as column[=header], header is optional, default is id[=Id]. Valid columns are activeInstances[="Active Instances"], containerImage[="Container Image URL"], helmChart[="Helm Chart URL"], helmChartServiceName[="Helm Chart Service Name"], models[="Model URLs"], id[="Function ID"], description[=Description], functionType[="Function Type"], status[=Status], version[="Version ID"]. Use quotes with spaces. Multiple column arguments are allowed.
info¶
Info about a version
ngc cloud-function function info [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
remove (rm, delete)¶
Delete a version
ngc cloud-function function remove [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
update-secret¶
Update a function's secret values.
ngc cloud-function function update-secret [--debug] [--format_type <fmt>]
[--json-secret-file <filename>]
[--org <name>]
[--secret <name:value>]
[--team <name>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --secret
Secret name/value pair
- --json-secret-file
Takes in a file, assumes the key as the filename, value is the file contents. Only works with json files
update¶
Update a function version. Rate-limit and LLM configuration fields are mutable.
ngc cloud-function function update [--debug] [--format_type <fmt>]
[--llm-default-priority <priority>]
[--llm-model <name>:<uri,uri...>[:<tokenizer>][:<tokenRateLimit>][:<routingMethod>]]
[--llm-per-account-priority <nca-id>:<priority>]
[--org <name>]
[--rate-limit-exempt-nca-id <rate-limit-exempt-id>]
[--rate-limit-pattern <rate-limit-pattern>]
[--rate-limit-per-nca-id-rate <nca-id>:<rate-limit-pattern>]
[--rate-limit-sync-check | --no-rate-limit-sync-check]
[--team <name>] [-h]
<function-id>:<function-version-id>
Positional Arguments¶
- <function-id>:<function-version-id>
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --rate-limit-pattern
Rate limit pattern, format NUMBER-S|M|H|D (e.g. 3-S). Comma-separated for multiple cadences (e.g. '100-S, 1000-D'). Pass an empty string to clear the rate limit.
- --rate-limit-exempt-nca-id
Exempt NCA id (repeatable).
- --rate-limit-per-nca-id-rate
Per-NCA-id override rate (repeatable). The pattern follows the same format as --rate-limit-pattern.
- --rate-limit-sync-check, --no-rate-limit-sync-check
Rate limit sync check.
- --llm-model
Update an LLM model's mutable config. Format: <name>:<uri,uri...>[:<tokenizer>][:<tokenRateLimit>][:<routingMethod>], leaving immutable fields (uris, tokenizer) empty. At least one of tokenRateLimit or routingMethod (one of [round_robin,power_of_two,groq_multiregion,pulsar,random]) is required. E.g. 'meta/llama-3.1-8b-instruct:::1000-S:power_of_two'. Repeatable.
- --llm-default-priority
Function-level default request priority. Lower is higher priority, 0 is highest.
- --llm-per-account-priority
Per-account request priority override (repeatable).
update-rate-limit¶
DEPRECATED: prefer 'ngc cf function update --rate-limit-pattern ...'.
ngc cloud-function function update-rate-limit [--debug] [--format_type <fmt>]
[--org <name>]
[--rate-limit-exempt-nca-id <rate-limit-exempt-id>]
--rate-limit-pattern
<rate-limit-pattern>
[--rate-limit-sync-check]
[--team <name>] [-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --rate-limit-exempt-nca-id
Exempt NCA id.
- --rate-limit-sync-check
Rate limit sync check.
Required named arguments¶
- --rate-limit-pattern
Rate limit pattern, format NUMBER-S|M|H|D (e.g. 3-S). Comma-separated for multiple cadences (e.g. '100-S, 1000-D').
remove-rate-limit¶
Delete a function's rate limit. 'ngc cf function update' cannot clear a rate limit, so this command is the only way to remove one.
ngc cloud-function function remove-rate-limit [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>]
[-h]
<function-id>:[<function-version-id>]
Positional Arguments¶
- <function-id>:[<function-version-id>]
Function. Format: function-id:[version]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
create¶
Create a new function
ngc cloud-function function create [--api-body-format <api-body-format>]
[--container-args <container-args>]
[--container-environment-variable <key:value>]
[--container-image <org>/[<team>/]<image>:<tag>]
[--debug] [--description <description>]
[--format_type <fmt>]
[--function-type <function-type>]
[--health-expected-status-code <health-expected-status-code>]
[--health-port <health-port>]
[--health-protocol <health-protocol>]
[--health-timeout <health-timeout>]
[--health-uri <health-uri>]
[--helm-chart <org>/[<team>/]<helm-chart>:<tag>]
[--helm-chart-service <helm-chart-service-name>]
[--inference-port <inference-port>]
--inference-url <inference-url>
[--json-secret-file <filename>]
[--llm-default-priority <priority>]
[--llm-model <name>:<uri,uri...>[:<tokenizer>][:<tokenRateLimit>][:<routingMethod>]]
[--llm-per-account-priority <nca-id>:<priority>]
[--logs-telemetry-id <logs-telemetry-id>]
[--metrics-telemetry-id <metrics-telemetry-id>]
[--model [<override-name>:]<org>/[<team>/]<image>:version]
--name <name> [--no-default-created-by-tag]
[--org <name>]
[--rate-limit-exempt-nca-id <rate-limit-exempt-id>]
[--rate-limit-pattern <rate-limit-pattern>]
[--rate-limit-per-nca-id-rate <nca-id>:<rate-limit-pattern>]
[--rate-limit-sync-check | --no-rate-limit-sync-check]
[--resource <org>/[<team>/]<resource>:version]
[--secret <name:value>] [--tag <tag>]
[--team <name>]
[--traces-telemetry-id <traces-telemetry-id>]
[-h]
[[<function-id>]]
Positional Arguments¶
- [<function-id>]
Function ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --helm-chart
Helm Chart in NGC used for deployment.
- --helm-chart-service
Must be provided if a helm chart function.
- --health-uri
Health endpoint for inferencing
- --health-port
Health port for inferencing
- --health-timeout
Health timeout for inferencing
- --health-protocol
Possible choices: HTTP, gRPC
Health protocol for inferencing
- --health-expected-status-code
Health status code.
- --inference-port
Optional port number where the inference listener is running - defaults to 8000 for HTTPS, 8001 for GRPC
- --container-args
Args to be passed in for inferencing
- --container-environment-variable
Environment settings for inferencing
- --secret
Secret name/value pair
- --json-secret-file
Takes in a file, assumes the key as the filename, value is the file contents. Only works with json files
- --model
List of models - could be empty with custom container, can accept multiple
- --resource
Optional List of resources, can accept multiple
- --tag
Tag to identify the function by, can accept multiple
- --no-default-created-by-tag
Skip adding the default 'created_by:<user-email>' tag.
- --description
Optional description for function/version
- --container-image
Custom container Image
- --api-body-format
Possible choices: CUSTOM, PREDICT_V2
Information about the request body format
- --function-type
Possible choices: DEFAULT, LLM, STREAMING
Function type, one of [DEFAULT,STREAMING,LLM]. Defaults to DEFAULT.
- --llm-model
LLM model served by the function. Format: <name>:<uri,uri...>[:<tokenizer>][:<tokenRateLimit>][:<routingMethod>], with comma-separated OpenAI-compatible URI paths (at least one required) and routingMethod one of [round_robin,power_of_two,groq_multiregion,pulsar,random]. E.g. 'meta/llama-3.1-8b-instruct:/v1/chat/completions,/v1/completions::1000-S:power_of_two'. Requires --function-type LLM. Repeatable.
- --llm-default-priority
Function-level default request priority. Lower is higher priority, 0 is highest.
- --llm-per-account-priority
Per-account request priority override (repeatable).
- --metrics-telemetry-id
UUID representing the metrics telemetry.
- --logs-telemetry-id
UUID representing the logs telemetry.
- --traces-telemetry-id
UUID representing the traces telemetry.
- --rate-limit-pattern
Rate limit pattern, format NUMBER-S|M|H|D (e.g. 3-S). Comma-separated for multiple cadences (e.g. '100-S, 1000-D').
- --rate-limit-exempt-nca-id
Exempt NCA id (repeatable).
- --rate-limit-per-nca-id-rate
Per-NCA-id override rate (repeatable).
- --rate-limit-sync-check, --no-rate-limit-sync-check
Rate limit sync check.
Required named arguments¶
- --name
Function name must start with lowercase/uppercase/digit and can only contain lowercase, uppercase, digit, hyphen, and underscore characters
- --inference-url
Serves as entrypoint for Triton to Custom container
cluster¶
Cloud function cluster commands.
ngc cloud-function cluster [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
{create,delete,info,list,rotate-key,update} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
cluster¶
- {create,delete,info,list,rotate-key,update}
Possible choices: create, delete, info, list, ls, register, remove, rm, rotate-key, update
Sub-commands¶
list (ls)¶
List clusters.
ngc cloud-function cluster list [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
info¶
Info about a cluster
ngc cloud-function cluster info [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<cluster-id>
Positional Arguments¶
- <cluster-id>
Cluster Id
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
delete (rm, remove)¶
Delete a cluster
ngc cloud-function cluster delete [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<cluster-id>
Positional Arguments¶
- <cluster-id>
Cluster Id
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
create (register)¶
Register a new cluster with the specified details
ngc cloud-function cluster create [--attribute ATTRIBUTE]
[--authorized-nca-id <nca-id>]
[--capability <capability>] --cloud-provider
{AWS,AZURE,DGX-CLOUD,GCP,NCP,OCI,ON-PREM}
[--cluster-description CLUSTER_DESCRIPTION]
--cluster-group-name CLUSTER_GROUP_NAME
--cluster-name CLUSTER_NAME
[--custom-attribute CUSTOM_ATTRIBUTE]
[--debug] [--format_type <fmt>]
[--management-mode <management-mode>]
[--nvca-version NVCA_VERSION] [--org <name>]
--region
{ap-east-1,eu-central-1,eu-north-1,eu-south-1,eu-west-1,us-east-1,us-west-1,us-west-2}
[--ssa-client-id SSA_CLIENT_ID]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --cluster-description
Optional description providing additional context about the cluster.
- --ssa-client-id, --oauth-client-id
SSA client ID (needed for internal NVIDIA organizations only). --oauth-client-id is an alias.
- --capability
Possible choices: BYOObservability, CachingSupport, DynamicGPUDiscovery, DynamoOperatorSupport, KAIScheduler, LogPosting
Capabilities. Options: DynamicGPUDiscovery, LogPosting, CachingSupport, BYOObservability, KAIScheduler, DynamoOperatorSupport. Default: DynamicGPUDiscovery
- --attribute
Attributes
- --custom-attribute
Custom attributes
- --authorized-nca-id
Authorize an NVIDIA Cloud Account ID to use this cluster ('*' for any account). Repeatable.
- --nvca-version
NVCA version
- --management-mode
Possible choices: helm-managed, ngc-managed
How the cluster is configured after registration: through NGC (default) or through helm values. Options: ngc-managed, helm-managed.
Required named arguments¶
- --cluster-name
The name for the cluster. This field is not changeable once configured.
- --cluster-group-name
The name of the cluster group, typically identical to the cluster name,allowing function deployment across grouped clusters.
- --cloud-provider
Possible choices: AWS, AZURE, DGX-CLOUD, GCP, NCP, OCI, ON-PREM
The cloud platform on which the cluster is deployed.
- --region
Possible choices: ap-east-1, eu-central-1, eu-north-1, eu-south-1, eu-west-1, us-east-1, us-west-1, us-west-2
The region where the cluster is deployed.
update¶
Update mutable cluster fields: management mode, authorized NCA IDs, capabilities, NVCA version. Changes that the cluster must apply through helm print the helm command to run and ask for confirmation before the record is written.
ngc cloud-function cluster update [--add-authorized-nca-id <nca-id>] [--debug]
[--disable-capability <capability>]
[--enable-capability <capability>]
[--format_type <fmt>]
[--management-mode <management-mode>]
[--nvca-version <nvca-version>]
[--org <name>]
[--remove-authorized-nca-id <nca-id>]
[--team <name>] [-h] [-y]
<cluster-id>
Positional Arguments¶
- <cluster-id>
Cluster Id
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --management-mode
Possible choices: helm-managed, ngc-managed
Switch the cluster management mode. Prints the helm command that performs the switch on the cluster.
- --add-authorized-nca-id
Authorize an NVIDIA Cloud Account ID to use this cluster ('*' for any account). Repeatable.
- --remove-authorized-nca-id
Revoke an authorized NVIDIA Cloud Account ID. Repeatable.
- --enable-capability
Possible choices: BYOObservability, CachingSupport, DynamicGPUDiscovery, DynamoOperatorSupport, KAIScheduler, LogPosting
Enable a cluster capability. Repeatable.
- --disable-capability
Possible choices: BYOObservability, CachingSupport, DynamicGPUDiscovery, DynamoOperatorSupport, KAIScheduler, LogPosting
Disable a cluster capability. Repeatable.
- --nvca-version
Upgrade the Cluster Agent (NVCA) to this version. Prints the helm command to run first when the operator must be upgraded too, or when the cluster is helm-managed.
- -y, --yes
Automatically say yes to all interactive questions.
rotate-key¶
Create a new cluster service key, attach it to the cluster, revoke the old one, and print the helm command that switches the cluster to the new key.
ngc cloud-function cluster rotate-key [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<cluster-id>
Positional Arguments¶
- <cluster-id>
Cluster Id
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
gpu (instance-type)¶
Cloud function gpu commands.
ngc cloud-function gpu [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
{capacity,info,list,quota} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
gpu¶
- {capacity,info,list,quota}
Possible choices: capacity, get, info, list, ls, quota
Sub-commands¶
list (ls)¶
List allocated gpus.
ngc cloud-function gpu list [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
capacity¶
Show a summary of GPU capacity allocation.
ngc cloud-function gpu capacity [--debug] [--format_type <fmt>] [--gpu GPU]
[--org <name>]
[--region {ap-east-1,eu-central-1,eu-north-1,eu-south-1,eu-west-1,us-east-1,us-west-1,us-west-2}]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --region
Possible choices: ap-east-1, eu-central-1, eu-north-1, eu-south-1, eu-west-1, us-east-1, us-west-1, us-west-2
Filter by specific region. Can specify multiple times.
- --gpu
Filter by specific GPU. Can specify multiple times.
quota¶
Get GPU Quota information.
ngc cloud-function gpu quota [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
info (get)¶
Info for a given gpu.
ngc cloud-function gpu info [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
Positional Arguments¶
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
registry-credential¶
Cloud function registry-credential commands.
ngc cloud-function registry-credential [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
{create,info,list,remove,update} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
registry-credential¶
- {create,info,list,remove,update}
Possible choices: add, create, delete, info, list, ls, remove, rm, update
Sub-commands¶
info¶
Info about registry credential.
ngc cloud-function registry-credential info [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>]
[-h]
<registry-credential-id>
Positional Arguments¶
- <registry-credential-id>
Id of registry credential to get
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
list (ls)¶
List registry credentials.
ngc cloud-function registry-credential list [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>]
[-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
create (add)¶
Create a new registry credential.
ngc cloud-function registry-credential create [--aws-access-key <username>]
[--debug]
[--description <description>]
[--format_type <fmt>] --hostname
<hostname> --key <key> --name
<name> [--org <name>]
[--tag <tag>] [--team <name>]
--type
<registry-credential-type>
[--username <username>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --username
Username for credentials
- --aws-access-key
Access key for AWS credentials
- --description
Description for credentials
- --tag
Tag for credentials
Required named arguments¶
- --name
Registry Credentials name
- --key
Registry credentials key; for AWS, it equals the secret access key
- --hostname
Registry Hostname of credentials
- --type
Possible choices: CONTAINER, HELM, MODEL, RESOURCE
Set of registry credential data types
update¶
Update an existing registry credential .
ngc cloud-function registry-credential update [--aws-access-key <username>]
[--debug] [--format_type <fmt>]
--key <key> [--org <name>]
[--team <name>]
[--username <username>] [-h]
<registry-credential-id>
Positional Arguments¶
- <registry-credential-id>
Id of registry credential to update
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --username
Username for credentials
- --aws-access-key
Access key for AWS credentials
Required named arguments¶
- --key
Registry credentials key
remove (delete, rm)¶
Delete a registry credential.
ngc cloud-function registry-credential remove [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>]
[-h]
<registry-credential-id>
Positional Arguments¶
- <registry-credential-id>
Id of registry credential to delete
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
task¶
Cloud function task commands.
ngc cloud-function task [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
{cancel,create,delete,events,info,instance,list,logs,results,update-secret}
...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
task¶
- {cancel,create,delete,events,info,instance,list,logs,results,update-secret}
Possible choices: cancel, create, delete, events, info, instance, list, logs, ls, remove, results, rm, update-secret
Sub-commands¶
instance¶
Task instance commands
ngc cloud-function task instance [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
{execute,list,logs} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
instance¶
- {execute,list,logs}
Possible choices: exec, execute, list, logs, ls
Sub-commands¶
list (ls)¶
List specific task instances
ngc cloud-function task instance list [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
execute (exec)¶
Execute command on target container
ngc cloud-function task instance execute --command <command> --container-name
<container-name> [--debug]
[--format_type <fmt>] --instance-id
<instance-id> [--org <name>]
--pod-name <pod-name> [--team <name>]
[--timeout <timeout>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --timeout
Timeout for the command execution
Required named arguments¶
- --container-name
Target container to execute container
- --pod-name
Target pod which including the target container
- --instance-id
Target instance id.
- --command
Command to execute
logs¶
Read the target instance logs
ngc cloud-function task instance logs [--container-name <container-name>]
[--debug] [--format_type <fmt>]
--instance-id <instance-id>
[--org <name>] [--pod-name <pod-name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --pod-name
Target task pod which including the target container. Must be used with --container-name
- --container-name
Target task container to query logs. Must be used with --pod-name
Required named arguments¶
- --instance-id
Target task instance id.
create¶
Create a new task
ngc cloud-function task create [--configuration CONFIGURATION]
[--configuration-file <file.yaml>]
[--container-args <container-args>]
[--container-environment-variable <key:value>]
[--container-image <org>/[<team>/]<image>:<tag>]
[--debug] [--description <description>]
[--format_type <fmt>] --gpu-specification
<gpu:instance_type[:backend][:cluster_1,
cluster_2]>
[--helm-chart <org>/[<team>/]<helm-chart>:<tag>]
[--logs-telemetry-id <logs-telemetry-id>]
[--max-queued-duration <max-queued-duration>]
[--max-runtime-duration <max-runtime-duration>]
[--metrics-telemetry-id <metrics-telemetry-id>]
[--model <model>] --name <name>
[--no-default-created-by-tag] [--org <name>]
[--resource <resource>]
[--result-handling-strategy <result-handling-strategy>]
[--result-location <result-location>]
[--secret <name:value>]
[--set <key1.key2=value>] [--tag <tag>]
[--team <name>]
[--termination-grace-period-duration <termination-grace-period-duration>]
[--traces-telemetry-id <traces-telemetry-id>]
[-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --configuration
Helm chart overrides json will apply cross the functions, only for Helm chart deployment
- --configuration-file, -f
Helm chart overrides yaml. Can apply multiple times.The priority will be given to the last (right-most) file specified.
- --set
Set configuration values (e.g., --set image.tag=latest). Can apply multiple times.
- --container-image
Custom container image
- --container-args
Args to be passed in for inferencing
- --container-environment-variable
Environment settings for inferencing
- --model
Name of model(s) to use in the task.
- --resource
Name of resource(s) to use in the task.
- --tag
Tags to identify the function by, can accept multiple
- --no-default-created-by-tag
Skip adding the default 'created_by:<user-email>' tag.
- --description
Optional description for function/version
- --max-runtime-duration
Specifies the maximum runtime duration. Format: [nD][nH][nM][nS].
- --max-queued-duration
Specifies the maximum queued duration. Format: [nD][nH][nM][nS].
- --termination-grace-period-duration
Specifies the duration for grace period after termination. Format: [nD][nH][nM][nS].
- --result-handling-strategy
Possible choices: NONE, UPLOAD
Specifies how results will be handled. Choices: UPLOAD, NONE
- --result-location
Specifies where result will be stored.}
- --secret
Secret name/value pair
- --helm-chart
Helm Chart in NGC used for deployment.
- --metrics-telemetry-id
UUID representing the metrics telemetry.
- --logs-telemetry-id
UUID representing the logs telemetry.
- --traces-telemetry-id
UUID representing the traces telemetry.
Required named arguments¶
- --name
Task name
- --gpu-specification
GPU specification. Format: gpu:instance_type[:backend][:cluster_1, cluster_2]
update-secret¶
Update a task's secret value. Can specify multiple times.
ngc cloud-function task update-secret [--debug] [--format_type <fmt>]
[--org <name>] --secret <name:value>
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
Required named arguments¶
- --secret
Secret name/value pair
list (ls)¶
List of tasks
ngc cloud-function task list [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
info¶
Info about a task
ngc cloud-function task info [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
delete (remove, rm)¶
Delete a task
ngc cloud-function task delete [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
cancel¶
Cancel a task
ngc cloud-function task cancel [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
events¶
List of events for a task
ngc cloud-function task events [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
results¶
List of results for a task
ngc cloud-function task results [--debug] [--format_type <fmt>] [--org <name>]
[--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
logs¶
List of logs for a task
ngc cloud-function task logs [--debug] [--duration <t>] [--end-time <t>]
[--format_type <fmt>] [--org <name>]
[--start-time <t>] [--team <name>] [-h]
<task-id>
Positional Arguments¶
- <task-id>
Task ID
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --start-time
Specifies the start time for querying logs. Format: [yyyy-MM-dd::HH:mm:ss].
- --end-time
Specifies the end time for querying logs. Format: [yyyy-MM-dd::HH:mm:ss]. Default: now
- --duration
Specifies the duration of time, either after begin-time or before end-time. Format: [nD][nH][nM][nS]. Default 1 hour, doesn't respect decimal measurements
telemetry-endpoint¶
Cloud function telemetry-endpoint commands.
ngc cloud-function telemetry-endpoint [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
{create,list,remove} ...
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
telemetry-endpoint¶
- {create,list,remove}
Possible choices: add, create, delete, list, ls, remove, rm
Sub-commands¶
list (ls)¶
List telemetries.
ngc cloud-function telemetry-endpoint list [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
create (add)¶
Create a new telemetry endpoint.
ngc cloud-function telemetry-endpoint create
[--application-id <application-id>]
[--client-cert <client-cert>]
[--debug] --endpoint <endpoint>
[--format_type <fmt>]
[--instance <instance>] --key
<key>
[--live-endpoint <live-endpoint>]
--name <name> [--org <name>]
--protocol <protocol> --provider
<provider>
[--self-cert <self-cert>]
[--team <name>] --type
<telemetry-type> [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
- --instance
Instance id for GRAFANA endpoints.
- --client-cert
Client cert for PROMETHEUS endpoints.
- --self-cert
Optional self certificate for PROMETHEUS endpoints.
- --live-endpoint
Live endpoint for AZURE MONITOR endpoints.
- --application-id
Application ID AZURE MONITOR endpoints.
Required named arguments¶
- --name
Telemetry Name
- --key
Telemetry key
- --endpoint
Telemetry endpoint URL
- --protocol
Possible choices: GRPC, HTTP
Protocol used for communication
- --provider
Possible choices: AZURE_MONITOR, DATADOG, GRAFANA_CLOUD, OTEL_COLLECTOR, PROMETHEUS, SERVICENOW
Provider name for telemetry endpoint
- --type
Possible choices: LOGS, METRICS, TRACES
Set of telemetry data types
remove (delete, rm)¶
Delete a telemetry endpoint.
ngc cloud-function telemetry-endpoint remove [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>]
[-h]
<telemetry-id>
Positional Arguments¶
- <telemetry-id>
Id of telemtry endpoint to delete
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
available-gpus¶
List available GPUs in your Org, Admin Only
ngc cloud-function available-gpus [--debug] [--format_type <fmt>]
[--org <name>] [--team <name>] [-h]
Named Arguments¶
- --debug
Enable debug mode.
- --format_type
Possible choices: ascii, csv, json
Specify the output format type. Supported formats are: ['ascii', 'csv', 'json']. Only commands that produce tabular data support csv format. Default: ascii
- --org
Specify the organization name. Use "--org no-org" to override other sources and specify no org (no-org cannot be used if API key is set). Default: current configuration
- --team
Specify the team name. Use "--team no-team" to override other sources and specify no team. Default: current configuration
Examples¶
Functions¶
How to list functions?¶
$ ngc cf fn list
+-----------------------+------------------------+------------------------+------------------------+------------------------+----------+
| Name | Id | Version | Container | Helm Chart | Status |
+-----------------------+------------------------+------------------------+------------------------+------------------------+----------+
| example-helm | 33352d58-10ae-4a38-ade | 28980bb7-844e-40e6-a05 | | https://helm.ngc.nvidi | ACTIVE |
| | 9-ca2468798315 | b-411236a6cc9a | | a.com/lpeb1gw2rozl/cha | |
| | | | | rts/peek-helm-chart- | |
| | | | | test-0.1.0.tgz | |
| example-container | 74510cba-6d29-43f6-88b | 6094e300-375f-42b2-9e7 | canary.nvcr.io/org | | ACTIVE |
| | 4-e1c3d055f3ab | b-f3d87ce59353 | /example_container | | |
+-----------------------+------------------------+------------------------+------------------------+------------------------+----------+
How to create a function?¶
$ ngc cf function create --name peek-test-e2e --inference-url /health --container-image lpeb1gw2rozl/pytriton_echo_sample:latest --secret key:value --secret key2:value2 --description "test-function"
--------------------------------------------------------------------
Function Information
Name: peek-test-e2e
Version: 6698aa4d-1ea3-4f95-b6cb-4bf879fe538a
ID: a7346e6f-4755-47cf-9163-6212e8db0900
Status: INACTIVE
Inference URL: /health
Description: test-function
Container Image: nvcr.io/lpeb1gw2rozl/pytriton_echo_sample:latest
Secret Keys: key2, key
Health Endpoint Information
URI: /v2/health/ready
Port: 8000
Timeout: PT10S
Protocol: HTTP
Expected Status Code: 200
--------------------------------------------------------------------
How to create a function with a json secret?¶
$ ngc cf function create --name peek-test-e2e --inference-url /health --container-image lpeb1gw2rozl/pytriton_echo_sample:latest --json-secret-file key.json
--------------------------------------------------------------------
Function Information
Name: peek-test-e2e
Version: 6698aa4d-1ea3-4f95-b6cb-4bf879fe538a
ID: a7346e6f-4755-47cf-9163-6212e8db0900
Status: INACTIVE
Inference URL: /health
Description: test-function
Container Image: nvcr.io/lpeb1gw2rozl/pytriton_echo_sample:latest
Secret Keys: key
Health Endpoint Information
URI: /v2/health/ready
Port: 8000
Timeout: PT10S
Protocol: HTTP
Expected Status Code: 200
--------------------------------------------------------------------
How to allow others to invoke your function across all versions?¶
$ ngc cf fn auth add 7835a350-921e-4fd3-8004-554b84c1367b --authorized-party "2lxz_k9-GnzBXPFt0OCagYnVXWwoUxNMJ3-IeT-qSYA"
-----------------------------------------------------------------------
Function Authorization Information
Function ID: 7835a350-921e-4fd3-8004-554b84c1367b
Author NCA ID: CMYBKSNNjtg1TQmSke-gHNGgMlFvA-dCRAI8gcHOBcw
Authorized Party NCAIds: 2lxz_k9-GnzBXPFt0OCagYnVXWwoUxNMJ3-IeT-qSYA
-----------------------------------------------------------------------
Tasks¶
How to create a task?¶
$ ngc cf task create --name task-name --container-image stg.nvcr.io/ax3ysqem02xw/testing-jet/server:main_11178569 --gpu-specification T10:g6.full:GFN --description description --max-runtime-duration 1H --max-queued-duration 1H --termination-grace-period-duration 1H --result-handling-strategy NONE
-------------------------------------------------------------------------------------------------------------------
Task Information
Name: task-name
ID: f2a41879-1ee4-46bf-9b51-5f3da7dd139b
Status: QUEUED
Description: description
Created: 2024-12-04T20:15:13.910726527Z
Last Updated:
Configuration Details
Container: stg.nvcr.io/ax3ysqem02xw/testing-jet/server:main_11178569
GPU Type: T10
Instance Type: g6.full
Max Runtime Duration: 1h
Max Queued Duration: 1h
Termination Grace Period: 1h
Result Handling: NONE
Result Path:
-------------------------------------------------------------------------------------------------------------------
How to list tasks?¶
$ ngc cf task list
+-----------------------+-------------------------------+--------------------------------------+--------------------------+--------------------------+
| Name | Status | Id | Created At | Last Updated |
+-----------------------+-------------------------------+--------------------------------------+--------------------------+--------------------------+
| yura-task-3 | EXCEEDED_MAX_RUNTIME_DURATION | 15bc4cb5-e09b-4b2c-af7d-19750f342b59 | 2024-11-27T20:03:50.017Z | 2024-11-27T20:16:34.447Z |
| yura-task-2 | CANCELED | 93847acd-1604-4c57-a293-f43265eb5cc0 | 2024-11-21T19:19:03.801Z | 2024-11-22T18:30:20.695Z |
+-----------------------+-------------------------------+--------------------------------------+--------------------------+--------------------------+
How to view a task's information?¶
$ ngc cf task info 1bd74059-c494-4b1f-bc6a-f028677546f4
--------------------------------------------------------------------
Task Information
Name: randomcontainer
ID: 1bd74059-c494-4b1f-bc6a-f028677546f4
Status: QUEUED
Description:
Created: 2024-10-16T19:09:08.006Z
Last Updated:
Configuration Details
Container: stg.nvcr.io/ax3ysqemo02xw/testing-jet/server:main_11257917
GPU Type: L40
Instance Type: fl40_1.br20_2xlarge
Max Runtime Duration: 10 minutes
Max Queued Duration: 3 days
Termination Grace Period: 10 minutes
Result Handling: NONE
Result Path:
--------------------------------------------------------------------
How to delete a task?¶
$ ngc cf task delete 1bd74059-c494-4b1f-bc6a-f028677546f4
Successfully deleted task 1bd74059-c494-4b1f-bc6a-f028677546f4.
How to cancel a task?¶
$ ngc cf task delete 1bd74059-c494-4b1f-bc6a-f028677546f4
Successfully canceled task: 1bd74059-c494-4b1f-bc6a-f028677546f4.
How to view a task's events?¶
$ ngc cf task events 1bd74059-c494-4b1f-bc6a-f028677546f4
+-----------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Event Id | Created At | Message |
+--------------------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------------------+
| 21027187-9426-4b26-b907-c5432943ff7f | 2024-10-19T10:13:46.213Z | Changing status from 'QUEUED' to 'ERRORED' with error 'SIS Request Id cbcc58d5-d580-45d8-ad62-8dbae76f1a4d: Request State |
| | | CANCELED; Instance Id UNKNOWN: Instance State UNKNOWN; Error Message - Backend 'null', GPU 'L40', Instance Type |
| | | 'gl40_1.br20_2xlarge': Capacity not available at this time' |
+--------------------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------------------+
How to view a task's results?¶
$ ngc cf task events 1bd74059-c494-4b1f-bc6a-f028677546f4
How to view a task's logs?¶
$ ngc cf task events 1bd74059-c494-4b1f-bc6a-f028677546f4
+-----------------------------+----------------------------------------------------------------------------------------------------------------------------------+-----------+
| Timestamp | Message | Source |
+-----------------------------+----------------------------------------------------------------------------------------------------------------------------------+-----------+
| 2024-12-17 18:39:22.769 UTC | 2024-12-17T18:39:22.768Z INFO 1 --- [gdn-nvct-service] [io-8080-exec-10] c.n.k.n.s.task.TaskManagementService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Creating and launching new task with name 'yura-task-12' in account 'CMYBKSNNjtg1TQmSke- | |
| | gHNGgMlFvA-dCRAI8gcHOBcw' | |
| 2024-12-17 18:39:23.259 UTC | 2024-12-17T18:39:23.258Z DEBUG 1 --- [gdn-nvct-service] [io-8080-exec-10] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Saving task | |
| 2024-12-17 18:39:23.321 UTC | 2024-12-17T18:39:23.321Z DEBUG 1 --- [gdn-nvct-service] [io-8080-exec-10] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Saved task | |
| 2024-12-17 18:39:23.726 UTC | 2024-12-17T18:39:23.726Z INFO 1 --- [gdn-nvct-service] [io-8080-exec-10] c.n.kaizen.nvct.service.sis.SisService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210', SIS requestId 'b84f3b26-2ffe-489a-9489-1cf78518d138', GPU 'A100': Requested '1' instance | |
| 2024-12-17 18:39:24.471 UTC | 2024-12-17T18:39:24.471Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Fetching task | |
| 2024-12-17 18:39:24.476 UTC | 2024-12-17T18:39:24.476Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Fetched task | |
| 2024-12-17 18:39:24.476 UTC | 2024-12-17T18:39:24.476Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Fetching task | |
| 2024-12-17 18:39:24.476 UTC | 2024-12-17T18:39:24.476Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.n.s.task.TaskManagementService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Retrieving | |
| 2024-12-17 18:39:24.478 UTC | 2024-12-17T18:39:24.478Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Fetching task metadata | |
| 2024-12-17 18:39:24.478 UTC | 2024-12-17T18:39:24.478Z DEBUG 1 --- [gdn-nvct-service] [nio-8080-exec-7] c.n.k.nvct.service.task.TaskService : Task id | inference |
| | '7e06ca3a-b907-4a71-8ae9-ba69f44d0210': Fetched task | |
+-----------------------------+----------------------------------------------------------------------------------------------------------------------------------+-----------+
How to update a task's secrets?¶
$ ngc cf task update-secret 1bd74059-c494-4b1f-bc6a-f028677546f4 --secret secret1:new-value
Successfully updated secret.