octomizer package

Submodules

octomizer.auth module

Helper classes for authenticating via gRPC.

class octomizer.auth.AuthInterceptor(access_token)

Bases: grpc.UnaryUnaryClientInterceptor

This class intercepts outbound gRPC calls and adds an Authorization: Bearer header carrying an access token.

Parameters

access_token – The token to send in the Authorization header.

intercept_unary_unary(continuation, client_call_details, request)

Invoked by gRPC when issuing a request using this interceptor.

octomizer.client module

Python client for the OctoML Platform API.

octomizer.client.OCTOMIZER_API_HOST = 'api.octoml.ai'

Default value for the OctoML API port, if not specified manually or via the environment.

class octomizer.client.OctomizerClient(host: Optional[str] = None, port: Optional[int] = None, insecure: bool = False, access_token: Optional[str] = None, check_connection: bool = True)

Bases: object

A client to the OctoML Platform service.

Parameters
  • host – The hostname of the OctoML RPC service. Note that this is typically not the same as the hostname used to access the OctoML web interface. The default is the value of the environment variable OCTOMIZER_API_HOST. This defaults to api.octoml.ai if not specified.

  • port – The port of the OctoML RPC serivce. The default is the value of the environment variable OCTOMIZER_API_PORT, and defaults to 443.

  • insecure – whether to create an insecure (non-TLS) channel to the gRPC server. This is typically used only for testing.

  • access_token – An OctoML access token. These can be obtained from the web interface or via the CreateAccessToken RPC call.

  • check_connection – If True, check that the connection is live when the client is created, raising a RuntimeError if the connection cannot be established.

add_user(given_name: str, family_name: str, email: str, active: bool = True, is_own_account_admin: bool = False, can_octomize: bool = True, account_uuid: Optional[str] = None)octoml.octomizer.v1.users_pb2.User

Creates and returns a user with the given information

Parameters
  • given_name – The given name of the new user.

  • family_name – The family name of the new user.

  • email – The email associated to the new user.

  • active – Whether the new user is active (not offboarded).

  • is_own_account_admin – Whether the new user is an admin of their own account.

  • can_octomize – Whether the new user has the ability to trigger octomizations.

  • account_uuid – uuid corresponding to the account that the new user should be associated to.

Returns

the newly created user

cancel_workflow(uuid: str)octomizer.workflow.Workflow

Cancels the workflow with the given id.

Parameters

uuid – the id of the workflow to cancel.

Returns

the requested workflow, if it exists.

property channel: grpc.Channel

Returns the underlying gRPC channel used by this OctomizerClient.

delete_model(uuid: str)

Deletes the model with the given id. This deletes all ModelVariants and Workflows associated with a model, and cannot be undone.

Parameters

uuid – the id of the model to delete.

get_account(uuid: str)octoml.octomizer.v1.users_pb2.Account

Returns the account with the given id.

Parameters

uuid – the id of the account to get.

Returns

the requested account, if it exists.

get_account_users(uuid: str)Iterator[octoml.octomizer.v1.users_pb2.User]

Returns all users associated with the account with the given uuid.

Parameters

uuid – the id of the account whose users to get.

Returns

all the users in the account

get_current_user()octoml.octomizer.v1.users_pb2.User

Returns the currently-authenticated user.

get_dataref(uuid: str)octoml.octomizer.v1.datarefs_pb2.DataRef

Returns the DataRef with the given id.

Parameters

uuid – the id of the DataRef to get.

Returns

the requested DataRef, if it exists.

get_hardware_targets()List[octoml.octomizer.v1.hardware_targets_pb2.HardwareTarget]

Gets the available hardware targets for the current user’s account.

Returns

the list of hardware targets available to the user’s account.

get_model(uuid: str)octomizer.model.Model

Returns the model with the given id.

Parameters

uuid – the id of the model to get.

Returns

the requested model, if it exists.

get_model_variant(uuid: str)octomizer.model_variant.ModelVariant

Returns the model variant with the given id.

Parameters

uuid – the id of the model variant to get.

Returns

the requested model variant, if it exists.

get_usage(start_time: Optional[datetime.datetime] = None, end_time: Optional[datetime.datetime] = None, account_uuid: Optional[str] = None)List[octoml.octomizer.v1.octomizer_service_pb2.HardwareUsage]

Returns the usage records associated with the user’s account.

Parameters
  • start_time – Timestamp of the beginning of when to check usage metrics. The default start_time is the first day of the month.

  • end_time – Timestamp of the end of when to check usage metrics. The default end_time is the first day of next month.

  • account_uuid – account_uuid to check account usage against. The default account_uuid is the uuid associated to the active user’s account.

Returns

a list of usage records associated with the given account.

get_user(uuid: str)octoml.octomizer.v1.users_pb2.User

Returns the user with the given id.

Parameters

uuid – the id of the user to get.

Returns

the requested user, if it exists.

get_workflow(uuid: str)octomizer.workflow.Workflow

Returns the workflow with the given id.

Parameters

uuid – the id of the workflow to get.

Returns

the requested workflow, if it exists.

property host: str

Returns the OctoML host that the OctomizerClient connects to.

list_models()Iterator[octomizer.model.Model]

Returns all Models associated with the current user.

Returns

all the current user’s Models.

list_users(account_uuid: Optional[str] = None)Iterator[octoml.octomizer.v1.users_pb2.User]

Returns all users associated with the account with the given uuid.

Parameters

uuid – the id of the account whose users to get.

Returns

all the users in the account

property stub: octoml.octomizer.v1.octomizer_service_pb2_grpc.OctomizerServiceStub

Return the underlying gRPC client stub used by this OctomizerClient. This is useful for cases where you wish to invoke gRPC calls directly.

update_user(user_uuid: str, given_name: Optional[str] = None, family_name: Optional[str] = None, email: Optional[str] = None, account_uuid: Optional[str] = None, active: Optional[bool] = None, is_own_account_admin: Optional[bool] = None, can_octomize: Optional[bool] = None)octoml.octomizer.v1.users_pb2.User

Updates a user’s data with the provided parameters

Parameters
  • user_uuid – Uuid corresponding to the user whose information should be updated.

  • given_name – The new given name for the user.

  • family_name – The new family name for the user.

  • email – A new email associated to the user.

  • account_uuid – The uuid corresponding to the new account that the user should be associated to.

  • active – A new value for whether the user is active (not offboarded).

  • is_own_account_admin – A new value for whether the user is an admin of their own account.

  • can_octomize – A new value for whether the user has the ability to trigger octomizations.

Returns

the updated user

octomizer.logging_ module

octomizer.model module

Generic wrapper for Models in the OctoML Platform.

octomizer.model.DEFAULT_JOB_POLL_INTERVAL_SECONDS = 5

The default number of seconds to wait between polling for job statuses.

octomizer.model.DEFAULT_JOB_TIMEOUT_SECONDS = 1200

The default number of seconds to wait for a job to finish.

class octomizer.model.Model(client: octomizer.client.OctomizerClient, name: Optional[str] = None, model: Optional[Union[bytes, str]] = None, description: Optional[str] = None, labels: Optional[List[str]] = None, uuid: Optional[str] = None, proto: Optional[octoml.octomizer.v1.models_pb2.Model] = None, model_format: Optional[str] = None, timeout: int = 1200)

Bases: object

Represents a generic Model in the OctoML Platform.

get_model_variant(uuid: str)octomizer.model_variant.ModelVariant

Retrieves the ModelVariant with the given id associated with this Model.

Parameters

uuid – the id of the ModelVariant to retrieve.

Returns

the ModelVariant associated with this Model that has the given id.

get_uploaded_model_variant()octomizer.model_variant.ModelVariant

Returns the original, uploaded ModelVariant for this Model.

get_workflow(uuid: str)octomizer.workflow.Workflow

Retrieves the Workflow with the given id associated with this Model.

Parameters

uuid – the id of the Workflow to retrieve.

Returns

the Workflow associated with this Model that has the given id.

property inputs: Tuple[Dict[str, List[int]], Dict[str, str]]

Return the input shapes and dtypes for this Model. Shapes are expected to be positive but -1 can be used as a sentinel when the dim is unknown and the user is expected to clarify.

list_model_variants()Iterator[octomizer.model_variant.ModelVariant]

Retrieves all ModelVariants associated with this Model.

Returns

all ModelVariants associated with this Model.

list_workflows()Iterator[octomizer.workflow.Workflow]

Retrieves all Workflows associated with this Model.

Returns

all Workflows associated with this Model.

property proto: octoml.octomizer.v1.models_pb2.Model

Return the underlying protobuf describing this Model.

property status: ingest_model_status_pb2.IngestModelStatus.Status

Return the status of the IngestModel job.

static upload_data(client: octomizer.client.OctomizerClient, model_bytes: bytes)
property uuid: str

Return the UUID for this Model.

wait_for_ingestion(timeout: int = 1200)octoml.octomizer.v1.ingest_model_status_pb2.IngestModelStatus
exception octomizer.model.ModelCreationError

Bases: RuntimeError

octomizer.model_inputs module

octomizer.model_inputs.inputs_are_dynamic(input_shapes: Dict[str, List[int]])bool

Returns True if inputs are dynamic, i.e. at least one shape is -1.

octomizer.model_inputs.inputs_are_valid(input_shapes: Dict[str, List[int]], input_dtypes: Dict[str, str])Optional[str]

Returns None when this model’s inputs are valid, otherwise returns a message indicating what updates are necessary to fix the inputs.

octomizer.model_inputs.inputs_to_input_proto(input_shapes: Optional[Dict[str, List[int]]] = None, input_dtypes: Optional[Dict[str, str]] = None)octoml.octomizer.v1.model_inputs_pb2.ModelInputs

Constructs ModelInputs message from input_shapes, input_dtypes dicts

Parameters
  • input_shapes – optional dict of input name to input shape.

  • input_dtypes – optional dict of input name to input dtype.

Returns

ModelInputs proto constructed from the input dicts.

octomizer.model_variant module

Generic wrapper for ModelVariants in the OctoML Platform.

class octomizer.model_variant.AutoTVMOptions(kernel_trials: int = 500, exploration_trials: int = 0, random_trials: int = 0, early_stopping_threshold: int = 100)

Bases: object

Specifies options for autotuning using AutoTVM.

early_stopping_threshold: int = 100

Threshold to terminate autotuning if results have not improved in this many iterations.

exploration_trials: int = 0

[experimental, use at your own risk] Minimum number of trials to tune from scratch during autotuning. Note for each tuning job, max(kernel_trials - cached trials, exploration_trials) number of trials are actively tuned.

kernel_trials: int = 500

Number of trials for each kernel during autotuning – records are pulled from a cache if available, and the remaining trials are actively tuned.

random_trials: int = 0

[experimental, use at your own risk] On top of any cached trials, this indicates the maximum additional random records from the cache to seed autotuning if available.

class octomizer.model_variant.AutoschedulerOptions(trials_per_kernel: int = 1000, early_stopping_threshold: int = 250)

Bases: object

Specifies options for autotuning using Autoscheduler.

early_stopping_threshold: int = 250

Threshold to terminate autotuning if results have not improved in this many iterations.

trials_per_kernel: int = 1000

Number of trials for each kernel during autotuning.

class octomizer.model_variant.ModelVariant(client: octomizer.client.OctomizerClient, model: octomizer.model.Model, uuid: Optional[str] = None, proto: Optional[octoml.octomizer.v1.models_pb2.ModelVariant] = None)

Bases: object

Represents a ModelVariant on the OctoML Platform.

benchmark(platform: str, num_benchmark_trials: int = 10, num_runs_per_trial: int = 3, relay_opt_lvl: int = 3, enable_profiler: bool = False, tvm_num_threads: int = 0, untuned_tvm: bool = False, input_shapes: Optional[Dict[str, List[int]]] = None, input_dtypes: Optional[Dict[str, str]] = None, create_package: bool = False, package_name: Optional[str] = None, use_onnx_engine: bool = False)octomizer.workflow.Workflow

Benchmark this ModelVariant. This is a convenience function that creates a Workflow consisting of a single benchmarking stage.

Parameters
  • platform – The hardware platform to target. Available platforms can be queried via the get_hardware_targets method on an OctomizerClient. If you would like to benchmark on other hardware platforms, please submit a feature request here.

  • num_benchmark_trials – Number of benchmarking trials to execute.

  • num_runs_per_trial – Number of benchmarks to run per trial.

  • relay_opt_lvl – The Relay optimization level to use, if the model format is Relay.

  • enable_profiler – Whether to enable the debug runtime when benchmarking, if the model format is Relay.

  • tvm_num_threads – Number of cores (not threads) that TVM runtime uses when running inference. Setting to 0 enables TVM to automatically select number of cores if the model format is Relay.

  • untuned_tvm – Whether this is a baseline untuned TVM benchmark.

  • create_package – Whether a package should be created or not. Defaults to False.

  • package_name – The name of the package. If unset or empty, will default to the name of the model. Note: Non-alphanumeric characters in the name will be replaced with underscores (‘_’) and trailing/leading underscores will be stripped. Valid package names must only contain lower case letters, numbers, and single (non leading/trailing) underscores (‘_’).

:return A Workflow instance.

create_workflow(workflow_spec: octoml.octomizer.v1.workflows_pb2.Workflow)octomizer.workflow.Workflow

Creates a new Workflow for this ModelVariant.

Parameters

workflow_spec – the specification for the Workflow to be created.

Returns

the new Workflow.

property format: octomizer.model_variant.ModelVariantFormat

Returns the ModelVariantFormat of this ModelVariant.

Returns

the ModelVariantFormat of this ModelVariant.

property inputs: Tuple[Dict[str, List[int]], Dict[str, str]]

Return the input shapes and dtypes for this ModelVariant. Shapes are expected to be positive but -1 can be used as a sentinel when the dim is unknown and the user is expected to clarify.

octomize(platform: str, relay_opt_lvl: int = 3, enable_profiler: bool = False, tvm_num_threads: int = 0, kernel_trials: Optional[int] = None, exploration_trials: Optional[int] = None, random_trials: Optional[int] = None, early_stopping_threshold: Optional[int] = None, num_benchmark_trials: int = 10, num_runs_per_trial: int = 3, package_type: Optional[octomizer.package_type.PackageType] = None, input_shapes: Optional[Dict[str, List[int]]] = None, input_dtypes: Optional[Dict[str, str]] = None, tuning_options: Optional[Union[octomizer.model_variant.AutoTVMOptions, octomizer.model_variant.AutoschedulerOptions]] = None, create_package: bool = True, package_name: Optional[str] = None)octomizer.workflow.Workflow

Accelerate this ModelVariant. This is a convenience function that creates a Workflow consisting of autotuning, benchmarking, and (optional) packaging stages.

Parameters
  • platform

    The hardware platform to target. Available platforms can be queried via the get_hardware_targets method on an OctomizerClient. If you would like to benchmark on other hardware platforms, please submit a feature request here.

  • relay_opt_lvl – The Relay optimization level to use.

  • enable_profiler – Whether to enable the debug runtime when benchmarking.

  • tvm_num_threads – Number of threads that TVM runtime uses when running inference. Setting to 0 allows TVM to decide.

  • kernel_trials – deprecated, specify tuning_options instead.

  • exploration_trials – deprecated, specify tuning_options instead.

  • random_trials – deprecated, specify tuning_options instead.

  • early_stopping_threshold – deprecated, specify tuning_options instead.

  • num_benchmark_trials – Number of benchmarking trials to execute.

  • num_runs_per_trial – Number of benchmarks to run per trial.

  • package_type – Deprecated, all available package types for the engine will be packaged.

  • input_shapes – dict mapping input name to shape. Must be provided if input_dtypes is provided.

  • input_dtypes – dict mapping input name to dtype. Must be provied if input_shapes is provided.

  • tuning_options – options to control the autotuning search. Provide either AutoTVMOptions or AutoschedulerOptions.

  • create_package – Whether a package should be created or not. Defaults to True.

  • package_name – The name of the package. If unset or empty, will default to the name of the model. Note: Non-alphanumeric characters in the name will be replaced with underscores (‘_’) and trailing/leading underscores will be stripped. Valid package names must only contain lower case letters, numbers, and single (non leading/trailing) underscores (‘_’).

:return A Workflow instance.

:raises ValueError when the package name is not a valid package name.

package(platform: str, relay_opt_lvl: Optional[int] = None, tvm_num_threads: Optional[int] = None, package_type: Optional[octomizer.package_type.PackageType] = None, input_shapes: Optional[Dict[str, List[int]]] = None, input_dtypes: Optional[Dict[str, str]] = None, package_name: Optional[str] = None, package_options: Optional[octomizer.model_variant.PackageOptions] = None)octomizer.workflow.Workflow

Package this ModelVariant. This is a convenience function that creates a Workflow consisting of a single packaging stage.

Parameters
  • platform

    The hardware platform to target. Available platforms can be queried via the get_hardware_targets method on an OctomizerClient. If you would like to benchmark on other hardware platforms, please submit a feature request here.

  • relay_opt_lvl – Deprecated, specify relay optimization level with package_options

  • package_type – Deprecated, all available package types will be packaged for the runtime specified in package_options.

  • input_shapes – dict mapping input name to shape. Must be provided if input_dtypes is provided.

  • input_dtypes – dict mapping input name to dtype. Must be provied if input_shapes is provided.

  • package_name – The name of the package. If unset or empty, will default to the name of the model. Note: Non-alphanumeric characters in the name will be replaced with underscores (‘_’) and trailing/leading underscores will be stripped. Valid package names must only contain lower case letters, numbers, and single (non leading/trailing) underscores (‘_’).

  • package_options – Selects and configures runtime: TVMPackageOptions (TVM), or OnnxPackageOption (ONNX). All available package types for the specified runtime engine will be created.

:return A Workflow instance.

:raises ValueError when the package name is not a valid package name.

property uuid: str

Return the UUID for this Model.

class octomizer.model_variant.ModelVariantFormat(value)

Bases: enum.Enum

An enum representing the formats a ModelVariant can be in.

ONNX = 'onnx'
RELAY = 'relay'
TENSORFLOW = 'tensorflow'
TFLITE = 'tflite'
class octomizer.model_variant.OnnxPackageOptions

Bases: octomizer.model_variant.PackageOptions

Specifies packaging to the ONNX runtime

class octomizer.model_variant.PackageOptions

Bases: abc.ABC

class octomizer.model_variant.TVMPackageOptions(relay_opt_lvl: int = 3)

Bases: octomizer.model_variant.PackageOptions

Specifies packaging options for TVM runtime

relay_opt_lvl: int = 3

The Relay optimization level to use.

octomizer.octomizer_cli module

octomizer.octomizer_cli.print_cli_result(result)

Prints an OctoML object or a collection of OctoML objects, optionally using the JSON representation of the backing protobuf message.

octomizer.package_type module

class octomizer.package_type.PackageType(value)

Bases: enum.Enum

An enum representing the types of possible packages.

LINUX_SHARED_OBJECT = 'LINUX_SHARED_OBJECT'
PYTHON_PACKAGE = 'PYTHON_PACKAGE'

octomizer.user module

class octomizer.user.User(client: octomizer.client.OctomizerClient, given_name: Optional[str] = None, family_name: Optional[str] = None, email: Optional[str] = None, account_uuid: Optional[str] = None, active: Optional[bool] = None, is_own_account_admin: Optional[bool] = None, can_octomize: Optional[bool] = None, uuid: Optional[str] = None, proto: Optional[octoml.octomizer.v1.users_pb2.User] = None)

Bases: object

property account_uuid: str

Return the account UUID for this User.

property active: bool

Return a value for if this User is active (not offboarded).

property family_name: str

Return the family name for this User.

property given_name: str

Return the given name for this User.

property permissions: octoml.octomizer.v1.users_pb2.Permissions

Returns the permissions this User has.

property proto: octoml.octomizer.v1.users_pb2.User

Return the underlying protobuf describing this User.

property uuid: str

Return the UUID for this User.

exception octomizer.user.UserCreationError

Bases: RuntimeError

octomizer.workflow module

Resource representing a Workflow.

octomizer.workflow.DEFAULT_WORKFLOW_POLL_INTERVAL_SECONDS = 10

The default number of seconds to wait between polling for Workflow statuses.

octomizer.workflow.DEFAULT_WORKFLOW_TIMEOUT_SECONDS = 300

The default number of seconds to wait for a Workflow to finish.

class octomizer.workflow.Workflow(client: octomizer.client.OctomizerClient, model: octomizer.model.Model, uuid: Optional[str] = None, proto: Optional[octoml.octomizer.v1.workflows_pb2.Workflow] = None)

Bases: object

Represents an OctoML Workflow.

cancel()octomizer.workflow.Workflow

Cancel this Workflow.

completed()bool

Returns True if this Workflow has completed successfully.

done()bool

Returns True if this Workflow has finished.

has_benchmark_stage()bool

Returns True when the workflow contains a benchmark stage, False otherwise.

metrics()octoml.octomizer.v1.benchmark_pb2.BenchmarkMetrics

Return the BenchmarkMetrics for this Workflow.

The result is only valid if the Workflow had a Benchmark stage, and the Workflow state is WorkflowState.COMPLETED.

package_url(package_type: Optional[octomizer.package_type.PackageType] = <PackageType.PYTHON_PACKAGE: 'PYTHON_PACKAGE'>)str

Return the URL of the package output for this Workflow.

Parameters

package_type – The package type we want to get the url for. Defaults to Python wheel.

The result is only valid if the Workflow had a Package stage, the package type is available for the specified runtime engine, and the Workflow state is WorkflowState.COMPLETED.

:raises ValueError when a package type isn’t available or when there is

no package DataRef UUID.

progress()octoml.octomizer.v1.workflows_pb2.WorkflowProgress

Returns the progress of this Workflow.

property proto: octoml.octomizer.v1.workflows_pb2.Workflow

Return the raw representation for this Workflow.

refresh()octomizer.workflow.Workflow

Get the latest status of this Workflow from the OctoML Platform.

result()octoml.octomizer.v1.workflows_pb2.WorkflowResult

Returns the result of this Workflow. The result is only valid if the Workflow has finished running.

save_package(out_dir: str, package_type: Optional[octomizer.package_type.PackageType] = <PackageType.PYTHON_PACKAGE: 'PYTHON_PACKAGE'>)str

Download the package result for this Workflow to the given directory, and returns the full path to the package.

Out_dir

Where the package will be saved to.

Parameters

package_type – The package type we want to save. Defaults to Python wheel.

The result is only valid if the Workflow had a Package stage, and the Workflow state is WorkflowState.COMPLETED.

state()workflows_pb2.WorkflowStatus.WorkflowState

Returns the current state of this Workflow.

status()octoml.octomizer.v1.workflows_pb2.WorkflowStatus

Returns the current status of this Workflow.

property uuid: str

Return the UUID for this Workflow.

wait(timeout: typing.Optional[int] = 300, poll_interval: int = 10, poll_callback: typing.Optional[typing.Callable[[Workflow], None]] = None)workflows_pb2.WorkflowStatus.WorkflowState

Waits until this Workflow has finished or the given timeout has elapsed.

Parameters
  • timeout – the number of seconds to wait for this Workflow.

  • poll_interval – the number of seconds to wait between polling for the status of this workflow.

  • poll_callback – Optional callback invoked with self as an argument each time the Workflow status is polled.

Returns

the terminal state of this workflow.

Raise

WorkflowTimeoutError if timeout seconds elapse before the Workflow reaches a terminal state.

exception octomizer.workflow.WorkflowTimeoutError

Bases: Exception

Indicates a Workflow timed out while being awaited.

Module contents