datasets

Use datasets to curate spans from your LLM applications for testing. Read our quickstart guide.

To use in your code, import the following:

from arize.experimental.datasets import ArizeDatasetsClient

client

class ArizeDatasetsClient(developer_key, api_key, host='flight.arize.com', port=443, scheme='grpc+tls', otlp_endpoint='https://otlp.arize.com/v1')

Bases: object

ArizeDatasetsClient is a client for interacting with the Arize Datasets API.

Parameters:

• developer_key (str, required) – Arize provided developer key associated with your user profile, located on the space settings page.

• api_key (str, required) – Arize provided API key associated with your user profile, located on the space settings page.

• host (str, optional) – URI endpoint host to send your export request to Arize AI. Defaults to “{DEFAULT_ARIZE_FLIGHT_HOST}”.

• port (int, optional) – URI endpoint port to send your export request to Arize AI. Defaults to {DEFAULT_ARIZE_FLIGHT_PORT}.

• scheme (str, optional) – Transport scheme to use for the connection. Defaults to “{DEFAULT_TRANSPORT_SCHEME}”.

• otlp_endpoint (str, optional) – OTLP endpoint to send experiment traces to Arize. Defaults to “{DEFAULT_ARIZE_OTLP_ENDPOINT}”.

create_dataset(space_id, dataset_name, dataset_type, data, convert_dict_to_json=True)

Create a new dataset.

Parameters:

• space_id (str) – The ID of the space where the dataset will be created.

• dataset_name (str) – The name of the dataset.

• dataset_type (DatasetType) – The type of the dataset.

• data (pd.DataFrame) – The data to be included in the dataset.

• convert_dict_to_json (bool, optional) – Convert dictionary columns to JSON strings for default JSON str columns per Open Inference. Defaults to True.

Returns:

The ID of the created dataset, or None if the creation failed.

Return type:

str

delete_dataset(space_id, dataset_id)

Delete a dataset.

Parameters:

• space_id (str) – The ID of the space where the dataset is located.

• dataset_id (str) – The ID of the dataset to delete.

Returns:

True if the dataset was successfully deleted, False otherwise.

Return type:

bool

get_dataset(space_id, dataset_id=None, dataset_name=None, dataset_version=None, convert_json_str_to_dict=True)

Get the data of a dataset.

Parameters:

• space_id (str) – The ID of the space where the dataset is located.

• dataset_id (str) – Dataset id. Required if dataset_name is not provided.

• dataset_name (str) – Dataset name. Required if dataset_id is not provided.

• dataset_version (str, optional) – version name of the dataset Defaults to “” and gets the latest version by based on creation time

• convert_json_str_to_dict (bool, optional) – Convert JSON strings to Python dictionaries. For default JSON str columns per Open Inference. Defaults to True.

Returns:

The data of the dataset.

Return type:

pd.DataFrame

get_dataset_versions(space_id, dataset_id)

Get the versions of a dataset.

Parameters:

• space_id (str) – The ID of the space where the dataset is located.

• dataset_id (str) – The ID of the dataset to get versions info for.

Returns:

A list of dictionaries representing the versions of the dataset.

Return type:

List[Dict[str, Any]]

get_experiment(space_id, experiment_name=None, dataset_name=None, experiment_id=None)

Retrieve experiment data from Arize.

Parameters:

• space_id (str) – The ID of the space where the experiment is located.

• experiment_name (Optional[str]) – The name of the experiment. Required if experiment_id is not provided.

• dataset_name (Optional[str]) – The name of the dataset associated with the experiment. Required if experiment_id is not provided.

• experiment_id (Optional[str]) – The ID of the experiment. Required if experiment_name and dataset_name are not provided.

Returns:

A pandas DataFrame containing the experiment data,

or None if the retrieval fails.

Return type:

Optional[pd.DataFrame]

Raises:

• ValueError – If neither experiment_id nor both experiment_name and dataset_name are provided.

• RuntimeError – If the experiment retrieval fails.

Note

You must provide either the experiment_id or both the experiment_name and dataset_name.

list_datasets(space_id)

List all datasets in a space.

Parameters:

space_id (str) – The ID of the space to list datasets for.

Returns:

A list of dictionaries representing the datasets in the space.

Return type:

List[Dict[str, Any]]

run_experiment(space_id, experiment_name, task, dataset_df=None, dataset_id=None, dataset_name=None, evaluators=None, dry_run=False, concurrency=3, set_global_tracer_provider=False, exit_on_error=False)

Run an experiment on a dataset and upload the results.

This function initializes an experiment, retrieves or uses a provided dataset, runs the experiment with specified tasks and evaluators, and uploads the results.

Parameters:

• space_id (str) – The ID of the space where the experiment will be run.

• experiment_name (str) – The name of the experiment.

• task (ExperimentTask) – The task to be performed in the experiment.

• dataset_df (Optional[pd.DataFrame], optional) – The dataset as a pandas DataFrame. If not provided, the dataset will be downloaded using dataset_id or dataset_name. Defaults to None.

• dataset_id (Optional[str], optional) – The ID of the dataset to use. Required if dataset_df and dataset_name are not provided. Defaults to None.

• dataset_name (Optional[str], optional) – The name of the dataset to use. Used if dataset_df and dataset_id are not provided. Defaults to None.

• evaluators (Optional[Evaluators], optional) – The evaluators to use in the experiment. Defaults to None.

• dry_run (bool) – If True, the experiment result will not be uploaded to Arize. Defaults to False.

• concurrency (int) – The number of concurrent tasks to run. Defaults to 3.

• set_global_tracer_provider (bool) – If True, sets the global tracer provider for the experiment. Defaults to False.

• exit_on_error (bool) – If True, the experiment will stop running on first occurrence of an error.

Returns:

A tuple of experiment ID and experiment result DataFrame. If dry_run is True, the experiment ID will be an empty string.

Return type:

Tuple[str, pd.DataFrame]

Raises:

• ValueError – If dataset_id and dataset_name are both not provided, or if the dataset is empty.

• RuntimeError – If experiment initialization, dataset download, or result upload fails.

update_dataset(space_id, dataset_id, data)

Update an existing dataset by creating a new version.

Parameters:

• space_id (str) – The ID of the space where the dataset is located.

• dataset_id (str) – The ID of the dataset to update.

• data (pd.DataFrame) – The updated data to be included in the dataset.

Returns:

The ID of the updated dataset, or None if the update failed.

Return type:

str