OpenAI Embedder

Overview

  • Transform-type Snap
  • Works in Ultra Pipelines when the batch size is set to 1.

You can use this Snap to generate an embedding vector based on the provided input data. The Snap processes the input document to generate the corresponding embedding for the output document, regardless of batch size.


OpenAI Embedder Overview

Prerequisites

None.

Limitations

  • The free tier allows up to 3 requests per minute, totaling 200 requests per day. The first paid tier offers 500 requests per minute with no daily limit.

Snap Views

View Description Examples of Upstream and Downstream Snaps
Input This Snap has at the most one document input view. The Snap requires the text to generate the embedding vector. Mapper
Output This Snap has at the most one document output view. The Snap provides the embedded vectors, and the original input document. Mapper
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when the Snap encounters an error.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

Note:
  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
  • Expression icon (): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
  • Add icon (): Indicates that you can add fields in the field set.
  • Remove icon (): Indicates that you can remove fields from the field set.
Field / Field Set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if there are more than one of the same Snap in the pipeline.

Default value: OpenAI Embedder

Example: Embedding data
Model name String/Expression/Suggestion

Specify the model name to use for the embeddings vector calculation. Learn more about the list of models from OpenAI that are compatible with the embeddings API.

Default value: N/A

Example: text-embedding-3-small
Batch size Integer/Expression
Required. Specify the number of documents batched per request. If running in Ultra mode, this must be set to 1.
Note:
  • This field does not support input values from the upstream Snap.
  • When you configure the batch size with a value greater than 1, the Snap accumulates input documents until the specified batch size is attained. Subsequently, a single request is made to the endpoint with the accumulated batch size.
  • Output documents are sent one at a time in the same order they were received.

Maximum value: 2048

Default value: N/A

Example: 50
Text to embed String/Expression

Required. Specify the text to generate the embedding vector.

Default value: N/A

Example: $msg
Model parameters

Appears when you enable expression for the Model name and/or specify the value as text-embedding-3.

Specify the parameters to tune the model runtime.
Dimensions Number/Expression

Specify the output dimensions of the embedding vector for the text-embedding-3 models. Value must be between 1 and the maximum dimension for the model. Learn more about how to get embeddings.

Minimum value: 1

Default value: N/A

Example: 1536
Snap execution Dropdown list
Select one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute. Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only. Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled. Disables the Snap and all Snaps that are downstream from it.

Default value: Validate & Execute

Example: Execute only

Examples