OpenAI Add Vector Store File

Overview

You can use this Snap to add an existing file from OpenAI storage to the specified vector store with the specific vector store ID and file ID, converting it into a collection of vectors. Configure the chunk size and overlap size to optimize the file's division and indexing into vectors for the most efficient retrieval and analysis.


OpenAI Add Vector Store File Snap Overview

Prerequisites

Known issues

None.

Limitations

  • When you add a file to the vector store, a successful response does not guarantee that OpenAI has completed processing the file. The file's status will be marked as in-progress when it is being processed, during which specific operations, such as removing the vector store file, fail with the error message: 409 The vector store was updated by another process. Please reload and try again.

Snap views

View Description Examples of upstream and downstream Snaps
Input This Snap supports a maximum of one document input view. You must provide vector store and file ID.
Output This Snap has a maximum of one document output view. The Snap provides the details of the added file from the Create vector store file API.
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Note:
  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
  • Expression icon (): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
  • Add icon (Plus Icon): Indicates that you can add fields in the field set.
  • Remove icon (Minus Icon): Indicates that you can remove fields from the field set.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: OpenAI Add Vector Store File

Example: Add product embeddings
Vector store ID String/Expression/Suggestion

Required. Specify the ID of the vector store.

Default value: N/A

Example: vs_XYZ12345ABCDTest123
File ID String/Expression/Suggestion

Required. Specify the ID of the file.

Default value: N/A

Example: file-123ABC456DEF789GHIJKL
Chunking strategy Configure the chunking strategy to optimize how the file is divided and indexed into vectors for the most efficient retrieval and analysis.
Important: Both the Max chunk size tokens and Chunk overlap tokens must be either null or non-null.
Max chunk size tokens Integer/Expression
Specify the maximum number of tokens in each chunk.
Tip: This field must either be configured or blank to avoid an error.
Default value: 800

Example: 50
Chunk overlap tokens Integer/Expression
Specify the number of tokens to overlap between chunks.
Tip:
  • This field must either be configured or blank to avoid an error.
  • Ensure the Chunk overlap tokens is no more than half of the Max chunk size tokens.
Default value: 400

Example: 50
Snap execution Dropdown list
Select one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Example: Validate & Execute

Troubleshooting

Invalid chunking strategy configuration

Both Max chunk size tokens and Chunk overlap tokens must be either null or non-null.

Ensure both properties are configured or leave them blank.

Invalid chunk overlap tokens

Chunk overlap tokens must not exceed half of the Max chunk size tokens.

Ensure the Chunk overlap tokens is no more than half of the Max chunk size tokens.

409 The vector store was updated by another process. Please reload and try again.

The file being updated or modified is still being processed by OpenAI.

Wait until the file’s status is no longer marked as in-progress and marked completed before attempting to remove or update it. You can verify the file's status using the OpenAI List Vector Store Files Snap.

Examples