Set up an OpenSearch Index

You must set up the index and the index mapping (schema) in AWS to use the OpenSearch Snaps. The OpenSearch Upsert Snap fetches data according to the current mapping structure, which means that fields not defined in the mapping will not be uploaded, even if they exist in the input stream.

Note: For OpenSearch Query Snap, you can use all the vector search methods only when you set the index correctly, especially Approximate k-NN search.

Here is an example of how to create an index and set its mapping:

   PUT snap-markdown-doc{
  "settings": {
  "index": {
  "knn": true,
  "knn.algo_param.ef_search": 100
  }
  },
  "mappings" : {
  "properties": {
  "index": { "type": "integer" },
  "vector": { 
  "type": "knn_vector", 
  "dimension": 1536,
  "method": {
  "name": "hnsw",
  "space_type": "cosinesimil",
  "engine": "nmslib"
  }
  },
  "text": { "type": "text" },
  "source": { "type": "text" }
  }
  }
  }

Note: Without setting the index knn to true, you cannot do Approximate k-NN search. However, you may still use the Brute force search method such as Script score k-NN, which could be slower when the search space is large.


      "settings": {
      "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
      }   
      }

Note: For the vector field, the type should be knn_vector and the dimension should exactly match the vector you upload. Learn more: k-NN index.

 "vector": { 
        "type": "knn_vector", 
        "dimension": 1536,
        "method": {
        "name": "hnsw",
        "space_type": "cosinesimil",
        "engine": "nmslib"
        }
        }

Note: The OpenSearch Upsert Snap only upserts fields defined in the mapping in the database (and their field types must also match). Any additional fields are ignored. Ensure that all the fields you want to upsert are defined in the mapping. In the example above, we defined these fields and their types:


    {
    "index": "integer",
    "vector": "knn",
    "text": "text",
    "source":"text"
    }