AI-TokenHub

English

简体中文

English

简体中文

Appearance

Request
Response

Multimodal Embedding API ​

When you need to process video, images, and text semantically — such as image-to-image search, semantic retrieval, etc. — you can call the multimodal embedding service to convert video, images, and text into vectors for analyzing their semantic relationships. This document provides detailed parameter descriptions for the interface.

 

POST https://ai-tokenhub.com/api/v3/embeddings/multimodal

Request Parameters ​

Request Information ​

ItemValue
Request MethodPOST
Official Request URLhttps://ai-tokenhub.com/api/v3/embeddings/multimodal
Content-Typeapplication/json
AuthenticationAPI Key Authentication (Bearer Token)

Request Headers ​

Field NameRequiredDescription
Authorization✓Format: Bearer <$YOUR_API_KEY>, where <$YOUR_API_KEY> is the API key you obtained from the TokenHub platform
Content-Type✓Fixed value: application/json

Request Body ​

model string Required ​

The Model ID of the model you need to call.

input object[] Required ​

The list of content to be vectorized. List elements support text, image, and video information.

Text Information object ​

The text content portion of the input to be vectorized by the model.

input.type string Required ​

The type of input content, should be text here.

input.text string Required ​

The text content input to the model. A single text is encoded in utf-8 and its length does not exceed the maximum input token count of the model.

Image Information object ​

The image information portion of the input to be vectorized by the model.

input.type string Required ​

The type of input content, should be image_url here.

input.image_url object Required ​

The image object input to the model.

input.image_url.url string Required ​

Image information, can be an image URL or image Base64 encoding.

  • Image URL: Please ensure the image URL is accessible.
  • Base64 encoding: Please follow the format data:image/{image_format};base64,{image_base64_encoding}.
Video Information object ​

The video information portion of the input to be vectorized by the model.

Note
  • Format: .mp4, .avi, .mov (lowercase).
  • Single video file must be under 50MB.
  • Audio information in video files is not supported for understanding.
input.type string Required ​

The type of input content, should be video_url here.

input.video_url object Required ​

The video object input to the model.

input.video_url.url string Required ​

Supports video links or video Base64 encoding.

input.video_url.fps number ​

Range: [0.2, 5]. The number of images extracted per second from the video. Higher values provide finer understanding of visual changes; lower values reduce perception of changes but use fewer tokens and are faster.

input.video_url.max_video_tokens integer ​

Range: [10240, 204800]. Maximum video tokens. Larger values preserve more video information. When exceeded, frames are truncated to the upper limit.

input.video_url.min_frame_tokens integer ​

Range: [16, 128]. Minimum tokens per frame, controlling the lower bound of per-frame compression. Smaller values allow more compression but reduce per-frame detail.

input.video_url.max_frame_tokens integer ​

Range: [128, 640], and must be greater than or equal to min_frame_tokens. Maximum tokens per frame, controlling the upper bound of per-frame quality.

input.video_url.min_frames integer ​

Range: [5, 16]. Minimum number of extracted frames, ensuring sufficient processing frames for short videos.

Note
  • The video custom frame extraction parameters fps, max_video_tokens, min_frame_tokens, max_frame_tokens, min_frames are all optional.
  • When these fields are not provided, default values based on the model's frame extraction strategy are used.
  • The frame extraction strategy works synergistically: after extracting by fps, each frame's token limit is controlled by max_frame_tokens, total tokens do not exceed max_video_tokens, and actual frame count is no less than min_frames.
  • Frame count directly affects video embedding token consumption, but more frames is not always better; if per-frame tokens are too few, video embedding quality will also decrease.

encoding_format string / null ​

Default value float. Range: float, base64, null. The return format of the embedding.

dimensions integer ​

Default value 2048. Range: 1024 or 2048. Used to specify the output vector dimension.

instructions string ​

Inference prompt. When provided by the user, it is used directly; when not provided, a default value is generated based on the input modality.

sparse_embedding object ​

Sparse vector switch configuration. Only supported for pure text input.

sparse_embedding.type string ​

Default value disabled. Range: enabled, disabled. Controls whether to output sparse vectors.

  • disabled: Only output dense vectors.
  • enabled: Output both dense and sparse vectors.

multi_embedding object ​

Multi-vector output configuration, controlling whether to output multi-vectors and their compression method. When enabled, each input returns an additional set of token-level vectors (2D array) in addition to the dense vector.

multi_embedding.type string ​

Default value disabled. Range: enabled, disabled. Multi-vector output switch.

  • type="disabled": Do not output multi-vectors.
  • type="enabled": In addition to dense vectors, return data.multi_embedding.
multi_embedding.compression string ​

The compression method for multi-vectors, only effective when type="enabled". When provided, data.multi_embedding returns a compressed and base64-encoded string, significantly reducing transmission size. The client needs to decode according to the decoding protocol.

Range: blosc2, zstd.

  • blosc2: Optimized specifically for numeric arrays, compresses the same vectors to a smaller size, saving more bandwidth.
  • zstd: General-purpose high-speed compression method, with ready-made decoding libraries available in almost all languages, easy to integrate and fast to decompress.
Note
  • When multi_embedding.type="disabled", the compression field is ignored.
  • When multi_embedding.type="enabled" and compression is not provided, uncompressed 2D arrays are returned.
  • multi_embedding can be enabled simultaneously with sparse_embedding, but sparse_embedding still only supports pure text input.
  • multi_embedding does not allow empty object {}, otherwise returns 400 MissingParameter.

Response Parameters ​

id string ​

The unique identifier of this request.

model string ​

The actual model name and version used for this request.

created integer ​

The Unix timestamp (seconds) of when this request was created.

object string ​

Fixed as list.

data object ​

The algorithm output content of this request.

data.embedding float[] ​

The vectorization result of the corresponding content.

data.sparse_embedding array ​

Sparse vector, only returned when sparse_embedding.type="enabled". Each member has the structure {"index": dimension_index, "value": non_zero_value}, returning only non-zero elements.

data.multi_embedding float[][] / string ​

Multi-vector, only returned when multi_embedding.type="enabled".

  • When compression is not enabled, returns float[][], where the outer dimension represents tokens and the inner dimension represents each token's sub-vector.
  • When compression=blosc2/zstd is enabled, returns a compressed and Base64-encoded string.
Note
  • When data.multi_embedding is returned as a compressed string, the client should decode as follows:
  • 1. Decode the Base64 string and decompress using the algorithm specified by compression.
  • 2. Parse the decompressed binary data as fp16 (2 bytes, little-endian) into a 1D array.
  • 3. Reshape the 1D array into a 2D array as [num_tokens, dimensions].
  • In compressed return scenarios, the response will not separately return shape or dtype fields.
data.object string ​

Fixed as embedding.

usage object ​

The token usage of this request.

usage.prompt_tokens integer ​

The number of input content tokens.

usage.total_tokens integer ​

The total number of tokens consumed by this request (input + output).

usage.prompt_tokens_details object ​

Detailed information about the token usage of the input content.

usage.prompt_tokens_details.text_tokens integer ​

The token count corresponding to text content in the input, and the tokens generated by the video content timeline. To ensure model performance, when images or videos are provided, a small number of preset text tokens are generated, producing additional text_tokens.

usage.prompt_tokens_details.image_tokens integer ​

The token count corresponding to image content and video frame extraction in the input.