docs: Update README and compilation guides for clarity and consistency, including path corrections and improved formatting. Add copyright notices to source files and adjust file permissions for several scripts and directories.
This commit is contained in:
dian.yuan
parent
f960c5030d
commit
bd891a96dd
136 changed files with 14413 additions and 9399 deletions
|
|
@ -1,159 +1,159 @@
|
|||
# CLIP
|
||||
|
||||
## 1. Overview
|
||||
|
||||
This demo demonstrates how to run CLIP (Contrastive Language-Image Pre-Training) image-text matching using AMLNNLite. The CLIP model consists of two parts: a vision encoder and a text encoder, which work together to compute similarity between images and text descriptions.
|
||||
|
||||
## 2. Model Download
|
||||
|
||||
TO DO
|
||||
|
||||
## 3. Model Conversion
|
||||
|
||||
TO DO
|
||||
|
||||
## 4. Demo Run
|
||||
|
||||
### CPP
|
||||
|
||||
#### 1. Compile
|
||||
|
||||
**Prerequisites:**
|
||||
- Android NDK (r25e recommended)
|
||||
- `ANDROID_NDK_PATH` environment variable set
|
||||
|
||||
**Build:**
|
||||
```bash
|
||||
# Build for arm64-v8a
|
||||
cd examples/clip/cpp
|
||||
AMLNN_HOME=/path/to/amlnn-toolkit ./build-android.sh -a arm64-v8a
|
||||
```
|
||||
|
||||
The executable will be generated at `build/android_arm64-v8a/clip_demo`.
|
||||
|
||||
#### 2. Run
|
||||
|
||||
```bash
|
||||
# Push executable and resources to device
|
||||
adb push build/android_arm64-v8a/clip_demo /data/local/tmp/
|
||||
adb push model/vision_model_int8_S905X5.adla /data/local/tmp/
|
||||
adb push model/text_model_int8_S905X5.adla /data/local/tmp/
|
||||
adb push tokenizer_path/ /data/local/tmp/
|
||||
|
||||
# Run on device
|
||||
adb shell
|
||||
cd /data/local/tmp
|
||||
chmod +x clip_demo
|
||||
export LD_LIBRARY_PATH=/vendor/lib64 or (/vendor/lib)
|
||||
|
||||
# Usage: ./clip_demo <vision_model> <text_model> <tokenizer_path> [--profiling]
|
||||
./clip_demo vision_model_int8_S905X5.adla text_model_int8_S905X5.adla ./tokenizer_path/
|
||||
```
|
||||
|
||||
The program will prompt for image paths and text descriptions interactively. Enter the path to an image file, then enter comma-separated text descriptions (or `skip` to use defaults). Type `exit` to quit.
|
||||
|
||||
**Argument Descriptions:**
|
||||
|
||||
| Argument | Description |
|
||||
| -------------- | ------------------------------------------------------------ |
|
||||
| vision_model | Path to vision encoder .adla model (required) |
|
||||
| text_model | Path to text encoder .adla model (required) |
|
||||
| tokenizer_path | Path to directory containing `vocab.json` and `merges.txt` (required) |
|
||||
| --profiling | Enable performance profiling output (optional) |
|
||||
|
||||
**Note:** The `tokenizer_path` should contain `vocab.json` and `merges.txt` files from the CLIP tokenizer (e.g., from `openai/clip-vit-base-patch32`).
|
||||
|
||||
### Python
|
||||
|
||||
**Prerequisites:**
|
||||
- Python 3.10
|
||||
- Required packages: `numpy`, `Pillow`, `transformers`, `amlnnlite`
|
||||
|
||||
**Install dependencies:**
|
||||
```bash
|
||||
pip install numpy Pillow transformers amlnnlite-1.0.0-cp310-cp310-linux_aarch64.whl
|
||||
```
|
||||
|
||||
**Run on device:**
|
||||
```bash
|
||||
python clip.py \
|
||||
--vision-model ./vision_model_int8_S905X5.adla \
|
||||
--text-model ./text_model_int8_S905X5.adla \
|
||||
--tokenizer-dir ./tokenizer_path \
|
||||
--image-path ./000000004505.jpg \
|
||||
--texts "a red handbag" "a blue jacket" "a red bus"
|
||||
```
|
||||
|
||||
**Interactive Mode (Recommended):**
|
||||
|
||||
If you don't provide `--image-path`, the program will run in interactive mode:
|
||||
|
||||
```bash
|
||||
python clip.py \
|
||||
--vision-model ./vision_model_int8_S905X5.adla \
|
||||
--text-model ./text_model_int8_S905X5.adla \
|
||||
--tokenizer-dir ./tokenizer_path
|
||||
```
|
||||
|
||||
The program will prompt for image paths and text descriptions. Enter an image path to process, then enter comma-separated texts to compare. Type `exit` to quit.
|
||||
|
||||
**Argument Descriptions:**
|
||||
|
||||
| Argument | Description |
|
||||
| ---------------- | ------------------------------------------------------------ |
|
||||
| --vision-model | Path to vision encoder .adla model (required) |
|
||||
| --text-model | Path to text encoder .adla model (required) |
|
||||
| --tokenizer-dir | Path to CLIPTokenizer directory (required) |
|
||||
| --image-path | Path to input image (.jpg, .png) - optional, will prompt if not provided |
|
||||
| --texts | List of text descriptions to compare (space-separated) |
|
||||
| --max-len | Maximum token sequence length, default is 64 |
|
||||
| --logit-scale | Logit scale factor, default is 100.0 |
|
||||
|
||||
**Note:** The `--tokenizer-dir` should point to the directory containing the CLIPTokenizer files. You can use a Hugging Face model ID (e.g., `openai/clip-vit-base-patch32`) or a local directory.
|
||||
|
||||
## 5. Results
|
||||
|
||||
**Performance Feedback**
|
||||
|
||||
By using the `--profiling` flag (C++) or setting the loglevel to INFO, the program provides real-time performance metrics upon completion. The console log will display essential hardware and execution details, including:
|
||||
- Hardware Information: System and ADLA library versions.
|
||||
- Model Overview: Basic input/output configurations.
|
||||
- NPU Metrics: Total inference time (latency) and total DRAM bandwidth consumption.
|
||||
|
||||
**Interactive Mode Example:**
|
||||
|
||||
```bash
|
||||
$ ./clip_demo vision_model_int8_S905X5.adla text_model_int8_S905X5.adla ./tokenizer_path
|
||||
|
||||
[Info] Models initialized successfully.
|
||||
|
||||
============================================================
|
||||
[Info] Image Path (or 'exit' to quit):
|
||||
000000004505.jpg
|
||||
[Info] Enter text descriptions (comma-separated, or 'skip' for defaults):
|
||||
a red handbag, a blue jacket, a red bus
|
||||
|
||||
[Info] Processing image: 000000004505.jpg
|
||||
[Info] Image embedding size: 512
|
||||
[Info] Processing 3 text(s)...
|
||||
[Info] Text embeddings size: 3 x 512
|
||||
|
||||
============================================================
|
||||
CLIP Image-Text Matching Results
|
||||
============================================================
|
||||
Image: 000000004505.jpg
|
||||
logit_scale: 100.000000
|
||||
------------------------------------------------------------
|
||||
[1] prob=0.999975 sim=0.327895 text='a red bus'
|
||||
[2] prob=0.000016 sim=0.217690 text='a red handbag'
|
||||
[3] prob=0.000008 sim=0.211029 text='a blue jacket'
|
||||
============================================================
|
||||
|
||||
============================================================
|
||||
[Info] Image Path (or 'exit' to quit):
|
||||
exit
|
||||
[Info] Exiting...
|
||||
Free vision model memory.
|
||||
Free text model memory.
|
||||
[Info] Done.
|
||||
```
|
||||
# CLIP
|
||||
|
||||
## 1. Overview
|
||||
|
||||
This demo demonstrates how to run CLIP (Contrastive Language-Image Pre-Training) image-text matching using AMLNNLite. The CLIP model consists of two parts: a vision encoder and a text encoder, which work together to compute similarity between images and text descriptions.
|
||||
|
||||
## 2. Model Download
|
||||
|
||||
TO DO
|
||||
|
||||
## 3. Model Conversion
|
||||
|
||||
TO DO
|
||||
|
||||
## 4. Demo Run
|
||||
|
||||
### CPP
|
||||
|
||||
#### 1. Compile
|
||||
|
||||
**Prerequisites:**
|
||||
- Android NDK (r25e recommended)
|
||||
- `ANDROID_NDK_PATH` environment variable set
|
||||
|
||||
**Build:**
|
||||
```bash
|
||||
# Build for arm64-v8a
|
||||
cd examples/clip/cpp
|
||||
AMLNN_HOME=/path/to/amlnn-toolkit ./build-android.sh -a arm64-v8a
|
||||
```
|
||||
|
||||
The executable will be generated at `build/android_arm64-v8a/clip_demo`.
|
||||
|
||||
#### 2. Run
|
||||
|
||||
```bash
|
||||
# Push executable and resources to device
|
||||
adb push build/android_arm64-v8a/clip_demo /data/local/tmp/
|
||||
adb push model/vision_model_int8_S905X5.adla /data/local/tmp/
|
||||
adb push model/text_model_int8_S905X5.adla /data/local/tmp/
|
||||
adb push tokenizer_path/ /data/local/tmp/
|
||||
|
||||
# Run on device
|
||||
adb shell
|
||||
cd /data/local/tmp
|
||||
chmod +x clip_demo
|
||||
export LD_LIBRARY_PATH=/vendor/lib64 or (/vendor/lib)
|
||||
|
||||
# Usage: ./clip_demo <vision_model> <text_model> <tokenizer_path> [--profiling]
|
||||
./clip_demo vision_model_int8_S905X5.adla text_model_int8_S905X5.adla ./tokenizer_path/
|
||||
```
|
||||
|
||||
The program will prompt for image paths and text descriptions interactively. Enter the path to an image file, then enter comma-separated text descriptions (or `skip` to use defaults). Type `exit` to quit.
|
||||
|
||||
**Argument Descriptions:**
|
||||
|
||||
| Argument | Description |
|
||||
| -------------- | ------------------------------------------------------------ |
|
||||
| vision_model | Path to vision encoder .adla model (required) |
|
||||
| text_model | Path to text encoder .adla model (required) |
|
||||
| tokenizer_path | Path to directory containing `vocab.json` and `merges.txt` (required) |
|
||||
| --profiling | Enable performance profiling output (optional) |
|
||||
|
||||
**Note:** The `tokenizer_path` should contain `vocab.json` and `merges.txt` files from the CLIP tokenizer (e.g., from `openai/clip-vit-base-patch32`).
|
||||
|
||||
### Python
|
||||
|
||||
**Prerequisites:**
|
||||
- Python 3.10
|
||||
- Required packages: `numpy`, `Pillow`, `transformers`, `amlnnlite`
|
||||
|
||||
**Install dependencies:**
|
||||
```bash
|
||||
pip install numpy Pillow transformers amlnnlite-1.0.0-cp310-cp310-linux_aarch64.whl
|
||||
```
|
||||
|
||||
**Run on device:**
|
||||
```bash
|
||||
python clip.py \
|
||||
--vision-model ./vision_model_int8_S905X5.adla \
|
||||
--text-model ./text_model_int8_S905X5.adla \
|
||||
--tokenizer-dir ./tokenizer_path \
|
||||
--image-path ./000000004505.jpg \
|
||||
--texts "a red handbag" "a blue jacket" "a red bus"
|
||||
```
|
||||
|
||||
**Interactive Mode (Recommended):**
|
||||
|
||||
If you don't provide `--image-path`, the program will run in interactive mode:
|
||||
|
||||
```bash
|
||||
python clip.py \
|
||||
--vision-model ./vision_model_int8_S905X5.adla \
|
||||
--text-model ./text_model_int8_S905X5.adla \
|
||||
--tokenizer-dir ./tokenizer_path
|
||||
```
|
||||
|
||||
The program will prompt for image paths and text descriptions. Enter an image path to process, then enter comma-separated texts to compare. Type `exit` to quit.
|
||||
|
||||
**Argument Descriptions:**
|
||||
|
||||
| Argument | Description |
|
||||
| ---------------- | ------------------------------------------------------------ |
|
||||
| --vision-model | Path to vision encoder .adla model (required) |
|
||||
| --text-model | Path to text encoder .adla model (required) |
|
||||
| --tokenizer-dir | Path to CLIPTokenizer directory (required) |
|
||||
| --image-path | Path to input image (.jpg, .png) - optional, will prompt if not provided |
|
||||
| --texts | List of text descriptions to compare (space-separated) |
|
||||
| --max-len | Maximum token sequence length, default is 64 |
|
||||
| --logit-scale | Logit scale factor, default is 100.0 |
|
||||
|
||||
**Note:** The `--tokenizer-dir` should point to the directory containing the CLIPTokenizer files. You can use a Hugging Face model ID (e.g., `openai/clip-vit-base-patch32`) or a local directory.
|
||||
|
||||
## 5. Results
|
||||
|
||||
**Performance Feedback**
|
||||
|
||||
By using the `--profiling` flag (C++) or setting the loglevel to INFO, the program provides real-time performance metrics upon completion. The console log will display essential hardware and execution details, including:
|
||||
- Hardware Information: System and ADLA library versions.
|
||||
- Model Overview: Basic input/output configurations.
|
||||
- NPU Metrics: Total inference time (latency) and total DRAM bandwidth consumption.
|
||||
|
||||
**Interactive Mode Example:**
|
||||
|
||||
```bash
|
||||
$ ./clip_demo vision_model_int8_S905X5.adla text_model_int8_S905X5.adla ./tokenizer_path
|
||||
|
||||
[Info] Models initialized successfully.
|
||||
|
||||
============================================================
|
||||
[Info] Image Path (or 'exit' to quit):
|
||||
000000004505.jpg
|
||||
[Info] Enter text descriptions (comma-separated, or 'skip' for defaults):
|
||||
a red handbag, a blue jacket, a red bus
|
||||
|
||||
[Info] Processing image: 000000004505.jpg
|
||||
[Info] Image embedding size: 512
|
||||
[Info] Processing 3 text(s)...
|
||||
[Info] Text embeddings size: 3 x 512
|
||||
|
||||
============================================================
|
||||
CLIP Image-Text Matching Results
|
||||
============================================================
|
||||
Image: 000000004505.jpg
|
||||
logit_scale: 100.000000
|
||||
------------------------------------------------------------
|
||||
[1] prob=0.999975 sim=0.327895 text='a red bus'
|
||||
[2] prob=0.000016 sim=0.217690 text='a red handbag'
|
||||
[3] prob=0.000008 sim=0.211029 text='a blue jacket'
|
||||
============================================================
|
||||
|
||||
============================================================
|
||||
[Info] Image Path (or 'exit' to quit):
|
||||
exit
|
||||
[Info] Exiting...
|
||||
Free vision model memory.
|
||||
Free text model memory.
|
||||
[Info] Done.
|
||||
```
|
||||
|
|
|
|||
0
examples/clip/cpp/.gitkeep
Normal file → Executable file
0
examples/clip/cpp/.gitkeep
Normal file → Executable file
|
|
@ -1,38 +1,38 @@
|
|||
cmake_minimum_required(VERSION 3.10...3.27)
|
||||
project(clip_demo)
|
||||
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
|
||||
list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/../../../../cmake")
|
||||
find_package(AMLNN REQUIRED)
|
||||
include_directories(${AMLNN_INCLUDE_DIR})
|
||||
link_directories(${AMLNN_LIBRARY_DIR})
|
||||
|
||||
include_directories(${CMAKE_SOURCE_DIR}/../../../../common)
|
||||
|
||||
# Set 3rdparty path
|
||||
set(3RDPARTY_DIR "${CMAKE_SOURCE_DIR}/../../../../dependency")
|
||||
|
||||
# Include directories for stb_image and json
|
||||
# Note: code uses #include "stb_image.h" and #include "json.hpp"
|
||||
include_directories(${3RDPARTY_DIR}/stb_image)
|
||||
include_directories(${3RDPARTY_DIR}/json)
|
||||
|
||||
if(CMAKE_SYSTEM_NAME STREQUAL "Android")
|
||||
# Android needs log
|
||||
link_libraries(log)
|
||||
endif()
|
||||
|
||||
add_executable(${PROJECT_NAME}
|
||||
main.cpp
|
||||
model_invoke.cpp
|
||||
pre_postprocess.cpp
|
||||
clip_tokenizer.cpp
|
||||
)
|
||||
|
||||
target_link_libraries(${PROJECT_NAME}
|
||||
${AMLNN_LIBRARY}
|
||||
dl
|
||||
m
|
||||
)
|
||||
|
||||
cmake_minimum_required(VERSION 3.10...3.27)
|
||||
project(clip_demo)
|
||||
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
|
||||
list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/../../../../cmake")
|
||||
find_package(AMLNN REQUIRED)
|
||||
include_directories(${AMLNN_INCLUDE_DIR})
|
||||
link_directories(${AMLNN_LIBRARY_DIR})
|
||||
|
||||
include_directories(${CMAKE_SOURCE_DIR}/../../../../common)
|
||||
|
||||
# Set 3rdparty path
|
||||
set(3RDPARTY_DIR "${CMAKE_SOURCE_DIR}/../../../../dependency")
|
||||
|
||||
# Include directories for stb_image and json
|
||||
# Note: code uses #include "stb_image.h" and #include "json.hpp"
|
||||
include_directories(${3RDPARTY_DIR}/stb_image)
|
||||
include_directories(${3RDPARTY_DIR}/json)
|
||||
|
||||
if(CMAKE_SYSTEM_NAME STREQUAL "Android")
|
||||
# Android needs log
|
||||
link_libraries(log)
|
||||
endif()
|
||||
|
||||
add_executable(${PROJECT_NAME}
|
||||
main.cpp
|
||||
model_invoke.cpp
|
||||
pre_postprocess.cpp
|
||||
clip_tokenizer.cpp
|
||||
)
|
||||
|
||||
target_link_libraries(${PROJECT_NAME}
|
||||
${AMLNN_LIBRARY}
|
||||
dl
|
||||
m
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -26,8 +26,8 @@
|
|||
// Initialize network from file
|
||||
void* init_network_file(const char *model_path);
|
||||
|
||||
// Run vision model inference
|
||||
std::vector<float> run_vision_model(void* context, const std::vector<float>& input_data);
|
||||
// Run image model inference
|
||||
std::vector<float> run_image_model(void* context, const std::vector<float>& input_data);
|
||||
|
||||
// Run text model inference
|
||||
std::vector<float> run_text_model(void* context, const std::vector<int64_t>& input_ids);
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ struct ProfilingTimer
|
|||
{
|
||||
uint64_t init_start, init_end;
|
||||
uint64_t preprocess_start, preprocess_end;
|
||||
uint64_t vision_infer_start, vision_infer_end;
|
||||
uint64_t image_infer_start, image_infer_end;
|
||||
uint64_t text_infer_start, text_infer_end;
|
||||
};
|
||||
|
||||
|
|
@ -71,10 +71,10 @@ std::vector<std::string> parse_texts(const std::string& input)
|
|||
|
||||
void print_usage(const char* prog_name)
|
||||
{
|
||||
printf("Usage: %s <vision_model> <text_model> <tokenizer_dir> [--profiling]\n", prog_name);
|
||||
printf("Usage: %s <image_model> <text_model> <tokenizer_dir> [--profiling]\n", prog_name);
|
||||
printf("\n");
|
||||
printf("Arguments:\n");
|
||||
printf(" vision_model: Path to vision model (.adla)\n");
|
||||
printf(" image_model: Path to image model (.adla)\n");
|
||||
printf(" text_model: Path to text model (.adla)\n");
|
||||
printf(" tokenizer_dir: Path to directory containing vocab.json and merges.txt\n");
|
||||
printf(" --profiling: Enable performance profiling output (optional)\n");
|
||||
|
|
@ -96,7 +96,7 @@ int main(int argc, char ** argv)
|
|||
return -1;
|
||||
}
|
||||
|
||||
const char* vision_model_path = argv[1];
|
||||
const char* image_model_path = argv[1];
|
||||
const char* text_model_path = argv[2];
|
||||
const char* tokenizer_dir = argv[3];
|
||||
|
||||
|
|
@ -119,11 +119,11 @@ int main(int argc, char ** argv)
|
|||
}
|
||||
|
||||
// Initialize models
|
||||
printf("[Info] Initializing vision model: %s\n", vision_model_path);
|
||||
printf("[Info] Initializing image model: %s\n", image_model_path);
|
||||
timer.init_start = get_time_count();
|
||||
void* vision_context = init_network_file(vision_model_path);
|
||||
if (vision_context == NULL) {
|
||||
printf("[Error] Failed to initialize vision model.\n");
|
||||
void* image_context = init_network_file(image_model_path);
|
||||
if (image_context == NULL) {
|
||||
printf("[Error] Failed to initialize image model.\n");
|
||||
return -1;
|
||||
}
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ int main(int argc, char ** argv)
|
|||
void* text_context = init_network_file(text_model_path);
|
||||
if (text_context == NULL) {
|
||||
printf("[Error] Failed to initialize text model.\n");
|
||||
destroy_network(vision_context);
|
||||
destroy_network(image_context);
|
||||
return -1;
|
||||
}
|
||||
timer.init_end = get_time_count();
|
||||
|
|
@ -218,14 +218,14 @@ int main(int argc, char ** argv)
|
|||
}
|
||||
timer.preprocess_end = get_time_count();
|
||||
|
||||
// Run vision model
|
||||
timer.vision_infer_start = get_time_count();
|
||||
std::vector<float> image_embedding = run_vision_model(vision_context, image_input);
|
||||
// Run image model
|
||||
timer.image_infer_start = get_time_count();
|
||||
std::vector<float> image_embedding = run_image_model(image_context, image_input);
|
||||
if (image_embedding.empty()) {
|
||||
printf("[Error] Vision model inference failed.\n");
|
||||
printf("[Error] Image model inference failed.\n");
|
||||
continue;
|
||||
}
|
||||
timer.vision_infer_end = get_time_count();
|
||||
timer.image_infer_end = get_time_count();
|
||||
|
||||
// L2 normalize image embedding
|
||||
image_embedding = l2_normalize(image_embedding);
|
||||
|
|
@ -264,7 +264,8 @@ int main(int argc, char ** argv)
|
|||
continue;
|
||||
}
|
||||
|
||||
printf("[Info] Text embeddings size: %zu x %zu\n", text_embeddings.size(),
|
||||
printf("[Info] Text embeddings size: %zu x %zu\n",
|
||||
text_embeddings.size(),
|
||||
text_embeddings.empty() ? 0 : text_embeddings[0].size());
|
||||
|
||||
// ==================== Compute Similarity ====================
|
||||
|
|
@ -302,11 +303,11 @@ int main(int argc, char ** argv)
|
|||
|
||||
if (profiling) {
|
||||
uint64_t preprocess_time = (timer.preprocess_end - timer.preprocess_start) / 1000000;
|
||||
uint64_t vision_time = (timer.vision_infer_end - timer.vision_infer_start) / 1000000;
|
||||
uint64_t image_time = (timer.image_infer_end - timer.image_infer_start) / 1000000;
|
||||
uint64_t text_total_time = (timer.text_infer_end - timer.text_infer_start) / 1000000;
|
||||
printf("\n[Profiling]\n");
|
||||
printf(" Image preprocess: %lums\n", preprocess_time);
|
||||
printf(" Vision inference: %lums\n", vision_time);
|
||||
printf(" Image inference: %lums\n", image_time);
|
||||
for (size_t i = 0; i < texts.size() && i < text_infer_times.size(); ++i) {
|
||||
printf(" Text inference[%zu]: %lums '%s'\n", i, text_infer_times[i], texts[i].c_str());
|
||||
}
|
||||
|
|
@ -316,9 +317,9 @@ int main(int argc, char ** argv)
|
|||
}
|
||||
|
||||
// Cleanup
|
||||
ret = destroy_network(vision_context);
|
||||
ret = destroy_network(image_context);
|
||||
if (ret != 0) {
|
||||
printf("[Error] Failed to destroy vision model.\n");
|
||||
printf("[Error] Failed to destroy image model.\n");
|
||||
}
|
||||
|
||||
ret = destroy_network(text_context);
|
||||
|
|
@ -328,4 +329,4 @@ int main(int argc, char ** argv)
|
|||
|
||||
printf("[Info] Done.\n");
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
|
@ -27,9 +27,9 @@
|
|||
#include "nn_sdk.h"
|
||||
|
||||
// Global DMA config for models
|
||||
static aml_memory_config_t vision_mem_config;
|
||||
static aml_memory_data_t vision_mem_data;
|
||||
static void* vision_context_flag = nullptr;
|
||||
static aml_memory_config_t image_mem_config;
|
||||
static aml_memory_data_t image_mem_data;
|
||||
static void* image_context_flag = nullptr;
|
||||
|
||||
static aml_memory_config_t text_mem_config;
|
||||
static aml_memory_data_t text_mem_data;
|
||||
|
|
@ -48,7 +48,7 @@ void* init_network_file(const char *model_path)
|
|||
|
||||
/* set omp, If you are considering high CPU usage during operation,
|
||||
you can turn off this api, set_openmp_opt_flag = false */
|
||||
aml_openmp_opt_t openmp_opt[] =
|
||||
aml_openmp_opt_t openmp_opt[] =
|
||||
{
|
||||
{
|
||||
.operator_type = AML_Unknown,
|
||||
|
|
@ -62,7 +62,7 @@ void* init_network_file(const char *model_path)
|
|||
config.forward_ctrl.softop_info.openmp_opt = openmp_opt;
|
||||
|
||||
/* set neon */
|
||||
aml_neon_opt_t neon_opt[] =
|
||||
aml_neon_opt_t neon_opt[] =
|
||||
{
|
||||
{
|
||||
.operator_type = AML_Unknown,
|
||||
|
|
@ -84,10 +84,11 @@ void* init_network_file(const char *model_path)
|
|||
return qcontext;
|
||||
}
|
||||
|
||||
std::vector<float> run_vision_model(void* qcontext, const std::vector<float>& input_data)
|
||||
std::vector<float> run_image_model(void* qcontext, const std::vector<float>& input_data)
|
||||
{
|
||||
int ret = 0;
|
||||
nn_input inData;
|
||||
|
||||
nn_output *outdata = NULL;
|
||||
aml_output_config_t outconfig;
|
||||
|
||||
|
|
@ -96,19 +97,19 @@ std::vector<float> run_vision_model(void* qcontext, const std::vector<float>& in
|
|||
inData.size = input_data.size() * sizeof(float);
|
||||
|
||||
// Use DMA
|
||||
if (!vision_context_flag) {
|
||||
vision_mem_config.cache_type = AML_WITH_CACHE;
|
||||
vision_mem_config.memory_type = AML_VIRTUAL_ADDR;
|
||||
vision_mem_config.direction = AML_MEM_DIRECTION_READ_WRITE;
|
||||
vision_mem_config.index = 0;
|
||||
vision_mem_config.mem_size = inData.size;
|
||||
aml_util_mallocBuffer(qcontext, &vision_mem_config, &vision_mem_data);
|
||||
aml_util_swapExternalInputBuffer(qcontext, &vision_mem_config, &vision_mem_data);
|
||||
vision_context_flag = qcontext;
|
||||
if (!image_context_flag) {
|
||||
image_mem_config.cache_type = AML_WITH_CACHE;
|
||||
image_mem_config.memory_type = AML_VIRTUAL_ADDR;
|
||||
image_mem_config.direction = AML_MEM_DIRECTION_READ_WRITE;
|
||||
image_mem_config.index = 0;
|
||||
image_mem_config.mem_size = inData.size;
|
||||
aml_util_mallocBuffer(qcontext, &image_mem_config, &image_mem_data);
|
||||
aml_util_swapExternalInputBuffer(qcontext, &image_mem_config, &image_mem_data);
|
||||
image_context_flag = qcontext;
|
||||
}
|
||||
|
||||
inData.input_type = INPUT_DMA_DATA;
|
||||
memcpy(vision_mem_data.viraddr, input_data.data(), vision_mem_config.mem_size);
|
||||
memcpy(image_mem_data.viraddr, input_data.data(), image_mem_config.mem_size);
|
||||
inData.input = NULL;
|
||||
|
||||
memset(&outconfig, 0, sizeof(aml_output_config_t));
|
||||
|
|
@ -117,7 +118,7 @@ std::vector<float> run_vision_model(void* qcontext, const std::vector<float>& in
|
|||
outdata = (nn_output*)aml_module_output_get(qcontext, outconfig);
|
||||
|
||||
if (outdata == NULL || outdata->out[0].buf == NULL) {
|
||||
printf("Vision model inference failed.\n");
|
||||
printf("Image model inference failed.\n");
|
||||
return {};
|
||||
}
|
||||
|
||||
|
|
@ -178,10 +179,10 @@ int destroy_network(void *qcontext)
|
|||
{
|
||||
int ret = 0;
|
||||
|
||||
if (vision_context_flag == qcontext) {
|
||||
printf("Free vision model memory.\n");
|
||||
aml_util_freeBuffer(qcontext, &vision_mem_config, &vision_mem_data);
|
||||
vision_context_flag = nullptr;
|
||||
if (image_context_flag == qcontext) {
|
||||
printf("Free image model memory.\n");
|
||||
aml_util_freeBuffer(qcontext, &image_mem_config, &image_mem_data);
|
||||
image_context_flag = nullptr;
|
||||
} else if (text_context_flag == qcontext) {
|
||||
printf("Free text model memory.\n");
|
||||
aml_util_freeBuffer(qcontext, &text_mem_config, &text_mem_data);
|
||||
|
|
@ -199,4 +200,4 @@ int destroy_network(void *qcontext)
|
|||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
|
@ -1,13 +1,29 @@
|
|||
#ifndef MODEL_INVOKE_H
|
||||
#define MODEL_INVOKE_H
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <map>
|
||||
|
||||
void* init_network_file(const char *model_path);
|
||||
std::vector<std::string> process_image_dir(void *context_model, const std::string& json_path, const std::string& base_dir = "", const std::string& json_filename = "");
|
||||
int destroy_network(void *qcontext);
|
||||
|
||||
#endif // MODEL_INVOKE_H
|
||||
|
||||
/*
|
||||
* Copyright (C) 2026 Amlogic, Inc. All rights reserved.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#ifndef MODEL_INVOKE_H
|
||||
#define MODEL_INVOKE_H
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <map>
|
||||
|
||||
void* init_network_file(const char *model_path);
|
||||
std::vector<std::string> process_image_dir(void *context_model, const std::string& json_path, const std::string& base_dir = "", const std::string& json_filename = "");
|
||||
int destroy_network(void *qcontext);
|
||||
|
||||
#endif // MODEL_INVOKE_H
|
||||
|
||||
|
|
|
|||
|
|
@ -102,7 +102,7 @@ std::vector<float> preprocess_image(const std::string& image_path) {
|
|||
}
|
||||
}
|
||||
|
||||
// Return NHWC format (batch dimension will be added in caller)
|
||||
// get NHWC
|
||||
return cropped;
|
||||
}
|
||||
|
||||
|
|
|
|||
0
examples/clip/model/.gitkeep
Normal file → Executable file
0
examples/clip/model/.gitkeep
Normal file → Executable file
0
examples/clip/py/.gitkeep
Normal file → Executable file
0
examples/clip/py/.gitkeep
Normal file → Executable file
|
|
@ -1,339 +1,336 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Copyright (C) 2024–2025 Amlogic, Inc. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
"""
|
||||
|
||||
# This inference script is designed for CLIP model using AMLNNLite.
|
||||
|
||||
import os
|
||||
import argparse
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from transformers import CLIPTokenizer
|
||||
from amlnnlite.api import AMLNNLite
|
||||
|
||||
# ==================== Utility Functions ====================
|
||||
|
||||
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
|
||||
"""Compute softmax values for array x."""
|
||||
x = x - np.max(x, axis=axis, keepdims=True)
|
||||
e = np.exp(x)
|
||||
return e / np.sum(e, axis=axis, keepdims=True)
|
||||
|
||||
|
||||
def l2_normalize(x: np.ndarray, axis: int = -1, eps: float = 1e-12) -> np.ndarray:
|
||||
"""L2 normalize array x along specified axis."""
|
||||
return x / (np.linalg.norm(x, axis=axis, keepdims=True) + eps)
|
||||
|
||||
# ==================== Vision Preprocessing ====================
|
||||
|
||||
def preprocess_image(image_path: str, target_size: int = 224) -> np.ndarray:
|
||||
"""
|
||||
Preprocess image for CLIP model.
|
||||
|
||||
Args:
|
||||
image_path (str): Path to input image
|
||||
target_size (int): Target image size (default: 224)
|
||||
|
||||
Returns:
|
||||
np.ndarray: Preprocessed image data with shape (1, target_size, target_size, 3) in NHWC format
|
||||
"""
|
||||
image = Image.open(image_path).convert("RGB")
|
||||
width, height = image.size
|
||||
|
||||
# Scale the shorter side
|
||||
scale = target_size / min(width, height)
|
||||
new_width = int(width * scale)
|
||||
new_height = int(height * scale)
|
||||
image_resized = image.resize((new_width, new_height), resample=Image.BICUBIC)
|
||||
|
||||
# Center crop
|
||||
left = (new_width - target_size) // 2
|
||||
top = (new_height - target_size) // 2
|
||||
right = left + target_size
|
||||
bottom = top + target_size
|
||||
image_cropped = image_resized.crop((left, top, right, bottom))
|
||||
|
||||
# Convert to numpy array and normalize to [0, 1]
|
||||
image_np = np.array(image_cropped).astype(np.float32) / 255.0
|
||||
|
||||
# CLIP normalization
|
||||
mean = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
|
||||
std = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)
|
||||
image_np = (image_np - mean) / std
|
||||
|
||||
# Add batch dimension: HWC -> NHWC
|
||||
image_np = np.expand_dims(image_np, axis=0)
|
||||
|
||||
return image_np.astype(np.float32) # [1, 224, 224, 3]
|
||||
|
||||
# ==================== Text Preprocessing ====================
|
||||
|
||||
def preprocess_text(tokenizer: CLIPTokenizer, text: str, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Preprocess text for CLIP model using CLIPTokenizer.
|
||||
|
||||
Args:
|
||||
tokenizer: CLIPTokenizer instance
|
||||
text (str): Input text string
|
||||
max_len (int): Maximum sequence length (default: 64)
|
||||
|
||||
Returns:
|
||||
np.ndarray: Tokenized text with shape (1, max_len) as int64
|
||||
"""
|
||||
enc = tokenizer(
|
||||
text,
|
||||
padding="max_length",
|
||||
truncation=True,
|
||||
max_length=max_len,
|
||||
return_tensors="np",
|
||||
)
|
||||
# text model input: int64[1, max_len]
|
||||
input_ids = enc["input_ids"].astype(np.int64)
|
||||
return input_ids
|
||||
|
||||
# ==================== Model Inference ====================
|
||||
|
||||
def compute_image_embedding(vision_amlnn: AMLNNLite, image_path: str) -> np.ndarray:
|
||||
"""
|
||||
Compute image embedding using vision model.
|
||||
|
||||
Args:
|
||||
vision_amlnn: AMLNNLite instance for vision model
|
||||
image_path (str): Path to input image
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized image embedding with shape (1, embed_dim)
|
||||
"""
|
||||
input_data = preprocess_image(image_path) # [1, 224, 224, 3]
|
||||
|
||||
outputs = vision_amlnn.inference(
|
||||
inputs=[input_data],
|
||||
inputs_data_format='NHWC',
|
||||
outputs_data_format='NHWC'
|
||||
)
|
||||
|
||||
feats = outputs[0].astype(np.float32)
|
||||
feats = feats.reshape(1, -1) # Squeeze to [1, embed_dim]
|
||||
return l2_normalize(feats, axis=1)
|
||||
|
||||
def compute_text_embedding(text_amlnn: AMLNNLite, tokenizer: CLIPTokenizer, text: str, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Compute text embedding using text model.
|
||||
|
||||
Args:
|
||||
text_amlnn: AMLNNLite instance for text model
|
||||
tokenizer: CLIPTokenizer instance
|
||||
text (str): Input text string
|
||||
max_len (int): Maximum sequence length
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized text embedding with shape (1, embed_dim)
|
||||
"""
|
||||
input_ids = preprocess_text(tokenizer, text, max_len) # [1, max_len]
|
||||
print(f"input_ids: {input_ids}")
|
||||
|
||||
# AMLNNLite requires 4D input, reshape to (1, 1, 1, max_len)
|
||||
input_ids_4d = input_ids[:, None, None, :] # [1, 1, 1, max_len]
|
||||
|
||||
outputs = text_amlnn.inference(
|
||||
inputs=[input_ids_4d],
|
||||
inputs_data_format='NHWC',
|
||||
outputs_data_format='NHWC'
|
||||
)
|
||||
|
||||
feats = outputs[0].astype(np.float32)
|
||||
feats = feats.reshape(1, -1) # Squeeze to [1, embed_dim]
|
||||
return l2_normalize(feats, axis=1)
|
||||
|
||||
def compute_text_embeddings_batch(text_amlnn: AMLNNLite, tokenizer: CLIPTokenizer, texts: list, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Compute text embeddings for multiple texts.
|
||||
|
||||
Args:
|
||||
text_amlnn: AMLNNLite instance for text model
|
||||
tokenizer: CLIPTokenizer instance
|
||||
texts (list): List of input text strings
|
||||
max_len (int): Maximum sequence length
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized text embeddings with shape (num_texts, embed_dim)
|
||||
"""
|
||||
embeddings = []
|
||||
for text in texts:
|
||||
emb = compute_text_embedding(text_amlnn, tokenizer, text, max_len)
|
||||
embeddings.append(emb[0]) # Remove batch dimension
|
||||
return np.stack(embeddings, axis=0) # [num_texts, embed_dim]
|
||||
|
||||
# ==================== Similarity Calculation ====================
|
||||
|
||||
def compute_similarity(image_embedding: np.ndarray, text_embeddings: np.ndarray, logit_scale: float = 100.0) -> tuple:
|
||||
"""
|
||||
Compute similarity between image and text embeddings.
|
||||
|
||||
Args:
|
||||
image_embedding (np.ndarray): Image embedding with shape (1, embed_dim)
|
||||
text_embeddings (np.ndarray): Text embeddings with shape (num_texts, embed_dim)
|
||||
logit_scale (float): Scale factor for logits
|
||||
|
||||
Returns:
|
||||
tuple: (similarities, logits, probabilities)
|
||||
"""
|
||||
# Cosine similarity (embeddings are already L2-normalized)
|
||||
sims = text_embeddings @ image_embedding[0] # [num_texts]
|
||||
logits = sims * logit_scale # [num_texts]
|
||||
probs = softmax(logits, axis=0) # [num_texts]
|
||||
|
||||
return sims, logits, probs
|
||||
|
||||
# ==================== Main Function ====================
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='CLIP Image-Text Matching Demo using AMLNNLite')
|
||||
parser.add_argument('--vision-model', required=True, help='Path to vision model (.adla)')
|
||||
parser.add_argument('--text-model', required=True, help='Path to text model (.adla)')
|
||||
parser.add_argument('--tokenizer-dir', required=True, help='Path to CLIPTokenizer directory')
|
||||
parser.add_argument('--image-path', default=None, help='Path to input image (optional, will prompt if not provided)')
|
||||
parser.add_argument('--texts', nargs='+', default=None, help='List of text descriptions to compare')
|
||||
parser.add_argument('--max-len', type=int, default=64, help='Maximum token sequence length (default: 64)')
|
||||
parser.add_argument('--logit-scale', type=float, default=100.0, help='Logit scale factor (default: 100.0)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate model paths
|
||||
if not os.path.exists(args.vision_model):
|
||||
print(f"[Error] Vision model not found: {args.vision_model}")
|
||||
return -1
|
||||
|
||||
if not os.path.exists(args.text_model):
|
||||
print(f"[Error] Text model not found: {args.text_model}")
|
||||
return -1
|
||||
|
||||
# Load tokenizer
|
||||
print(f"[Info] Loading CLIPTokenizer from: {args.tokenizer_dir}")
|
||||
tokenizer = CLIPTokenizer.from_pretrained(args.tokenizer_dir)
|
||||
|
||||
# Initialize vision model
|
||||
print(f"[Info] Initializing vision model: {args.vision_model}")
|
||||
vision_amlnn = AMLNNLite()
|
||||
vision_amlnn.config(model_path=args.vision_model, run_cycles=1)
|
||||
vision_amlnn.init()
|
||||
|
||||
# Initialize text model
|
||||
print(f"[Info] Initializing text model: {args.text_model}")
|
||||
text_amlnn = AMLNNLite()
|
||||
text_amlnn.config(model_path=args.text_model, run_cycles=1)
|
||||
text_amlnn.init()
|
||||
|
||||
print("[Info] Models initialized successfully.\n")
|
||||
|
||||
try:
|
||||
# Interactive loop
|
||||
while True:
|
||||
# Get image path
|
||||
if args.image_path:
|
||||
image_path = args.image_path
|
||||
args.image_path = None # Clear for next iteration
|
||||
else:
|
||||
print("=" * 60)
|
||||
print("[Info] Image Path (or 'exit' to quit):")
|
||||
image_path = input().strip()
|
||||
|
||||
# Check for exit
|
||||
if image_path.lower() == 'exit':
|
||||
print("[Info] Exiting...")
|
||||
break
|
||||
|
||||
# Validate image path
|
||||
if not image_path:
|
||||
print("[Warning] Please enter an image path.")
|
||||
continue
|
||||
|
||||
if not os.path.exists(image_path):
|
||||
print(f"[Error] Image not found: {image_path}")
|
||||
continue
|
||||
|
||||
# Get texts to compare
|
||||
if args.texts:
|
||||
texts = args.texts
|
||||
args.texts = None # Clear for next iteration
|
||||
else:
|
||||
print("[Info] Enter text descriptions (comma-separated, or 'skip' to use defaults):")
|
||||
text_input = input().strip()
|
||||
|
||||
if text_input.lower() == 'skip' or not text_input:
|
||||
# Default texts for demo
|
||||
texts = [
|
||||
"a red handbag",
|
||||
"a blue jacket",
|
||||
"a red bus",
|
||||
]
|
||||
print(f"[Info] Using default texts: {texts}")
|
||||
else:
|
||||
texts = [t.strip() for t in text_input.split(',') if t.strip()]
|
||||
|
||||
if not texts:
|
||||
print("[Warning] No texts provided.")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Compute image embedding
|
||||
print(f"\n[Info] Processing image: {image_path}")
|
||||
image_embedding = compute_image_embedding(vision_amlnn, image_path)
|
||||
print(f"[Info] Image embedding shape: {image_embedding.shape}")
|
||||
|
||||
# Compute text embeddings
|
||||
print(f"[Info] Processing {len(texts)} text(s)...")
|
||||
text_embeddings = compute_text_embeddings_batch(text_amlnn, tokenizer, texts, args.max_len)
|
||||
print(f"[Info] Text embeddings shape: {text_embeddings.shape}")
|
||||
|
||||
# Compute similarity
|
||||
sims, logits, probs = compute_similarity(image_embedding, text_embeddings, args.logit_scale)
|
||||
|
||||
# Print results
|
||||
print("\n" + "=" * 60)
|
||||
print("CLIP Image-Text Matching Results")
|
||||
print("=" * 60)
|
||||
print(f"Image: {image_path}")
|
||||
print(f"logit_scale: {args.logit_scale:.6f}")
|
||||
print("-" * 60)
|
||||
|
||||
# Sort by probability (descending)
|
||||
sorted_indices = np.argsort(probs)[::-1]
|
||||
for rank, i in enumerate(sorted_indices):
|
||||
print(f"[{rank + 1}] prob={probs[i]:.6f} sim={float(sims[i]):.6f} text='{texts[i]}'")
|
||||
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Error] Processing failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
continue
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n[Info] Interrupted by user. Exiting...")
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
vision_amlnn.uninit()
|
||||
text_amlnn.uninit()
|
||||
|
||||
print("[Info] Done.")
|
||||
return 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(main())
|
||||
#
|
||||
# Copyright (C) 2026 Amlogic, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import os
|
||||
import argparse
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from transformers import CLIPTokenizer
|
||||
from amlnnlite.api import AMLNNLite
|
||||
|
||||
# ==================== Utility Functions ====================
|
||||
|
||||
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
|
||||
"""Compute softmax values for array x."""
|
||||
x = x - np.max(x, axis=axis, keepdims=True)
|
||||
e = np.exp(x)
|
||||
return e / np.sum(e, axis=axis, keepdims=True)
|
||||
|
||||
|
||||
def l2_normalize(x: np.ndarray, axis: int = -1, eps: float = 1e-12) -> np.ndarray:
|
||||
"""L2 normalize array x along specified axis."""
|
||||
return x / (np.linalg.norm(x, axis=axis, keepdims=True) + eps)
|
||||
|
||||
# ==================== Vision Preprocessing ====================
|
||||
|
||||
def preprocess_image(image_path: str, target_size: int = 224) -> np.ndarray:
|
||||
"""
|
||||
Preprocess image for CLIP model.
|
||||
|
||||
Args:
|
||||
image_path (str): Path to input image
|
||||
target_size (int): Target image size (default: 224)
|
||||
|
||||
Returns:
|
||||
np.ndarray: Preprocessed image data with shape (1, target_size, target_size, 3) in NHWC format
|
||||
"""
|
||||
image = Image.open(image_path).convert("RGB")
|
||||
width, height = image.size
|
||||
|
||||
# Scale the shorter side
|
||||
scale = target_size / min(width, height)
|
||||
new_width = int(width * scale)
|
||||
new_height = int(height * scale)
|
||||
image_resized = image.resize((new_width, new_height), resample=Image.BICUBIC)
|
||||
|
||||
# Center crop
|
||||
left = (new_width - target_size) // 2
|
||||
top = (new_height - target_size) // 2
|
||||
right = left + target_size
|
||||
bottom = top + target_size
|
||||
image_cropped = image_resized.crop((left, top, right, bottom))
|
||||
|
||||
# Convert to numpy array and normalize to [0, 1]
|
||||
image_np = np.array(image_cropped).astype(np.float32) / 255.0
|
||||
|
||||
# CLIP normalization
|
||||
mean = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
|
||||
std = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)
|
||||
image_np = (image_np - mean) / std
|
||||
|
||||
# Add batch dimension: HWC -> NHWC
|
||||
image_np = np.expand_dims(image_np, axis=0)
|
||||
|
||||
return image_np.astype(np.float32) # [1, 224, 224, 3]
|
||||
|
||||
# ==================== Text Preprocessing ====================
|
||||
|
||||
def preprocess_text(tokenizer: CLIPTokenizer, text: str, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Preprocess text for CLIP model using CLIPTokenizer.
|
||||
|
||||
Args:
|
||||
tokenizer: CLIPTokenizer instance
|
||||
text (str): Input text string
|
||||
max_len (int): Maximum sequence length (default: 64)
|
||||
|
||||
Returns:
|
||||
np.ndarray: Tokenized text with shape (1, max_len) as int64
|
||||
"""
|
||||
enc = tokenizer(
|
||||
text,
|
||||
padding="max_length",
|
||||
truncation=True,
|
||||
max_length=max_len,
|
||||
return_tensors="np",
|
||||
)
|
||||
# text model input: int64[1, max_len]
|
||||
input_ids = enc["input_ids"].astype(np.int64)
|
||||
return input_ids
|
||||
|
||||
# ==================== Model Inference ====================
|
||||
|
||||
def compute_image_embedding(vision_amlnn: AMLNNLite, image_path: str) -> np.ndarray:
|
||||
"""
|
||||
Compute image embedding using vision model.
|
||||
|
||||
Args:
|
||||
vision_amlnn: AMLNNLite instance for vision model
|
||||
image_path (str): Path to input image
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized image embedding with shape (1, embed_dim)
|
||||
"""
|
||||
input_data = preprocess_image(image_path) # [1, 224, 224, 3]
|
||||
|
||||
outputs = vision_amlnn.inference(
|
||||
inputs=[input_data],
|
||||
inputs_data_format='NHWC',
|
||||
outputs_data_format='NHWC'
|
||||
)
|
||||
|
||||
feats = outputs[0].astype(np.float32)
|
||||
feats = feats.reshape(1, -1) # Squeeze to [1, embed_dim]
|
||||
return l2_normalize(feats, axis=1)
|
||||
|
||||
def compute_text_embedding(text_amlnn: AMLNNLite, tokenizer: CLIPTokenizer, text: str, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Compute text embedding using text model.
|
||||
|
||||
Args:
|
||||
text_amlnn: AMLNNLite instance for text model
|
||||
tokenizer: CLIPTokenizer instance
|
||||
text (str): Input text string
|
||||
max_len (int): Maximum sequence length
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized text embedding with shape (1, embed_dim)
|
||||
"""
|
||||
input_ids = preprocess_text(tokenizer, text, max_len) # [1, max_len]
|
||||
print(f"input_ids: {input_ids}")
|
||||
|
||||
# AMLNNLite requires 4D input, reshape to (1, 1, 1, max_len)
|
||||
input_ids_4d = input_ids[:, None, None, :] # [1, 1, 1, max_len]
|
||||
|
||||
outputs = text_amlnn.inference(
|
||||
inputs=[input_ids_4d],
|
||||
inputs_data_format='NHWC',
|
||||
outputs_data_format='NHWC'
|
||||
)
|
||||
|
||||
feats = outputs[0].astype(np.float32)
|
||||
feats = feats.reshape(1, -1) # Squeeze to [1, embed_dim]
|
||||
return l2_normalize(feats, axis=1)
|
||||
|
||||
def compute_text_embeddings_batch(text_amlnn: AMLNNLite, tokenizer: CLIPTokenizer, texts: list, max_len: int = 64) -> np.ndarray:
|
||||
"""
|
||||
Compute text embeddings for multiple texts.
|
||||
|
||||
Args:
|
||||
text_amlnn: AMLNNLite instance for text model
|
||||
tokenizer: CLIPTokenizer instance
|
||||
texts (list): List of input text strings
|
||||
max_len (int): Maximum sequence length
|
||||
|
||||
Returns:
|
||||
np.ndarray: L2-normalized text embeddings with shape (num_texts, embed_dim)
|
||||
"""
|
||||
embeddings = []
|
||||
for text in texts:
|
||||
emb = compute_text_embedding(text_amlnn, tokenizer, text, max_len)
|
||||
embeddings.append(emb[0]) # Remove batch dimension
|
||||
return np.stack(embeddings, axis=0) # [num_texts, embed_dim]
|
||||
|
||||
# ==================== Similarity Calculation ====================
|
||||
|
||||
def compute_similarity(image_embedding: np.ndarray, text_embeddings: np.ndarray, logit_scale: float = 100.0) -> tuple:
|
||||
"""
|
||||
Compute similarity between image and text embeddings.
|
||||
|
||||
Args:
|
||||
image_embedding (np.ndarray): Image embedding with shape (1, embed_dim)
|
||||
text_embeddings (np.ndarray): Text embeddings with shape (num_texts, embed_dim)
|
||||
logit_scale (float): Scale factor for logits
|
||||
|
||||
Returns:
|
||||
tuple: (similarities, logits, probabilities)
|
||||
"""
|
||||
# Cosine similarity (embeddings are already L2-normalized)
|
||||
sims = text_embeddings @ image_embedding[0] # [num_texts]
|
||||
logits = sims * logit_scale # [num_texts]
|
||||
probs = softmax(logits, axis=0) # [num_texts]
|
||||
|
||||
return sims, logits, probs
|
||||
|
||||
# ==================== Main Function ====================
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='CLIP Image-Text Matching Demo using AMLNNLite')
|
||||
parser.add_argument('--vision-model', required=True, help='Path to vision model (.adla)')
|
||||
parser.add_argument('--text-model', required=True, help='Path to text model (.adla)')
|
||||
parser.add_argument('--tokenizer-dir', required=True, help='Path to CLIPTokenizer directory')
|
||||
parser.add_argument('--image-path', default=None, help='Path to input image (optional, will prompt if not provided)')
|
||||
parser.add_argument('--texts', nargs='+', default=None, help='List of text descriptions to compare')
|
||||
parser.add_argument('--max-len', type=int, default=64, help='Maximum token sequence length (default: 64)')
|
||||
parser.add_argument('--logit-scale', type=float, default=100.0, help='Logit scale factor (default: 100.0)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate model paths
|
||||
if not os.path.exists(args.vision_model):
|
||||
print(f"[Error] Vision model not found: {args.vision_model}")
|
||||
return -1
|
||||
|
||||
if not os.path.exists(args.text_model):
|
||||
print(f"[Error] Text model not found: {args.text_model}")
|
||||
return -1
|
||||
|
||||
# Load tokenizer
|
||||
print(f"[Info] Loading CLIPTokenizer from: {args.tokenizer_dir}")
|
||||
tokenizer = CLIPTokenizer.from_pretrained(args.tokenizer_dir)
|
||||
|
||||
# Initialize vision model
|
||||
print(f"[Info] Initializing vision model: {args.vision_model}")
|
||||
vision_amlnn = AMLNNLite()
|
||||
vision_amlnn.config(model_path=args.vision_model, run_cycles=1)
|
||||
vision_amlnn.init()
|
||||
|
||||
# Initialize text model
|
||||
print(f"[Info] Initializing text model: {args.text_model}")
|
||||
text_amlnn = AMLNNLite()
|
||||
text_amlnn.config(model_path=args.text_model, run_cycles=1)
|
||||
text_amlnn.init()
|
||||
|
||||
print("[Info] Models initialized successfully.\n")
|
||||
|
||||
try:
|
||||
# Interactive loop
|
||||
while True:
|
||||
# Get image path
|
||||
if args.image_path:
|
||||
image_path = args.image_path
|
||||
args.image_path = None # Clear for next iteration
|
||||
else:
|
||||
print("=" * 60)
|
||||
print("[Info] Image Path (or 'exit' to quit):")
|
||||
image_path = input().strip()
|
||||
|
||||
# Check for exit
|
||||
if image_path.lower() == 'exit':
|
||||
print("[Info] Exiting...")
|
||||
break
|
||||
|
||||
# Validate image path
|
||||
if not image_path:
|
||||
print("[Warning] Please enter an image path.")
|
||||
continue
|
||||
|
||||
if not os.path.exists(image_path):
|
||||
print(f"[Error] Image not found: {image_path}")
|
||||
continue
|
||||
|
||||
# Get texts to compare
|
||||
if args.texts:
|
||||
texts = args.texts
|
||||
args.texts = None # Clear for next iteration
|
||||
else:
|
||||
print("[Info] Enter text descriptions (comma-separated, or 'skip' to use defaults):")
|
||||
text_input = input().strip()
|
||||
|
||||
if text_input.lower() == 'skip' or not text_input:
|
||||
# Default texts for demo
|
||||
texts = [
|
||||
"a red handbag",
|
||||
"a blue jacket",
|
||||
"a red bus",
|
||||
]
|
||||
print(f"[Info] Using default texts: {texts}")
|
||||
else:
|
||||
texts = [t.strip() for t in text_input.split(',') if t.strip()]
|
||||
|
||||
if not texts:
|
||||
print("[Warning] No texts provided.")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Compute image embedding
|
||||
print(f"\n[Info] Processing image: {image_path}")
|
||||
image_embedding = compute_image_embedding(vision_amlnn, image_path)
|
||||
print(f"[Info] Image embedding shape: {image_embedding.shape}")
|
||||
|
||||
# Compute text embeddings
|
||||
print(f"[Info] Processing {len(texts)} text(s)...")
|
||||
text_embeddings = compute_text_embeddings_batch(text_amlnn, tokenizer, texts, args.max_len)
|
||||
print(f"[Info] Text embeddings shape: {text_embeddings.shape}")
|
||||
|
||||
# Compute similarity
|
||||
sims, logits, probs = compute_similarity(image_embedding, text_embeddings, args.logit_scale)
|
||||
|
||||
# Print results
|
||||
print("\n" + "=" * 60)
|
||||
print("CLIP Image-Text Matching Results")
|
||||
print("=" * 60)
|
||||
print(f"Image: {image_path}")
|
||||
print(f"logit_scale: {args.logit_scale:.6f}")
|
||||
print("-" * 60)
|
||||
|
||||
# Sort by probability (descending)
|
||||
sorted_indices = np.argsort(probs)[::-1]
|
||||
for rank, i in enumerate(sorted_indices):
|
||||
print(f"[{rank + 1}] prob={probs[i]:.6f} sim={float(sims[i]):.6f} text='{texts[i]}'")
|
||||
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Error] Processing failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
continue
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n[Info] Interrupted by user. Exiting...")
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
vision_amlnn.uninit()
|
||||
text_amlnn.uninit()
|
||||
|
||||
print("[Info] Done.")
|
||||
return 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(main())
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue