่ฝป้็บง้ซๆง่ฝๅ้็ดขๅผๅผๆ ยท ็บฏ Python ้ถไพ่ตๅฎ็ฐ
็ฎไฝไธญๆ ยท ็น้ซไธญๆ ยท English
pip install git+https://github.com/gitstq/VectorForge.git
VectorForge ๆฏไธไธช่ฝป้็บงใ้ซๆง่ฝ็ๅ้็ดขๅผๅผๆ๏ผๅฎๅ จไฝฟ็จ็บฏ Python ๅฎ็ฐ๏ผ้ถๅค้จไพ่ต๏ผNumPy ไธบๅฏ้ๅ ้ไพ่ต๏ผใๅฎไธบๅ้็ธไผผๅบฆๆ็ดขๆไพไบไธๅฅๅฎๆด็ๅทฅๅ ท้พ๏ผๆถต็ๅค็ง็ดขๅผ็ฎๆณใ่ท็ฆปๅบฆ้ใๆไน ๅๅญๅจใๅฝไปค่กๅทฅๅ ทๅๅฏ่งๅไปช่กจ็ใ
ๆ ่ฎบไฝ ๆฏๅจๆๅปบ่ฏญไนๆ็ดขๅผๆใๆจ่็ณป็ปใๅพๅๆฃ็ดข๏ผ่ฟๆฏๅๆบๅจๅญฆไน ๅๅ้ช่ฏ๏ผVectorForge ้ฝ่ฝ่ฎฉไฝ ไปฅๆ็ฎๅ็ๆนๅผๅฟซ้ไธๆๅ้ๆ็ดขใ
| ็นๆง | ๆ่ฟฐ |
|---|---|
| ๐ง ๅ็ง็ดขๅผ็ฎๆณ | HNSW๏ผๅๅฑๅฏผ่ชๅฐไธ็ๅพ๏ผใIVF-Flat๏ผๅๆๆไปถ๏ผใLSH๏ผๅฑ้จๆๆๅๅธ๏ผใBrute-Force๏ผๆดๅๆ็ดข๏ผ |
| ๐ ไธ็ง่ท็ฆปๅบฆ้ | ไฝๅผฆ็ธไผผๅบฆ๏ผCosine๏ผใๆฌงๆฐ่ท็ฆป๏ผEuclidean๏ผใๅ ็งฏ๏ผInner Product๏ผ |
| ๐พ ็ดขๅผๆไน ๅ | ๆฏๆ JSON ๆ ผๅผๅบๅๅ/ๅๅบๅๅ๏ผ็ดขๅผๅฏไฟๅญๅฐ็ฃ็้ๆถๅ ่ฝฝ |
| ๐ ๅคๆ ผๅผ IO | ๆฏๆ JSON ๅ CSV ๆ ผๅผ็ๅ้ๆฐๆฎๅฏผๅ ฅๅฏผๅบ |
| ๐ TUI ็ป็ซฏไปช่กจ็ | ๅจ็ป็ซฏไธญๅฏ่งๅๆฅ็็ดขๅผ็ถๆใ็ป่ฎกไฟกๆฏๅๆง่ฝๆๆ |
| โก ๆง่ฝๅบๅๆต่ฏ | ๅ ็ฝฎ benchmark ๆจกๅ๏ผไธ้ฎๅฏนๆฏไธๅ็ดขๅผ็ฎๆณ็ๆๅปบไธๆฅ่ฏขๆง่ฝ |
| ๐ฅ๏ธ CLI ๅฝไปค่กๅทฅๅ ท | ๅฎๆด็ๅฝไปค่กๆฅๅฃ๏ผๆฏๆๆๅปบใๆ็ดขใ่ฝฌๆขใ็ๆใๅบๅๆต่ฏ็ญๆไฝ |
| ๐ ้ซ็บงๆ็ดข API | ็ฎๆดไผ้ ็ Python API๏ผๅ ่กไปฃ็ ๅณๅฏๅฎๆๅ้็ดขๅผไธๆฃ็ดข |
# ้่ฟ pip ไป GitHub ๅฎ่ฃ
pip install git+https://github.com/gitstq/VectorForge.git
# ๆ่
็ดๆฅๅ
้ๆบ็ ไฝฟ็จ
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# 1. ๅๅคๅ้ๆฐๆฎ
vectors = [Vector(id=i, values=[float(j) for j in range(128)]) for i in range(100)]
# 2. ๅๅปบๆ็ดขๅผๆ๏ผ้ซ็บง API๏ผ
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
engine.index(vectors)
# 3. ๆ็ดขๆ่ฟ้ป
query = Vector(id=-1, values=[float(j) for j in range(128)])
results = engine.search(query, k=10)
for r in results:
print(f"ID: {r.id}, ่ท็ฆป: {r.distance:.4f}")VectorForge ๆไพไบๅ่ฝๅฎๆด็ๅฝไปค่กๅทฅๅ
ท vectorforge๏ผ่ฆ็ไปๆๅปบๅฐๆ็ดข็ๅฎๆดๅทฅไฝๆต๏ผ
# ๐๏ธ ๆๅปบ็ดขๅผ
vectorforge build -t hnsw -d 128 -n 1000 -o my_index.json
# ๐ ๆ็ดขๆ่ฟ้ป
vectorforge search my_index.json -q "0.1,0.2,0.3,..." -k 10
# โก ่ฟ่กๆง่ฝๅบๅๆต่ฏ
vectorforge benchmark -n 1000 -d 64
# ๐ ๆๅผ็ป็ซฏไปช่กจ็
vectorforge dashboard my_index.json
# ๐ฒ ็ๆ้ๆบๅ้ๆฐๆฎ
vectorforge generate -n 500 -d 128 -o vectors.json
# ๐ ๆ ผๅผ่ฝฌๆข๏ผCSV โ JSON๏ผ
vectorforge convert -i vectors.csv -o vectors.json้ซ็บง API๏ผๆจ่๏ผ
from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# ๅๅปบๆ็ดขๅผๆ
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
# ๆน้็ดขๅผๅ้
engine.index(vectors)
# ๆ็ดข k ไธชๆ่ฟ้ป
results = engine.search(query_vector, k=10)ๅบๅฑ API๏ผ็ฒพ็ปๆงๅถ๏ผ
from vectorforge.core import Vector, IndexConfig
from vectorforge.index import HNSWIndex
# ๅๅปบ HNSW ็ดขๅผ
index = HNSWIndex(IndexConfig(dimension=128, metric="cosine"))
# ๆๅปบ็ดขๅผ
index.build(vectors)
# ๆ็ดข
results = index.search(query, k=10)
# ๆไน
ๅๅฐ็ฃ็
index.save("my_index.json")
# ไป็ฃ็ๅ ่ฝฝ
index.load("my_index.json")| ็ฎๆณ | ้็จๅบๆฏ | ็น็น |
|---|---|---|
| HNSW | ๅคง่งๆจก้ซ็ฒพๅบฆๆ็ดข | ๅๅฑๅพ็ปๆ๏ผๅฌๅ็้ซ๏ผๆฅ่ฏข้ๅบฆๅฟซ |
| IVF-Flat | ไธญๅคง่งๆจกๅนณ่กกๆ็ดข | ๅๆ็ดขๅผ + ็ฒพ็กฎ้ๆ๏ผ้ๅบฆไธ็ฒพๅบฆๅนณ่กก |
| LSH | ่ถ ๅคง่งๆจก่ฟไผผๆ็ดข | ๅๅธๆ ๅฐ๏ผไบ็บฟๆงๆฅ่ฏขๆถ้ด |
| Brute-Force | ๅฐ่งๆจก็ฒพ็กฎๆ็ดข | ้ไธๆฏๅฏน๏ผ100% ๅฌๅ็ |
from vectorforge.core import IndexConfig
# ไฝๅผฆ็ธไผผๅบฆ๏ผ้ป่ฎค๏ผ้ๅๆๆฌ/่ฏญไนๆ็ดข๏ผ
config_cosine = IndexConfig(dimension=128, metric="cosine")
# ๆฌงๆฐ่ท็ฆป๏ผ้ๅๅพๅ็นๅพๆ็ดข๏ผ
config_euclidean = IndexConfig(dimension=128, metric="euclidean")
# ๅ
็งฏ๏ผ้ๅๅทฒๅฝไธๅ็ๅ้๏ผ
config_inner = IndexConfig(dimension=128, metric="inner_product")from vectorforge.io import export_json, import_json, export_csv, import_csv
# ๅฏผๅบไธบ JSON
export_json(vectors, "vectors.json")
# ไป JSON ๅฏผๅ
ฅ
vectors = import_json("vectors.json")
# ๅฏผๅบไธบ CSV
export_csv(vectors, "vectors.csv")
# ไป CSV ๅฏผๅ
ฅ
vectors = import_csv("vectors.csv")- ้ถไพ่ตไผๅ ๏ผๆ ธๅฟๅ่ฝ็บฏ Python ๅฎ็ฐ๏ผไธไพ่ตไปปไฝ็ฌฌไธๆนๅบ๏ผ้ไฝ้จ็ฝฒๅคๆๅบฆ
- API ๅๅฑ่ฎพ่ฎก๏ผ้ซ็บง API ็ฎๆดๆ็จ๏ผๅบๅฑ API ็ตๆดปๅฏๆง๏ผๆปก่ถณไธๅๅบๆฏ้ๆฑ
- ๅฏๆฉๅฑๆถๆ๏ผๅบไบๆฝ่ฑกๅบ็ฑป็็ดขๅผๆฅๅฃ๏ผ่ฝปๆพๆทปๅ ๆฐ็็ดขๅผ็ฎๆณๅ่ท็ฆปๅบฆ้
- ๅทฅๅ ท้พๅฎๆด๏ผไปๆฐๆฎ็ๆใๆ ผๅผ่ฝฌๆขใ็ดขๅผๆๅปบใๆ็ดขๆฅ่ฏขๅฐๆง่ฝๆต่ฏ๏ผไธๆก้พ่ฆ็
- v1.0.0 โ โ ๆ ธๅฟๅผๆๅๅธ๏ผๅ็ง็ดขๅผ็ฎๆณใไธ็ง่ท็ฆปๅบฆ้ใCLI ๅทฅๅ ทใTUI ไปช่กจ็
- v1.1.0 ๐ฎ โ ๆฐๅข PQ๏ผไน็งฏ้ๅ๏ผ็ฎๆณใๆฏๆ mmap ๅ ๅญๆ ๅฐใๆๅๅคง่งๆจกๆฐๆฎๆง่ฝ
- v1.2.0 ๐ฎ โ ๅผๅ ฅ่ฟๆปคๆ็ดข๏ผFilter Search๏ผใๆฏๆๅ ๆฐๆฎๅ ณ่ๆฅ่ฏข
- v2.0.0 ๐ฎ โ ๅๅธๅผ็ดขๅผๆฏๆใgRPC ๆๅก็ซฏๆจกๅผใWeb ็ฎก็็้ข
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .# ๅจ requirements.txt ไธญๆทปๅ
vectorforge @ git+https://github.com/gitstq/VectorForge.git- Python 3.8 ๆๆด้ซ็ๆฌ
- ๆ ้ไปปไฝๅค้จไพ่ต๏ผNumPy ไธบๅฏ้๏ผๅฎ่ฃ ๅๅฏ่ชๅจๅ ้่ฎก็ฎ๏ผ
ๆไปฌๆฌข่ฟๅนถๆ่ฐขๆๆๅฝขๅผ็่ดก็ฎ๏ผๆ ่ฎบๆฏๆไบค Bugใๆน่ฟๆๆกฃ๏ผ่ฟๆฏ่ดก็ฎไปฃ็ ใ
- ๐ด Fork ๆฌไปๅบ
- ๐ฟ ๅๅปบ็นๆงๅๆฏ๏ผ
git checkout -b feature/my-new-feature - ๐พ ๆไบคไฝ ็ๆนๅจ๏ผ
git commit -m 'Add some feature' - ๐ ๆจ้ๅฐ่ฟ็จๅๆฏ๏ผ
git push origin feature/my-new-feature - ๐ ๆไบค Pull Request
่ฏท็กฎไฟๆๆๆไบค้่ฟ CI ๆต่ฏ๏ผๅนถ้ตๅพช้กน็ฎ็ไปฃ็ ่ง่ใ
ๆฌ้กน็ฎๅบไบ MIT License ๅผๆบใ
MIT License
Copyright (c) 2024 VectorForge Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
pip install git+https://github.com/gitstq/VectorForge.git
VectorForge ๆฏไธๅ่ผ้็ดใ้ซๆ่ฝ็ๅ้็ดขๅผๅผๆ๏ผๅฎๅ จไฝฟ็จ็ด Python ๅฏฆ็พ๏ผ้ถๅค้จไพ่ณด๏ผNumPy ็บๅฏ้ธๅ ้ไพ่ณด๏ผใๅฎ็บๅ้็ธไผผๅบฆๆๅฐๆไพไบไธๅฅๅฎๆด็ๅทฅๅ ท้๏ผๆถต่ๅค็จฎ็ดขๅผๆผ็ฎๆณใ่ท้ขๅบฆ้ใๆไน ๅๅฒๅญใๅฝไปคๅๅทฅๅ ทๅ่ฆ่ฆบๅๅ่กจๆฟใ
็ก่ซไฝ ๆฏๅจๅปบๆง่ชๆๆๅฐๅผๆใๆจ่ฆ็ณป็ตฑใๅๅๆชข็ดข๏ผ้ๆฏๅๆฉๅจๅญธ็ฟๅๅ้ฉ่ญ๏ผVectorForge ้ฝ่ฝ่ฎไฝ ไปฅๆ็ฐกๅฎ็ๆนๅผๅฟซ้ไธๆๅ้ๆๅฐใ
| ็นๆง | ๆ่ฟฐ |
|---|---|
| ๐ง ๅ็จฎ็ดขๅผๆผ็ฎๆณ | HNSW๏ผๅๅฑคๅฐ่ชๅฐไธ็ๅ๏ผใIVF-Flat๏ผๅๆๆชๆก๏ผใLSH๏ผๅฑ้จๆๆ้ๆน๏ผใBrute-Force๏ผๆดๅๆๅฐ๏ผ |
| ๐ ไธ็จฎ่ท้ขๅบฆ้ | ้คๅผฆ็ธไผผๅบฆ๏ผCosine๏ผใๆญๆฐ่ท้ข๏ผEuclidean๏ผใๅ ง็ฉ๏ผInner Product๏ผ |
| ๐พ ็ดขๅผๆไน ๅ | ๆฏๆด JSON ๆ ผๅผๅบๅๅ/ๅๅบๅๅ๏ผ็ดขๅผๅฏๅฒๅญๅฐ็ฃ็ข้จๆ่ผๅ ฅ |
| ๐ ๅคๆ ผๅผ IO | ๆฏๆด JSON ๅ CSV ๆ ผๅผ็ๅ้่ณๆๅฏๅ ฅๅฏๅบ |
| ๐ TUI ็ต็ซฏๅ่กจๆฟ | ๅจ็ต็ซฏไธญ่ฆ่ฆบๅๆฅ็็ดขๅผ็ๆ ใ็ตฑ่จ่ณ่จๅๆ่ฝๆๆจ |
| โก ๆ่ฝๅบๆบๆธฌ่ฉฆ | ๅ งๅปบ benchmark ๆจก็ต๏ผไธ้ตๅฐๆฏไธๅ็ดขๅผๆผ็ฎๆณ็ๅปบๆง่ๆฅ่ฉขๆ่ฝ |
| ๐ฅ๏ธ CLI ๅฝไปคๅๅทฅๅ ท | ๅฎๆด็ๅฝไปคๅไป้ข๏ผๆฏๆดๅปบๆงใๆๅฐใ่ฝๆใ็ๆใๅบๆบๆธฌ่ฉฆ็ญๆไฝ |
| ๐ ้ซ้ๆๅฐ API | ็ฐกๆฝๅช้ ็ Python API๏ผๅนพ่ก็จๅผ็ขผๅณๅฏๅฎๆๅ้็ดขๅผ่ๆชข็ดข |
# ้้ pip ๅพ GitHub ๅฎ่ฃ
pip install git+https://github.com/gitstq/VectorForge.git
# ๆ่
็ดๆฅๅ
้ๅๅง็ขผไฝฟ็จ
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# 1. ๆบๅๅ้่ณๆ
vectors = [Vector(id=i, values=[float(j) for j in range(128)]) for i in range(100)]
# 2. ๅปบ็ซๆๅฐๅผๆ๏ผ้ซ้ API๏ผ
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
engine.index(vectors)
# 3. ๆๅฐๆ่ฟ้ฐ
query = Vector(id=-1, values=[float(j) for j in range(128)])
results = engine.search(query, k=10)
for r in results:
print(f"ID: {r.id}, ่ท้ข: {r.distance:.4f}")VectorForge ๆไพไบๅ่ฝๅฎๆด็ๅฝไปคๅๅทฅๅ
ท vectorforge๏ผ่ฆ่ๅพๅปบๆงๅฐๆๅฐ็ๅฎๆดๅทฅไฝๆต็จ๏ผ
# ๐๏ธ ๅปบๆง็ดขๅผ
vectorforge build -t hnsw -d 128 -n 1000 -o my_index.json
# ๐ ๆๅฐๆ่ฟ้ฐ
vectorforge search my_index.json -q "0.1,0.2,0.3,..." -k 10
# โก ๅท่กๆ่ฝๅบๆบๆธฌ่ฉฆ
vectorforge benchmark -n 1000 -d 64
# ๐ ้ๅ็ต็ซฏๅ่กจๆฟ
vectorforge dashboard my_index.json
# ๐ฒ ็ๆ้จๆฉๅ้่ณๆ
vectorforge generate -n 500 -d 128 -o vectors.json
# ๐ ๆ ผๅผ่ฝๆ๏ผCSV โ JSON๏ผ
vectorforge convert -i vectors.csv -o vectors.json้ซ้ API๏ผๆจ่ฆ๏ผ
from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# ๅปบ็ซๆๅฐๅผๆ
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
# ๆน้็ดขๅผๅ้
engine.index(vectors)
# ๆๅฐ k ๅๆ่ฟ้ฐ
results = engine.search(query_vector, k=10)ๅบๅฑค API๏ผ็ฒพ็ดฐๆงๅถ๏ผ
from vectorforge.core import Vector, IndexConfig
from vectorforge.index import HNSWIndex
# ๅปบ็ซ HNSW ็ดขๅผ
index = HNSWIndex(IndexConfig(dimension=128, metric="cosine"))
# ๅปบๆง็ดขๅผ
index.build(vectors)
# ๆๅฐ
results = index.search(query, k=10)
# ๆไน
ๅๅฐ็ฃ็ข
index.save("my_index.json")
# ๅพ็ฃ็ข่ผๅ
ฅ
index.load("my_index.json")| ๆผ็ฎๆณ | ้ฉ็จๅ ดๆฏ | ็น้ป |
|---|---|---|
| HNSW | ๅคง่ฆๆจก้ซ็ฒพๅบฆๆๅฐ | ๅๅฑคๅ็ตๆง๏ผๅฌๅ็้ซ๏ผๆฅ่ฉข้ๅบฆๅฟซ |
| IVF-Flat | ไธญๅคง่ฆๆจกๅนณ่กกๆๅฐ | ๅๆ็ดขๅผ + ็ฒพ็ขบ้ๆ๏ผ้ๅบฆ่็ฒพๅบฆๅนณ่กก |
| LSH | ่ถ ๅคง่ฆๆจก่ฟไผผๆๅฐ | ้ๆนๆ ๅฐ๏ผไบ็ทๆงๆฅ่ฉขๆ้ |
| Brute-Force | ๅฐ่ฆๆจก็ฒพ็ขบๆๅฐ | ้ไธๆฏๅฐ๏ผ100% ๅฌๅ็ |
from vectorforge.core import IndexConfig
# ้คๅผฆ็ธไผผๅบฆ๏ผ้ ่จญ๏ผ้ฉๅๆๅญ/่ชๆๆๅฐ๏ผ
config_cosine = IndexConfig(dimension=128, metric="cosine")
# ๆญๆฐ่ท้ข๏ผ้ฉๅๅๅ็นๅพตๆๅฐ๏ผ
config_euclidean = IndexConfig(dimension=128, metric="euclidean")
# ๅ
ง็ฉ๏ผ้ฉๅๅทฒๆญธไธๅ็ๅ้๏ผ
config_inner = IndexConfig(dimension=128, metric="inner_product")from vectorforge.io import export_json, import_json, export_csv, import_csv
# ๅฏๅบ็บ JSON
export_json(vectors, "vectors.json")
# ๅพ JSON ๅฏๅ
ฅ
vectors = import_json("vectors.json")
# ๅฏๅบ็บ CSV
export_csv(vectors, "vectors.csv")
# ๅพ CSV ๅฏๅ
ฅ
vectors = import_csv("vectors.csv")- ้ถไพ่ณดๅชๅ ๏ผๆ ธๅฟๅ่ฝ็ด Python ๅฏฆ็พ๏ผไธไพ่ณดไปปไฝ็ฌฌไธๆนๅฝๅผๅบซ๏ผ้ไฝ้จ็ฝฒ่ค้ๅบฆ
- API ๅๅฑค่จญ่จ๏ผ้ซ้ API ็ฐกๆฝๆ็จ๏ผๅบๅฑค API ้ๆดปๅฏๆง๏ผๆปฟ่ถณไธๅๅ ดๆฏ้ๆฑ
- ๅฏๆดๅฑๆถๆง๏ผๅบๆผๆฝ่ฑกๅบ้ก็็ดขๅผไป้ข๏ผ่ผ้ฌๆฐๅขๆฐ็็ดขๅผๆผ็ฎๆณๅ่ท้ขๅบฆ้
- ๅทฅๅ ท้ๅฎๆด๏ผๅพ่ณๆ็ๆใๆ ผๅผ่ฝๆใ็ดขๅผๅปบๆงใๆๅฐๆฅ่ฉขๅฐๆ่ฝๆธฌ่ฉฆ๏ผไธๆข้พ่ฆ่
- v1.0.0 โ โ ๆ ธๅฟๅผๆ็ผๅธ๏ผๅ็จฎ็ดขๅผๆผ็ฎๆณใไธ็จฎ่ท้ขๅบฆ้ใCLI ๅทฅๅ ทใTUI ๅ่กจๆฟ
- v1.1.0 ๐ฎ โ ๆฐๅข PQ๏ผไน็ฉ้ๅ๏ผๆผ็ฎๆณใๆฏๆด mmap ่จๆถ้ซๆ ๅฐใๆๅๅคง่ฆๆจก่ณๆๆ่ฝ
- v1.2.0 ๐ฎ โ ๅผๅ ฅ้ๆฟพๆๅฐ๏ผFilter Search๏ผใๆฏๆดๅ ่ณๆ้่ฏๆฅ่ฉข
- v2.0.0 ๐ฎ โ ๅๆฃๅผ็ดขๅผๆฏๆดใgRPC ไผบๆ็ซฏๆจกๅผใWeb ็ฎก็ไป้ข
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .# ๅจ requirements.txt ไธญๆทปๅ
vectorforge @ git+https://github.com/gitstq/VectorForge.git- Python 3.8 ๆๆด้ซ็ๆฌ
- ็ก้ไปปไฝๅค้จไพ่ณด๏ผNumPy ็บๅฏ้ธ๏ผๅฎ่ฃๅพๅฏ่ชๅๅ ้่จ็ฎ๏ผ
ๆๅๆญก่ฟไธฆๆ่ฌๆๆๅฝขๅผ็่ฒข็ป๏ผ็ก่ซๆฏๆไบค Bugใๆน้ฒๆไปถ๏ผ้ๆฏ่ฒข็ป็จๅผ็ขผใ
- ๐ด Fork ๆฌๅๅบซ
- ๐ฟ ๅปบ็ซ็นๆงๅๆฏ๏ผ
git checkout -b feature/my-new-feature - ๐พ ๆไบคไฝ ็ๆนๅ๏ผ
git commit -m 'Add some feature' - ๐ ๆจ้ๅฐ้ ็ซฏๅๆฏ๏ผ
git push origin feature/my-new-feature - ๐ ๆไบค Pull Request
่ซ็ขบไฟๆๆๆไบค้้ CI ๆธฌ่ฉฆ๏ผไธฆ้ตๅพชๅฐๆก็็จๅผ็ขผ่ฆ็ฏใ
ๆฌๅฐๆกๅบๆผ MIT License ้ๆบใ
MIT License
Copyright (c) 2024 VectorForge Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
pip install git+https://github.com/gitstq/VectorForge.git
VectorForge is a lightweight, high-performance vector index engine built entirely in pure Python with zero external dependencies (NumPy is optional for acceleration). It provides a complete toolkit for vector similarity search, featuring multiple index algorithms, distance metrics, persistent storage, CLI tools, and a visual dashboard.
Whether you're building semantic search engines, recommendation systems, image retrieval pipelines, or prototyping ML models, VectorForge gets you up and running with vector search in the simplest way possible.
| Feature | Description |
|---|---|
| ๐ง Four Index Algorithms | HNSW (Hierarchical Navigable Small World), IVF-Flat (Inverted File), LSH (Locality-Sensitive Hashing), Brute-Force |
| ๐ Three Distance Metrics | Cosine Similarity, Euclidean Distance, Inner Product |
| ๐พ Index Persistence | Serialize/deserialize indexes to/from disk in JSON format |
| ๐ Multi-Format I/O | Import and export vector data in JSON and CSV formats |
| ๐ TUI Dashboard | Visualize index status, statistics, and performance metrics right in your terminal |
| โก Benchmarking Module | Built-in benchmark tool to compare build and query performance across index algorithms |
| ๐ฅ๏ธ CLI Tool | Full-featured command-line interface for building, searching, converting, generating, and benchmarking |
| ๐ Advanced Search API | Clean and elegant Python API โ index and search vectors in just a few lines of code |
# Install via pip from GitHub
pip install git+https://github.com/gitstq/VectorForge.git
# Or clone the source and install locally
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# 1. Prepare vector data
vectors = [Vector(id=i, values=[float(j) for j in range(128)]) for i in range(100)]
# 2. Create a search engine (high-level API)
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
engine.index(vectors)
# 3. Search for nearest neighbors
query = Vector(id=-1, values=[float(j) for j in range(128)])
results = engine.search(query, k=10)
for r in results:
print(f"ID: {r.id}, Distance: {r.distance:.4f}")VectorForge ships with a full-featured CLI tool vectorforge that covers the entire workflow from index building to querying:
# ๐๏ธ Build an index
vectorforge build -t hnsw -d 128 -n 1000 -o my_index.json
# ๐ Search for nearest neighbors
vectorforge search my_index.json -q "0.1,0.2,0.3,..." -k 10
# โก Run performance benchmarks
vectorforge benchmark -n 1000 -d 64
# ๐ Open the terminal dashboard
vectorforge dashboard my_index.json
# ๐ฒ Generate random vector data
vectorforge generate -n 500 -d 128 -o vectors.json
# ๐ Convert formats (CSV โ JSON)
vectorforge convert -i vectors.csv -o vectors.jsonHigh-Level API (Recommended)
from vectorforge.core import Vector, IndexConfig
from vectorforge.search import VectorSearch
# Create a search engine
engine = VectorSearch(index_type="hnsw", config=IndexConfig(dimension=128))
# Index vectors in bulk
engine.index(vectors)
# Search for k nearest neighbors
results = engine.search(query_vector, k=10)Low-Level API (Fine-Grained Control)
from vectorforge.core import Vector, IndexConfig
from vectorforge.index import HNSWIndex
# Create an HNSW index
index = HNSWIndex(IndexConfig(dimension=128, metric="cosine"))
# Build the index
index.build(vectors)
# Search
results = index.search(query, k=10)
# Persist to disk
index.save("my_index.json")
# Load from disk
index.load("my_index.json")| Algorithm | Best For | Characteristics |
|---|---|---|
| HNSW | Large-scale, high-accuracy search | Hierarchical graph structure, high recall, fast queries |
| IVF-Flat | Medium-to-large scale, balanced search | Inverted index + exact re-ranking, balanced speed and accuracy |
| LSH | Ultra-large scale, approximate search | Hash-based mapping, sub-linear query time |
| Brute-Force | Small-scale, exact search | Exhaustive comparison, 100% recall guarantee |
from vectorforge.core import IndexConfig
# Cosine similarity (default, ideal for text/semantic search)
config_cosine = IndexConfig(dimension=128, metric="cosine")
# Euclidean distance (ideal for image feature search)
config_euclidean = IndexConfig(dimension=128, metric="euclidean")
# Inner product (ideal for pre-normalized vectors)
config_inner = IndexConfig(dimension=128, metric="inner_product")from vectorforge.io import export_json, import_json, export_csv, import_csv
# Export to JSON
export_json(vectors, "vectors.json")
# Import from JSON
vectors = import_json("vectors.json")
# Export to CSV
export_csv(vectors, "vectors.csv")
# Import from CSV
vectors = import_csv("vectors.csv")- Zero Dependencies First: Core functionality is implemented in pure Python with no third-party libraries, minimizing deployment complexity
- Layered API Design: The high-level API is simple and intuitive, while the low-level API offers fine-grained control for advanced use cases
- Extensible Architecture: Abstract base classes for index interfaces make it easy to add new algorithms and distance metrics
- Complete Toolchain: From data generation and format conversion to index building, querying, and benchmarking โ everything is covered end-to-end
- v1.0.0 โ โ Core engine release: four index algorithms, three distance metrics, CLI tool, TUI dashboard
- v1.1.0 ๐ฎ โ Add PQ (Product Quantization) algorithm, mmap memory mapping support, improved large-scale data performance
- v1.2.0 ๐ฎ โ Introduce filtered search, metadata-associated queries
- v2.0.0 ๐ฎ โ Distributed index support, gRPC server mode, web management UI
git clone https://github.com/gitstq/VectorForge.git
cd VectorForge
pip install -e .# Add to requirements.txt
vectorforge @ git+https://github.com/gitstq/VectorForge.git- Python 3.8 or later
- No external dependencies required (NumPy is optional and enables automatic compute acceleration when installed)
We welcome and appreciate contributions of all kinds โ bug reports, documentation improvements, or code contributions.
- ๐ด Fork this repository
- ๐ฟ Create a feature branch:
git checkout -b feature/my-new-feature - ๐พ Commit your changes:
git commit -m 'Add some feature' - ๐ Push to the remote branch:
git push origin feature/my-new-feature - ๐ Submit a Pull Request
Please make sure all submissions pass CI tests and follow the project's code conventions.
This project is licensed under the MIT License.
MIT License
Copyright (c) 2024 VectorForge Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
Made with ๐ฅ by VectorForge Contributors