Semi-supervised inverted file index approach for approximate nearest neighbor search
Вантажиться...
Дата
2023
Автори
Науковий керівник
Назва журналу
Номер ISSN
Назва тому
Видавець
КПІ ім. Ігоря Сікорського
Анотація
This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc.
Опис
Ключові слова
approximate nearest neighbor search, inverted file index, highdimensional data, machine learning, пошук наближених найближчих сусідів, інвертований файловий індекс, дані високої розмірності, машинне навчання
Бібліографічний опис
Bazdyrev, A. Semi-supervised inverted file index approach for approximate nearest neighbor search / A. Bazdyrev // Системні дослідження та інформаційні технології : міжнародний науково-технічний журнал. – 2023. – № 4. – С. 69-75. – Бібліогр.: 10 назв.