IDEA index
IDEA is a research inverted index that maps terms to shared chunks and then maps those chunks to the files that reference them.
Whitespace-aligned chunking keeps textual terms inside chunks. The index separates term-to-chunk, chunk-to-file and file-to-path mappings. Its prototype integrates Lucene-based indexing with Destor. [1]
Already-chunked inputs and compressed documents need additional handling; arbitrary byte-string search is not interchangeable with whitespace-aligned term indexing. The evaluation uses Linux source versions and Wikipedia dumps. Its ranking prototype supports single-term queries; proposed multi-term ranking is not an implemented result. [1]
Relations
- related
- Data deduplication, Destor, DedupSearch
Citations
Further reading
Sources & caveats
- Full-text source review, especially sections 4-8 and the artifact appendix; no native IDEA benchmark or paper-result reproduction.