first commit (FALCONN-LIB#62)

* first commit * final version * add ssnpp-1B in benchmarks * fix no new line in algo.yaml * modify benchmark * modify new line * Update algos.yaml add missing "}" to make msturing-1B work * modify algos.yaml to make all CI test pass * remove new line in algos.yaml * add new line in algo.yaml * modify new line * algo.yaml new line * fix bigann * fix comma problem in ssnpp-1B * Count ties using extended ground truth list (FALCONN-LIB#59) * Count ties using extended ground truth list * added credit to ann-benchmarks.com * added diskann-v0.3 and medoids to component list * Bump pyyaml from 5.3.1 to 5.4 Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3.1 to 5.4. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](yaml/pyyaml@5.3.1...5.4) --- updated-dependencies: - dependency-name: pyyaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Change default search radius for SSNPP-1B to 96237 * Update metrics.py * a proposed recall-with-ties impl with related unit tests (FALCONN-LIB#74) * a new impl of recall-with-ties with careful attention determining true_ids subset and inner product distance; unit tests adjusted accordingly. * more compact and readable code based on martin's suggestion * validated on existing recall unit tests, in addition approved by T3 participants who noticed the ties issue. * First T3 track leaderboard (FALCONN-LIB#75) * adding latest leaderboard and all readme's that link to it * sync tasks from gw/T3 branch * syncing from gw/T3 * The markdown rel paths didnt work with the last PR to main. Switching to absolute paths which is not ideal, but should be ok for now. (FALCONN-LIB#76) * updated leaderboard with cuanns updates (FALCONN-LIB#77) * cumulative updates forcing new LEADERBOARD;updated TASKS as well (FALCONN-LIB#78) * added float32 coercion to data before faiss range search call; locked python version to 369 for docker build (FALCONN-LIB#79) * new leaderboard based on eval of new gemini indexes for ssnpp and text2image (FALCONN-LIB#80) * Store individual search times and extend CSV export capability for search times and power metrics. (FALCONN-LIB#81) * storing the list of search times as h5py attr during query evaluation * support for power sensor and search_times extraction to CSV in data_export.py * added --detect_caching and --sensor_times flags * added simple logic to detect possible response caching * added leaderboard history directory with first snapshot (FALCONN-LIB#82) * Update the last LB rankings and TASKs for public query set (FALCONN-LIB#83) * updated the last LB rankings and TASK for public query set * fixing a few typos * added logic to generate a leaderboard version that rejects anomalous query sets * T1/T2 public query set results * T1/T2 public query set results * T1/T2 public query set results * T1/T2 public query set results * refactored LM into multi LB READMEs and new navigation logic (FALCONN-LIB#86) * Adding more private_query functionality and fixes (FALCONN-LIB#87) * added some private query sets * added paths to heldout queries and GT100 for all datasets except msspacev * added spacev private queries * updates * isolating pqs changes for merge * need to update main with the baseline config and algo from T3 for testing * fix issue with failed extraction of search times * using hyphens in cmd flags * undoing/commenting the deep1b incorrect header HACK * adding a priv run script for testing * removed debug print Co-authored-by: Harsha Vardhan Simhadri <[email protected]> * Fix T1/T2 results * Fix typo in T1/T2 readme * Adding winners to the T3 public data leaderboard (FALCONN-LIB#88) * adding co-winner announcements * synced LB that fixed wording * synced LB changes from gw/T3 * synced LB fix with missing msspacev * synced LB with public link fix * updated both T3 LBs (FALCONN-LIB#89) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: qiaoyu <[email protected]> Co-authored-by: Harsha Vardhan Simhadri <[email protected]> Co-authored-by: Martin Aumueller <[email protected]> Co-authored-by: Harsha Vardhan Simhadri <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: C. George Williams <[email protected]> Co-authored-by: Harsha Vardhan Simhadri <[email protected]>
duckth · Apr 6, 2023 · b475f10 · b475f10
1 parent afcabb0
commit b475f10
Show file tree

Hide file tree

Showing 4 changed files with 405 additions and 0 deletions.
diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml
@@ -36,6 +36,12 @@ jobs:
           - algorithm: httpann_example
             dataset: random-range-xs
             library: httpann_example
+          - algorithm: kota-t2
+            dataset: random-xs
+            library: kota
+          - algorithm: kota-t2
+            dataset: random-range-xs
+            library: kota 
           - algorithm: team11
             dataset: random-range-xs
             library: faissconda

diff --git a/algos.yaml b/algos.yaml
@@ -9,6 +9,17 @@ any:
           args: [[1024,2048,4096,8192]]
           query-args: [[1, 5, 10, 50, 100, 200]]
 random-range-xs:
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":32, "L":32, "B":0.0001, "M":1}]
+          query-args: |
+            [{"Lmin":3, "Lmax":100, "BW":4, "T":16}]
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1
@@ -78,6 +89,19 @@ random-range-xs:
           query-args: |
             [{"rangeSearchProbeCount":20, "efSearch":100, "K1":2, "blockSize":4096, "radiusFactor":1.00, "identifier":"bbann"}]
 random-xs:
+
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":32, "L":32, "B":0.0001, "M":1}]
+          query-args: |
+              [{"Ls":10, "BW":4, "T":16}]
+
     team11:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1_plus
@@ -102,6 +126,7 @@ random-xs:
               "nprobe=64,quantizer_efSearch=128"
               ]
 
+
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1
@@ -282,6 +307,28 @@ deep-1B:
               {"Ls":80, "BW":4, "T":16},
               {"Ls":100, "BW":4, "T":16}]
 msspacev-1B:
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":100, "L":100, "B":47, "M":100,
+                "url": "https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/msspacev-1B/R100_L100_B47_M100"
+                }]
+          query-args: |
+              [{"Ls":60, "BW":4, "T":16, "HE":3},
+              {"Ls":70, "BW":4, "T":16, "HE":3},
+              {"Ls":80, "BW":4, "T":16, "HE":3},
+              {"Ls":90, "BW":4, "T":16, "HE":3},
+              {"Ls":95, "BW":4, "T":16, "HE":3},
+              {"Ls":100, "BW":4, "T":16, "HE":3},
+              {"Ls":105, "BW":4, "T":16, "HE":3},
+              {"Ls":110, "BW":4, "T":16, "HE":3},
+              {"Ls":120, "BW":4, "T":16, "HE":3},
+              {"Ls":130, "BW":4, "T":16, "HE":3}]
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1
@@ -367,6 +414,28 @@ msspacev-1B:
                 {"nProbe":180, "efSearch":360, "K1":128, "blockSize":4096, "identifier": "bbann"}
               ]
 msturing-1B:
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":100, "L":100, "B":50, "M":80,
+                "url": "https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/msturing-1B/R100_L100_B50_M80"
+                }]
+          query-args: |
+              [{"Ls":30, "BW":4, "T":16},
+              {"Ls":40, "BW":4, "T":16},
+              {"Ls":50, "BW":4, "T":16},
+              {"Ls":55, "BW":4, "T":16},
+              {"Ls":57, "BW":4, "T":16},
+              {"Ls":59, "BW":4, "T":16},
+              {"Ls":60, "BW":4, "T":16},
+              {"Ls":70, "BW":4, "T":16},
+              {"Ls":80, "BW":4, "T":16},
+              {"Ls":100, "BW":4, "T":16}]
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1
@@ -450,6 +519,28 @@ msturing-1B:
               {"Ls":80, "BW":4, "T":16},
               {"Ls":100, "BW":4, "T":16}]
 bigann-1B:
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":100, "L":100, "B":50, "M":80,
+                "url": "https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/bigann-1B/R100_L100_B50_M80"
+                }]
+          query-args: |
+              [{"Ls":30, "BW":4, "T":16},
+              {"Ls":40, "BW":4, "T":16},
+              {"Ls":50, "BW":4, "T":16},
+              {"Ls":55, "BW":4, "T":16},
+              {"Ls":60, "BW":4, "T":16},
+              {"Ls":62, "BW":4, "T":16},
+              {"Ls":65, "BW":4, "T":16},
+              {"Ls":70, "BW":4, "T":16},
+              {"Ls":80, "BW":4, "T":16},
+              {"Ls":100, "BW":4, "T":16}]   
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1
@@ -537,6 +628,28 @@ bigann-1B:
               {"Ls":80, "BW":4, "T":16},
               {"Ls":100, "BW":4, "T":16}]
 ssnpp-1B:
+    kota-t2:
+      docker-tag: billion-scale-benchmark-kota
+      module: benchmark.algorithms.kota-t2
+      constructor: KoTa
+      base-args: ["@metric"]
+      run-groups:
+        base:
+          args: |
+              [{"R":100, "L":100, "B":60, "M":100, "C":500000, "CM":2,
+                "url": "https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/fbssnpp-1B/R100_L100_B60_M100"
+                }]
+          query-args: |
+              [{"Lmin":9, "Lmax":10000, "BW":4, "T":16},
+               {"Lmin":14, "Lmax":10000, "BW":4, "T":16},
+               {"Lmin":9, "Lmax":20000, "BW":4, "T":16},
+               {"Lmin":14, "Lmax":20000, "BW":4, "T":16},
+               {"Lmin":19, "Lmax":20000, "BW":4, "T":16},
+               {"Lmin":39, "Lmax":20000, "BW":4, "T":16},
+               {"Lmin":7, "Lmax":30000, "BW":4, "T":16},
+               {"Lmin":14, "Lmax":30000, "BW":4, "T":16},
+               {"Lmin":29, "Lmax":30000, "BW":4, "T":16},
+               {"Lmin":58, "Lmax":30000, "BW":4, "T":16}]
     faiss-t1:
       docker-tag: billion-scale-benchmark-faissconda
       module: benchmark.algorithms.faiss_t1