Skip to content

get_table Improvment - #1145

Merged
tsmathis merged 1 commit into
materialsproject:mainfrom
bfoley12:table-search-optimization
Nov 5, 2026
Merged

tsmathis merged 1 commit into
materialsproject:mainfrom
bfoley12:table-search-optimization

Conversation

@bfoley12

@bfoley12 bfoley12 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Response time for getting larger tables was large (~10sec for ~9k rows). This was due to serial pagination. Now get_table concurrently finds all pages then assembles them into a table. This leads to a 1.8x speed-up during testing.

Major Changes

get_table now fetches pages >=2 concurrently, using page 1 to size its concurrent requests.

…ally

Response time for getting larger tables was large (~10sec for ~9k rows).
This was due to serial pagination. Now get_table concurrently finds all
pages then assembles them into a table. This leads to a 1.8x speed-up
during testing.
@tsmathis

tsmathis commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

which table/query were you testing?

@bfoley12

bfoley12 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

which table/query were you testing?

A_Lab's tables are being loaded into an app, so I did a quick pass at speeding up loading tables. Wouldn't call this "optimzation", since I am just using concurrency to speed up the original process.

Getting tables with md5s: ['b785f29bef8655ee75bfb98ec95c6022', 'f11c5e78483d8818c39bc99afa576de0', 'fdb6e4eb3f52376aa0eb5a47b2211abd']

Running on a single thread the third table took ~7 seconds. With concurrency it took ~3 seconds.

from mpships_infra import get_rester 
doc_client = get_rester(use_document_model=False, project="A_Lab").contribs

raw = doc_client.query_contributions(
    query={"identifier": "RG-5694494-5"}, fields=["_all"]
)["data"][0]
tables = raw.get("tables")
md5s = [t["md5"] for t in tables]

for m in md5s:
    doc_client.get_table(m)

@tsmathis
tsmathis merged commit a827c33 into materialsproject:main Nov 5, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants