Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance for CLI with database format output #3

Open
wtlin1228 opened this issue Oct 12, 2024 · 1 comment
Open

Improve performance for CLI with database format output #3

wtlin1228 opened this issue Oct 12, 2024 · 1 comment
Assignees

Comments

@wtlin1228
Copy link
Owner

wtlin1228 commented Oct 12, 2024

The database version takes 3m while the in-memory version only takes 5s to parse 5,000 modules and create the portable. Maybe there is some optimization could speed up the database version. Like:

  • Use in-memory https://www.sqlite.org/inmemorydb.html SQLite then serialize it in the end.
  • Use the original in-memory approach to collect the portable then batch insert the records into SQLite database.
  • Enable multiple threads database modification by (1) replacing the "retrieve then create" with UNIQUE table constraint and (2) removing last_insert_rowid().
  • Add index for tables.
@wtlin1228 wtlin1228 self-assigned this Oct 12, 2024
@wtlin1228 wtlin1228 moved this to Todo in Leo's Work Oct 15, 2024
@wtlin1228
Copy link
Owner Author

Maybe async/await is better choice since the bottleneck is I/O bound, not CPU bound.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant