Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning: set the 17th bit of the txn_source to indicate the write source is lightning physical mode import #57706

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

lightning: set the 17th bit of the txn_source to indicate the write source is lightning physical mode import #57706

wants to merge 5 commits into from
+24 −11

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented Nov 26, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #57714

Problem Summary:

  • TiCDC changefeed need to filter the lightning physical mode imported data, but there is no clue to indicate whether one row is inserted by the lightning or not.

What changed and how does it work?

  • set the 17th bit of the field txn_source, to indicates that the row is inserted by the lightning physical mode, so the TiKV-CDC can skip this row

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 26, 2024
@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

Hi @3AceShowHand. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 26, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lance6716 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov Codecov
Copy link

codecov bot commented Nov 26, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.6033%. Comparing base (b11d034) to head (25d2a9a).

Additional details and impacted files 
@@               Coverage Diff                @@
##             master     #57706        +/-   ##
================================================
+ Coverage   72.8126%   73.6033%   +0.7907%     
================================================
  Files          1677       1707        +30     
  Lines        463954     471981      +8027     
================================================
+ Hits         337817     347394      +9577     
+ Misses       105277     103109      -2168     
- Partials      20860      21478       +618
Flag Coverage Δ
integration 46.2283% <100.0000%> (?)
unit 72.1950% <100.0000%> (-0.0200%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.7673% <ø> (ø)
parser ∅ <ø> (∅)
br 45.5082% <ø> (+0.0060%) ⬆️

@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 26, 2024
@3AceShowHand 3AceShowHand changed the title lightning: set the 17th to indicate the write source lightning: set the 17th bit of the txn_source to indicate the write source is lightning physical mode import Nov 26, 2024
lance6716
lance6716 reviewed Nov 26, 2024
View reviewed changes
pkg/lightning/backend/local/region_job.go
ResourceGroupName: ResourceGroupName,
},
RequestSource: util.BuildRequestSource(true, kv.InternalTxnLightning, TaskType),
TxnSource: kv.LightningPhysicalImportTxnSource,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please post manual test result. I'm not sure TiKV import service will use this value.

And maybe we can add a SST table / block property to skip scanning the whole SST to reduce IO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can add a SST table / block property to skip scanning the whole SST to reduce IO.

It sounds great to do so, but looks more complicated since we have to consider the case such as region merge or split? Also, is it possible that there is one sst that contains lightning physical imported data and normal inserted data at the same time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please post manual test result. I'm not sure TiKV import service will use this value.

Yes, this PR should be get merged after test, I will post the manual test result.

https://github.com/tikv/tikv/pull/17895/files This PR set the txn_source to initialize the txn_sst_writer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can add a SST table / block property to skip scanning the whole SST to reduce IO.

It sounds great to do so, but looks more complicated since we have to consider the case such as region merge or split? Also, is it possible that there is one sst that contains lightning physical imported data and normal inserted data at the same time?

Yes, just as a fast path. If SST merging happened we have to scan all KV.

@3AceShowHand
Copy link
Contributor Author

/retest

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@3AceShowHand: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@purelind
Copy link
Contributor

/retest

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@purelind: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@3AceShowHand
Copy link
Contributor Author

/retest

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@3AceShowHand: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@wuhuizuo
Copy link
Contributor

/test build

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@wuhuizuo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@3AceShowHand
Copy link
Contributor Author

/retest

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@3AceShowHand: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@3AceShowHand 3AceShowHand mentioned this pull request Nov 26, 2024
Open
9 tasks
@3AceShowHand
Copy link
Contributor Author

/retest

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@3AceShowHand: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@wuhuizuo
Copy link
Contributor

/test build

@tiprow tiprow
Copy link

tiprow bot commented Nov 26, 2024

@wuhuizuo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 26, 2024 •
edited
Loading

@3AceShowHand: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
idc-jenkins-ci-tidb/build 25d2a9a link true /test build

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lance6716 lance6716 lance6716 left review comments

@glorv glorv Awaiting requested review from glorv

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Lightning set write source when import data in the physical mode
4 participants
@3AceShowHand @purelind @wuhuizuo @lance6716