chore(middleware-flexible-checksums): use RequestChecksumCalculation and ResponseChecksumValidation

[trivikr]

Issue

• Internal JS-5396
• Alternative to chore(middleware-flexible-checksums): use RequestChecksumCalculation and ResponseChecksumValidation without interceptor middleware #6484 with flexibleChecksumsInterceptorMiddleware

Description

• Uses values in RequestChecksumCalculation and ResponseChecksumValidation to decide flexible checksums algorithm.
• Adds flexibleChecksumsInterceptorMiddleware to populate input[requestValidationModeMember]
• Deprecates MD5, and sets CRC32 as default.

Testing

Unit testing in CI

Also, verified that checksums are computed by default

import { S3 } from "../aws-sdk-js-v3/clients/client-s3/dist-cjs/index.js";
import { NodeHttpHandler } from "@smithy/node-http-handler";
import { equal } from "assert";

// Prints checksum headers for request and response.
class CustomHandler extends NodeHttpHandler {
  constructor() {
    super();
  }

  printChecksumHeaders(prefix, headers) {
    for (const [header, value] of Object.entries(headers)) {
      if (header.startsWith("x-amz-checksum-") || header.startsWith("x-amz-sdk-checksum-")) {
        console.log(`${prefix}['${header}']: '${value}'`);
      }
    }
  }

  async handle(request, options) {
    this.printChecksumHeaders("request", request.headers);
    const response = await super.handle(request, options);
    this.printChecksumHeaders("response", response.response.headers);
    return response;
  }
}

// WHEN_SUPPORTED is default.
const client = new S3({
  requestHandler: new CustomHandler(),
});
const Bucket = "test-flexible-checksums-v2"; // Replace with your test bucket name.
const Key = "hello-world.txt";
const Body = "Hello World"; // Replace with the content you want to test.

console.log("Put Object");
// ChecksumAlgorithm is not explicitly required to be specified.
await client.putObject({ Bucket, Key, Body });

console.log("\nGet Object");
// ChecksumMode is not explicitly required to be specified.
const response = await client.getObject({ Bucket, Key });

equal(Body, await response.Body.transformToString());

Output

Put Object
request['x-amz-sdk-checksum-algorithm']: 'CRC32'
request['x-amz-checksum-crc32']: 'ShexVg=='
response['x-amz-checksum-crc32']: 'ShexVg=='

Get Object
request['x-amz-checksum-mode']: 'ENABLED'
response['x-amz-checksum-crc32']: 'ShexVg=='

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[trivikr]

The PR is ready for review. It's marked as draft, as it shouldn't be merged at this time.

[trivikr]

The default checksum is changed from CRC32 to SHA256 based on benchmarks provided in #6499

[trivikr]

The default checksum is switched back to CRC32 based on internal discussions with other SDK maintainers

[trivikr]

Feature flags were added in d7c329a (#6492)

[trivikr]

Code to skip checksum computation when user has set checksum header field was added in bc74a9a (#6492)

[trivikr]

The checksum computation is not skipped for x-amz-checksum-algorithm in 9204de2 (#6492)

Behavior reverted in internal communications

[trivikr]

Commit to set input[requestAlgorithmMember] to default checksum algorithm if it's not set and either requestChecksumCalculation is supported or requestChecksumRequired is set to true: 317926c (#6492)

[trivikr]

Commit to remove the case from flexibleChecksumsMiddleware where the input[requestAlgorithmMember] is not set, as that case is now handled in flexibleChecksumsInputMiddleware: dc5b5ae (#6492)

[trivikr]

The tests are failing because the input is processed from middlewareConfig sent from and not args.
Since it's not related to using RequestChecksumCalculation and ResponseChecksumValidation, I'll fix it outside of PR and rebase.

[trivikr]

100+ integration tests were added in ae9d5e9 (#6492)

We also fixed an edge case to skip checksum if input[requestAlgorithmMember] is not set

[trivikr]

An edge case to leave input.requestAlgorithmMember if checksum header is set was implemented in 969be81 (#6492)

Reverted in 2d70aed (#6492) since request is not available before serialization.

[trivikr]

To handle previous ask, the input[requestAlgorithmMember] (if added by SDK) was removed in 5f86077 (#6492) if checksum header is provided by the user.

[kuhe]

this should state the condition more completely, since

Remove input[requestAlgorithmMember] and header, if it was added by flexibleChecksumsInputMiddleware

sounds like an unconditional removal

it should perhaps say "... and the user has set the checksum via a modeled field"?

[trivikr]

Instead of populating input[requestAlgorithmMember] in InputMiddleware and attempting to remove it from main middleware, we switched to using newly introduced requestAlgorithmMemberHttpHeader to set httpHeader in b614dbe (#6492)