Code Chronicles
Inconsistent Accept-Encoding Request Header

Inconsistent Accept-Encoding Request Header

in Chrome and Chromium (113.0.5672.85)

Jeff Tong's photo
Jeff Tong
·

2 min read

There is inconsistent behavior with the output generated by the Chrome/Chromium Developer Tools for cUrl and Powershell (Invoke-WebRequest).

I'm using httpbin.org/post as an example:

curl -X POST 'https://httpbin.org/post' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/json' \
  -H 'Origin: https://share.getawair.com' \
  -H 'Referer: https://share.getawair.com/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  --data-raw '{"something":"something_else"}' \
  --compressed


Invoke-WebRequest -UseBasicParsing -Uri "https://httpbin.org/post" `
-Method "POST" `
-Headers @{
"Accept"="application/json, text/plain, */*"
  "Accept-Encoding"="gzip, deflate, br"
  "Accept-Language"="en-US,en;q=0.9"
  "Origin"="https://share.getawair.com"
  "Referer"="https://share.getawair.com/"
  "Sec-Fetch-Dest"="empty"
  "Sec-Fetch-Mode"="cors"
  "Sec-Fetch-Site"="cross-site"
  "sec-ch-ua"="`"Chromium`";v=`"112`", `"Google Chrome`";v=`"112`", `"Not:A-Brand`";v=`"99`""
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
} `
-ContentType "application/json" `
-Body "{`"something`":`"something_else`"}"

The response back from Powershell has garbled response content:

At first, I thought it might be a Powershell encoding issue and tried to re-encode the content to UTF-8, ASCII, UTF-16, and ISO-8859-1 to no avail.

Comparing the cUrl request vs the Invoke-WebRequest, you can see the differences exist in the Accept-Encoding request header with "Accept-Encoding"="gzip, deflate, br"

The br indicates Brotli compression. By removing this, I was able to get the same response as the cUrl request. Hitting other APIs with Invoke-WebRequest and that request header value will yield the expected response.

Why do Chrome and Chromium generate the cUrl request without Accept-Encoding while Powershell Invoke-WebRequest does? Firefox includes the 'Accept-Encoding: gzip, deflate, br' request header.

curl 'https://httpbin.org/post' -X POST -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://share.getawair.com/' -H 'Content-Type: application/json' -H 'Origin: https://share.getawair.com' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'x-ak-sbux-debug: x-ak-sbux-debug' --data-raw '{"something":"something_else"}'

I had to add the --compressed flag to have it try to output the response and using version 7.68.0-1ubuntu2.18:

curl: (23) Unrecognized content encoding type. libcurl understands deflate, gzip content encodings.

Supposedly Brotli compression is supported by cUrl, which seems to be available in the default build.

References

chromiumcurlPowershell