Upload of wheel to Artifactory Failing
curl: (56) Recv failure: Connection was reset
A team member was having issues with a curl command to upload .whl file to a JFrog Artifactory instance. This had been working the previous day. Initially, I thought this could have been an SSL error, but it was not. Seems that connection was dropped.
authorization: Basic xyz*******************************************************************************=
2023-06-08T02:34:48.5721634Z > user-agent: curl/7.88.1
2023-06-08T02:34:48.5721851Z
2023-06-08T02:34:48.5722085Z > accept: */*
2023-06-08T02:34:48.5722284Z
2023-06-08T02:34:48.5722623Z > content-length: 280197
2023-06-08T02:34:48.5722818Z
2023-06-08T02:34:48.5723054Z >
2023-06-08T02:34:48.5723208Z
2023-06-08T02:34:48.5723450Z { [5 bytes data]
2023-06-08T02:34:48.5723617Z
2023-06-08T02:34:49.5708670Z 46 273k 0 0 46 128k 0 94854 0:00:02 0:00:01 0:00:01 94910
2023-06-08T02:34:50.5710965Z 46 273k 0 0 46 128k 0 55030 0:00:05 0:00:02 0:00:03 55049
2023-06-08T02:34:51.5708602Z 46 273k 0 0 46 128k 0 38757 0:00:07 0:00:03 0:00:04 38767
2023-06-08T02:34:52.5708652Z 46 273k 0 0 46 128k 0 29912 0:00:09 0:00:04 0:00:05 29918
2023-06-08T02:34:53.5709281Z 46 273k 0 0 46 128k 0 24354 0:00:11 0:00:05 0:00:06 26892
2023-06-08T02:34:54.5712684Z 46 273k 0 0 46 128k 0 20538 0:00:13 0:00:06 0:00:07 0
2023-06-08T02:34:55.5710293Z 46 273k 0 0 46 128k 0 17756 0:00:15 0:00:07 0:00:08 0
2023-06-08T02:34:56.5710444Z 46 273k 0 0 46 128k 0 15637 0:00:17 0:00:08 0:00:09 0
2023-06-08T02:34:57.5710586Z 46 273k 0 0 46 128k 0 13970 0:00:20 0:00:09 0:00:11 0
2023-06-08T02:34:58.5710720Z 46 273k 0 0 46 128k 0 12625 0:00:22 0:00:10 0:00:12 0
2023-06-08T02:34:59.5710919Z 46 273k 0 0 46 128k 0 11515 0:00:24 0:00:11 0:00:13 0
2023-06-08T02:35:00.5710539Z 46 273k 0 0 46 128k 0 10585 0:00:26 0:00:12 0:00:14 0
2023-06-08T02:35:01.5710664Z 46 273k 0 0 46 128k 0 9794 0:00:28 0:00:13 0:00:15 0
2023-06-08T02:35:02.5711744Z 46 273k 0 0 46 128k 0 9113 0:00:30 0:00:14 0:00:16 0
2023-06-08T02:35:03.5710683Z 46 273k 0 0 46 128k 0 8521 0:00:32 0:00:15 0:00:17 0
2023-06-08T02:35:04.5711418Z 46 273k 0 0 46 128k 0 8001 0:00:35 0:00:16 0:00:19 0
2023-06-08T02:35:05.5711752Z 46 273k 0 0 46 128k 0 7540 0:00:37 0:00:17 0:00:20 0
2023-06-08T02:35:06.5711897Z 46 273k 0 0 46 128k 0 7130 0:00:39 0:00:18 0:00:21 0
2023-06-08T02:35:07.5712981Z 46 273k 0 0 46 128k 0 6762 0:00:41 0:00:19 0:00:22 0
2023-06-08T02:35:08.5711947Z 46 273k 0 0 46 128k 0 6430 0:00:43 0:00:20 0:00:23 0
2023-06-08T02:35:09.5051403Z 46 273k 0 0 46 128k 0 6130 0:00:45 0:00:21 0:00:24 0* Recv failure: Connection was reset
2023-06-08T02:35:09.5052004Z * OpenSSL SSL_read: Connection was reset, errno 10054
2023-06-08T02:35:09.5052354Z * Failed receiving HTTP2 data
2023-06-08T02:35:09.5052507Z
2023-06-08T02:35:09.5052832Z 46 273k 0 0 46 128k 0 5876 0:00:47 0:00:22 0:00:25 0
2023-06-08T02:35:09.5054011Z * Connection #0 to host jfrog.artifactory.fake.com left intact
2023-06-08T02:35:09.5054279Z curl: (56) Recv failure: Connection was reset
The curl command used:
curl -v -u $(user):$(token)-X PUT https://jfrog.artifactory.fake.com/artifactory/group/service/pre-prod/1.1.1/service-1.1.1-py3-none-any.whl -T ./output/service-1.1.1-py3-none-any.whl
Other team members had tested locally and shown that the curl command works. I cloned the Azure DevOps pipeline and was trying some logging of the variables to make sure that they were valid. The secrets such as $(user) and $(token) won't show up in the logs when echoed, but you can retrieve them.
Fun Fact 1: You can retrieve them by encoding the string as hexadecimal:
secret_in_hex=$(xxd -p <<< $jfrog-artifactory-token -c 256)
echo $secret_in_hex
This will output the ASCII string to hexadecimal bypassing the masking logic. Use xxd to encode it back to ASCII:
echo $secret | xxd -r -p
Fun Fact 2: You can also print out secrets from a Spark cluster using Databricks notebook with just substring/slicing:
print(some_secret[:-2])
print(some_secret[-1])
Fun Fact 3: The curl -v output will show you the authentication header:
authorization: Basic xyz*******************************************************************************=
This is just a base64 encoded string of the curl -u value of username:token.
I tried making the cURL call locally with VPN connected. It returned a 201 response with no issues. I then disconnected the VPN and ran ipconfig /flushdns and re-ran the same curl command. I noticed that the upload was much slower and seemed to hang around here:
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Length: 280197
>
* We are completely uploaded and fine
This made me suspect that the curl command run on Azure DevOps pipeline was timing out. After adding the following flags --connect-timeout 60 --max-time 60 --retry 3 --retry-all-errors, I was greeted with a successful run of that task.
curl --connect-timeout 60 --max-time 60 --retry 3 --retry-all-errors -u $(user):$(token)-X PUT https://jfrog.artifactory.fake.com/artifactory/group/service/pre-prod/1.1.1/service-1.1.1-py3-none-any.whl -T ./output/service-1.1.1-py3-none-any.whl
We'll be reaching out to the appropriate teams to see if anything can be done to not hinder these uploads outside of the VPN.
Summary
If you see a drop in bytes transferred, you might want to extend the connection time and add retries.