7 February 2019

debugging http/https

by mo


This week I’ve been working on integrating with a service from another team. I’ve learned a few new things during this integration so I thought I would list some of those things here.

In the example code below the client is sending a request to the server with an absolute URI.

require 'json'
require 'logger'
require 'net/http'
require 'uri'

headers = { "Accept" => 'application/json', "Content-type" => 'application/json' }

uri = URI.parse("https://www.example.org/trade/services/RPSScreening")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.set_debug_output(Logger.new(STDOUT))

post = Net::HTTP::Post.new(uri.to_s, headers)
post.basic_auth('username', 'password')
post.body = JSON.generate({})
puts http.request(post).inspect

The raw logger output is as follows.

モ ruby _includes/rps-absolute.rb
opening connection to www.example.org:443...
opened
starting SSL for www.example.org:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256
<- "POST https://www.example.org/trade/services/RPSScreening HTTP/1.1\r\nAccept: application/json\r\nContent-Type: application/json\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nUser-Agent: Ruby\r\nAuthorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=\r\nConnection: close\r\nHost: www.example.org\r\nContent-Length: 2\r\n\r\n"
<- "{}"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Cache-Control: max-age=604800\r\n"
-> "Content-Type: text/html; charset=UTF-8\r\n"
-> "Date: Thu, 07 Feb 2019 22:49:49 GMT\r\n"
-> "Expires: Thu, 14 Feb 2019 22:49:49 GMT\r\n"
-> "Server: EOS (vny006/0452)\r\n"
-> "Content-Length: 61\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 61 bytes...
-> ""
-> "{\"success\": false, \"errors\": [{ \"message\": \"\", \"code\": 404}]}"
read 61 bytes
Conn close
#<Net::HTTPNotFound 404 Not Found readbody=true>

The HTTP request looks like this:

POST https://www.example.org/trade/services/RPSScreening HTTP/1.1
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
User-Agent: Ruby
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Connection: close
Host: www.example.org
Content-Length: 2

{}

The HTTP Response looks like this:

HTTP/1.1 404 Not Found
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Thu, 07 Feb 2019 22:49:49 GMT
Expires: Thu, 14 Feb 2019 22:49:49 GMT
Server: EOS (vny006/0452)
Content-Length: 61
Connection: close


{"success": false, "errors": [{ "message": "", "code": 404}]}

In RFC-2616 there is a section describing the different ways a request URI can be formatted.

   The absoluteURI form is REQUIRED when the request is being made to a
   proxy. The proxy is requested to forward the request or service it
   from a valid cache, and return the response. Note that the proxy MAY
   forward the request on to another proxy or directly to the server
   specified by the absoluteURI. In order to avoid request loops, a
   proxy MUST be able to recognize all of its server names, including
   any aliases, local variations, and the numeric IP address. An example
   Request-Line would be:

       GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1

   To allow for transition to absoluteURIs in all requests in future
   versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI
   form in requests, even though HTTP/1.1 clients will only generate
   them in requests to proxies.

   The authority form is only used by the CONNECT method (section 9.9).

   The most common form of Request-URI is that used to identify a
   resource on an origin server or gateway. In this case the absolute
   path of the URI MUST be transmitted (see section 3.2.1, abs_path) as
   the Request-URI, and the network location of the URI (authority) MUST
   be transmitted in a Host header field. For example, a client wishing
   to retrieve the resource above directly from the origin server would
   create a TCP connection to port 80 of the host "www.w3.org" and send
   the lines:

       GET /pub/WWW/TheProject.html HTTP/1.1
       Host: www.w3.org

   followed by the remainder of the Request. Note that the absolute path
   cannot be empty; if none is present in the original URI, it MUST be
   given as "/" (the server root).

Although, it is recommended that all server support the absolute URI for future versions of HTTP it is still possible to run into older services out in the wild that do not handle the absolute URI.

To conver to the relative form of the URI we will change the following line:

# post = Net::HTTP::Post.new(uri.to_s, headers)
post = Net::HTTP::Post.new(uri.path, headers)

This time the request to the server looks like:

POST /trade/services/RPSScreening HTTP/1.1
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
User-Agent: Ruby
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Connection: close
Host: www.example.org
Content-Length: 2

{}

This particular server is able to process either the absolute or relative URI. So the response is:

HTTP/1.1 404 Not Found
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Thu, 07 Feb 2019 23:03:28 GMT
Expires: Thu, 14 Feb 2019 23:03:28 GMT
Server: EOS (vny006/044F)
Content-Length: 61
Connection: close

{"success": false, "errors": [{ "message": "", "code": 404}]}

Beware that not all servers can respond to both forms.

The next issue started with this log line.

opening connection to wsgx-test.example.org:443... 
opened 
starting SSL for wsgx-test.example.org:443... 
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1) 
#<OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)> 

So it looked lik a peer verification error. I wanted to see the certificate that was being served from the server to make sure the hostname matched.

$ echo | openssl s_client -showcerts -connect wsgx-test.example.org:443 < /dev/null | openssl x509 -text
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = Thawte RSA CA 2018
verify return:1
depth=0 C = US, ST = California, L = San Jose, O = "Disco Systems, Inc.", OU = PMA, CN = *.example.com
verify return:1
DONE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            09:87:5e:62:58:8d:55:89:18:18:fb:48:1f:d7:85:51
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=Thawte RSA CA 2018
        Validity
            Not Before: Jan 11 00:00:00 2018 GMT
            Not After : Feb 16 12:00:00 2019 GMT
        Subject: C=US, ST=California, L=San Jose, O=Disco Systems, Inc., OU=PMA, CN=*.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    <SNIP>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:A3:C8:5E:65:54:E5:30:78:C1:05:EA:07:0A:6A:59:CC:B9:FE:DE:5A
            X509v3 Subject Key Identifier:
                AA:6B:55:C1:95:62:F2:2E:26:CF:91:44:27:A5:16:1D:71:76:50:D9
            X509v3 Subject Alternative Name:
                DNS:*.example.com, DNS:example.com
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points:                Full Name:
                  URI:http://cdp.thawte.com/ThawteRSACA2018.crl            X509v3 Certificate Policies:
                Policy: 2.16.840.1.114412.1.1
                  CPS: https://www.digicert.com/CPS
                Policy: 2.23.140.1.2.2            Authority Information Access:
                OCSP - URI:http://status.thawte.com
                CA Issuers - URI:http://cacerts.thawte.com/ThawteRSACA2018.crt            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         <SNIP>
-----BEGIN CERTIFICATE-----
<SNIP>
-----END CERTIFICATE-----

Using openssl I can dump the x509 cert and extensions to see that the CN that the cert was issued for was not the target domain I was trying to connect to. In this case it was one of our own servers and not one of the destination teams services. I wondered if perhaps DNS was set to respond with an internal IP address instead of an external IP and just happened to collide with an internal IP in our VPC.

モ dig wsgx-test.example.com

; <<>> DiG 9.10.6 <<>> wsgx-test.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36523
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 7

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;wsgx-test.example.com.           IN      A

;; ANSWER SECTION:
wsgx-test.example.com.    300     IN      A       173.38.125.152

;; AUTHORITY SECTION:
wsgx-test.example.com.    1800    IN      NS      sngdc01-ab07-dcz01n-gss1.example.com.
wsgx-test.example.com.    1800    IN      NS      mtv5-ap10-dcz06n-gss1.example.com.
wsgx-test.example.com.    1800    IN      NS      alln01-ag09-dcz03n-gss1.example.com.
wsgx-test.example.com.    1800    IN      NS      aer01-r4c25-dcz01n-gss1.example.com.
wsgx-test.example.com.    1800    IN      NS      rcdn9-14p-dcz05n-gss1.example.com.
wsgx-test.example.com.    1800    IN      NS      rtp5-dmz-gss1.example.com.

;; ADDITIONAL SECTION:
rtp5-dmz-gss1.example.com. 1800   IN      A       64.102.246.5
mtv5-ap10-dcz06n-gss1.example.com. 1800 IN A      173.36.224.100
rcdn9-14p-dcz05n-gss1.example.com. 1800 IN A      72.163.4.28
aer01-r4c25-dcz01n-gss1.example.com. 1800 IN A    173.38.212.108
alln01-ag09-dcz03n-gss1.example.com. 1800 IN A    173.37.144.100
sngdc01-ab07-dcz01n-gss1.example.com. 1800 IN A   173.39.112.68

;; Query time: 64 msec
;; SERVER: 171.70.168.183#53(171.70.168.183)
;; WHEN: Thu Feb 07 16:20:54 MST 2019
;; MSG SIZE  rcvd: 375

This output shows a few things. The AUTHORITY SECTION identifies the nameservers that are authorized to respond with an IP for this hostname.

In the end I was totally wrong and if I have started with a curl request and looked at the IP address the request was actually going to then I probably could have skipped dig.

$ curl https://wsgx-test.example.com -vvv
* About to connect() to wsgx-test.example.com port 443 (#0)
*   Trying 0.0.0.0...
* Connected to wsgx-test.example.com (0.0.0.0) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=*.example.com,OU=PMA,O="Disco Systems, Inc.",L=San Jose,ST=California,C=US
*       start date: Jan 11 00:00:00 2018 GMT
*       expire date: Feb 16 12:00:00 2019 GMT
*       common name: *.example.com
*       issuer: CN=Thawte RSA CA 2018,OU=www.digicert.com,O=DigiCert Inc,C=US
* NSS error -12276 (SSL_ERROR_BAD_CERT_DOMAIN)
* Unable to communicate securely with peer: requested domain name does not match the server's certificate.
* Closing connection 0
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.

All thanks to this little gem.

$ tail -n1 /etc/hosts
0.0.0.0 wsgx-test.example.com

Happy hacking!

devops