http-client

An HTTP client for Clojure and Babashka built on java.net.http.

API

See API.md.

NOTE: The babashka.http-client library is built-in as of babashka version 1.1.171.

TIP: We test and support babashka.http-client on Clojure v1.10 and above.

Installation

Use as a dependency in deps.edn or bb.edn:

org.babashka/http-client {:mvn/version "0.3.11"}

Rationale

Babashka has several built-in options for making HTTP requests, including:

• babashka.curl
• http-kit
• java.net.http

In addition, it allows to use several libraries to be used as a dependency:

• java-http-clj
• hato
• clj-http-lite

The built-in clients come with their own trade-offs. E.g. babashka.curl shells out to curl which on Windows requires your local curl to be updated. Http-kit buffers the entire response in memory. Using java.net.http directly can be a bit verbose.

Babashka's http-client aims to be a good default for most scripting use cases and is built on top of java.net.http and can be used as a dependency-free JVM library as well. The API is mostly compatible with babashka.curl so it can be used as a drop-in replacement. The other built-in solutions will not be removed any time soon.

Usage

The APIs in this library are mostly compatible with babashka.curl, which is in turn inspired by libraries like clj-http.

(require '[babashka.http-client :as http])
(require '[clojure.java.io :as io]) ;; optional
(require '[cheshire.core :as json]) ;; optional

GET

Simple GET request:

(http/get "https://httpstat.us/200")
;;=> {:status 200, :body "200 OK", :headers { ... }}

Headers

Passing headers:

(def resp (http/get "https://httpstat.us/200" {:headers {"Accept" "application/json"}}))
(json/parse-string (:body resp)) ;;=> {"code" 200, "description" "OK"}

Headers may be provided as keywords as well:

{:headers {:content-type "application/json"}}

Query parameters

Query parameters:

(->
  (http/get "https://postman-echo.com/get" {:query-params {"q" "clojure"}})
  :body
  (json/parse-string true)
  :args)
;;=> {:q "clojure"}

To send multiple params to the same key:

;; https://postman-echo.com/get?q=clojure&q=curl

(-> (http/get "https://postman-echo.com/get" {:query-params {:q ["clojure" "curl"]}})
    :body (json/parse-string true) :args)
;;=> {:q ["clojure" "curl"]}

POST

A POST request with a :body:

(def resp (http/post "https://postman-echo.com/post" {:body "From Clojure"}))
(json/parse-string (:body resp)) ;;=> {"args" {}, "data" "From Clojure", ...}

A POST request with a JSON :body:

(def resp (http/post "https://postman-echo.com/post"
                     {:headers {:content-type "application/json"}
                      :body (json/encode {:a 1 :b "2"})}))
(:data (json/parse-string (:body resp) true)) ;;=> {:a 1, :b "2"}

Posting a file as a POST body:

(:status (http/post "https://postman-echo.com/post" {:body (io/file "README.md")}))
;; => 200

Posting a stream as a POST body:

(:status (http/post "https://postman-echo.com/post" {:body (io/input-stream "README.md")}))
;; => 200

Posting form params:

(:status (http/post "https://postman-echo.com/post" {:form-params {"name" "Michiel"}}))
;; => 200

Basic auth

Basic auth:

(:body (http/get "https://postman-echo.com/basic-auth" {:basic-auth ["postman" "password"]}))
;; => "{\"authenticated\":true}"

Oauth token

Oauth token:

(:body (http/get "https://httpbin.org/bearer" {:oauth-token "qwertyuiop"}))
;; => "{\n  \"authenticated\": true, \n  \"token\": \"qwertyuiop\"\n}\n"

Streaming

With :as :stream:

(:body (http/get "https://github.com/babashka/babashka/raw/master/logo/icon.png"
    {:as :stream}))

will return the raw input stream.

Download binary

Download a binary file:

(io/copy
  (:body (http/get "https://github.com/babashka/babashka/raw/master/logo/icon.png"
    {:as :stream}))
  (io/file "icon.png"))
(.length (io/file "icon.png"))
;;=> 7748

To obtain an in-memory byte array you can use :as :bytes.

URI construction

Using the verbose :uri API for fine grained (and safer) URI construction:

(-> (http/request {:uri {:scheme "https"
                           :host   "httpbin.org"
                           :port   443
                           :path   "/get"
                           :query  "q=test"}})
    :body
    (json/parse-string true))
;;=>
{:args {:q "test"},
 :headers
 {:Accept "*/*",
  :Host "httpbin.org",
  :User-Agent "Java-http-client/11.0.17"
  :X-Amzn-Trace-Id
  "Root=1-5e63989e-7bd5b1dba75e951a84d61b6a"},
 :origin "46.114.35.45",
 :url "https://httpbin.org/get?q=test"}

Custom client

The default client in babashka.http-client is constructed conceptually as follows:

(def client (http/client http/default-client-opts))

To pass more options in addition to the default options, you can use http/default-client-opts and associate more options:

(def client (http/client (assoc-in http/default-client-opts [:ssl-context :insecure] true)))

Then use the custom client with HTTP requests:

(http/get "https://clojure.org" {:client client})

Redirects

The default client is configured to always follow redirects. To opt out of this behaviour, construct a custom client:

(:status (http/get "https://httpstat.us/302" {:client (http/client {:follow-redirects :never})}))
;; => 302
(:status (http/get "https://httpstat.us/302" {:client (http/client {:follow-redirects :always})}))
;; => 200

Exceptions

An ExceptionInfo will be thrown for all HTTP response status codes other than #{200 201 202 203 204 205 206 207 300 301 302 303 304 307}.

user=> (http/get "https://httpstat.us/404")
Execution error (ExceptionInfo) at babashka.http-client.interceptors/fn (interceptors.clj:194).
Exceptional status code: 404

To opt out of an exception being thrown, set :throw to false.

(:status (http/get "https://httpstat.us/404" {:throw false}))
;;=> 404

Multipart

To perform a multipart request, supply :multipart with a sequence of maps with the following options:

• :name: The name of the param
• :part-name: Override for :name
• :content: The part's data. May be string or something that can be fed into clojure.java.io/input-stream
• :file-name: The part's file name. If the :content is a file, the name of the file will be used, unless :file-name is set.
• :content-type: The part's content type. By default, if :content is a string it will be text/plain; charset=UTF-8; if :content is a file it will attempt to guess the best content type or fallback to application/octet-stream.

An example request:

(http/post "https://postman-echo.com/post"
           {:multipart [{:name "title" :content "My Title"}
                        {:name "Content/type" :content "image/jpeg"}
                        {:name "file" :content (io/file "foo.jpg") :file-name "foobar.jpg"}]})

Compression

To accept gzipped or zipped responses, use:

(http/get "https://api.stackexchange.com/2.2/sites"
  {:headers {"Accept-Encoding" ["gzip" "deflate"]}})

The above server only serves compressed responses, so if you remove the header, the request will fail. Accepting compressed responses may become the default in a later version of this library.

Interceptors

Babashka http-client interceptors are similar to Pedestal interceptors. They are maps of :name (a string), :request (a function), :response (a function). An example is shown in this test:

(deftest interceptor-test
  (let [json-interceptor
        {:name ::json
         :description
         "A request with `:as :json` will automatically get the
         \"application/json\" accept header and the response is decoded as JSON."
         :request (fn [request]
                    (if (= :json (:as request))
                      (-> (assoc-in request [:headers :accept] "application/json")
                          ;; Read body as :string
                          ;; Mark request as amenable to json decoding
                          (assoc :as :string ::json true))
                      request))
         :response (fn [response]
                     (if (get-in response [:request ::json])
                       (update response :body #(json/parse-string % true))
                       response))}
        ;; Add json interceptor add beginning of chain
        ;; It will be the first to see the request and the last to see the response
        interceptors (cons json-interceptor interceptors/default-interceptors)
        ]
    (testing "interceptors on request"
      (let [resp (http/get "https://httpstat.us/200"
                             {:interceptors interceptors
                              :as :json})]
        (is (= 200 (-> resp :body
                       ;; response as JSON
                       :code)))))))

A :request function is executed when the request is built and the :response function is executed on the response. Default interceptors are in babashka.http-client.interceptors/default-interceptors. Interceptors can be configured on the level of requests by passing a modified :interceptors chain.

Changing an existing interceptor

In this example we change the throw-on-exceptional-status-code interceptor to not throw on a 404 status code:

(require '[babashka.http-client :as http]
         '[babashka.http-client.interceptors :as i])

(def unexceptional-statuses
  (conj #{200 201 202 203 204 205 206 207 300 301 302 303 304 307}
        ;; we also don't throw on 404
        404))

(def my-throw-on-exceptional-status-code
  "Response: throw on exceptional status codes"
  {:name ::throw-on-exceptional-status-code
   :response (fn [resp]
               (if-let [status (:status resp)]
                 (if (or (false? (some-> resp :request :throw))
                         (contains? unexceptional-statuses status))
                   resp
                   (throw (ex-info (str "Exceptional status code: " status) resp)))
                 resp))})

(def my-interceptors
  (mapv (fn [i]
         (if (= ::i/throw-on-exceptional-status-code
                (:name i))
           my-throw-on-exceptional-status-code
           i))
        i/default-interceptors))

(def my-response
  (http/get "https://postman-echo.com/get/404" {:interceptors my-interceptors}))

(prn (:status my-response)) ;; 404

Testing interceptors

For testing interceptors it can be useful to use the :client option in combination with a Clojure function. When passing a function, the request won't be converted to a java.net.http.Request but just passed as a ring request to the function. The function is expected to return a ring response:

(http/get "https://clojure.org" {:client (fn [req] {:body 200})})

Async

To execute request asynchronously, use :async true. The response will be a CompletableFuture with the response map.

(-> (http/get "https://clojure.org" {:async true}) deref :status)
;;=> 200

Timeouts

Two different timeouts can be set:

• The connection timeout, :connect-timeout, in http/client
• The request :timeout in http/request

Alternatively you can use :async + deref with a timeout + default value:

(let [resp (http/get "https://httpstat.us/200?sleep=5000" {:async true})] (deref resp 1000 ::too-late))
;;=> :user/too-late

Logging and debug

If you need to debug HTTP requests you need to add a JVM system property with some debug options: "-Djdk.httpclient.HttpClient.log=errors,requests,headers,frames[:control:data:window:all..],content,ssl,trace,channel"

One way to handle that with tools-deps is to add an alias with :jvm-opts option.

Here is a code snippet for deps.edn

{
;; REDACTED
:aliases {
 :debug
 {:jvm-opts
  [;; enable logging for java.net.http
  "-Djdk.httpclient.HttpClient.log=errors,requests,headers,frames[:control:data:window:all..],content,ssl,trace,channel"]}
}}

Test

$ bb test:clj
$ bb test:bb

Credits

This library has borrowed liberally from java-http-clj and hato, both available under the MIT license.

License

Copyright © 2022 - 2023 Michiel Borkent

Distributed under the MIT License. See LICENSE.