Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kong 0.6.0 RC2 - Issues #843

Closed
samek opened this issue Jan 7, 2016 · 33 comments
Closed

kong 0.6.0 RC2 - Issues #843

samek opened this issue Jan 7, 2016 · 33 comments

Comments

@samek
Copy link

samek commented Jan 7, 2016

  • While using the 0.6.0rc1 - when I add a new api It's not working until I restart kong. Same goes for changes/deletion of apis.
curl -XPUT http://10.0.0.48:8001/apis/ -H "Content-Type: application/json" --data '{"name":"testapi","request_path":"/test/","upstream_url":"http://google.com","strip_request_path":true}'
{"upstream_url":"http:\/\/google.com","request_path":"\/test","id":"ff75b06a-3c2c-4889-a3ab-19c3f829ccb8","created_at":1452172005000,"name":"testapi","strip_request_path":true}

list apis

curl http://10.0.0.48:8001/apis/
{"data":[{"upstream_url":"http:\/\/google.com","request_path":"\/test","id":"ff75b06a-3c2c-4889-a3ab-19c3f829ccb8","strip_request_path":true,"name":"testapi","created_at":1452172005000}],"total":1}

call new created api

curl http://10.0.0.48:8000/test/
{"request_path":"\/test\/","message":"API not found with these values","request_host":["10.0.0.48"]}

after restart:

curl http://10.0.0.48:8000/test/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.si/?gws_rd=cr&amp;ei=V2OOVunXHuPOygOk-YqYDQ">here</A>.
</BODY></HTML>

  • When using cassandra multinode cluster when migrations are run it does't wait until the changes are propagated. I needed to re-run it for almost all tables.
[INFO] Kong 0.6.0rc1
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74,10.0.0.200,10.0.0.194 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong60
[INFO] Database not initialized. Running migrations...
[INFO] Migrating core on keyspace "kong60" (cassandra)
[INFO] core migrated up to: 2015-01-12-175310_skeleton
[INFO] Leaving cluster..
[ERR] Error executing migration for "core": Cassandra error: ResponseError: [Invalid] unconfigured columnfamily consumers
[ERR] Could not start Kong

  • Cluster serf doesn't get started ??
kong cluster members
[INFO] Using configuration: /etc/kong/kong.yml
[ERR] serf is not running

kong status
[INFO] Using configuration: /etc/kong/kong.yml
[ERR] Some services required by Kong are not running. Please execute "kong restart"!

My config:

databases_available:
  cassandra:
    contact_points:
        - "10.0.0.74"
        - "10.0.0.200"
        - "10.0.0.194"

    keyspace: kong60
    replication_strategy: SimpleStrategy
    replication_factor: 3
cluster:

  ######
  ## Instructs this node to auto-join the nodes that are sharing the same datastore on startup.
 auto-join: true

  ######
  ## The advertise flag is used to change the address that we advertise to other nodes in the
  ## cluster. By default, the bind address is advertised. However, in some cases (specifically 
  ## NAT traversal), there may be a routable address that cannot be bound to. This flag enables 
  ## gossiping a different address to support this
 advertise: "10.0.0.48"

  ######
  ## Key for encrypting network traffic within Kong. Must be a base64-encoded 16-byte key.
 encrypt: "736RR8FlRywwPKAwGFZTmQ=="
...
@subnetmarco
Copy link
Member

@samek what system are you running?

@thibaultcha
Copy link
Member

(Thanks a lot for giving the RC a try btw)

@subnetmarco
Copy link
Member

@samek I tried to replicate the same commands in a brand new CentOS 6 machine, and it seems to work. If you pulled Kong from Docker, I am suspecting that the container for 0.6.0rc1 is broken.

It may also be something else, but before I keep testing I would like to understand if you are indeed using the Docker container.

@samek
Copy link
Author

samek commented Jan 8, 2016

I took Dockerfile from https://github.com/Mashape/docker-kong.git
I've changed the kong.yml and build my docker.
Anyway I've tried now with published docker and my config and it's the same.

docker run -ti  -p 8000:8000 -p 8001:8001 -v /tmp/docker-kong/kong.yml:/etc/kong/kong.yml  mashape/kong:0.6.0rc1 

My Cassandra version is 2.1.4.

I've also tried with --net=host (just guessing maybe it's all connected to serf), same.

my whole config:

databases_available:
  cassandra:
    contact_points:
        - "10.0.0.74"
        - "10.0.0.200"
        - "10.0.0.194"
    keyspace: kong60
    replication_strategy: SimpleStrategy
    replication_factor: 3
cluster:

  ######
  ## Instructs this node to auto-join the nodes that are sharing the same datastore on startup.
 auto-join: true

  ######
  ## The advertise flag is used to change the address that we advertise to other nodes in the
  ## cluster. By default, the bind address is advertised. However, in some cases (specifically 
  ## NAT traversal), there may be a routable address that cannot be bound to. This flag enables 
  ## gossiping a different address to support this
 advertise: "10.0.0.48"

  ######
  ## Key for encrypting network traffic within Kong. Must be a base64-encoded 16-byte key.
 encrypt: "736RR8FlRywwPKAwGFZTmQ=="
nginx: |
  worker_processes auto;
  error_log logs/error.log error;
  daemon off;

  worker_rlimit_nofile {{auto_worker_rlimit_nofile}};

  env KONG_CONF;
  env PATH;

  events {
    worker_connections {{auto_worker_connections}};
    multi_accept on;
  }

  http {
    resolver {{dns_resolver}} ipv6=off;
    charset UTF-8;

    access_log logs/access.log;
    access_log off;

    # Timeouts
    keepalive_timeout 60s;
    client_header_timeout 60s;
    client_body_timeout 60s;
    send_timeout 60s;

    # Proxy Settings
    proxy_buffer_size 128k;
    proxy_buffers 4 256k;
    proxy_busy_buffers_size 256k;
    proxy_ssl_server_name on;

    # IP Address
    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;
    real_ip_recursive on;

    # Other Settings
    client_max_body_size 0;
    underscores_in_headers on;
    reset_timedout_connection on;
    tcp_nopush on;

    ################################################
    #  The following code is required to run Kong  #
    # Please be careful if you'd like to change it #
    ################################################

    # Lua Settings
    lua_package_path ';;';
    lua_code_cache on;
    lua_max_running_timers 4096;
    lua_max_pending_timers 16384;
    lua_shared_dict reports_locks 100k;
    lua_shared_dict cluster_locks 100k;
    lua_shared_dict cache {{memory_cache_size}}m;
    lua_shared_dict cassandra 1m;
    lua_shared_dict cassandra_prepared 5m;
    lua_socket_log_errors off;
    {{lua_ssl_trusted_certificate}}

    init_by_lua '
      kong = require "kong"
      local status, err = pcall(kong.init)
      if not status then
        ngx.log(ngx.ERR, "Startup error: "..err)
        os.exit(1)
      end
    ';

    init_worker_by_lua 'kong.exec_plugins_init_worker()';

    server {
      server_name _;
      listen {{proxy_port}};
      listen {{proxy_ssl_port}} ssl;

      ssl_certificate_by_lua 'kong.exec_plugins_certificate()';

      ssl_certificate {{ssl_cert}};
      ssl_certificate_key {{ssl_key}};
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;# omit SSLv3 because of POODLE (CVE-2014-3566)

      location / {
        default_type 'text/plain';

        # These properties will be used later by proxy_pass
        set $upstream_host nil;
        set $upstream_url nil;

        # Authenticate the user and load the API info
        access_by_lua 'kong.exec_plugins_access()';

        # Proxy the request
        # Proxy the request
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $upstream_host;
        proxy_pass $upstream_url;
        proxy_pass_header Server;

        # Add additional response headers
        header_filter_by_lua 'kong.exec_plugins_header_filter()';

        # Change the response body
        body_filter_by_lua 'kong.exec_plugins_body_filter()';

        # Log the request
        log_by_lua 'kong.exec_plugins_log()';
      }

      location /robots.txt {
        return 200 'User-agent: *\nDisallow: /';
      }

      error_page 500 /500.html;
      location = /500.html {
        internal;
        content_by_lua '
          local responses = require "kong.tools.responses"
          responses.send_HTTP_INTERNAL_SERVER_ERROR("An unexpected error occurred")
        ';
      }
    }

    server {
      listen {{admin_api_port}};

      client_max_body_size 10m;
      client_body_buffer_size 10m;

      location / {
        default_type application/json;
        content_by_lua '
          ngx.header["Access-Control-Allow-Origin"] = "*"
          if ngx.req.get_method() == "OPTIONS" then
            ngx.header["Access-Control-Allow-Methods"] = "GET,HEAD,PUT,PATCH,POST,DELETE"
            ngx.header["Access-Control-Allow-Headers"] = "Content-Type"
            ngx.exit(204)
          end
          local lapis = require "lapis"
          lapis.serve("kong.api.app")
        ';
      }

      location /nginx_status {
        internal;
        access_log off;
        stub_status;
      }

      location /robots.txt {
        return 200 'User-agent: *\nDisallow: /';
      }
    }
  }

@thibaultcha
Copy link
Member

Quick question: are all your Cassandra nodes available from your containers (not only the contact points but all of them)?

@samek
Copy link
Author

samek commented Jan 9, 2016

Actually I’ve entered all nodes in.

I have a 3 node cluster. And yes they are all visible .

On Jan 9, 2016, at 4:59 AM, Thibault Charbonnier [email protected] wrote:

Quick question: are all your Cassandra nodes available from your containers (not only the contact points but all of them)?


Reply to this email directly or view it on GitHub #843 (comment).

@subnetmarco
Copy link
Member

@samek - can you try again with mashape/kong:0.6.0rc2 (just released) ?

I also fixed Docker.

@samek
Copy link
Author

samek commented Jan 14, 2016

Sure, first thing when i get in the offi

@samek samek closed this as completed Jan 14, 2016
@samek samek reopened this Jan 14, 2016
@samek
Copy link
Author

samek commented Jan 14, 2016

Sure. I need couple of hours to get to the office.

@samek
Copy link
Author

samek commented Jan 14, 2016

OK.
I've used rc2 docker. All I've changes in config:

database: cassandra
cassandra:
  contact_points:
    - "10.0.0.74:9042"
  keyspace: kong60
  replication_strategy: SimpleStrategy
  replication_factor: 3

If the keyspace doesn't exists it still does not create one (using cluster) probably since it tries to use it on a node not propagated yet - so for the first time you need to start it several times until it inserts all the tables. took me 6 times.

eg:

[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong60
[INFO] Database not initialized. Running migrations...
[INFO] Migrating core on keyspace "kong60" (cassandra)
[INFO] core migrated up to: 2015-01-12-175310_skeleton
[INFO] Leaving cluster..
[ERR] Error executing migration for "core": Cassandra error: ResponseError: [Invalid] unconfigured columnfamily consumers
[ERR] Could not start Kong

Then when it starts there are ngx.timer error in cluster.ua

[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong60
[INFO] Migrating basic-auth on keyspace "kong60" (cassandra)
[INFO] basic-auth migrated up to: 2015-08-03-132400_init_basicauth
[INFO] Migrating response-ratelimiting on keyspace "kong60" (cassandra)
[INFO] response-ratelimiting migrated up to: 2015-08-21_init_response-rate-limiting
[INFO] Migrating hmac-auth on keyspace "kong60" (cassandra)
[INFO] hmac-auth migrated up to: 2015-09-16-132400_init_hmacauth
[INFO] dnsmasq............address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] Auto-generating the default SSL certificate and key...
[INFO] nginx .............admin_api_listen=0.0.0.0:8001 proxy_listen=0.0.0.0:8000 proxy_listen_ssl=0.0.0.0:8443
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -node=f89ef57cfa0c_0.0.0.0:7946 -log-level=err
[INFO] Trying to auto-join Kong nodes, please wait..
[WARN] Cannot auto-join the cluster because no nodes were found
[OK] Started
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Refreshing local and peers info
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Local info retrieved
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Adding host 10.0.0.200
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Adding host 10.0.0.194
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Peers info retrieved
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Cluster infos retrieved in shared dict cassandra
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Closing connection to 10.0.0.74:9042.
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Query not prepared in cluster yet. Preparing.
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Query prepared for host 10.0.0.200
2016/01/14 09:59:45 [notice] 58#0: [lua] log.lua:43: print(): INFO -- Closing connection to 10.0.0.200.
2016/01/14 09:59:48 [error] 74#0: lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/core/cluster.lua:57: attempt to compare nil with number
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/core/cluster.lua: in function </usr/local/share/lua/5.1/kong/core/cluster.lua:23>, context: ngx.timer
2016/01/14 09:59:48 [error] 79#0: lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/core/cluster.lua:57: attempt to compare nil with number
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/core/cluster.lua: in function </usr/local/share/lua/5.1/kong/core/cluster.lua:23>, context: ngx.timer
2016/01/14 09:59:48 [error] 69#0: lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/core/cluster.lua:57: attempt to compare nil with number

It registers in cassandra (I can see it in the nodes table)

But it seams that something with the dnsmasq is again different.

when I've inserted some of our apps in
Api's are inserted and kong knows about them (not like in rc1 when I had to restart the kong).
Problem is that I get bad gateway

502 Bad Gateway

openresty/1.9.3.1

Error log in kong:

2016/01/14 10:49:16 [error] 103#0: *477 haproxy.marathon.mesos could not be resolved (3: Host not found), client: 10.0.0.50, server: _, request: "GET /v0/news/ HTTP/1.1", host: "10.0.0.48:8000"

If I go inside of the docker and ping haproxy.marathon.mesos it works.

ping haproxy.marathon.mesos
PING haproxy.marathon.mesos (10.0.0.208) 56(84) bytes of data.
64 bytes from 10.0.0.208: icmp_seq=1 ttl=63 time=0.515 ms
64 bytes from 10.0.0.208: icmp_seq=2 ttl=63 time=0.263 ms

This did work in rc1.

@subnetmarco
Copy link
Member

@samek thanks for your test. I fixed the timer problem, although in your scenario is irrelevant (it's for async cluster auto-joins).

I will leave the Cassandra problems to @thibaultcha - The dnsmasq is an interesting one, since we didn't change anything between RC1 and RC2. Uhm, it could be Docker specific, and not about Kong. Let me investigate.

@subnetmarco subnetmarco changed the title kong 0.6.0rc1 - Issues kong 0.6.0 RC2 - Issues Jan 14, 2016
@thibaultcha
Copy link
Member

@samek May I ask which Cassandra version you are using?

@samek
Copy link
Author

samek commented Jan 14, 2016

@thibaultcha apache-cassandra-2.1.4 (on mesos as a framework)
@thefosk As for dnsmasq I've tried again with rc1 same kong keyspace and it resolves it just fine while rc2 doesnt.

@subnetmarco
Copy link
Member

So after a better look, RC2 explicitly binds dnsmasq to 127.0.0.1 using the --listen-address=127.0.0.1 argument.

This should not cause any problems since dnsmasq is being used only locally by the local nginx server - but that's the only change I found.

Anyways, I have created a new Docker distribution mashape/kong:samek which fixes the cluster problem and also reverts the dnsmasq change, could you try it? @samek

@thibaultcha
Copy link
Member

I was unable to reproduce any failure on a 2.1.4 cluster of four nodes running locally. In fact all the tests regarding the new driver have always been done on a multiple nodes setup since #660.

probably since it tries to use it on a node not propagated yet

Actually one of the improvements of the new driver compared to the old one is that it does wait for a schema consensus between all nodes on each DDL query, such as keyspace or table creation. The driver waits before returning a result from such a query until all nodes have the same schema version. If the waiting time exceeds 5 seconds (default value), then the waiting period times-out and the driver returns. This timeout is configurable.

Could it be that this timeout is too short for your use case? If you want to give this possibility a try, this option is currently not configurable from Kong. You could test this possibility by applying the following patch to Kong (with a new value of 15 seconds):

diff --git a/kong/dao/cassandra/factory.lua b/kong/dao/cassandra/factory.lua
index 783dbbd..a61dd56 100644
--- a/kong/dao/cassandra/factory.lua
+++ b/kong/dao/cassandra/factory.lua
@@ -119,6 +119,9 @@ function CassandraFactory:get_session_options()
     query_options = {
       prepare = true
     },
+    protocol_options = {
+      max_schema_consensus_wait = 15000
+    },
     username = self.properties.username,
     password = self.properties.password,
     ssl_options = {

Or maybe you prefer a Docker image with this patch?

@subnetmarco
Copy link
Member

If you give me 20/30 mins I can recreate the Docker image including the Cassandra patch.

@samek
Copy link
Author

samek commented Jan 14, 2016

@thefosk I did and It doesn't resolve :(

both rc1 and samek tag dnsmasq bind to 127.0.0.1 but rc1 resolves it rc2/samek doesnt.

rc1

[INFO] Kong 0.6.0rc1
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74,10.0.0.200,10.0.0.194 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong60
[INFO] dnsmasq ...........address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] Auto-generating the default SSL certificate and key...
^C[INFO] nginx .............admin_api_port=8001 proxy_port=8000 proxy_ssl_port=8443
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -encrypt=736RR8FlRywwPKAwGFZTmQ== -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -advertise=10.0.0.48 -node=1c157b16c6aa_7946 -log-level=err
[INFO] Auto-joining cluster, please wait..
[INFO] Successfully auto-joined 172.17.0.28:7946
[OK] Started

samek (rc2)

mashape/kong:samek
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong60
[INFO] dnsmasq............address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] Auto-generating the default SSL certificate and key...
[INFO] nginx .............admin_api_listen=0.0.0.0:8001 proxy_listen=0.0.0.0:8000 proxy_listen_ssl=0.0.0.0:8443
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -node=5b6e28161c7d_0.0.0.0:7946 -log-level=err
[INFO] Trying to auto-join Kong nodes, please wait..
[INFO] Successfully auto-joined 172.17.0.28:7946
[OK] Started
    /usr/local/share/lua/5.1/kong/core/cluster.lua: in function </usr/local/share/lua/5.1/kong/core/cluster.lua:23>, context: ngx.timer
2016/01/14 22:39:19 [error] 104#0: lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/core/cluster.lua:34: attempt to index local 'lock' (a nil value)
stack traceback:
coroutine 0:
....
....
2016/01/14 22:40:54 [error] 104#0: *15 haproxy.marathon.mesos could not be resolved (3: Host not found), client: 10.0.0.50, server: _, request: "GET /v0/news/oldFront/get/1 HTTP/1.1", host: "10.0.0.48:8000"

I'll dig a bit and try to find out more why doesn't it resolve and let you know.

@samek
Copy link
Author

samek commented Jan 14, 2016

@thibaultcha If the default timeout is 5 seconds than it has to be somethig else.

I did patch the factory.lua but it exits most of the time before 15 seconds are passed (or 5 which is default)

time kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong62
[INFO] Migrating oauth2 on keyspace "kong62" (cassandra)
[INFO] Leaving cluster..
[ERR] Error executing migration for "oauth2": Cassandra error: ResponseError: [Invalid] unconfigured columnfamily oauth2_tokens
[ERR] Could not start Kong

real    0m2.842s
user    0m0.073s
sys 0m0.119s

time kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong62
[INFO] Migrating oauth2 on keyspace "kong62" (cassandra)
[INFO] oauth2 migrated up to: 2015-08-03-132400_init_oauth2
[INFO] oauth2 migrated up to: 2015-08-24-215800_cascade_delete_index
[INFO] Migrating jwt on keyspace "kong62" (cassandra)
[INFO] Leaving cluster..
[ERR] Error executing migration for "jwt": Cassandra error: ResponseError: [Invalid] unconfigured columnfamily jwt_secrets
[ERR] Could not start Kong

real    0m4.096s
user    0m0.083s
sys 0m0.133s

@samek
Copy link
Author

samek commented Jan 14, 2016

@thefosk
What I did was started kong, killed dnsmasq and started it with -q (log queries) and -d (no daemon).
Looks like dnsmasq is resolving it ..

dnsmasq -p 8053 --pid-file=/usr/local/kong/dnsmasq.pid -N -o -d -q
dnsmasq: started, version 2.66 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 10.0.0.235#53
dnsmasq: using nameserver 10.0.0.137#53
dnsmasq: using nameserver 10.0.0.136#53
dnsmasq: read /etc/hosts - 7 addresses
dnsmasq: query[A] haproxy.marathon.mesos from 127.0.0.1
dnsmasq: forwarded haproxy.marathon.mesos to 10.0.0.136
dnsmasq: query[AAAA] haproxy.marathon.mesos from 127.0.0.1
dnsmasq: forwarded haproxy.marathon.mesos to 10.0.0.136
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.128
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.191
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.135
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.192
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.74
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.129
dnsmasq: reply haproxy.marathon.mesos is 10.0.0.86

errorlog

2016/01/14 23:15:01 [error] 1610#0: *41 haproxy.marathon.mesos could not be resolved (3: Host not found), client: 10.0.0.50, server: _, request: "GET /v0/news/oldFront/get/1 HTTP/1.1", host: "10.0.0.48:8000"

Now I also tested a "real" domain upstream and that one works.
Curl to the same upstream (haproxy.marathon.mesos) works, also ping works.

@thibaultcha
Copy link
Member

@samek Just to be sure, are you deleting your keyspace in between each try? Since I don't see the core migrations being executed first in your output?

If you mind, it would really help if you could execute the following steps:

  • Drop your test keyspace (from cqlsh or kong migrations reset -c kong.yml.
  • Start Kong (expecting migrations will fail again, the output of this would be useful)
  • cqlsh into the cluster and run:
    • DESCRIBE keyspaces; (I don't expect to see the result of this, simply checking if the keyspace is cirrectly created)
    • USE <keyspace>;
    • DESCRIBE tables; (The output of this would be useful)
    • SELECT * FROM schema_migrations; (The result would be really useful too)

Are you willing to try this? I do not understand why your migrations are failing tbh.

@samek
Copy link
Author

samek commented Jan 14, 2016

@thibaultcha sure..

I just changed to a new keyspace in config which was not created before (kong65)
kong start

root@5ea7e2c907cb share]# kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false contact_points=10.0.0.74:9042 data_centers= replication_strategy=SimpleStrategy replication_factor=3 keyspace=kong65
[INFO] Database not initialized. Running migrations...
[INFO] Migrating core on keyspace "kong65" (cassandra)
[INFO] core migrated up to: 2015-01-12-175310_skeleton
[INFO] Leaving cluster..
[ERR] Error executing migration for "core": Cassandra error: ResponseError: [Invalid] unconfigured columnfamily consumers
[ERR] Could not start Kong

Describe keyspaces

describe keyspaces;

kong53  system  "OpsCenter"  kong  kong62  kong61  kong60  kong65  system_traces

describe tables

cqlsh> use kong65
   ... ;
cqlsh:kong65> DESCRIBE tables;

schema_migrations  consumers

cqlsh:kong65> 

schema migrations

cqlsh:kong65> SELECT * FROM schema_migrations;

 id   | migrations
------+--------------------------------
 core | ['2015-01-12-175310_skeleton']

(1 rows)
cqlsh:kong65> 

@subnetmarco
Copy link
Member

@samek so basically everything works, dnsmasq is properly being queried and is resolving the address, yet nginx is still unable to resolve the host?

@thibaultcha
Copy link
Member

I just setup a 3 nodes Cassandra cluster running in Docker containers and used the Mashape/kong:samek Dockerfile and yet I was still unable to reproduce the issue. All migrations run smoothly and propagate to all 3 nodes.

$ docker run -d -p 9040:9042 --name cass1 poklet/cassandra start
$ docker run -d -p 9041:9042 --name cass2 --link cass1:seed poklet/cassandra start seed
$ docker run -d -p 9042:9042 --name cass3 --link cass1:seed poklet/cassandra start seed
$ docker run -it --rm --net container:cass1 poklet/cassandra nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.3  111.49 KB  256          ?       ecd2ec6a-7ac1-4704-b4c0-b4cfac03a9f1  rack1
UN  172.17.0.2  111.52 KB  256          ?       bcc512ae-177e-4f60-bb7a-e3e98bdec466  rack1
UN  172.17.0.4  111.47 KB  256          ?       8f726455-e296-46ec-b0f3-e11c617e1385  rack1

$ docker run -d -p 8000:8000 -p 8001:8001 -v /path/to/kong/config/:/etc/kong --name kong --link cass1:cass1 --link cass2:cass2 --link cass3:cass3 mashape/kong:samek

Any major difference with such a setup in yours?

@samek
Copy link
Author

samek commented Jan 15, 2016

@thibaultcha Differences:

  • Mine is on 3 separate machines
  • Not in the docker
  • Setup of nodes is done by the mesos framework.

nodetool

./nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns    Host ID                               Rack
UN  10.0.0.194  1.5 MB     256     ?       f4b52afe-2c3b-469f-9a10-42dd12be91f6  RAC1
UN  10.0.0.74   773.98 KB  256     ?       4ed31a7c-95ff-4bc2-9a4e-9c83ea3c1595  RAC1
UN  10.0.0.200  1.23 MB    256     ?       8b7061dc-e539-4eac-9fb5-00f8e88f6613  RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

./nodetool status kong65
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.0.0.194  1.5 MB     256     100.0%            f4b52afe-2c3b-469f-9a10-42dd12be91f6  RAC1
UN  10.0.0.74   774.58 KB  256     100.0%            4ed31a7c-95ff-4bc2-9a4e-9c83ea3c1595  RAC1
UN  10.0.0.200  1.23 MB    256     100.0%            8b7061dc-e539-4eac-9fb5-00f8e88f6613  RAC1

root@nw6:/var/lib/mesos-slave/slaves/3f5b0e40-fda6-4b75-84bb-a70c06a59bd3-S14/frameworks/53b1988b-bf59-4317-927a-7c9c3014376e-0000/executors/cassandra.kong-cluster.node.3.executor/runs/d7ae3ea5-92c8-4d56-bcb1-57b6843dfcc4/apache-cassandra-2.1.4/bin# 

In couple of hours I'll make another cluster by hand (in docker) and try on that, then I'll make another one with mesos framework.
I'll let you know.

@samek
Copy link
Author

samek commented Jan 15, 2016

@thibaultcha So,
I've made a 3 node cassandra cluster with

  • cassandra:2.1.12 - not working
  • cassandra:2.2.4 - not working
  • cassandra:latest (which is 3.1.1) - different error (it doesn't migrate at all)
kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false keyspace=kong replication_factor=1 replication_strategy=SimpleStrategy contact_points=10.0.0.208:9042 data_centers=
[INFO] Leaving cluster..
[ERR] Cassandra error: ResponseError: [Invalid] unconfigured table schema_keyspaces
[ERR] Could not start Kong

I've started nodes like this:
First node , wait until it settles (IP 10.0.0.208)

docker run -ti -e CASSANDRA_BROADCAST_ADDRESS=10.0.0.208 -p 7000:7000 -p 9042:9042 cassandra:2.1.12

Second node, wait until it actually joins (IP 10.0.0.197)

docker run -ti -e CASSANDRA_BROADCAST_ADDRESS=10.0.0.197 -p 7000:7000  -p 9042:9042 -e CASSANDRA_SEEDS=10.0.0.208 cassandra:2.1.12

Third node (IP 10.0.0.48)

ocker run -ti -e CASSANDRA_BROADCAST_ADDRESS=10.0.0.48 -p 7000:7000  -p 9042:9042 -e CASSANDRA_SEEDS=10.0.0.208 cassandra:2.1.12

I've done this with config provided in the docker (just change of contact point) - so leaving out replication settings keyspace ..
Can you guys try on multi node cluster as well ?

@samek
Copy link
Author

samek commented Jan 15, 2016

@thefosk about non-resolvable hosts ..
The change from 0.6.0rc1 and rc2 is
ipv6=off

 http {
    resolver {{dns_resolver}} ipv6=off;

If I add ipv6=off then it works like it should.

@subnetmarco
Copy link
Member

@samek thanks so much for spotting this - somehow it went missing and I am pretty sure it would have been very hard for me to notice it.

@thibaultcha
Copy link
Member

Yes Cassandra 3.0 is not supported. Let me try your setup.

@thibaultcha
Copy link
Member

I am very much unable to reproduce your setup:

  • How come you are binding the containers' ports to the same host port? 9042 and 7000 every time? Are your containers running in separate machines?
  • Even by changing them, I always get a java.lang.RuntimeException: Unable to gossip with any seeds exception when starting nodes 2 and 3, which cannot contact the seed node (the containers are not linked in your example?)
  • Even if I link the nodes to the seed node, I still get java.lang.IllegalStateException: Unable to contact any seeds!

Maybe one thing to consider is that I am running docker from docker-machine (being on OS X). But I don't think it should cause any problem to inter-containers communication.

Can you guys try on multi node cluster as well ?

As I said, many of us have been running Kong on multi node cluster since 0.6 and the driver is always tested against a multi-node cluster as well. Unless I misunderstand your question?

@thibaultcha
Copy link
Member

After a lot of digging, I did found an issue with the driver. It seems that somehow, I was silly enough to implement the check on this consensus timeout the other way around. Sadly, this is something very hard to test and I never noticed the error because the use cases never happened to me or in the CI :(

Anyways, this particular change should improve the driver. It might not fix your issue (since I was not able to reproduce it), but it might be worth a try. It is very nice to have spotted this issue! So thanks again for giving 0.6 a try.

@samek
Copy link
Author

samek commented Jan 16, 2016

@thibaultcha Yes I was running them on separate machines - CoreOS.
I'll try with the change and let you know.

@samek
Copy link
Author

samek commented Jan 16, 2016

@thibaultcha I've took rc2, and replaced
cassandra.lua
cassandra/options.lua
in /usr/local/share/lua/5.1

Tried it 2 times in two different keyspaces and it always worked.
Before It never (for me) went through migrations in one go before, so I guess this solves the problem.

[root@e8482902aba6 cassandra]# kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false replication_factor=1 keyspace=kongfix2 replication_strategy=SimpleStrategy contact_points=10.0.0.74:9042 data_centers=
[INFO] Database not initialized. Running migrations...
[INFO] Migrating core on keyspace "kongfix2" (cassandra)
[INFO] core migrated up to: 2015-01-12-175310_skeleton
[INFO] core migrated up to: 2015-01-12-175310_init_schema
[INFO] core migrated up to: 2015-11-23-817313_nodes
[INFO] Migrating key-auth on keyspace "kongfix2" (cassandra)
[INFO] key-auth migrated up to: 2015-07-31-172400_init_keyauth
[INFO] Migrating rate-limiting on keyspace "kongfix2" (cassandra)
[INFO] rate-limiting migrated up to: 2015-08-03-132400_init_ratelimiting
[INFO] Migrating mashape-analytics on keyspace "kongfix2" (cassandra)
[INFO] mashape-analytics migrated up to: 2015-12-03-161400_mashape-analytics-config
[INFO] Migrating acl on keyspace "kongfix2" (cassandra)
[INFO] acl migrated up to: 2015-08-25-841841_init_acl
[INFO] Migrating oauth2 on keyspace "kongfix2" (cassandra)
[INFO] oauth2 migrated up to: 2015-08-03-132400_init_oauth2
[INFO] oauth2 migrated up to: 2015-08-24-215800_cascade_delete_index
[INFO] Migrating jwt on keyspace "kongfix2" (cassandra)
[INFO] jwt migrated up to: 2015-06-09-jwt-auth
[INFO] Migrating basic-auth on keyspace "kongfix2" (cassandra)
[INFO] basic-auth migrated up to: 2015-08-03-132400_init_basicauth
[INFO] Migrating response-ratelimiting on keyspace "kongfix2" (cassandra)
[INFO] response-ratelimiting migrated up to: 2015-08-21_init_response-rate-limiting
[INFO] Migrating hmac-auth on keyspace "kongfix2" (cassandra)
[INFO] hmac-auth migrated up to: 2015-09-16-132400_init_hmacauth
[INFO] dnsmasq............address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] nginx .............admin_api_listen=0.0.0.0:8001 proxy_listen=0.0.0.0:8000 proxy_listen_ssl=0.0.0.0:8443
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -node=e8482902aba6_0.0.0.0:7946 -log-level=err
[INFO] Trying to auto-join Kong nodes, please wait..
[WARN] Cannot auto-join the cluster because no nodes were found
[OK] Started

@thibaultcha
Copy link
Member

Amazing! That's great news. I will merge the patch, release the driver and update the dependency today. Thanks again!

On Jan 16, 2016, at 9:13 AM, samek [email protected] wrote:

@thibaultcha I've took rc2, and replaced
cassandra.lua
cassandra/options.lua
in /usr/local/share/lua/5.1

Tried it 2 times in two different keyspaces and it always worked.
Before It never (for me) went through migrations in one go before, so I guess this solves the problem.

[root@e8482902aba6 cassandra]# kong start
[INFO] Kong 0.6.0rc2
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] database...........cassandra ssl=verify=false enabled=false replication_factor=1 keyspace=kongfix2 replication_strategy=SimpleStrategy contact_points=10.0.0.74:9042 data_centers=
[INFO] Database not initialized. Running migrations...
[INFO] Migrating core on keyspace "kongfix2" (cassandra)
[INFO] core migrated up to: 2015-01-12-175310_skeleton
[INFO] core migrated up to: 2015-01-12-175310_init_schema
[INFO] core migrated up to: 2015-11-23-817313_nodes
[INFO] Migrating key-auth on keyspace "kongfix2" (cassandra)
[INFO] key-auth migrated up to: 2015-07-31-172400_init_keyauth
[INFO] Migrating rate-limiting on keyspace "kongfix2" (cassandra)
[INFO] rate-limiting migrated up to: 2015-08-03-132400_init_ratelimiting
[INFO] Migrating mashape-analytics on keyspace "kongfix2" (cassandra)
[INFO] mashape-analytics migrated up to: 2015-12-03-161400_mashape-analytics-config
[INFO] Migrating acl on keyspace "kongfix2" (cassandra)
[INFO] acl migrated up to: 2015-08-25-841841_init_acl
[INFO] Migrating oauth2 on keyspace "kongfix2" (cassandra)
[INFO] oauth2 migrated up to: 2015-08-03-132400_init_oauth2
[INFO] oauth2 migrated up to: 2015-08-24-215800_cascade_delete_index
[INFO] Migrating jwt on keyspace "kongfix2" (cassandra)
[INFO] jwt migrated up to: 2015-06-09-jwt-auth
[INFO] Migrating basic-auth on keyspace "kongfix2" (cassandra)
[INFO] basic-auth migrated up to: 2015-08-03-132400_init_basicauth
[INFO] Migrating response-ratelimiting on keyspace "kongfix2" (cassandra)
[INFO] response-ratelimiting migrated up to: 2015-08-21_init_response-rate-limiting
[INFO] Migrating hmac-auth on keyspace "kongfix2" (cassandra)
[INFO] hmac-auth migrated up to: 2015-09-16-132400_init_hmacauth
[INFO] dnsmasq............address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] nginx .............admin_api_listen=0.0.0.0:8001 proxy_listen=0.0.0.0:8000 proxy_listen_ssl=0.0.0.0:8443
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -node=e8482902aba6_0.0.0.0:7946 -log-level=err
[INFO] Trying to auto-join Kong nodes, please wait..
[WARN] Cannot auto-join the cluster because no nodes were found
[OK] Started

Reply to this email directly or view it on GitHub.

kikito pushed a commit that referenced this issue Apr 23, 2024
### Summary

#### 2.6.0
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

#### 2.6.1
```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

#### 2.6.2
```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

Signed-off-by: Aapo Talvensaari <[email protected]>
bungle added a commit that referenced this issue Apr 23, 2024
### Summary

#### 2.6.0
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

#### 2.6.1
```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

#### 2.6.2
```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

Signed-off-by: Aapo Talvensaari <[email protected]>
tysoekong pushed a commit that referenced this issue Apr 26, 2024
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

KAG-4331

Signed-off-by: Aapo Talvensaari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants