getaddrinfo EAI_AGAIN error, then backend container crashes

[brandonhinds]

Describe the bug

After authenticating with Spotify I get a triggerUncaughtException error, and then the backend container crashes and restarts. After turning on debug logging I found a getaddrinfo EAI_AGAIN accounts.spotify.com error message right before the container crash.

Expected behavior

After authenticating with Spotify you are turned to the frontend site.

Additional context

The output of the debug logs for the backend container are below:

Debug Log

yarn run v1.22.19
$ node ./lib/migrations.js
[info]  Trying to connect to database at mongodb://mongo-service.your-spotify:27017/your_spotify
[info]  Connected to database !
[info]  Starting migrations
[info]  Migrations successfully ran
Done in 0.47s.
yarn run v1.22.19
$ node lib/bin/www
[info]  Trying to connect to database at mongodb://mongo-service.your-spotify:27017/your_spotify
[info]  Connected to database !
[debug]  Locking DB_LongWriteLock
[info]  Checking database...
[debug]  Listening on port 8080
[warn]  No user is admin, cannot auto fix database
[debug]  Unlocking DB_LongWriteLock
[info]  Starting loop for 0 users
GET /me 200 4.610 ms - 16
GET /global/preferences 200 14.415 ms - 68
[info]  Starting loop for 0 users
GET /me 304 0.700 ms - -
GET /global/preferences 304 4.729 ms - -
GET /oauth/spotify 302 7.736 ms - 636

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^
AxiosError: getaddrinfo EAI_AGAIN accounts.spotify.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
  hostname: 'accounts.spotify.com',
  syscall: 'getaddrinfo',
  code: 'EAI_AGAIN',
  errno: -3001,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: {
      FormData: [Function: FormData] {
        LINE_BREAK: '\r\n',
        DEFAULT_CONTENT_TYPE: 'application/octet-stream'
      }
    },
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/x-www-form-urlencoded',
      'User-Agent': 'axios/0.27.2'
    },
    params: {
      grant_type: 'authorization_code',
      code: '<AUTHORISATION_CODE>',
      redirect_uri: 'https://your-spotify-api.<DOMAIN.COM>/oauth/spotify/callback',
      client_id: '<CLIENT_ID>',
      client_secret: '<CLIENT_SECRET>'
    },
    method: 'post',
    url: 'https://accounts.spotify.com/api/token',
    data: null
  },
  request: <ref *3> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError],
      socket: [Function: handleRequestSocket]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'https:',
      path: '/api/token?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>',
      method: 'POST',
      headers: {
        Accept: 'application/json, text/plain, */*',
        'Content-Type': 'application/x-www-form-urlencoded',
        'User-Agent': 'axios/0.27.2'
      },
      agent: undefined,
      agents: { http: undefined, https: undefined },
      auth: undefined,
      hostname: 'accounts.spotify.com',
      port: null,
      nativeProtocols: {
        'http:': {
          _connectionListener: [Function: connectionListener],
          METHODS: [
            'ACL',         'BIND',       'CHECKOUT',
            'CONNECT',     'COPY',       'DELETE',
            'GET',         'HEAD',       'LINK',
            'LOCK',        'M-SEARCH',   'MERGE',
            'MKACTIVITY',  'MKCALENDAR', 'MKCOL',
            'MOVE',        'NOTIFY',     'OPTIONS',
            'PATCH',       'POST',       'PROPFIND',
            'PROPPATCH',   'PURGE',      'PUT',
            'REBIND',      'REPORT',     'SEARCH',
            'SOURCE',      'SUBSCRIBE',  'TRACE',
            'UNBIND',      'UNLINK',     'UNLOCK',
            'UNSUBSCRIBE'
          ],
          STATUS_CODES: {
            '100': 'Continue',
            '101': 'Switching Protocols',
            '102': 'Processing',
            '103': 'Early Hints',
            '200': 'OK',
            '201': 'Created',
            '202': 'Accepted',
            '203': 'Non-Authoritative Information',
            '204': 'No Content',
            '205': 'Reset Content',
            '206': 'Partial Content',
            '207': 'Multi-Status',
            '208': 'Already Reported',
            '226': 'IM Used',
            '300': 'Multiple Choices',
            '301': 'Moved Permanently',
            '302': 'Found',
            '303': 'See Other',
            '304': 'Not Modified',
            '305': 'Use Proxy',
            '307': 'Temporary Redirect',
            '308': 'Permanent Redirect',
            '400': 'Bad Request',
            '401': 'Unauthorized',
            '402': 'Payment Required',
            '403': 'Forbidden',
            '404': 'Not Found',
            '405': 'Method Not Allowed',
            '406': 'Not Acceptable',
            '407': 'Proxy Authentication Required',
            '408': 'Request Timeout',
            '409': 'Conflict',
            '410': 'Gone',
            '411': 'Length Required',
            '412': 'Precondition Failed',
            '413': 'Payload Too Large',
            '414': 'URI Too Long',
            '415': 'Unsupported Media Type',
            '416': 'Range Not Satisfiable',
            '417': 'Expectation Failed',
            '418': "I'm a Teapot",
            '421': 'Misdirected Request',
            '422': 'Unprocessable Entity',
            '423': 'Locked',
            '424': 'Failed Dependency',
            '425': 'Too Early',
            '426': 'Upgrade Required',
            '428': 'Precondition Required',
            '429': 'Too Many Requests',
            '431': 'Request Header Fields Too Large',
            '451': 'Unavailable For Legal Reasons',
            '500': 'Internal Server Error',
            '501': 'Not Implemented',
            '502': 'Bad Gateway',
            '503': 'Service Unavailable',
            '504': 'Gateway Timeout',
            '505': 'HTTP Version Not Supported',
            '506': 'Variant Also Negotiates',
            '507': 'Insufficient Storage',
            '508': 'Loop Detected',
            '509': 'Bandwidth Limit Exceeded',
            '510': 'Not Extended',
            '511': 'Network Authentication Required'
          },
          Agent: [Function: Agent] { defaultMaxSockets: Infinity },
          ClientRequest: [Function: ClientRequest],
          IncomingMessage: [Function: IncomingMessage],
          OutgoingMessage: [Function: OutgoingMessage],
          Server: [Function: Server],
          ServerResponse: [Function: ServerResponse],
          createServer: [Function: createServer],
          validateHeaderName: [Function: __node_internal_],
          validateHeaderValue: [Function: __node_internal_],
          get: [Function: get],
          request: [Function: request],
          maxHeaderSize: [Getter],
          globalAgent: [Getter/Setter]
        },
        'https:': {
          Agent: [Function: Agent],
          globalAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          Server: [Function: Server],
          createServer: [Function: createServer],
          get: [Function: get],
          request: [Function: request]
        }
      },
      pathname: '/api/token',
      search: '?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: <ref *1> ClientRequest {
      _events: [Object: null prototype] {
        response: [Function: bound onceWrapper] {
          listener: [Function (anonymous)]
        },
        abort: [Function (anonymous)],
        aborted: [Function (anonymous)],
        connect: [Function (anonymous)],
        error: [Function (anonymous)],
        socket: [Function (anonymous)],
        timeout: [Function (anonymous)]
      },
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: <ref *2> TLSSocket {
        _tlsOptions: {
          allowHalfOpen: undefined,
          pipe: false,
          secureContext: SecureContext { context: SecureContext {} },
          isServer: false,
          requestCert: true,
          rejectUnauthorized: true,
          session: undefined,
          ALPNProtocols: undefined,
          requestOCSP: undefined,
          enableTrace: undefined,
          pskCallback: undefined,
          highWaterMark: undefined,
          onread: undefined,
          signal: undefined
        },
        _secureEstablished: false,
        _securePending: false,
        _newSessionPending: false,
        _controlReleased: true,
        secureConnecting: true,
        _SNICallback: null,
        servername: null,
        alpnProtocol: null,
        authorized: false,
        authorizationError: null,
        encrypted: true,
        _events: [Object: null prototype] {
          close: [
            [Function: onSocketCloseDestroySSL],
            [Function],
            [Function: onClose],
            [Function: socketCloseListener]
          ],
          end: [ [Function: onConnectEnd], [Function: onReadableStreamEnd] ],
          newListener: [Function: keylogNewListener],
          connect: [ [Function], [Function], [Function] ],
          secure: [Function: onConnectSecure],
          session: [Function (anonymous)],
          free: [Function: onFree],
          timeout: [Function: onTimeout],
          agentRemove: [Function: onRemove],
          error: [Function: socketErrorListener],
          drain: [Function: ondrain]
        },
        _eventsCount: 11,
        connecting: false,
        _hadError: true,
        _parent: null,
        _host: 'accounts.spotify.com',
        _readableState: ReadableState {
          objectMode: false,
          highWaterMark: 16384,
          buffer: BufferList { head: null, tail: null, length: 0 },
          length: 0,
          pipes: [],
          flowing: true,
          ended: false,
          endEmitted: false,
          reading: true,
          constructed: true,
          sync: false,
          needReadable: true,
          emittedReadable: false,
          readableListening: false,
          resumeScheduled: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          destroyed: true,
          errored: Error: getaddrinfo EAI_AGAIN accounts.spotify.com
              at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
            errno: -3001,
            code: 'EAI_AGAIN',
            syscall: 'getaddrinfo',
            hostname: 'accounts.spotify.com'
          },
          closed: true,
          closeEmitted: true,
          defaultEncoding: 'utf8',
          awaitDrainWriters: null,
          multiAwaitDrain: false,
          readingMore: false,
          dataEmitted: false,
          decoder: null,
          encoding: null,
          [Symbol(kPaused)]: false
        },
        _maxListeners: undefined,
        _writableState: WritableState {
          objectMode: false,
          highWaterMark: 16384,
          finalCalled: false,
          needDrain: false,
          ending: false,
          ended: false,
          finished: false,
          destroyed: true,
          decodeStrings: false,
          defaultEncoding: 'utf8',
          length: 755,
          writing: true,
          corked: 0,
          sync: false,
          bufferProcessing: false,
          onwrite: [Function: bound onwrite],
          writecb: [Function: bound onFinish],
          writelen: 755,
          afterWriteTickInfo: null,
          buffered: [],
          bufferedIndex: 0,
          allBuffers: true,
          allNoop: true,
          pendingcb: 1,
          constructed: true,
          prefinished: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          errored: Error: getaddrinfo EAI_AGAIN accounts.spotify.com
              at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
            errno: -3001,
            code: 'EAI_AGAIN',
            syscall: 'getaddrinfo',
            hostname: 'accounts.spotify.com'
          },
          closed: true,
          closeEmitted: true,
          [Symbol(kOnFinished)]: []
        },
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: 'POST /api/token?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> HTTP/1.1\r\n' +
          'Accept: application/json, text/plain, */*\r\n' +
          'Content-Type: application/x-www-form-urlencoded\r\n' +
          'User-Agent: axios/0.27.2\r\n' +
          'Host: accounts.spotify.com\r\n' +
          'Connection: close\r\n' +
          'Content-Length: 0\r\n' +
          '\r\n',
        _pendingEncoding: 'latin1',
        server: undefined,
        _server: null,
        ssl: null,
        _requestCert: true,
        _rejectUnauthorized: true,
        parser: null,
        _httpMessage: [Circular *1],
        [Symbol(res)]: TLSWrap {
          _parent: TCP {
            reading: [Getter/Setter],
            onconnection: null,
            [Symbol(owner_symbol)]: [Circular *2],
            [Symbol(handle_onclose)]: [Function: done]
          },
          _parentWrap: undefined,
          _secureContext: SecureContext { context: SecureContext {} },
          reading: false,
          onkeylog: [Function: onkeylog],
          onhandshakestart: {},
          onhandshakedone: [Function (anonymous)],
          onocspresponse: [Function: onocspresponse],
          onnewsession: [Function: onnewsessionclient],
          onerror: [Function: onerror],
          [Symbol(owner_symbol)]: [Circular *2]
        },
        [Symbol(verified)]: false,
        [Symbol(pendingSession)]: null,
        [Symbol(async_id_symbol)]: 537,
        [Symbol(kHandle)]: null,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBuffer)]: null,
        [Symbol(kBufferCb)]: null,
        [Symbol(kBufferGen)]: null,
        [Symbol(kCapture)]: false,
        [Symbol(kSetNoDelay)]: false,
        [Symbol(kSetKeepAlive)]: true,
        [Symbol(kSetKeepAliveInitialDelay)]: 60,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 0,
        [Symbol(connect-options)]: {
          rejectUnauthorized: true,
          ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA',
          checkServerIdentity: [Function: checkServerIdentity],
          minDHSize: 1024,
          maxRedirects: 21,
          maxBodyLength: 10485760,
          protocol: 'https:',
          path: null,
          method: 'POST',
          headers: {
            Accept: 'application/json, text/plain, */*',
            'Content-Type': 'application/x-www-form-urlencoded',
            'User-Agent': 'axios/0.27.2'
          },
          agent: undefined,
          agents: { http: undefined, https: undefined },
          auth: undefined,
          hostname: 'accounts.spotify.com',
          port: 443,
          nativeProtocols: { 'http:': [Object], 'https:': [Object] },
          pathname: '/api/token',
          search: '?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>',
          _defaultAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          host: 'accounts.spotify.com',
          servername: 'accounts.spotify.com',
          _agentKey: 'accounts.spotify.com:443:::::::::::::::::::::',
          encoding: null,
          singleUse: true
        }
      },
      _header: 'POST /api/token?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/x-www-form-urlencoded\r\n' +
        'User-Agent: axios/0.27.2\r\n' +
        'Host: accounts.spotify.com\r\n' +
        'Connection: close\r\n' +
        'Content-Length: 0\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object: null prototype] { path: null },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {
          'accounts.spotify.com:443:::::::::::::::::::::': [ [TLSSocket] ]
        },
        freeSockets: [Object: null prototype] {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 1,
        maxCachedSessions: 100,
        _sessionCache: { map: {}, list: [] },
        [Symbol(kCapture)]: false
      },
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/api/token?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'accounts.spotify.com',
      protocol: 'https:',
      _redirectable: [Circular *3],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype] {
        accept: [ 'Accept', 'application/json, text/plain, */*' ],
        'content-type': [ 'Content-Type', 'application/x-www-form-urlencoded' ],
        'user-agent': [ 'User-Agent', 'axios/0.27.2' ],
        host: [ 'Host', 'accounts.spotify.com' ]
      }
    },
    _currentUrl: 'https://accounts.spotify.com/api/token?grant_type=authorization_code&code=<AUTHORISATION_CODE>&redirect_uri=https:%2F%2Fyour-spotify-api.<DOMAIN.COM>%2Foauth%2Fspotify%2Fcallback&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>',
    [Symbol(kCapture)]: false
  }
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I also don't use docker-compose to deploy this app - I use Kubernetes. Those files can be found below in case they are helpful:

Kubernetes Manifests

---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: v1
kind: Secret
metadata:
  labels:
    app: your-spotify
  name: your-spotify-secret
  namespace: your-spotify
data:
  SPOTIFY_SECRET: <CLIENT_SECRET>
---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: your-spotify
  name: your-spotify-configmap
  namespace: your-spotify
data:
  API_ENDPOINT: https://your-spotify-api.<DOMAIN.COM>              # Redirect to port 8080
  CLIENT_ENDPOINT: https://your-spotify.<DOMAIN.COM>               # Redirect to port 3000
  SPOTIFY_PUBLIC: <CLIENT_ID>
  CORS: all
  MONGO_ENDPOINT: mongodb://mongo-service.your-spotify:27017/your_spotify
  LOG_LEVEL: debug
---
# Source: your-spotify/templates/mongo.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mongo
  name: mongo-service
  namespace: your-spotify
spec:
  ports:
  - port: 27017
    protocol: TCP
    targetPort: 27017
  selector:
    app: mongo
  type: ClusterIP
---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: frontend
  name: frontend-service
  namespace: your-spotify
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: frontend
  type: ClusterIP
---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: backend
  name: backend-service
  namespace: your-spotify
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: backend
  type: ClusterIP
---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: backend
  name: backend
  namespace: your-spotify
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
      annotations:
        timestamp: "2022-09-17 08:10:57.1530524 +1000 AEST m=+0.141836701"
    spec:
      containers:
      - image: yooooomi/your_spotify_server:latest
        name: backend
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        env:
        - name: SPOTIFY_SECRET
          valueFrom:
            secretKeyRef:
              name: your-spotify-secret
              key: SPOTIFY_SECRET
        envFrom:
        - configMapRef:
            name: your-spotify-configmap
---
# Source: your-spotify/templates/yourSpotify.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: frontend
  name: frontend
  namespace: your-spotify
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
      annotations:
        timestamp: "2022-09-17 08:10:57.1530524 +1000 AEST m=+0.141836701"
    spec:
      containers:
      - image: yooooomi/your_spotify_client:latest
        name: frontend
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
          name: http
          protocol: TCP
        envFrom:
        - configMapRef:
            name: your-spotify-configmap
---
# Source: your-spotify/templates/mongo.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: mongo
  name: mongo
  namespace: your-spotify
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo
  serviceName: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - image: mongo:5.0.12
        name: mongo
        imagePullPolicy: Always
        args: ["--dbpath","/data/db"]
        livenessProbe:
          exec:
            command:
              - mongo
              - --disableImplicitSessions
              - --eval
              - "db.adminCommand('ping')"
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 6
        readinessProbe:
          exec:
            command:
              - mongo
              - --disableImplicitSessions
              - --eval
              - "db.adminCommand('ping')"
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 6
        volumeMounts:
        - name: mongo-data-dir
          mountPath: "/data/dir"
      volumes:
      - name: mongo-data-dir
        hostPath:
          path: /volumes/your-spotify/mongo-data-dir

Thanks for sharing what looks like an awesome app. I am very keen to get it setup.

[Yooooomi]

Hello! Thanks for the clear bug report and the nice comment. Awesome to see a k8s setup here :)
It feels like the final pod cannot resolve dns. This is nothing to do with the application itself in my opinion.
I don't know kubernetes enough to troubleshoot dns issues but I'll take a look.
In the meanwhile, you can check that your device has internet access.

EDIT: https://stackoverflow.com/questions/45805483/kubernetes-pods-cant-resolve-hostnames

[brandonhinds]

Thanks for the link - my coredns had restarted a couple of times. Redeploying it does not seem to have fixed the error though.

I have triple checked and the error presented above is still reproducable, but it looks like the backend container can resolve the DNS of accounts.spotify.com, as seen by the following command:

$ kubectl exec <BACKEND_POD> -n your-spotify -- nslookup accounts.spotify.com
Server:         10.152.183.10
Address:        10.152.183.10:53

Non-authoritative answer:
accounts.spotify.com    canonical name = edge-web.dual-gslb.spotify.com
Name:   edge-web.dual-gslb.spotify.com
Address: 35.186.224.25

Non-authoritative answer:
accounts.spotify.com    canonical name = edge-web.dual-gslb.spotify.com
Name:   edge-web.dual-gslb.spotify.com
Address: 2600:1901:1:c36::

I will deploy some other apps into my home lab (this one was actually the first one 😊 ) and see if I run into DNS issues in those. If I do, and it leads to me finding the fix, I will report back here too.

[brandonhinds]

Been doing a bit more investigating into this issue this evening but haven't had much luck. I did find this recent post which appears to describe the same issue, so hopefully someone responds to it with a solution.

There is also this post which suggest that Kubernetes is the root cause, but also no resolution there. That might be why I am the only one reporting this, if the vast majority of users deploy the app using the provided docker-compose script. This post supports that theory, saying that running an alpine image in Kubernetes can lead to these DNS issues.

I'll keep slowly ticking away at this when I have a chance.

[brandonhinds]

Alright I resolved it! The posts I linked to above had all the clues needed - the backend container didn't work due to a combination of the node package, the alpine base image, and Kubernetes. I forked the repo, modified the Dockerfile.production image to change the base image from alpine to debian, and then redeployed the stack and it worked!

I've raised a pull request with an additional dockerfile with my changes in case you are interested @Yooooomi , but no sweat if not. Regardless, I will close this ticket.

[samip5]

@brandonhinds I have had great success with deploying this on k8s without changing the base image.

For reference: https://github.com/samip5/k8s-cluster/tree/c028b26ef65a52706674299e49ef191c7ae3f1a3/archive/misc/my_spotify

[brandonhinds]

@samip5 interesting. Your manifest files look very similar to what I am using.

What "distro" of Kubernetes are you running? I am using Microk8s on a Ubuntu 22.04 server. I wonder if my issues are related to the cut-down set of typical services that Microk8s uses.

[samip5]

@samip5 interesting. Your manifest files look very similar to what I am using.

What "distro" of Kubernetes are you running? I am using Microk8s on a Ubuntu 22.04 server. I wonder if my issues are related to the cut-down set of typical services that Microk8s uses.

I'm using k3s on Ubuntu but I also have Kyverno doing fun things like tweaking DNS config in pods.

[brandonhinds]

Ok, that is probably where our differences are, as the issue seems to be related to DNS within the pods.

Re: the pull request, as this issue seems to be isolated to my setup specifically I think it is reasonable to have the PR denied, and this ticket and the accompanying PR can be used as a reference for anyone who has similar issues in the future. If someone is using Kubernetes they are likely also comfortable building their own Docker image.