Duplicate Calls to WAMP Procedures When crossbar roundrobin Is Used with Replica Pods

Our application makes use of a micro-services architecture.

A crossbar server (tested with version 18.9.2 and 22.6.1) is hosted in the stack (see below for config)

nodejs servers (using autobahn@18.10.2) are hosted in the same stack and register procedures (on server startup) with the crossbar server and also call procedures (registered by other servers). Config used for registration/calls is:

    "wamp": {
      "authid": "1",
      "realm": "realm1",
      "register": {
        "invoke": "roundrobin"
      },
      "secret": "THE-SECRET",
      "url": "ws://crossbar:8080/ws"
    }

When there is one pod for a workload (e.g., for user-server) everything is working as expected.

When we scale up a workload so that there are a number of replica pods (e.g., pods a, b, and c for user-server) we are encountering a duplicate procedure call issue.

I’ve looked at (and successfully run to see roundrobin working as expected) the example at crossbar-examples/sharedregs/python at master · crossbario/crossbar-examples · GitHub. Our implementation follows how this has been achieved.

Any ideas of what we might be missing? Is there any reason that this might not work in stack hosted in a k8s cluster in the way I’ve described?

Working Scenario:

  • user-server (workload has only 1 pod), registers ‘thecompany.user.get’
  • ‘thecompany.user.get’ call results in the call being handled 1 time by the one user-server pod.

Broken Scenario:

  • user-server (workload has only 3 replica pods), registers ‘thecompany.user.get’
  • ‘thecompany.user.get’ call results in the call being handled THREE times by one of the user-server pods (roundrobin is respected).
  • ‘thecompany.user.get’ call results in the call being handled THREE times by one (a different pod to previous call) of the user-server pods (roundrobin is respected).
number of replica user-server pods result of calling ‘thecompany.user.get’
1 user-server pod handles the ‘thecompany.user.get’ call once
2 user-server pod a or b (roundrobin is respected) handles the ‘thecompany.user.get’ call TWICE
3 user-server pod a, b or c (roundrobin is respected) handles the ‘thecompany.user.get’ call THREE times

The following is logging captured from the crossbar server (when there are 3 replica user-server pods in the stack), which shows the ‘thecompany.user.set’ being registered by each of the 3 replica user-server pods:

2022-08-25T07:01:14+0000 [Router         46 crossbar.router.router.Router] Authorized action 'register' for URI 'thecompany.user.set' by session 1212112427197046 with authid '1' and authrole 'backend' -> authorization: {'allow': True, 'cache': False, 'validate': None}
...
2022-08-25T07:01:15+0000 [Router         46 crossbar.router.router.Router] Authorized action 'register' for URI 'thecompany.user.set' by session 3880250571956818 with authid '1' and authrole 'backend' -> authorization: {'allow': True, 'cache': False, 'validate': None}
...
2022-08-25T07:01:15+0000 [Router         46 crossbar.router.router.Router] Authorized action 'register' for URI 'thecompany.user.set' by session 7174806059859627 with authid '1' and authrole 'backend' -> authorization: {'allow': True, 'cache': False, 'validate': None}

crossbar server config:

{
    "config": {
      "controller": {},
      "version": 2,
      "workers": [
        {
          "realms": [
            {
              "name": "realm1",
              "roles": [
                {
                  "name": "backend",
                  "permissions": [
                    {
                      "allow": {
                        "call": true,
                        "publish": true,
                        "register": true,
                        "subscribe": true
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": ""
                    }
                  ]
                },
                {
                  "name": "authenticated",
                  "permissions": [
                    {
                      "allow": {
                        "call": false,
                        "publish": false,
                        "register": false,
                        "subscribe": true
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": "thecompany.file.provision"
                    },
                    {
                      "allow": {
                        "call": false,
                        "publish": false,
                        "register": false,
                        "subscribe": true
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": "thecompany.notifications.updates"
                    },
                    {
                      "allow": {
                        "call": true,
                        "publish": false,
                        "register": false,
                        "subscribe": false
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": "thecompany.model"
                    },
                    {
                      "allow": {
                        "call": true,
                        "publish": true,
                        "register": false,
                        "subscribe": true
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": "public."
                    }
                  ]
                },
                {
                  "name": "anonymous",
                  "permissions": [
                    {
                      "allow": {
                        "call": true,
                        "publish": false,
                        "register": false,
                        "subscribe": false
                      },
                      "cache": true,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "exact",
                      "uri": "thecompany.app.meta"
                    },
                    {
                      "allow": {
                        "call": true,
                        "publish": false,
                        "register": false,
                        "subscribe": false
                      },
                      "cache": false,
                      "disclose": {
                        "caller": true,
                        "publisher": true
                      },
                      "match": "prefix",
                      "uri": "anon."
                    }
                  ]
                }
              ]
            }
          ],
          "transports": [
            {
              "endpoint": {
                "port": 8080,
                "type": "tcp"
              },
              "options": {
                "access_log": true,
                "hsts": true
              },
              "paths": {
                "/": {
                  "directory": "../web",
                  "type": "static"
                },
                "ws": {
                  "auth": {
                    "anonymous": {
                      "role": "anonymous",
                      "type": "static"
                    },
                    "ticket": {
                      "authenticator": "thecompany.auth.authenticate",
                      "type": "dynamic"
                    },
                    "wampcra": {
                      "type": "static",
                      "users": {
                        "1": {
                          "role": "backend",
                          "secret": "BACKEND-SECRET"
                        },
                        "monitor": {
                          "role": "devops",
                          "secret": "DEVOPS-SECRET"
                        }
                      }
                    }
                  },
                  "debug": false,
                  "options": {
                    "auto_fragment_size": 65536,
                    "auto_ping_interval": 10000,
                    "auto_ping_size": 12,
                    "auto_ping_timeout": 5000,
                    "close_handshake_timeout": 1000,
                    "echo_close_codereason": true,
                    "enable_webstatus": false,
                    "fail_by_drop": true,
                    "max_frame_size": 1048576,
                    "max_message_size": 1048576,
                    "open_handshake_timeout": 2500
                  },
                  "type": "websocket"
                }
              },
              "type": "web"
            }
          ],
          "type": "router"
        }
      ]
    }
  }

So you have a single pod with a single crossbar node, plus three pods with nodejs which run an identical wamp component connecting to the single node, and then you test using a caller connecting to that single crossbar node? That should work, at least I can’t see what could go wrong. You could also try to port the example (which is working for you) into your existing stack, and use the example client (or just run the working example inside k8 and test). This would allow you to bisect between the working non-k8’ed example, and only adding k8 to that …

Thanks for your response.

Yes, that’s our situation. 1 crossbar pod and 3 replica pods running nodejs for the same component (our user-server).

I mis-diagnosed in my original message and have now confirmed that calls are correctly respecting roundrobin invoke.

What is happening in our situation is that ‘thecompany.user.set’ (user-server) publishes ‘thecompany.stream.user.updated’ which is subscribed to by:

  • user-server (3 replica pods)
  • and other servers …

What I’d like to do is to be able to specify that each component handles a published procedure only once, so that user-server could handle ‘thecompany.stream.user.updated’ once instead of once for each replica pods (session)

Is there a pattern so that a crossbar publish would be handled is this way?

Note., we have other topics which we would like every session which has a subscription to respond to (e.g., clearing cache - we’d want every replica to handle that).