I got this exception when the load in increasing (pubsub) “Cancelling 2 outstanding requests”
How can I resolve this issue?
Regards
Hi Madazone … if you have any sample code or more detail re; what you are doing when you see the error, that would be great. In the meantime, it sounds like you might be closing your connection (or having it terminated by the server) before all queued requests have completed. If this is the case, the solution would be (I guess) to wait for in-flight requests to complete before closing the connection … or maybe typically just wait seconds before closing … ?
Session.leave() returns a future/deferred that you should await (which will ensure you’ve flushed everything and cleanly closed the WAMP session).
I don’t close the connection. It drops the connection automically, and I dont know what can cause that issue.
If the processing of a new topic message takes too much time, does the router or the client drops the connection automatically?
Thanks.
Mmm, I’m not too familiar with the inner workings, but I’m under the impression the WAMP client runs a ping-ping session with the server, and if a pong doesn’t arrive within a timeout period, then it drops the connection. Typically I think this will only happen if you don’t service your client-side event loop (or unless the server is totally overloaded) … is it possible you have a long-running client side process which isn’t yielding to the event loop at regular intervals?
It can happen that the server publishes messages in a millisecond intervall and the client takes more than one second.
Who is charge of the connection drop server or client? I can try to increase the timeout intervall so the connection doesnt drop. I just don’t know on which side I should configure this option.
Thanks.
Try here; https://crossbar.io/docs/WebSocket-Options/#production-settings
When I’ve seen this in my own code in the past, it’s always been a case of doing something that’s taking longer than I expected and thus starving the ping responder …
I already use this production configuration in my router and a new topic processing doesnt exceed 10 seconds.
Regards
Mmm, I think the default interval is 10s, but the timeout is 5s … but either way, if the connection is dropping, there will be a reason for it. If there’s nothing starving the ping-pong at either end, I’d start looking at the network connection.
I have more info in the log. Before the connection is closed, multiple socket.send() exceptions are raised
019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:18 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:19 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:20 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:21 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:22 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:23 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:24 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:25 socket.send() raised exception.
2019-06-10T14:33:26 socket.send() raised exception.
2019-06-10T14:33:26 socket.send() raised exception.
2019-06-10T14:33:26 socket.send() raised exception.
2019-06-10T14:33:26 socket.send() raised exception.
[2019-06-10 14:33:26.412261] INFO - ConnectorHandler: msg=‘left connection’ details=‘CloseDetails(reason=<wamp.close.transport_lost>, message=‘WAMP transport was lost without closing the session 1265807983731724 before’)’
[2019-06-10 14:33:26.413197] WARNING - ConnectorHandler: msg=‘disconnected’
Ok, so two things … are there any errors on the “other end” of the connection? And, are you seeing any errors on your network interface?
(also, are you running over a local connection, or via a wider network / the internet?)
2019-06-10T14:33:10+0000 [Router 14] dropping connection to peer tcp4:10.42.0.162:43560 with abort=True: WebSocket ping timeout (peer did not respond with pong in time)
The clients and the server are on the same machine. I suspect that there is something in my client code that creates a long running process.
I am going to investigate that.
Thanks for your help.
Le mar. 11 juin 2019 à 15:01, Gareth Bult postmaster@forum.crossbar.io a écrit :
When you say “processing takes 10 seconds” do you mean you’re pausing the reactor for that long? If so, that’s way too long to halt the reactor / event-loop. You can take however long you like to respond to messages, but you can’t just halt the reactor “forever” because then it can’t do anything else.
I am not pausing the reactor 10 seconds, it is the timeout intervall on the server side.
The topic processing on client side takes less than 1 second. I suspect that the connection drops when the client receives for example more than 10 messages in one second. (10 x 1sec > timeout interval).
Is the “1 second processing” pausing the reactor for that long?
(In general that is “too long” to pause the event-loop / reactor; for long-running loops etc you can either yield every view iterations to the event-loop or use a subprocess to do the long-running processing)
Ideally if you could provide a http://sscce.org style example that demonstrates your issue we could provide better help
There are obviously many takes on the best way to approach async coding, but for what it’s worth I tend to subscribe to the mantra “all jobs running in the main event loop should effectively complete immediately”. Now immediately is obviously subjective, for me once a routine starts to run into a time-frame measured in milliseconds, I start to think about moving it off the main event loop … otherwise you start building latency for other code that might expect to run in real-time …