Tuesday, 2016-06-07

shu-mutouflwang: sure! (^o^)/00:00
openstackgerritMerged openstack/zaqar-ui: Remove obsolote pot files  https://review.openstack.org/32375300:04
shu-mutouHi everyone, now we can translate Zaqar-UI on Zanata, please translate Zaqar-UI into your native language!! see also https://wiki.openstack.org/wiki/I18nTeam00:26
openstackgerritShu Muto proposed openstack/zaqar-ui: Fix typo  https://review.openstack.org/32618800:34
*** jasondotstar has quit IRC01:18
openstackgerritwangxiyuan proposed openstack/zaqar: Make the notifier max_workers configurable  https://review.openstack.org/30844301:18
*** tqtran has quit IRC01:19
*** Qiming_ has joined #openstack-zaqar01:34
flwangshu-mutou: awesome, thank you :)01:41
shu-mutouflwang: my pleasure. (^o^)/01:44
openstackgerritwangxiyuan proposed openstack/python-zaqarclient: Add update queue function in v2  https://review.openstack.org/29436801:45
*** Qiming_ has quit IRC01:46
openstackgerritEva Balycheva proposed openstack/zaqar: Make queues lazy in subscriptions  https://review.openstack.org/32074901:47
flwangwow, two new important patch sets :)01:48
Eva-i=)02:02
wxyjune 9-11 is the Chinese Dragon Boat Festival. Wanghao and I will be away from keyboard for 3 days.02:14
wxyflwang: Happy Dragon Boat Festival!02:15
flwangwxy: hah, thanks, are you guys in holiday?02:18
openstackgerritMerged openstack/zaqar: Show 'age' field in subscriptions (mongodb)  https://review.openstack.org/28890702:21
wxyflwang: from Thursday this week02:21
openstackgerritMerged openstack/zaqar: Show 'age' field in subscriptions (redis)  https://review.openstack.org/28890802:21
flwangwxy: ah, cool, we have brought some zongzi ;)02:25
wxyflwang: cool. Maybe Eva-i can have a try, zongzi is a kind of delicious chinese festival food. :)02:34
flwangwxy: i'm sure it's hard to get in Russia02:35
flwangEva-i: wxy: did you guys notice the issue reported by tripleO team?02:37
flwang(04:30:33) jtomasek: Hi, I have a question. I am connecting to a zaqar queue with web client using websockets. After connecting to socket, I authenticate, create queue and subscribe to it. Then when there is more than 10 subscriptions, the messages suddenly stop coming to the client. I can see them by calling zaqar api, but client won't receive them. Anone seen this?02:37
wxyEva-i: yes, could it be solved by this patch? https://review.openstack.org/#/c/308443/02:39
flwangwxy: nope, i don't think they're related02:41
flwangin other words, even there is only 1 or 2 workers, it should work02:41
flwangif it's a bug, then i don't think adding more workers is the solution02:41
wxyflwang: Eva-i: OK, I'll look into it today.02:45
openstackgerritMerged openstack/zaqar: Make the notifier max_workers configurable  https://review.openstack.org/30844302:45
flwangwxy: i'm trying to reproduce it and will update status here if there is any progress02:45
flwangjust a reminder since it may block tripleO's work02:46
Eva-iflwang: wxy: I know a restaurant nearby filled with chineze people. Maybe there I can find zongzi here. =)02:46
Eva-iwxy: I haven't inspected this problem, may look tomorrow02:46
Eva-iwxy: would be glad if you find something today02:47
wxyEva-i: If you can't find one. I can send you some by international express.:)02:47
wxyEva-i: flwang: OK, I'm try to reprodcue now.02:48
*** pcaruana has joined #openstack-zaqar04:11
*** pcaruana has quit IRC04:24
*** pcaruana has joined #openstack-zaqar04:34
*** pcaruana has quit IRC04:40
*** GB21 has joined #openstack-zaqar04:53
flwangwxy: any progress?04:54
*** flwang1 has quit IRC05:00
*** GB21 has quit IRC05:12
flaper87flwang: hey, apparently there are some issues in liberty with Zaqar because it's installing latest falcon05:20
flaper87http://logs.openstack.org/periodic-stable/periodic-zaqar-python27-liberty/2a60d92/console.html05:20
*** pt_15 has quit IRC05:26
*** itisha has quit IRC05:30
*** GB21 has joined #openstack-zaqar05:31
*** rcernin has joined #openstack-zaqar05:36
wxyflwang: I can't reproduce. I set max_notifier_workers=2, created 12 subscriptions for a queue. Then pose some messages to the queue. But everything works well.06:14
wxyflwang: used websocket06:16
wxyflwang: jomasek: I need more detail, a bug description on launchpd is better.06:17
wxyflaper87: https://review.openstack.org/#/c/320190/  I guess this patch could solve the problem.06:23
flaper87wxy: looking06:27
flaper87wxy: I don't think it will, tbh. I think the open cap on falcon's requirement is the issue06:27
wxyflaper87: emm, we fixed the falcon's issue at master branch by this patch: https://review.openstack.org/#/c/318529/  then should we cherry-pick it to liberty? or just update the liberty's requirement?06:30
flaper87wxy: lemme try to backport it06:30
flaper87wxy: did we backport that one to mitaka?06:31
flaper87yeah, it's invasive06:31
flaper87I'd prolly just cap it06:31
wxyflaper87: No, the only patch for mitaka is https://review.openstack.org/#/c/320193/06:34
flaper87wxy: I approved the one for liberty06:35
*** jtomasek has joined #openstack-zaqar06:48
openstackgerritOpenStack Proposal Bot proposed openstack/zaqar-ui: Imported Translations from Zanata  https://review.openstack.org/32626407:09
openstackgerritwangxiyuan proposed openstack/zaqar: Clean the auth app after authentication failure  https://review.openstack.org/32627207:31
*** mpanetta has joined #openstack-zaqar08:09
*** mpanetta has quit IRC08:13
*** tesseract has joined #openstack-zaqar08:50
openstackgerritwanghao proposed openstack/zaqar: Fix fail of getting claim after claim expire(Redis)  https://review.openstack.org/32628709:06
thervejtomasek, FWIW I tested 10 subscription over websocket and it seems to work fine for me09:41
thervejtomasek, Do you have a simple reproducer?09:41
jtomasektherve: I don't unfortunately. I'll try to provide one, eventually. Can you test more than 10, my webclient stops receiving messages when the subscriptions count is 11 and more09:42
thervejtomasek, Right I mean for than 1009:43
thervejtomasek, At that point you don't get any message at all, or not just the > 10 ones?09:43
jtomasektherve: ok, do you have a code example which you're using? I am doing this: https://review.openstack.org/#/c/319070/9/src/js/services/ZaqarWebSocketService.js09:44
therveAh, hum09:45
*** AAzza has quit IRC09:45
thervejtomasek, Do you use a browser as client?09:45
jtomasektherve: at that point I don't receive any messages at all (except for the messages from zaqar saying that authentication, queue creation and subscription has been created)09:45
jtomasektherve: yes09:45
thervejtomasek, it looks like this lib may create a connection per queue, no?09:45
*** AAzza has joined #openstack-zaqar09:46
jtomasektherve: init is used only once in the client lifetime (precisely when user logs in with keystone)09:46
therveOkay09:46
therveI use a python client FWIW09:46
jtomasektherve: if there is another instance of the client the client will create it's own connection, create/connec to queue, subscribe and will receive the same messages, so all client instances are in sync09:47
thervejtomasek, I was just wondering if you opened 10 connections to zaqar from the brower you may hit some limit09:48
jtomasektherve: I wonder how I could debug why the client does not receive the message. But I can safely say it starts happening when there is 11 and more subscriptions09:48
jtomasektherve: is it possible to list connections somehow?09:48
thervejtomasek, netstat on the server?09:49
jtomasektherve: hmm, only case I can think of is that if browser refreshed, the original connection would not terminate, but it seems improbable to me09:51
thervejtomasek, http://paste.openstack.org/show/508568/ is the script I used09:54
*** shu-mutou is now known as shu-mutou-AFK09:54
thervejtomasek, Also, do you use zaqar master?09:56
*** AAzza has quit IRC09:57
jtomasektherve: not master, It is version from RDO, ti believe it is mitaka based09:57
therveOkay09:58
jtomasektherve: anything to watch out for?09:58
thervejtomasek, The number 10 may come from https://github.com/openstack/zaqar/blob/master/zaqar/notification/notifier.py#L3609:58
therveThough it shouldn't and doesn't from my test09:58
jtomasektherve: I'll try to play with worker count and see if the behavior changes10:00
thervejtomasek, Note that this flag is not available in mitaka, you'll need to change the code10:01
*** AAzza has joined #openstack-zaqar10:01
jtomasektherve: yeah, looking at the code now, it still has TODO in it10:01
thervejtomasek, Let me know if there is anything I can try too10:05
jtomasektherve: thanks!10:05
jtomasektherve: is there a docs for message actions I can send from client?10:26
*** mdnadeem_ has joined #openstack-zaqar10:28
*** mdnadeem has quit IRC10:31
*** GB21 has quit IRC10:45
*** GB21 has joined #openstack-zaqar10:52
*** GB21 has quit IRC12:23
openstackgerritFlavio Percoco proposed openstack/zaqar: Move CLI options to cmd.server  https://review.openstack.org/32642112:25
*** AAzza_ has joined #openstack-zaqar12:28
*** AAzza- has joined #openstack-zaqar12:29
*** AAzza has quit IRC12:30
*** AAzza_ has quit IRC12:32
thervejtomasek, Not that I know about. https://github.com/openstack/zaqar/blob/master/zaqar/api/v1/request.py#L33 contains the actions12:33
*** AAzza- has quit IRC12:34
thervePlus what's in v212:34
*** AAzza has joined #openstack-zaqar12:34
*** Ravikiran_K has joined #openstack-zaqar12:54
*** ametts has joined #openstack-zaqar13:53
*** mpanetta has joined #openstack-zaqar13:56
*** mpanetta has quit IRC13:56
*** mpanetta has joined #openstack-zaqar13:57
jtomasektherve: thanks14:07
*** mpanetta_ has joined #openstack-zaqar14:20
*** mpanetta has quit IRC14:20
*** mpanetta_ is now known as mpanetta14:22
jtomasekwxy: Hi, thanks for a review, I've been following this example http://git.openstack.org/cgit/openstack/zaqar/tree/samples/javascript/websocket.html and it does not specify subscriber either. looking at the code https://github.com/openstack/zaqar/commit/13ffa248a19e3b1062f6025fb2dc8f8293abf8b7 I am not much smarter about what subscriber is...14:31
thervejtomasek, IIRC, on websocket subscriber is the connection itself by default14:54
wxyjtomasek: thanks for the link.14:59
thervehttps://github.com/openstack/zaqar/blob/master/zaqar/api/handler.py#L52 there you go15:01
wxyjtomasek: subscriber is the destination where the messages notify to.15:01
wxytherve: Thanks. I missed it.15:01
therveThat's somewhat a trick15:02
*** rcernin has quit IRC15:17
*** itisha has joined #openstack-zaqar15:20
*** GB21 has joined #openstack-zaqar15:45
jtomasektherve: this is how I can reproduce my issue: http://paste.openstack.org/show/508707/15:54
jtomasektherve: 2 clients, create 11 subscriptions by running and closing the first script. check subscriptions count by 'openstack subscription list tripleo --limit 20'15:55
jtomasektherve: then run the first script and second script to send a message. the first one will not receive any message15:56
jtomasektherve: if you delete the subscriptions to get less than 11 subscriptions, the runng clients will immediately start receiving incoming messages again15:57
*** GB21 has quit IRC16:01
*** tesseract has quit IRC16:13
*** flwang1 has joined #openstack-zaqar16:58
flwang1flaper87: around?17:09
thervejtomasek, Cool, will have a look18:20
*** ametts has quit IRC18:27
flwang1therve: can you show me the the chat log between jtomasek and you?18:27
flwang1i'm working on the issue reported by jtomasek yesterday18:27
jtomasek<jtomasek> therve: this is how I can reproduce my issue: http://paste.openstack.org/show/508707/18:27
jtomasek<jtomasek> therve: 2 clients, create 11 subscriptions by running and closing the first script. check subscriptions count by 'openstack subscription list tripleo --limit 20'18:27
jtomasek<jtomasek> therve: then run the first script and second script to send a message. the first one will not receive any message18:27
jtomasek<jtomasek> therve: if you delete the subscriptions to get less than 11 subscriptions, the runng clients will immediately start receiving incoming messages again18:27
therveflwang, http://eavesdrop.openstack.org/irclogs/%23openstack-zaqar/latest.log.html18:28
flwang1cool, thanks18:28
jtomasekflwang, therve: sorry that script is not very nice, but does the job18:29
flwang1jtomasek: no problem. so what's the zaqar version you're using?18:30
flwang1master?18:30
flwang1jtomasek: and can you reproduce it with wsgi?18:31
jtomasekflwang: openstack-zaqar package from here: http://trunk.rdoproject.org/centos7/current-passed-ci/18:32
jtomasekflwang: not sure if it is helpful18:32
jtomasekflwang1: I am not that knowlegeable about zaqar, so I am not sure how to reproduce it using wsgi18:33
*** Ravikiran_K has quit IRC18:34
flwang1jtomasek: ok, it's master branch18:35
*** Eva-i has quit IRC18:36
flwang1jtomasek: so you start two processes of your script http://paste.openstack.org/show/508707/18:36
*** Eva-i has joined #openstack-zaqar18:36
*** Eva-i has quit IRC18:36
flwang1with ws_test.py, you created 11 subscriptions18:37
flwang1then stop it18:37
*** Eva-i has joined #openstack-zaqar18:37
*** Eva-i has quit IRC18:37
*** Eva-i has joined #openstack-zaqar18:37
jtomasekflwang: I've updated the scripts here: http://paste.openstack.org/show/508744/18:38
jtomasekflwang: the first script creates the connection and subscription 10 times and closes the connection every time. then it connects (11th time) and stays connected18:39
jtomasekflwang: the second script then just sends the message to the same queue18:39
jtomasekflwang: the first script should receive that message, but does not18:39
flwang1and it works if the subscription's  number < 11?18:40
jtomasekyes18:40
*** ametts has joined #openstack-zaqar18:40
flwang1jtomasek: cool, then it should be related to the max workers, but i don't understand why18:41
jtomasekif some of those stays connected (e.g. I have my webclient as one of the first 10 subscriptions, then it receives the message fine, just not the 11 and up subscriptor18:41
flwang1jtomasek: would you mind opening a bug to track it?18:41
jtomasekflwang: I can, but I've tried to lowe the number of max workers to 5 and the behavior was the same18:41
jtomasek11th was not workning18:41
flwang1ha, interesting18:42
jtomasekflwang: if you then remove some subscription using 'openstack subscription delete tripleo <subs_id> then the originally eleventh starts working again18:42
jtomasekflwang: problem is that the subscription is not removed when the connection is closed. there is no way I can do that on browser refresh. Imo it needs to be done on server side - detect that connection has closed and remove that subscription too18:43
jtomasekflwang: or maybe there is more sofisticated solution, I am not sure18:43
jtomasekflwang: interesting discussion on the topic is here http://stackoverflow.com/questions/4812686/closing-websocket-correctly-html5-javascript18:44
jtomasekflwang: I'll open the bug tomorrow morning18:46
flwang1jtomasek: yep, i see. that's a known issue. the subscription can't be deleted after refresh18:46
flwang1but i don't understand why the old connection will impact the others18:46
flwang1jtomasek: is it possible to create a subscription with a known subscriber in your case?18:47
jtomasekflwang: I probably don't fully understand the subscriber concept, but if you tell me what would be the subscriber in my code, I can try to pass it to queue_subscribe message https://review.openstack.org/#/c/319070/9/src/js/services/ZaqarWebSocketService.js18:49
flwang1jtomasek: generally, subscriber could be a url18:50
jtomasekflwang1: oh, I see it now it is in the subscription list18:53
jtomasekflwang1: is this going to change anything? I'll still need to generate unique subscriber for every browser refresh18:53
flwang1jtomasek: yes, if you don't specify the subscriber in websocket, zaqar will use the default connection string/url18:54
flwang1jtomasek: yep, but that means you know what's the subscriber and then in close() you can delete the subscription18:54
jtomasekflwang1: because if I wouldn't then if user opened the web client twice in 2 tabs for example, then the subscriber would be the same... hmm I wonder if it is a problem or not18:54
flwang1though we still need to figure out why the 11th doesn't work18:54
flwang1jtomasek: i can see your point18:55
jtomasekflwang1: I see18:55
flwang1ok, we  still need to figure out what's the magic 10 or 1118:55
jtomasekflwang1: I'll try the subscriber workaround in the meantime18:56
flwang1maybe it's a default max connection number of the websocket lib using in zaqar18:56
flwang1jtomasek: keep in touch18:56
jtomasekflwang1: maybe, but the problem must come from the subscriptions, because I am sure that the connection count is less than 1018:56
jtomasekflwang1: something probably thinks that the old connections are still active because their subscriptions still exist18:57
jtomasek:)18:57
flwang1jtomasek: oh, so you mean even the coonnection's number less than 10, you will still see the issue?18:57
jtomasekflwang: see the script. I open, subscribe and close 10 connections. I expect that the result is 0 connections and 10 obsolete but still existent subscriptions18:58
jtomasekflwang: then I create 11th connection and subscription and leave that open18:59
jtomasekflwang: then I send a message. the only running connected client (11th subscriber) never receives the message18:59
flwang1jtomasek: i see19:00
jtomasekwhen subscriptions number lower (they timeout or are removed manually), then the 11th subscriber will start receiving messages19:01
jtomasekflwang: so it seems that only 10 subscriptions per queue are possible. that is fine, but the subscriptions without connection need to get removed because otherwise they block the communication19:02
flwang1jtomasek: so if there are more than 11 subscriptions and all of them are active, we will see run into problems19:03
flwang1that said, we still need to figure out what's the root cause, and yes, we need to detect the websocket close request can remove the subscription on server side19:03
jtomasekflwang: if by active you mean existent then yes (with or without connection)19:04
jtomaseksounds good, thanks! I'll try to cover it in bug report tomorrow19:05
flwang1jtomasek: cool, thank you19:07
flwang1jtomasek: btw, i'm opening the bug19:07
jtomasekflwang: even better19:07
flwang1and will add you as subscriber, feel free add comments19:07
jtomasekok19:07
flwang1jtomasek: thank you19:08
openstackgerritFei Long Wang proposed openstack/zaqar: Add release notes  https://review.openstack.org/32670419:11
therveflwang1, jtomasek https://github.com/openstack/zaqar/blob/master/zaqar/storage/base.py#L3719:18
therveSadness19:18
jtomasekdang19:19
flwang1therve: why will it impact?19:19
therveflwang1, Because that's the default limit when we do a list19:20
jtomasekhm, isn't it just for listing the subscriptions?19:20
flwang1therve: ah19:20
flwang1i see19:20
therveYeah we list them when we notify a queue19:20
thervehttps://github.com/openstack/zaqar/blob/master/zaqar/notification/notifier.py#L4319:20
jtomasekaha19:21
flwang1https://github.com/openstack/zaqar/blob/master/zaqar/notification/notifier.py#L4319:21
flwang1jtomasek: you will get a quick fix19:21
flwang1therve: thanks for the heads up19:21
flwang1but yep, we still need to fix the subscription delete issue19:22
flwang1when the connection closed19:22
jtomasekflwang1: max limit is 20, right? that will help just a bit19:22
jtomasekyeah19:22
flwang1jtomasek: we will provide a way to get all19:22
flwang1for sure :)19:22
jtomasekflwang: ah, cool19:22
jtomasek:)19:22
jtomasekthank you both!19:24
flwang1jtomasek: no problem, shame for the bug :D19:24
* flwang1 is thinking to nominate therve as Zaqar core again19:25
flwang1or a Honorary Zaqar Core degree to therve19:26
flwang1if he refused :)19:26
openstackgerritThomas Herve proposed openstack/zaqar: WIP: Fix subscription limit per queue  https://review.openstack.org/32671019:28
therveI wouldn't :)19:28
thervejtomasek, ^^^ I think that should do the trick19:29
jtomasektherve: thanks, will test it as first thing tomorrow19:30
flwang1therve: i prefer to use limit=019:35
flwang1therve: would you mind me posting patch set on your patch?19:35
therveflwang1, Kinda? :)19:35
therveI mean if you're going to put limit=0 you can recreate one patch19:36
flwang1therve: ok, i will add comments :D19:36
flwang1i'm investigating if redis respect limit=0 and i'm sure it's not :)19:37
flwang1with that, you may need a small change in redis driver https://github.com/openstack/zaqar/blob/master/zaqar/storage/redis/subscriptions.py#L7019:39
flwang1and we need a follow up patch to support this for all the other resources19:39
openstackgerritThai Tran proposed openstack/zaqar-ui: Ability to add and remove subscriptions  https://review.openstack.org/31185219:41
therveIf that's the case, that reinforces me into thinking my approach is better :)19:50
*** tqtran has joined #openstack-zaqar19:51
flwang1hmm... limit=0 maybe not the good idea19:56
flwang1if there are too much subscriptions19:56
flwang1maybe a bigger limit per page19:56
therveThat just pushes the problem though19:57
flwang1therve: ok, are you going to add an unit test today? i would like to merge it asap19:58
therveNot tonight no19:58
flwang1therve: mind me adding it?19:59
therveNot at all :)19:59
flwang1therve: awesome19:59
flwang1i will send the core nomination letter tomorrow19:59
openstackgerritMerged openstack/zaqar-ui: Fix typo  https://review.openstack.org/32618820:11
openstackgerritMerged openstack/python-zaqarclient: Add update queue function in v2  https://review.openstack.org/29436820:17
*** mpanetta has quit IRC21:50
openstackgerritMerged openstack/zaqar-ui: Imported Translations from Zanata  https://review.openstack.org/32626422:15
*** itisha has quit IRC22:30
*** ametts has quit IRC22:30

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!