Thursday, 2018-11-08

*** dhellmann has joined #openstack-meeting-400:07
*** Liang__ has joined #openstack-meeting-400:27
*** letrangg has joined #openstack-meeting-400:47
*** salmankhan has quit IRC00:48
*** Liang__ is now known as LiangFang01:03
*** dangtrinhnt_x has joined #openstack-meeting-401:12
*** bobh has joined #openstack-meeting-401:12
*** bobh has quit IRC01:28
*** mjturek has quit IRC01:43
*** yamamoto has joined #openstack-meeting-401:45
*** dangtrinhnt_x has quit IRC01:46
*** diablo_rojo has quit IRC01:53
*** hongbin has joined #openstack-meeting-402:13
*** markvoelker has joined #openstack-meeting-402:45
*** markvoelker has quit IRC02:45
*** markvoelker has joined #openstack-meeting-402:45
*** yamamoto has quit IRC02:49
*** bobh has joined #openstack-meeting-402:49
*** markvoelker has quit IRC02:50
*** markvoelker has joined #openstack-meeting-402:55
*** fungi has quit IRC03:06
*** fungi has joined #openstack-meeting-403:09
*** fungi has quit IRC03:10
*** iyamahat has quit IRC03:15
*** yamahata has quit IRC03:16
*** mrhillsman has joined #openstack-meeting-403:19
*** fungi has joined #openstack-meeting-403:40
*** fungi has quit IRC03:41
*** letrangg has quit IRC03:43
*** fungi has joined #openstack-meeting-403:45
*** bobh has quit IRC03:55
*** bobh has joined #openstack-meeting-404:08
*** bobh has quit IRC04:10
*** sambetts|afk has quit IRC04:11
*** sambetts_ has joined #openstack-meeting-404:18
*** janki has joined #openstack-meeting-404:36
*** radeks_ has joined #openstack-meeting-404:39
*** iyamahat has joined #openstack-meeting-404:48
*** yamahata has joined #openstack-meeting-405:27
*** yamamoto has joined #openstack-meeting-405:45
*** hongbin has quit IRC06:02
*** yboaron has joined #openstack-meeting-406:17
*** diablo_rojo has joined #openstack-meeting-406:20
*** pcaruana has joined #openstack-meeting-407:34
*** slaweq has joined #openstack-meeting-407:39
*** LiangFang has quit IRC07:43
*** Liang__ has joined #openstack-meeting-407:44
*** Liang__ is now known as LiangFang07:44
*** yamamoto has quit IRC07:46
*** yamamoto has joined #openstack-meeting-407:46
*** yamamoto has quit IRC07:47
*** bnemec has quit IRC08:00
*** diablo_rojo has quit IRC08:08
*** iyamahat has quit IRC08:13
*** bnemec has joined #openstack-meeting-408:26
*** Luzi has joined #openstack-meeting-408:30
*** yamamoto has joined #openstack-meeting-408:34
*** yamamoto has quit IRC08:40
*** yamamoto has joined #openstack-meeting-408:46
*** yamamoto has quit IRC08:47
*** yamamoto has joined #openstack-meeting-408:49
*** radeks has joined #openstack-meeting-409:11
*** radeks_ has quit IRC09:12
*** yamamoto has quit IRC09:13
*** iyamahat has joined #openstack-meeting-409:45
*** CrayZee has joined #openstack-meeting-409:48
*** CrayZee is now known as Guest3121209:48
*** Guest31212 has quit IRC09:49
*** e0ne has joined #openstack-meeting-409:50
*** snapiri- has joined #openstack-meeting-409:52
*** yamamoto has joined #openstack-meeting-409:54
*** LiangFang has quit IRC09:56
*** CrayZee- has joined #openstack-meeting-409:59
*** CrayZee- has quit IRC09:59
*** yamamoto has quit IRC10:00
*** yamamoto has joined #openstack-meeting-410:04
*** snapiri- has quit IRC10:19
*** snapiri- has joined #openstack-meeting-410:20
*** iyamahat has quit IRC10:23
*** salmankhan has joined #openstack-meeting-410:26
*** CrayZee has joined #openstack-meeting-410:29
*** snapiri- has quit IRC10:32
*** yamahata has quit IRC10:42
*** CrayZee_ has joined #openstack-meeting-411:00
*** CrayZee has quit IRC11:02
*** yamamoto has quit IRC11:10
*** yamamoto has joined #openstack-meeting-411:14
*** CrayZee_ has quit IRC11:31
*** Luzi has quit IRC12:05
*** pbourke has quit IRC12:11
*** pbourke has joined #openstack-meeting-412:11
*** Luzi has joined #openstack-meeting-412:20
*** janki has quit IRC12:20
*** seajay has joined #openstack-meeting-413:00
*** e0ne has quit IRC13:03
*** yamamoto has quit IRC13:07
*** yamamoto has joined #openstack-meeting-413:09
*** yamamoto has quit IRC13:14
*** yboaron has quit IRC13:28
*** janki has joined #openstack-meeting-413:29
*** e0ne has joined #openstack-meeting-413:37
*** rosmaita has joined #openstack-meeting-413:50
*** jokke_ has joined #openstack-meeting-413:57
*** yamamoto has joined #openstack-meeting-413:58
jokke_#startmeeting glance14:00
openstackMeeting started Thu Nov  8 14:00:26 2018 UTC and is due to finish in 60 minutes.  The chair is jokke_. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: glance)"14:00
openstackThe meeting name has been set to 'glance'14:00
jokke_#topic roll-call14:00
*** openstack changes topic to "roll-call (Meeting topic: glance)"14:00
jokke_o/14:00
rosmaitao/14:00
jokke_#link https://etherpad.openstack.org/p/glance-team-meeting-agenda14:01
jokke_Morning rosmaita14:01
rosmaitagood afternoon to you!14:01
*** pbourke has quit IRC14:02
*** salmankhan has quit IRC14:02
*** bnemec has quit IRC14:02
*** sambetts_ has quit IRC14:02
*** irenab has quit IRC14:02
*** tonyb has quit IRC14:02
*** dmellado has quit IRC14:02
*** gibi has quit IRC14:02
jokke_lets give couple of minutes to see who else is joining us14:02
rosmaitasounds good, full agenda, so we should have some more people soon14:02
*** yamamoto has quit IRC14:03
*** dmellado has joined #openstack-meeting-414:04
*** sambetts_ has joined #openstack-meeting-414:04
jokke_ok14:05
jokke_#topic updates14:05
*** openstack changes topic to "updates (Meeting topic: glance)"14:05
jokke_So Berlin Summit is in next week, no meeting14:05
rosmaitaok14:05
jokke_Spec deadline is 2 weeks after summit14:05
jokke_so 3 weeks to go to get them specs in14:05
rosmaitajokke_: that reminds me, did you ever send out the operator survey about changing default image visibility?14:06
jokke_FECK14:06
jokke_It's going out today!14:06
*** salmankhan has joined #openstack-meeting-414:06
rosmaitai was thinking that will require a spec lite14:06
rosmaitaok, cool14:06
*** tonyb has joined #openstack-meeting-414:07
jokke_totally forgot that14:07
*** gibi has joined #openstack-meeting-414:07
jokke_yeah14:07
rosmaitaalso, procedural question14:07
jokke_shoot14:07
rosmaitafor deprecated config opts, do we need a spec-lite to remove them?14:07
*** bnemec has joined #openstack-meeting-414:07
rosmaitai'm thinking of owner_is_tenant14:07
jokke_nah, if they are deprecated we clean them out and have reno for them14:08
rosmaitaok, that gives us some flexibility14:08
jokke_nice, something else on this?14:08
rosmaitanope, other than maybe i'll take an action item to make a list of deprecated opts that haven't been removed14:09
jokke_sounds good14:09
rosmaitacould be some low-hanging fruit14:09
jokke_#topic periodic jobs14:09
*** openstack changes topic to "periodic jobs (Meeting topic: glance)"14:09
rosmaita#action rosmaita compile list of deprecated options14:09
rosmaitaall green!14:09
jokke_I see note on the etherpad that everything is green :)14:09
rosmaitaexcept occasionally the keystone tips functional py3514:10
rosmaitahttp://zuul.openstack.org/builds?project=openstack%2Fglance&pipeline=periodic&job_name=glance-tox-functional-py35-keystone-tips14:10
jokke_yah, have you had any eye on those what times out?14:10
rosmaitano, i haven't looked14:10
jokke_kk14:10
rosmaitafor a week it looked like abhishek's change had fixed it14:10
rosmaitaand then it failed again yesterday14:10
jokke_yeah14:10
jokke_might be something us related, might be something test framework related, might be something infra related. Lets keep eye on them14:11
rosmaitayep14:11
rosmaitathat's all on periodic jobs14:11
rosmaitawe should wish Happy Diwali to everyone not here, though14:11
jokke_#topic smuggling validation data on location add/replace14:11
*** openstack changes topic to "smuggling validation data on location add/replace (Meeting topic: glance)"14:11
jokke_Soo I saw your review there and went ahead and merged the spec14:12
rosmaitagreat14:12
jokke_it obviously hit merge conflict but I think it finally merged this morn14:12
rosmaitaslightly off topic, the image_cache uses checksum for download validation14:13
rosmaitai don't know if we want to take a performance hit on that to use multihash?14:13
jokke_I think we do14:14
rosmaitaok, i will file a bug14:14
jokke_specially as it seems that the md5 is running away14:14
rosmaitawell, that's not completely clear14:14
rosmaitait's ok for non-security uses14:15
rosmaitalike making sure no bit flips during transfer14:15
jokke_yeah, but I rather be prepared for the environments where it's not available14:15
rosmaitai am still trying to get a timeline on that14:15
rosmaitayeah, we should probably assume not available14:16
*** bobh has joined #openstack-meeting-414:16
jokke_cool ... so this is not our first priority issue, just something we want to keep track of and be ready to act if needed14:16
rosmaita#action rosmaita file glance bug about image cache does not use multihash to verify download14:16
jokke_another good low hanging fruit for someone who suddenly has afternoon of free time :D14:16
rosmaitayeah, there's some example code in glanceclient14:17
rosmaitai'll try to carve out some time to test out imacdonn's patches, i have an S-1 devstack up14:18
jokke_same here, I need to just reroll my containers running my devstacks14:19
rosmaitai need to get with the containers program, i am still using VMs14:19
jokke_I just run lxc bare and simple :D14:20
jokke_ok lets move on14:20
jokke_#HTTP header encoding14:20
jokke_#topic HTTP header encoding14:20
*** openstack changes topic to "HTTP header encoding (Meeting topic: glance)"14:20
rosmaitai guess this is me14:20
rosmaitawell, it's my fault, anyway14:20
jokke_:o14:21
*** LiangFang has joined #openstack-meeting-414:21
rosmaitawe had that issue back in early rocky where some update to something broke something, and the glanceclient tests were breaking something, so the problem was sending unicode in headers which isn't strictly allowed, it's supposed to be 7bit ascii only14:22
jokke_yes, I recall that14:22
rosmaitaso i put up a change to use urlencoding to get unicode->7 bit ascii, and then sent that over in glanceclient headers14:22
rosmaitaproblem is, urlencoding may be a bit too much14:23
rosmaitait encodes some chars c where ord(c) < 128, which you don't need to do14:23
jokke_ok14:23
rosmaitabut the standards are a bit vague on how this should be done14:23
rosmaitaanyway, we are gradually finding a case here or there that this change breaks14:24
rosmaitai think wxy found sometimes the token was getting corrupted and fixed that14:24
jokke_I'm pretty sure, we're not the only ones dealing with this. Is there any safe-decode libs to just handle it?14:24
rosmaitaand ian found that the hostname part of urls are being hosed14:24
rosmaitai haven't found anything14:24
*** LiangFang is now known as LiangF14:25
rosmaitathe lib we are using allows you to specify a set of safe chars14:25
rosmaitathat's what wxy and ian do14:25
*** LiangF is now known as LiangFang14:25
rosmaitabut i don;'t know of a list anywhere14:25
rosmaitaanyway, here is the actual point14:25
rosmaitai was holding off on ian's patch until we figure out a thorough way to fix this14:25
rosmaitabut no one seems to have time ATM to tackle that14:26
rosmaitaso i have +2'd his patch to at least get this particular case he's running into fixed14:26
jokke_can we just tell the lib "do not touch anything ord(c) < 128?14:26
jokke_s/?/"//14:26
jokke_2?14:27
jokke_"?14:27
rosmaitai think we could, by passing it a list of 127 chars!14:27
jokke_:P14:27
jokke_ohhh :|14:27
rosmaitayeah, but my idea was to scan the string first to see if any chars are out of range before doing anything14:27
rosmaitaprobably the same complexity14:27
rosmaitaproblem is, we still don't know what to do when we have a string with out-fo-range chars14:28
jokke_Is there any reason why we shouldn't just reject it?14:28
rosmaitayeah, we need to support unicode14:29
rosmaitait's glancelient that had the tests that were using unicode, so i figured it was important to us at some point14:30
*** janki has quit IRC14:30
rosmaitai think i should take an action item to write up a new bug for this and be clear about exactly what the problem is14:30
jokke_Well it WAS at least with v114:30
rosmaitathat's true14:31
jokke_where all tag and property stuff happened over headers14:31
rosmaitaso i think we should either (a) find a library that handles this properly (there must be one if it's a real problem) or (b) take your string-rejection strategy14:32
jokke_now we are getting back to the same problem again14:32
rosmaitathe codec gives a nice exception if the string is out of range14:32
jokke_as for example the store IDs for where the image will be stored goes over headers again14:32
rosmaitaso we wont' need to scan or anything14:32
rosmaitaoh yeah14:33
rosmaitathat is still experimental14:33
rosmaitawe could require store IDs to be 7bit ascii14:33
rosmaitathe descriptions can be unicode14:33
jokke_yeah, that's a bit of bandaid but I guess so14:33
jokke_I think we need to keep this hard on mind before dumping any more crap to the headers14:34
jokke_and start being more strict that the stuff needs to happen before14:34
jokke_and use the json based calls14:34
rosmaita#action rosmaita propose patch to multi-backend config opts saying store ID must be 7-bit ASCII14:34
rosmaitayeah, problem is that it makes the actions extremely un-atomic14:35
rosmaitai guess we could do multipart uploads14:35
rosmaitajson+data14:35
jokke_jayy for HTTP based APIs ;)14:36
jokke_I see only Imperial fecktons of ways that would go so badly wrong :D14:36
rosmaitayeah, no kidding14:36
LiangFango/14:38
jokke_ok moving on14:40
jokke_#topic list --include-stores14:40
*** openstack changes topic to "list --include-stores (Meeting topic: glance)"14:40
rosmaitaanyway, back to the point, are you ok with imacdonn's quick-fix patch?14:40
rosmaitaoops, sorry about the slow typing14:40
jokke_#undo14:40
openstackRemoving item from minutes: #topic list --include-stores14:40
jokke_I will need to review the patch14:41
rosmaitasounds good, just wanted to verify you are not opposed in principle14:41
jokke_I'll put it on my list to do before heading to berlin14:41
rosmaitaty14:41
rosmaitai'm done14:41
jokke_#topic list --include-stores14:41
*** openstack changes topic to "list --include-stores (Meeting topic: glance)"14:41
jokke_there we go14:41
LiangFanghi Jokke14:41
jokke_The spec was also merged this morn14:42
LiangFangmy turn now?:)14:42
rosmaitathis is also on my list to test out14:42
rosmaitasorry LiangFang, the floor is yours14:43
LiangFangOK..14:43
LiangFangsorry14:43
rosmaitano my fault, i interrupted14:44
LiangFangmy English is not good, now is my topic?14:44
jokke_LiangFang: yes14:45
LiangFangOK..14:45
jokke_that's what we are talking about14:45
LiangFangsorry today I joined late14:45
LiangFangI have implemented the code14:45
rosmaita#link https://review.openstack.org/#/c/605014/14:46
LiangFangsomeone gave comment to add UT, and I have added14:46
LiangFanghope you can help to take a look about this14:46
LiangFanganother thing is that, I have about 10 patch to submit14:47
LiangFangso in the following days, I will submit one by one14:47
LiangFangsorry to bother you14:47
rosmaitano bother, but remember that next week is the summit, so things may be a bit slow14:48
LiangFangOk14:48
LiangFangsome colleague will join summit there14:48
LiangFangbut the network team14:49
LiangFangI'm the storage team14:49
rosmaitaso you are doing cinder and glance?14:49
LiangFangyes14:50
LiangFangstarlingx glance and cinder14:50
rosmaitaok14:51
LiangFangso I also have lots of cinder patches to submit14:51
jokke_ok lets continue on open discussion14:52
jokke_#topic open discussion14:52
*** openstack changes topic to "open discussion (Meeting topic: glance)"14:52
jokke_Anything else we run out of time?14:52
jokke_*before14:53
rosmaitayeah, just want to mention that gorka has a cinder patch up about greenthread starvation caused by the image-signature-verification code in cinder's image utils14:53
rosmaitai suspect we may be subject to a similar situation with the glance_store code14:54
jokke_oh, how did that happen?14:54
LiangFangglance support image cache, right?14:54
rosmaitawell, it depends on how much stuff you have running on the node14:54
rosmaitacinder uses rabbitmq for the scheduler and volume service interactions14:54
jokke_LiangFang: yes we have caching that can be enabled on the API nodes14:55
rosmaitathat's what was detecting starvation as service being down14:55
*** hongbin has joined #openstack-meeting-414:55
rosmaitaso we probably don't see that, but could see it in hyperconverged edge nodes14:55
rosmaitafound the link14:56
rosmaita#link https://review.openstack.org/#/c/615934/14:56
jokke_rosmaita: iirc our greenpool is by default 8000 per process ... I'd be surprised if that is the bottle neck14:56
rosmaitagorka has an interesting comment in there about greenthread and native thread interactions14:56
rosmaitayeah, i wonder why they're seeing this and we're not14:57
LiangFangjokke_: thanks14:57
jokke_rosmaita: likely because we do stream the data through all our validation rather than cache it and run it through the validators and what not14:59
rosmaitayeah, that makes sense14:59
jokke_like only place I really see us hitting heavily with these issue would be using fs store (which we fortunately do not support) with caching and loads of concurrent imports all going at the same time15:01
rosmaitaok, well that's a relief15:02
jokke_yeah, we rarely do blocking io15:03
jokke_most of our stuff is throttled by line speeds and buffers rather than having myriads of blocking interrupts15:04
*** mjturek has joined #openstack-meeting-415:04
rosmaitahooray for glance!15:04
jokke_ok, sorry, we're out of time15:04
jokke_#endmeeting15:04
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:04
openstackMeeting ended Thu Nov  8 15:04:48 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:04
openstackMinutes:        http://eavesdrop.openstack.org/meetings/glance/2018/glance.2018-11-08-14.00.html15:04
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/glance/2018/glance.2018-11-08-14.00.txt15:04
openstackLog:            http://eavesdrop.openstack.org/meetings/glance/2018/glance.2018-11-08-14.00.log.html15:04
*** hamzy has quit IRC15:36
*** yamahata has joined #openstack-meeting-415:45
*** pcaruana has quit IRC16:14
*** dklyle has quit IRC16:17
*** JamesBenson has joined #openstack-meeting-416:20
*** hamzy has joined #openstack-meeting-416:22
*** Luzi has quit IRC16:22
*** dklyle has joined #openstack-meeting-416:23
*** yboaron has joined #openstack-meeting-416:29
*** LiangFang has quit IRC16:32
*** yamamoto has joined #openstack-meeting-416:40
*** slaweq has quit IRC16:40
*** yamamoto has quit IRC16:44
*** e0ne has quit IRC16:48
*** iyamahat has joined #openstack-meeting-416:55
*** slaweq has joined #openstack-meeting-416:58
*** irclogbot_1 has quit IRC17:31
*** iyamahat has quit IRC17:39
*** yamahata has quit IRC17:40
*** diablo_rojo has joined #openstack-meeting-417:48
*** liuyulong has quit IRC17:50
*** iyamahat has joined #openstack-meeting-417:55
*** irclogbot_1 has joined #openstack-meeting-418:01
*** mjturek has quit IRC18:04
*** irclogbot_1 has quit IRC18:05
*** pcaruana has joined #openstack-meeting-418:10
*** salmankhan has quit IRC18:11
*** yamahata has joined #openstack-meeting-418:16
*** irclogbot_1 has joined #openstack-meeting-418:26
*** bobh has quit IRC18:29
*** irclogbot_1 has quit IRC18:32
*** irclogbot_1 has joined #openstack-meeting-418:36
*** diablo_rojo has quit IRC19:12
*** diablo_rojo has joined #openstack-meeting-419:15
*** salmankhan has joined #openstack-meeting-419:46
*** salmankhan has quit IRC19:51
*** e0ne has joined #openstack-meeting-420:19
*** e0ne has quit IRC20:22
*** rosmaita has left #openstack-meeting-420:26
*** radeks_ has joined #openstack-meeting-420:31
*** radeks has quit IRC20:34
*** radeks__ has joined #openstack-meeting-420:36
*** radeks_ has quit IRC20:39
*** dklyle has quit IRC20:45
*** dklyle has joined #openstack-meeting-420:45
*** dklyle has quit IRC20:46
*** radeks__ has quit IRC21:06
*** yboaron has quit IRC21:15
*** hamzy has quit IRC21:43
*** diablo_rojo has quit IRC21:50
*** iyamahat_ has joined #openstack-meeting-421:58
*** yamahata__ has joined #openstack-meeting-421:59
*** isq_ has quit IRC22:00
*** iyamahat has quit IRC22:00
*** diablo_rojo has joined #openstack-meeting-422:03
*** JamesBenson has quit IRC23:00
*** JamesBenson has joined #openstack-meeting-423:02
*** JamesBenson has quit IRC23:07
*** slaweq has quit IRC23:15
*** seajay has quit IRC23:32
*** dklyle has joined #openstack-meeting-423:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!