Tuesday, 2015-05-26

anteayaadd it to this file, I do believe: http://git.openstack.org/cgit/openstack/requirements/tree/projects.txt00:01
sdakenice thanks I'll give that a go ;)00:01
sdakemagic of infra :)00:01
anteayaat the very least you will learn where to actully put it, if I am wrong00:01
*** sdake_ has quit IRC00:01
anteayayou are too kind00:01
*** sdake_ has joined #openstack-infra00:02
sdake_anteaya looks like someone already did the job :)00:03
anteayayay, even better00:03
sdake_I haven't seen any requirements sync reviews00:03
anteayawhen did it get added?00:03
sdake_but I have to admit I've been really busy witht kolla and not on top of the magnum review queue00:03
anteayahas anyone edited global requirements since magnum was added?00:04
sdake_april 6th00:04
*** ddieterly has joined #openstack-infra00:04
sdake_ya added april 6th, last commit to global-requirements.txt is may 19th00:04
sdake_maybe I just missed it00:04
sdake_I know dims was doing some work to fix it up00:05
sdake_we had version caps on some of our requirements00:05
sdake_which was no beuno00:05
anteayaI'm not sure what to do to check the situation to ensure the files are accurate for your needs00:05
*** sdake has quit IRC00:06
sdake_anteaya let me rtfm for a bit to double check :)00:07
anteayaI hope you find something of use00:09
sdake_docs are good00:09
anteayathat they are00:10
sdake_we need a check job for requirements00:10
anteayaah ha00:10
sdake_i'll sort that out00:10
anteayasounds good00:10
anteayaI had two ducks in my driveway00:10
anteayawhich is unusual, they usually select the pond00:10
sdake_i used to live in flagstaff, ducks in teh neighboorhood walking around all the time00:11
sdake_in phoenix, no ducks :)00:11
anteayano idea what the attraction was for the wet asphalt00:11
anteayaha ha ha00:11
sdake_free human food00:11
anteayadoves and quail00:11
anteayaperhaps, for those who partake of ducks00:11
sdake_no i mean they eat our leftovers00:11
anteayaShrews: did you end up with any duck in Vancouver?00:11
anteayayes, I have a friend who feeds them and if she is delayed they walk into the garage and tap on the glass door to get her attention00:12
*** dkehn__ has joined #openstack-infra00:14
sdake_looks like our repo is up to date with the global requirements00:14
sdake_so I guess we are good to go iminus the check job :)00:14
anteayathat is a good development00:14
*** dkehn has quit IRC00:14
*** chlong has joined #openstack-infra00:15
*** dkehn_ has quit IRC00:15
*** salv-orlando has quit IRC00:16
*** asettle has joined #openstack-infra00:16
sdake_anteaya looks like we have a check-requirements job done as well00:17
* sdake_ wonders when all this work gets done :)00:17
*** dkehn has joined #openstack-infra00:17
anteayayou are blessed with elves00:17
*** koolhead17 has joined #openstack-infra00:22
*** koolhead17 has quit IRC00:22
*** koolhead17 has joined #openstack-infra00:22
*** baoli has joined #openstack-infra00:23
*** amotoki has joined #openstack-infra00:26
*** baoli has quit IRC00:27
*** sarob has joined #openstack-infra00:28
*** mestery has joined #openstack-infra00:38
*** mestery has quit IRC00:41
*** sarob has quit IRC00:42
*** sarob has joined #openstack-infra00:42
*** mestery has joined #openstack-infra00:43
*** Pacheco has quit IRC00:43
*** mahito has joined #openstack-infra00:47
*** woodster_ has quit IRC00:50
*** zhiwei has joined #openstack-infra00:52
*** freerunner has joined #openstack-infra00:54
*** yamamoto has quit IRC00:58
*** marzif has quit IRC01:00
*** krtaylor has quit IRC01:02
*** ddieterly has quit IRC01:06
*** ddieterly has joined #openstack-infra01:08
*** dimsum__ has joined #openstack-infra01:12
*** krtaylor has joined #openstack-infra01:15
*** dimsum__ has quit IRC01:15
*** dimsum__ has joined #openstack-infra01:15
*** freerunner has quit IRC01:19
openstackgerritBrant Knudson proposed openstack-infra/project-config: keystonemiddleware non-voting bandit job  https://review.openstack.org/18547801:20
*** ericksonsantos has quit IRC01:21
*** julim has joined #openstack-infra01:21
*** julim has quit IRC01:21
*** banix has joined #openstack-infra01:28
openstackgerritBrant Knudson proposed openstack-infra/project-config: Fix branch for keystone bandit job  https://review.openstack.org/18547901:29
*** baoli has joined #openstack-infra01:30
*** sarob has quit IRC01:30
*** ericksonsantos has joined #openstack-infra01:31
*** dimsum__ has quit IRC01:37
*** ivar-lazzaro has joined #openstack-infra01:41
*** dguerri`away has quit IRC01:41
openstackgerritBrant Knudson proposed openstack-infra/project-config: keystonemiddleware non-voting bandit job  https://review.openstack.org/18547801:41
*** tcammann has quit IRC01:42
*** dimsum__ has joined #openstack-infra01:43
*** otter768 has joined #openstack-infra01:45
*** dguerri`away has joined #openstack-infra01:45
*** dguerri`away is now known as dguerri01:45
*** rguillebert has quit IRC01:45
*** tcammann has joined #openstack-infra01:49
*** otter768 has quit IRC01:50
openstackgerritBrant Knudson proposed openstack-infra/project-config: keystoneclient non-voting bandit job  https://review.openstack.org/18548201:52
jamielennoxdhellmann: around?01:53
*** mtanino has quit IRC02:02
*** ayoung has joined #openstack-infra02:03
*** macjack has joined #openstack-infra02:04
*** salv-orlando has joined #openstack-infra02:04
*** zhiwei has quit IRC02:05
*** zhiwei has joined #openstack-infra02:06
*** ajmiller has quit IRC02:07
*** salv-orlando has quit IRC02:11
*** macjack has quit IRC02:18
*** zz_dimtruck is now known as dimtruck02:18
*** samueldmq has quit IRC02:20
*** baoli has quit IRC02:26
*** armax has joined #openstack-infra02:28
*** woodster_ has joined #openstack-infra02:30
*** puranamr has joined #openstack-infra02:31
*** macjack has joined #openstack-infra02:33
*** puranamr has quit IRC02:36
*** yamamoto has joined #openstack-infra02:39
*** dimsum__ has quit IRC02:45
*** dimsum__ has joined #openstack-infra02:46
*** sdake has joined #openstack-infra02:48
*** sdake_ has quit IRC02:50
*** sdake_ has joined #openstack-infra02:50
*** dimsum__ has quit IRC02:51
*** sdake has quit IRC02:54
*** asselin has quit IRC02:56
*** koolhead17 has quit IRC02:57
*** yamahata has joined #openstack-infra03:03
*** asettle has quit IRC03:07
*** notmyname has joined #openstack-infra03:13
*** ddieterly has quit IRC03:15
*** ociuhandu has joined #openstack-infra03:18
*** ayoung is now known as ayoung_ZZZZzzzz_03:19
*** koolhead17 has joined #openstack-infra03:19
*** panda has quit IRC03:21
*** panda has joined #openstack-infra03:21
*** sdake_ has quit IRC03:23
*** MarkAtwood has quit IRC03:23
*** MarkAtwood has joined #openstack-infra03:25
*** davideagnello has joined #openstack-infra03:32
*** Kennan2 has joined #openstack-infra03:33
*** Kennan has quit IRC03:34
*** banix has quit IRC03:36
*** davideagnello has quit IRC03:37
*** ivar-lazzaro has quit IRC03:42
*** unicell has joined #openstack-infra03:42
*** otter768 has joined #openstack-infra03:46
*** otter768 has quit IRC03:50
*** dimtruck is now known as zz_dimtruck04:01
*** dtantsur|afk is now known as dtantsur04:02
*** asettle has joined #openstack-infra04:05
*** alexpilotti has joined #openstack-infra04:06
*** yfried|afk is now known as yfried_04:12
*** armax has quit IRC04:12
*** achanda has joined #openstack-infra04:12
*** btully has joined #openstack-infra04:15
*** ddieterly has joined #openstack-infra04:16
*** puranamr has joined #openstack-infra04:16
*** ddieterly has quit IRC04:21
*** puranamr has quit IRC04:21
*** yfried_ has quit IRC04:24
*** links has joined #openstack-infra04:27
*** dboik has joined #openstack-infra04:27
*** sarob has joined #openstack-infra04:28
*** achanda has quit IRC04:30
*** hdd has joined #openstack-infra04:31
*** mohitsharma has joined #openstack-infra04:33
*** sputnik13 has joined #openstack-infra04:33
*** abregman has joined #openstack-infra04:36
*** sks has joined #openstack-infra04:36
*** deepakcs has joined #openstack-infra04:40
*** abregman has quit IRC04:40
*** alexpilotti has quit IRC04:41
*** abregman has joined #openstack-infra04:42
*** jpeeler has joined #openstack-infra04:49
*** jpeeler has quit IRC04:51
*** maishsk_afk has joined #openstack-infra04:52
*** maishsk_afk has quit IRC04:54
*** jpeeler has joined #openstack-infra04:54
*** davideagnello has joined #openstack-infra05:00
*** aswadr has joined #openstack-infra05:05
*** BharatK has joined #openstack-infra05:10
*** _nadya_ has joined #openstack-infra05:10
*** dboik has quit IRC05:14
*** dboik has joined #openstack-infra05:14
*** mohitsharma has quit IRC05:15
*** sarob has quit IRC05:15
*** puranamr has joined #openstack-infra05:15
*** davideagnello has quit IRC05:16
*** puranamr has quit IRC05:20
*** pal has joined #openstack-infra05:28
*** dboik has quit IRC05:30
*** dimsum__ has joined #openstack-infra05:36
*** ParsectiX has joined #openstack-infra05:36
*** dimsum__ has quit IRC05:41
*** e0ne has joined #openstack-infra05:46
*** yfried_ has joined #openstack-infra05:47
*** otter768 has joined #openstack-infra05:47
*** asettle has quit IRC05:50
*** otter768 has quit IRC05:51
openstackgerritIan Wienand proposed openstack-infra/nodepool: Convert to use latest statsd version  https://review.openstack.org/17189005:53
*** e0ne has quit IRC05:53
*** ildikov has quit IRC05:56
*** hdd has quit IRC05:56
palCan anyone tell me what all processes happen at backend when you launch an instance05:57
*** ParsectiX has quit IRC06:07
*** mohitsharma has joined #openstack-infra06:15
*** maishsk_afk has joined #openstack-infra06:16
*** ddieterly has joined #openstack-infra06:17
openstackgerrityolanda.robla proposed openstack-infra/system-config: Start using puppet-cgit to manage cgitrc  https://review.openstack.org/18183506:22
*** ddieterly has quit IRC06:22
*** ociuhandu has quit IRC06:23
*** ildikov has joined #openstack-infra06:24
*** chlong has quit IRC06:30
*** ujuc has joined #openstack-infra06:35
*** fedexo has joined #openstack-infra06:45
*** soren has joined #openstack-infra06:45
*** ZZelle has quit IRC06:48
*** maishsk_afk is now known as maishsk06:52
*** matbu has quit IRC06:58
*** matbu has joined #openstack-infra07:02
*** jtomasek has joined #openstack-infra07:03
*** achanda has joined #openstack-infra07:04
*** k4n0 has joined #openstack-infra07:08
*** cody-somerville has joined #openstack-infra07:10
*** cody-somerville has quit IRC07:10
*** cody-somerville has joined #openstack-infra07:10
*** yfried_ is now known as yfried|afk07:14
*** cody-somerville has quit IRC07:17
*** fijal has quit IRC07:20
*** yfried|afk is now known as yfried_07:20
*** fijal has joined #openstack-infra07:27
*** Ala has joined #openstack-infra07:29
*** btully has quit IRC07:29
*** achanda has quit IRC07:30
*** mohitsha_ has joined #openstack-infra07:33
*** mohitsharma has quit IRC07:33
*** jlanoux has joined #openstack-infra07:35
*** arxcruz has joined #openstack-infra07:35
*** HeOS has quit IRC07:38
*** jistr has joined #openstack-infra07:39
*** e0ne has joined #openstack-infra07:39
*** e0ne is now known as e0ne_07:40
*** _nadya_ has joined #openstack-infra07:40
*** markus_z has joined #openstack-infra07:44
*** otter768 has joined #openstack-infra07:47
*** achanda has joined #openstack-infra07:48
*** e0ne_ is now known as e0ne07:49
*** otter768 has quit IRC07:52
*** rcarrillocruz has quit IRC07:58
*** shardy has joined #openstack-infra07:58
*** ParsectiX has joined #openstack-infra07:58
openstackgerritMerged openstack/requirements: Add flag to include hacking  https://review.openstack.org/17049207:59
*** samuelBartel has joined #openstack-infra07:59
*** mpavone has joined #openstack-infra08:01
*** devvesa has joined #openstack-infra08:04
*** dizquierdo has joined #openstack-infra08:04
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Revert "Pin puppet-memcached"  https://review.openstack.org/18552408:06
electricalpal: you mean of openstack it self? I think you'll need the #openstack channel for that. this is mainly the openstack infra part08:08
openstackgerritFabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests  https://review.openstack.org/17597008:10
*** gema_ is now known as gema08:10
*** maishsk is now known as maishsk_afk08:12
*** fedexo has quit IRC08:13
*** Hal has joined #openstack-infra08:15
*** Hal is now known as Guest8893208:15
*** sarob has joined #openstack-infra08:15
*** ttx has quit IRC08:18
*** ttx has joined #openstack-infra08:19
*** ddieterly has joined #openstack-infra08:19
openstackgerritThomas Goirand proposed openstack-infra/project-config: Adds the 1st initial package for upstream infra packaging  https://review.openstack.org/18516408:19
*** sarob has quit IRC08:20
*** derekh has joined #openstack-infra08:20
*** bkero has quit IRC08:21
*** bkero has joined #openstack-infra08:21
zigoIs there something wrong in this ? https://review.openstack.org/#/c/185164/3/gerritbot/channels.yaml,unified08:22
*** ddieterly has quit IRC08:24
*** hichihara has quit IRC08:24
*** fijal_ has joined #openstack-infra08:25
electricalzigo: looks good to me.08:26
zigoelectrical: Any idea why it fails the zuul test then?08:26
*** fijal has quit IRC08:26
*** ZZelle has joined #openstack-infra08:27
openstackgerritFabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests  https://review.openstack.org/17597008:28
electricalzigo: https://review.openstack.org/#/c/185164/3/zuul/layout.yaml your change there is wrong.08:28
*** mpaolino has joined #openstack-infra08:28
electricalto many indents08:28
zigoNot sure where... :/08:29
electricalthe template key and the list under it should be 2 spaces back.08:30
electricallike the block of 'stackforge/poppy'  below it08:31
*** koolhead17 has quit IRC08:31
electricaland the irc access failure is that the channel is in gerritbot but not in accessbot.08:31
openstackgerritThomas Goirand proposed openstack-infra/project-config: Adds the 1st initial package for upstream infra packaging  https://review.openstack.org/18516408:31
zigoelectrical: Thanks a lot.08:32
electricalnp :-)08:32
electricalthe irc check will most likely still fail though.08:32
openstackgerritThomas Goirand proposed openstack-infra/project-config: Adds the 1st initial package for upstream infra packaging  https://review.openstack.org/18516408:33
*** hichihara has joined #openstack-infra08:33
electricaland should work now with your latest patch set ;-)08:33
*** hichihara has quit IRC08:33
*** hichihara has joined #openstack-infra08:33
*** Ala has quit IRC08:35
*** fhubik has joined #openstack-infra08:35
*** ttx has quit IRC08:35
*** ttx has joined #openstack-infra08:35
electricalwaynr: found the bug. if the  key exists in the puppet namespace of defaults but not in global it doesn't get picked up :-(  ( CC clarkb fungi ) should we consider that a bug or an undocumented feature? :-)08:36
*** fedexo has joined #openstack-infra08:40
*** e0ne is now known as e0ne_08:42
*** e0ne_ is now known as e0ne08:43
*** devvesa has quit IRC08:46
*** aswadr has quit IRC08:48
*** aswadr has joined #openstack-infra08:50
*** mpaolino has quit IRC08:50
*** sputnik13 has quit IRC08:56
*** mohitsha_ has quit IRC08:56
*** mohitsharma has joined #openstack-infra08:56
*** turul has joined #openstack-infra08:56
*** dalgaaf has joined #openstack-infra08:56
*** sputnik13 has joined #openstack-infra08:56
*** turul is now known as afazekas08:56
*** rlandy has joined #openstack-infra08:57
*** mohitsharma has quit IRC08:57
*** _nadya_ has quit IRC08:57
*** redixin has joined #openstack-infra08:59
electricalwaynr: also found a weird bug with setting defaults: to job-group or project. it is moaning it can't find some variables if i don't define them but they are not being applied. looks like the job-template part applying defaults is overriding it.08:59
electricalzigo: still got an error:  #openstack-pkg is not registered with ChanServ.09:00
electricali think that's something you or one of the admins will have to do :-)09:00
*** HeOS has joined #openstack-infra09:00
zigoelectrical: Who should I ask to do that?09:01
electricalzigo: i think any of the channel admins can help you with that. http://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml#n37 <-- list09:02
*** claudiub has joined #openstack-infra09:02
zigoelectrical: Ok, thanks.09:02
zigoclarkb: lifeless: jeblair: fungi: SergeyLukjanov: Can any of you register the #openstack-pkg channel?09:03
*** leopoldj has joined #openstack-infra09:04
*** rcarrillocruz has joined #openstack-infra09:06
*** fijal_ is now known as fijal09:08
*** ildikov has quit IRC09:14
*** fedexo has quit IRC09:15
*** fhubik is now known as fhubik_afk09:17
*** fhubik_afk is now known as fhubik09:17
*** david-t_ has joined #openstack-infra09:18
david-t_Hi infra team, could you have a look on https://review.openstack.org/#/c/185006/ please  in order to approuve my new project creation ? Thanks a lot09:19
*** abregman is now known as abregman|afk09:19
*** ddieterly has joined #openstack-infra09:20
*** ddieterly has quit IRC09:25
*** mcdonagm__ has quit IRC09:27
*** fijal has quit IRC09:29
yfried_AJaeger: could you please review redixin's patch https://review.openstack.org/#/c/185027/309:37
*** teran has quit IRC09:39
*** fijal has joined #openstack-infra09:40
*** ihrachyshka has joined #openstack-infra09:41
*** otter768 has joined #openstack-infra09:48
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Split slow rally job to several small ones  https://review.openstack.org/18502709:49
*** sputnik13 has quit IRC09:49
*** Ala has joined #openstack-infra09:49
*** fhubik is now known as fhubik_afk09:49
*** salv-orlando has joined #openstack-infra09:51
*** otter768 has quit IRC09:53
*** yfried_ is now known as yfried|afk09:57
*** sputnik13 has joined #openstack-infra10:00
*** freerunner has joined #openstack-infra10:00
*** panda is now known as panda|afk10:02
*** Ala has quit IRC10:03
*** yfried|afk is now known as yfried_10:03
*** jamespage_ has joined #openstack-infra10:06
*** e0ne is now known as e0ne_10:08
*** ujuc has quit IRC10:09
*** ildikov has joined #openstack-infra10:09
*** fhubik_afk is now known as fhubik10:16
*** pc_m has joined #openstack-infra10:16
*** Guest88932 has quit IRC10:17
*** e0ne_ has quit IRC10:19
*** rossella_s has joined #openstack-infra10:19
*** abregman|afk is now known as abregman10:20
*** ddieterly has joined #openstack-infra10:20
*** afazekas_ has joined #openstack-infra10:21
*** nikeshm has quit IRC10:23
*** mmedvede has joined #openstack-infra10:25
*** ddieterly has quit IRC10:25
*** _nadya_ has joined #openstack-infra10:28
*** e0ne has joined #openstack-infra10:28
*** jamespage_ has quit IRC10:29
*** dpaterson has joined #openstack-infra10:29
*** mmedvede has quit IRC10:29
*** jlanoux_ has joined #openstack-infra10:29
*** woodster_ has quit IRC10:30
*** jlanoux has quit IRC10:32
*** mahito has quit IRC10:34
*** samueldmq has joined #openstack-infra10:35
*** e0ne is now known as e0ne_10:39
*** marzif has joined #openstack-infra10:39
*** jlanoux has joined #openstack-infra10:41
*** jlanoux_ has quit IRC10:43
openstackgerritEkaterina Chernova proposed openstack-infra/project-config: Make 'gate-murano-devstack-dsvm' non-voting  https://review.openstack.org/18480910:44
*** teran has joined #openstack-infra10:45
*** zhiwei has quit IRC10:48
*** e0ne_ has quit IRC10:49
*** boris-42 has joined #openstack-infra10:49
*** teran has quit IRC10:51
electricalhiya samueldmq10:51
*** Hal has joined #openstack-infra10:55
*** Hal is now known as Guest660910:55
*** yamahata has quit IRC10:59
samueldmqelectrical, hi .. things still look to be moving a little bit slow here (as in other channels)  : )11:00
*** freerunner has quit IRC11:00
electricalsamueldmq: hehe yeah indeed11:00
electricalthe other side ( across the pond ) isn't awake yet11:01
samueldmqelectrical, hmm yes, this is important to notice as well ;)11:01
*** shardy_ has joined #openstack-infra11:02
*** shardy has quit IRC11:02
*** david-t_ has quit IRC11:04
*** teran has joined #openstack-infra11:04
*** _nadya_ has quit IRC11:05
electricalsamueldmq: did you ever work with JJB ? ( Jenkins Job Builder )11:05
*** dguitarbite has joined #openstack-infra11:06
*** nadya has joined #openstack-infra11:06
*** matbu is now known as matbu_afk11:07
*** nadya is now known as Guest4895611:07
samueldmqelectrical, well, I have a change that creates a job template and add it in the experimental job list of some projects11:07
samueldmqelectrical, that's all I know so far11:07
*** shardy_ has quit IRC11:07
*** fijal has quit IRC11:07
electricalah okay :-)11:07
*** yfried_ is now known as yfried|afk11:07
electricalFound a weird bug and still trying to chase it down.11:07
*** shardy has joined #openstack-infra11:08
samueldmqelectrical, what is that JJB ? looks like you know more than me on that :)11:08
*** Guest48956 is now known as _nadya_11:08
*** marcusvrn has joined #openstack-infra11:08
electricalJJB allows you to manage Jenkins jobs via yaml files instead via the web interface or ( even worse ) horrible xml files :-)11:08
samueldmqelectrical, nice, so actually JJB is what allows me to use that devstack-gate.yaml (yaml format) for example, in my change :)11:11
electricalsamueldmq: correct :-)11:11
samueldmqelectrical, cool, I'd be glad to dig a bit more on that with you .. but time is a bit tight here atm11:12
electricalhehe that's fine :-)11:12
*** fijal has joined #openstack-infra11:16
openstackgerritSam Betts proposed openstack-infra/project-config: Add project arecibo to stackforge  https://review.openstack.org/18556211:17
*** mohitsharma has joined #openstack-infra11:21
*** ddieterly has joined #openstack-infra11:21
*** yfried|afk is now known as yfried_11:21
*** yfried_ is now known as yfried11:22
*** sputnik13 has quit IRC11:22
*** ddieterly has quit IRC11:26
*** woodster_ has joined #openstack-infra11:30
*** jistr is now known as jistr|class11:33
*** ldnunes has joined #openstack-infra11:33
*** mohitsharma has quit IRC11:36
*** matbu_afk is now known as matbu11:40
*** mohitsharma has joined #openstack-infra11:42
*** fhubik is now known as fhubik_afk11:44
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repositories for Fuel plugins  https://review.openstack.org/18556611:47
*** freerunner has joined #openstack-infra11:48
*** otter768 has joined #openstack-infra11:49
*** panda|afk is now known as panda11:50
*** yamamoto has quit IRC11:50
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repositories for Fuel plugins  https://review.openstack.org/18556611:51
*** otter768 has quit IRC11:54
*** pal has quit IRC11:55
*** baoli has joined #openstack-infra11:57
*** koolhead17 has joined #openstack-infra11:59
*** e0ne has joined #openstack-infra12:00
*** mohitsharma has quit IRC12:00
*** koolhead17 has quit IRC12:00
*** baoli has quit IRC12:02
*** k4n0 has quit IRC12:03
*** dguerri is now known as dguerri`away12:10
*** yamamoto has joined #openstack-infra12:12
*** arxcruz has quit IRC12:12
*** doug-fish has joined #openstack-infra12:15
*** ekarlso has quit IRC12:16
*** ekarlso has joined #openstack-infra12:16
*** fhubik_afk is now known as fhubik12:17
*** dprince has joined #openstack-infra12:18
*** alla__ has joined #openstack-infra12:18
openstackgerritValeriy Ponomaryov proposed openstack/requirements: Bump ddt to min version 0.7.0  https://review.openstack.org/16616212:18
*** prad has joined #openstack-infra12:21
*** weshay has joined #openstack-infra12:21
*** ddieterly has joined #openstack-infra12:22
*** bswartz has quit IRC12:22
*** mohitsharma has joined #openstack-infra12:22
*** baoli has joined #openstack-infra12:24
*** mohitsharma has quit IRC12:25
*** mohitsharma has joined #openstack-infra12:26
*** mohitsharma has quit IRC12:26
*** ddieterly has quit IRC12:26
*** zz_dimtruck is now known as dimtruck12:29
*** dbite has joined #openstack-infra12:29
*** gordc has joined #openstack-infra12:31
russellblifeless: in case this doesn't show up in your queue, this patch could use your feedback: https://review.openstack.org/#/c/182746/12:32
*** achanda has quit IRC12:33
*** otter768 has joined #openstack-infra12:33
*** otter768 has quit IRC12:34
*** aysyanne has joined #openstack-infra12:35
*** baoli has quit IRC12:36
*** baoli_ has joined #openstack-infra12:38
*** mjturek1 has joined #openstack-infra12:41
*** xyang1 has joined #openstack-infra12:41
*** achanda has joined #openstack-infra12:41
openstackgerritSam Betts proposed openstack-infra/project-config: Add project arecibo to stackforge  https://review.openstack.org/18556212:45
*** sigmavirus24_awa is now known as sigmavirus2412:48
*** arxcruz has joined #openstack-infra12:48
*** dimsum__ has joined #openstack-infra12:49
*** btully has joined #openstack-infra12:49
*** bknudson has joined #openstack-infra12:49
*** deepakcs has quit IRC12:50
*** dguerri`away is now known as dguerri12:50
electricalzaro: i have starred a few changes in Gerrit that may be easy to land in the next version. If you have some time today would like to run through them with you for review.12:51
*** btully has quit IRC12:53
*** dimsum__ is now known as dims12:56
*** jistr|class is now known as jistr13:00
*** barnaby has quit IRC13:02
*** sigmavirus24 is now known as sigmavirus24_awa13:02
*** sks has quit IRC13:02
*** bswartz has joined #openstack-infra13:02
*** sigmavirus24_awa is now known as sigmavirus2413:02
*** achanda has quit IRC13:05
mordredzigo: http://docs.openstack.org/infra/system-config/irc.html#access13:05
electricalmorning mordred13:08
*** weshay is now known as weshay_brb13:08
*** bcourt has joined #openstack-infra13:09
*** e0ne is now known as e0ne_13:10
mordredmorning electrical13:11
*** changbl has quit IRC13:11
electricalHad a good trip back or didn't you go to the summit ?13:11
*** weshay_brb is now known as weshay13:12
*** mpaolino has joined #openstack-infra13:12
*** mpaolino has quit IRC13:12
*** alla__ has quit IRC13:12
*** julim has joined #openstack-infra13:12
*** stevemar has joined #openstack-infra13:14
mordredgood trip back ... and good to be home13:14
*** peristeri has joined #openstack-infra13:15
*** zaneb has joined #openstack-infra13:16
electricalhehe i can imagine.13:16
*** sks has joined #openstack-infra13:16
*** alla__ has joined #openstack-infra13:16
electricalI was away for 2 weeks for our user conference. was nice to be home after that.13:17
fungifyi, i'm not around much today due to errands, but should be here when the meeting block starts at 1900z13:20
*** e0ne_ has quit IRC13:21
electricallaterz fungi13:21
electricalmordred: i found a weird thing in JJB and i did find the issue but not sure if its a bug or an undocumented feature.13:22
*** ddieterly has joined #openstack-infra13:23
mordredprolly both13:24
electricalhehe :p13:24
fungiundocumented bugfeatures13:24
mordredwhat did that show up as fungi?13:24
mordred(my phone has an emoji button ... curious what it does)13:25
electricalin the defaults part. i have a global namespace and puppet namespace.. If i specify something only in the puppet namespace for those specific jobs, it will barf and say it can't find it. if i put an empty version of that var in the global namespace it works like expected..13:25
*** wenlock has joined #openstack-infra13:25
fungimordred: a blank message in my client/terminal13:25
*** wenlock1 has joined #openstack-infra13:26
electricalmordred: i got a smiling_imp image13:26
mordredfungi: how useful!13:26
*** e0ne has joined #openstack-infra13:26
fungilikely a missing glyph in my fontset13:26
mordredelectrical: intetesting13:26
fungii'm waiting to see what the meetbot does with it13:27
fungibut it hasn't flushed yet13:27
fungithere we go. my browser knew what to do with it. odd13:27
electrical1f608 is the id for that emoji ( according to irccloud )13:28
*** ddieterly has quit IRC13:28
*** dkranz has joined #openstack-infra13:28
fungii can't even paste it into my terminal to inspect it with python's unicodedata module13:29
*** links has quit IRC13:29
*** akshai has joined #openstack-infra13:29
*** tiswanso has joined #openstack-infra13:29
mordredelectrical: I'm not sure if your thing is bug or feature, btw13:31
electricalmordred: Nor do i.13:32
electricalcould be due to the way the defaults handling is done.13:32
electricalOh, and defaults can only be specified in a job-template but nowhere else. not sure if that's expected or not.13:32
*** hichihara has quit IRC13:33
fungithere we go... unicodedata.name(u'\U0001f608') says 'SMILING FACE WITH HORNS'13:33
electricalhehe :p13:33
fungibut yeah, trying to print it... ignored by my terminal13:34
*** koolhead17 has joined #openstack-infra13:37
*** mriedem has joined #openstack-infra13:41
*** dims has quit IRC13:45
*** btully has joined #openstack-infra13:45
*** woodster_ has quit IRC13:50
*** ayoung has joined #openstack-infra13:52
*** sdake has joined #openstack-infra13:56
electricalAnyone have time to go through a few changes for JJB before the next release? :-) would be nice to have a few things merged in before a new release is done.13:56
*** bnemec has joined #openstack-infra13:57
*** soren has quit IRC13:58
*** annegentle has joined #openstack-infra13:59
*** devvesa has joined #openstack-infra14:00
*** radez_g0n3 is now known as radez14:01
*** ayoung has quit IRC14:02
*** Kennan has joined #openstack-infra14:03
*** Kennan2 has quit IRC14:04
*** BharatK has quit IRC14:05
*** sks has quit IRC14:05
electricalI guess everyone went back to sleep or breakfast :p lol14:06
*** ddieterly has joined #openstack-infra14:06
*** mleroy1 has joined #openstack-infra14:07
*** mleroy has quit IRC14:08
*** spzala has joined #openstack-infra14:08
*** mattfarina has joined #openstack-infra14:10
*** maishsk_afk has quit IRC14:13
*** deepakcs has joined #openstack-infra14:14
*** spzala has quit IRC14:14
*** spzala has joined #openstack-infra14:15
*** gokrokve has joined #openstack-infra14:15
*** dimtruck is now known as zz_dimtruck14:15
*** freerunner has quit IRC14:16
*** zz_dimtruck is now known as dimtruck14:17
*** sks has joined #openstack-infra14:17
*** ayoung has joined #openstack-infra14:18
*** annegentle has quit IRC14:19
*** annegentle has joined #openstack-infra14:19
openstackgerritAlexis Lee proposed openstack-infra/elastic-recheck: gate-python-ironicclient-python34 #1454051  https://review.openstack.org/18562014:20
*** claudiub has quit IRC14:22
*** sks has quit IRC14:23
*** Bjoern__ has joined #openstack-infra14:24
*** timcline has joined #openstack-infra14:26
*** freerunner has joined #openstack-infra14:28
*** smccully has joined #openstack-infra14:33
*** otter768 has joined #openstack-infra14:34
*** dims has joined #openstack-infra14:35
openstackgerritAlexis Lee proposed openstack-infra/elastic-recheck: gate-python-ironicclient-python34 #1454051  https://review.openstack.org/18562014:35
*** claudiub has joined #openstack-infra14:38
*** annegentle has quit IRC14:38
*** otter768 has quit IRC14:40
*** emagana has joined #openstack-infra14:41
*** e0ne is now known as e0ne_14:41
*** e0ne_ has quit IRC14:41
pabelangertimrc, nibalizer: I wouldn't mind getting into etherpad and / or IRC for some high level dashboards you have going. See what's needed to import logic into grafyaml14:43
*** dguerri is now known as dguerri`away14:44
openstackgerritAlexis Lee proposed openstack-infra/elastic-recheck: gate-heat-python27 #1458399  https://review.openstack.org/18563214:44
KiallFolks, the question of Py3 support in Designate came up today.. My usual answer of "we'll do it once our deps support it" applied ;) But.. It left me curious as to how many of our deps don't support py3 yet.. So, before I go assemble a list for Designate, what versions are we as a community aiming to support.. 2.7 and 3.4 or?14:44
*** achanda has joined #openstack-infra14:45
*** cody-somerville has joined #openstack-infra14:45
*** cody-somerville has quit IRC14:45
*** cody-somerville has joined #openstack-infra14:45
*** mpavone has quit IRC14:47
electricalKiall: those are the 2 i would focus on yeah personally. truth be told. most distro's are still on 2.714:48
mriedemmordred: fungi: clarkb: jeblair: this is a fix for a bug blocking all stable/kilo changes: https://review.openstack.org/#/c/185073/14:48
mriedemcould use some review please14:48
KiallYea, 2.7 + 3.0 from memory can't actually be done without wayy more hassle than is necessary for 2.7+3.3ish.. Curious what Infra is seeing/expecting, as they probably see what every project is doing now by needing to provide the infra for it ;)14:49
*** armax has joined #openstack-infra14:49
electricalKiall: i know ubuntu 14.04 ships Version: 3.4.0-2ubuntu114:49
clarkbmriedem: it would be nice if the commit message explained the problem a bit more14:50
clarkbKiall: yes 2.7 and 3.4 for now14:50
electricalKiall: for Jenkins Job Builder ( tool that infra uses ) they focus on 2.7 and 3.414:50
electricallike clarkb said :p14:50
*** freerunner has quit IRC14:50
clarkbmriedem: but Iam reading the bug14:50
electricalmorning clarkb14:50
*** annegentle has joined #openstack-infra14:50
mriedemclarkb: basically sdague's change merged on 5/18 and logstash shows the spike in grenade failures on stable/kilo changes at the same time, in a keystone/apache change which is what this was14:51
clarkbmriedem: the bug doesn't really say its just a lot of "me too"14:51
mriedemwe are'nt currently running grenade on stable branches for tempest and d-g changes, which are both branchless, which is bad14:51
*** deepakcs has quit IRC14:51
clarkbmriedem: right but why would that affect kilo? the old side (juno) localrc will get the flag, the new side won't because its default14:51
*** freerunner has joined #openstack-infra14:51
mriedemclarkb: http://goo.gl/W8SqEN14:52
mriedemthe original change is specific to stable/kilo branch14:52
dstufftKiall: https://caremad.io/2015/04/a-year-of-pypi-downloads/ fwiw14:52
clarkbno the original change is on a branchless project, so it handle icehouse, and juno special and allows defaults to work in kilo and master14:52
mriedemclarkb: my only guess was a question in the original change https://review.openstack.org/#/c/175531/4/devstack-vm-gate.sh14:52
mriedemmtreinish: do you know specifically why that d-g change doesn't work on stable/kilo ^ ?14:53
Kiallclarkb: thanks, exactly what I wanted to hear :)14:53
*** bhunter71 has joined #openstack-infra14:53
clarkbmriedem: oh I see, the comment about not being able to upgrade is likely at fault, since we disable it in juno but don't do so in kilo14:54
clarkbKiall: also at this point I am not aware of any large openstack deps that pose a python3 problem14:54
KiallFor Designate, we have dnspython, which totally and utterly fails on py34 ;)14:54
clarkbmriedem: maybe you can record that on the bug?14:54
Kiall(Tests don't env get a chance to enumerate, as they have py2 only code as module init level stuff)14:55
mriedemclarkb: yeah i can update the bug14:56
mtreinishmriedem: that bug fix was only on kilo from what I saw, so when the kilo job tries to upgrade from juno to kilo it fails14:56
clarkbKiall: sure but the eventlet is no longer the blocker so now we can actually go through and fix things without that being the big wall14:56
mriedemmtreinish: do you mean https://review.openstack.org/#/c/118740/ ? b/c that's in stable/juno14:56
clarkbmriedem: ya in the old code it always set wsgi off so we could do not wsgi to not wsgi upgrades, with that change it would be not wsig + juno to wsgi + kilo and that fails14:56
*** matbu is now known as matbu_afk14:56
Kiallclarkb: yea, if dnspython is our only gottcha, I could easily stick someone on that for a few days to get it sorted - assuming upstream is friendly and happy to accept etc etc14:56
*** e0ne has joined #openstack-infra14:58
*** sputnik13 has joined #openstack-infra14:58
*** sputnik13 has quit IRC14:59
*** ayoung has quit IRC15:01
Kiallclarkb: Oh, wow. I just spotted this - dnspython and dnspython3 are both on pypi, same version #, same author. Ignoring that being really annoying, do we have any way to pick the right version etc in our tooling?15:01
*** arxcruz has quit IRC15:01
*** hemnafk is now known as hemna15:01
*** mtanino has joined #openstack-infra15:01
electricalKiall: that's annoying indeed. why can't they make 1 package work for both :-(15:01
*** matbu_afk is now known as matbu15:02
KiallI'm going to fire the author an email to ask ;)15:02
*** katyafervent_ is now known as katyafervent_awa15:03
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/"15:04
jeblairclarkb, fungi: i ordered parts to prototype a summit session streaming box; i should be able to start hacking on it next weekend15:05
taronerp, morning all15:05
electricalMorning taron15:06
*** yfried is now known as yfried|afk15:06
*** yfried|afk is now known as yfried15:07
fungijeblair: awesome!15:07
fungihowdy taron15:07
clarkbKiall: there are environment markers that we can use in pip reqs as of latest pip release aiui15:08
*** ayoung has joined #openstack-infra15:08
Kiallclarkb: oh, interesting.. I'll dig though the pip docs.. thanks15:09
Davieyjeblair: ~3 years ago stefanno tried the same using a bunch of rasp' pi's15:10
Davieyjeblair: might be worth finding out why he gave up?15:10
jeblairDaviey: i believe he didn't get around to actually getting started15:11
*** annegentle has quit IRC15:11
clarkbtaron: good morning15:12
taronworking on wrapping my head around go packaging, which probably doesn't require much input but I figured I should express my presence anyway15:12
bswartzwhat would a summit streaming box do exactly?15:12
* bswartz is interested15:12
jeblairbswartz: pipe fishbowl room audio to a conference server to facilitate remote participation15:13
*** mattfarina has quit IRC15:13
bswartzyeah -- for the Manila sessions we had a guy with a laptop, a google hangout, and a USB conference phone15:13
bswartzthe audio quality was HIGHLY variable15:13
bswartzgetting a decent microphone is a challenge15:14
Davieyjeblair: That is quite a blocker15:14
fungiDaviey: yeah, we _discussed_ something similar when we had so many issues with webex in san diego. the asterisk instance at pbx.openstack.org is part of what came out of those discussions but the dedicated hardware in the rooms has as of yet been unaddressed (so we've used laptops)15:14
*** fhubik is now known as fhubik_afk15:14
*** alla__ has quit IRC15:15
*** ParsectiX has quit IRC15:15
jeblairbswartz: yeah, i've ordered 3 mics for testing, since no one seems to have done real profiling of usb boundary mics15:15
fungialso, acoustics and audio engineering are super fun to geek out on ;)15:16
*** banix has joined #openstack-infra15:16
Davieyjeblair / fungi: As comparision, In Ubuntu Summits we found that two way audio was too slow to have a proper conversation and chose to use microphones streaming via icecast and IRC for feedback15:16
clarkbif other cores want to review https://review.openstack.org/#/c/168633/ today, my goal is to get that merged and have nodepool restarted today15:16
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/project-config: Add barbican-dogtag flag  https://review.openstack.org/18518315:16
clarkbthen I can update our ubuntu-trusty image config to upload to rax too15:16
*** dkehn__ is now known as dkehnx15:16
jeblairDaviey: yeah, i recall -- and clarkb seemed to prefer to type into etherpad chat rather than speak.  nibalizer spoke some, and it worked well enough as long as the conversation wasn't too fast-paced.15:17
fungiDaviey: for the most part this has been targeted at adding one or two necessary participants to an otherwise in-person conversation. it seemed to work well enough but i wouldn't want to try to virtualize the entire design summit that way15:17
clarkbya, using the chat window was easier for me15:17
clarkbbut having the audio to follow along was quite helpful15:17
*** woodster_ has joined #openstack-infra15:17
*** bdemers has joined #openstack-infra15:18
electricalclarkb: fungi found a ( what i think ) bug in JJB regarding defaults handing when specifying a defaults: foo in job-group and project. its not handing it in the same way when specifying defaults: foo in a job-template.15:19
clarkbjeblair: also, if you haven't started alread I was going to add a zuul-cloner step to the puppet beaker jobs15:19
*** rossella_s has quit IRC15:19
jeblairclarkb: i have not; i'm currently tracking down the eavesdrop/irc-meetings failure15:19
pabelangerfungi, Not this morning, but when time permits, I'd be interested in collaborating on the HTTP proxy (squid) stuff we talked about on Friday.15:20
*** vahidh has joined #openstack-infra15:20
*** rossella_s has joined #openstack-infra15:20
clarkbelectrical: its entirely possible that defaults are not handled properly at all in job groups, I don't think we use JJB that way so its uncovered in production as a use case15:20
clarkbelectrical: so likely need to fix, then add unittests to cover the behavior15:20
electricalclarkb: okay. i created some tests for it which are obviously failing atm.15:20
mriedemfungi: thanks15:20
fungipabelanger: ooh! i've already completely forgotten talking about a squid proxy15:21
*** panda has quit IRC15:21
pabelangerWell, not sure it was decided to use squid, but something to proxy HTTP requests to speed up yum / apt downloads15:21
*** panda has joined #openstack-infra15:22
*** dannywilson has joined #openstack-infra15:22
clarkbit should be noted that 168633 cannot be tested with devstack to use rax's upload system because rax has not upstreamed that stuff15:22
fungipabelanger: ahh... a caching proxy in each provider/region for job workers?15:22
*** achanda_ has joined #openstack-infra15:22
clarkbso I have tested that it doesn't break the hpcloud case15:22
clarkband intend on going from there15:22
*** achanda has quit IRC15:22
*** changbl has joined #openstack-infra15:23
*** krtaylor has quit IRC15:23
clarkbyum apt downloads should be fully cached on every slave image15:23
jeblairfungi: i'm reverse engineering our yaml2ical work -- i think we must have thought we wanted to put the output into its own dir and then symlink from the docroot into it.  does that sound right?  if so, i think we're missing the puppet change to create the symlinks15:23
fungipabelanger: i'd be marginally worried about piecemeal cache invalidation causing index inconsistencies for package repositories. i know custom proxies like approx take care of those sorts of issues15:23
clarkbthen you only deal with the delta between you and the mirror15:23
pabelangerfungi, Ya, that's what I'm thinking. To be honest, I don't have any metric on the time it takes to download packages from remote HTTP, however it could be beneficial once some of this packaging stuff starts landing15:23
fungijeblair: yes, that's where i think we got to. could have sworn jhesketh wrote the puppet change to add that symlink but can't remember whether i reviewed it15:24
pabelangerfungi, right. So, maybe the play is to just wait until we have some need to get a cache proxy vs starting with it out of the box.15:24
pabelangerhowever, that goes back to your issue with upstream HTTP sources that blink in and out15:24
clarkbfungi: pabelanger we shouldn't *need* that as long as we are already caching packages on nodes15:24
clarkbI feel like I am issing something15:24
fungipabelanger: i'm less worried about the time involved and more about intermittent uotages of remote repositories, so if proxies shield us from even some of that it's probably a great idea15:25
jeblairfungi: i think that was only the upload step15:25
clarkbok maybe thats the missing bit, proxies would help with the "sorry not home" errors15:25
jeblairoh i thought we were going to run mirrors for that15:25
pabelangerclarkb, this was more inline with having package build environments within infra. Which could have packages we don't currently cache15:26
clarkbjeblair: yes me too15:26
fungiclarkb: well, the package indexes get retrieved multiple times on every worker, and that's usually where we see job problems from connectivity15:26
clarkbpabelanger: no, the cache should always have packages we need to cache in it15:26
fungiand yes, i too thought we were still planning to have mirrors in each region15:26
pabelangereither way, I was mostly wanting to hash thinks out vs trying and go off an implement it15:26
*** dannywilson has quit IRC15:26
*** tim_o has joined #openstack-infra15:27
pabelangerclarkb, we currently cache yum / apt package downloads?  Or am I miss understanding15:27
fungithough honestly i don't know how mirrorable yum repos are (i remember red hat made it very hard to mirror their repos back in the bad old rhn_register days so you couldn't get security updates without paying for licenses)15:27
fungipabelanger: we pre-cache them on our job workers, yes15:27
pabelangerAh, I did not know that15:27
pabelangeroff to check the code15:27
*** patrickeast has joined #openstack-infra15:27
fungipabelanger: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/cache_devstack.py15:28
fungipabelanger: and http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/install.d/50-download-pkgs15:28
pabelangerOh, neat15:30
fungiand yeah, we don't currently cache everything you'd need to build a package building chroot, but we could easily add it. i'm working presently on generalizing our package caching mechanisms to cover more than just the devstack use case15:30
clarkbcrinkle: do you see any problems with https://review.openstack.org/#/c/185104/2 ?15:30
*** katyafervent_awa is now known as katyafervent15:30
clarkbcrinkle: specifically for the openstack puppet modules?15:30
pabelangeris that the bindep stuff mordred was talking about for ansible?15:30
fungipabelanger: but beyond that, we also wanted to run a set of package mirrors (likely backed by afs) in all our providers/regions similar to our current pypi mirrors15:31
fungipabelanger: yep15:31
clarkbfungi: right my point was more that if we need a package it belongs in the cache15:31
clarkbwe don't intentionally not cache things we need15:31
fungiclarkb: that i agree with15:31
*** dimtruck is now known as zz_dimtruck15:31
pabelangerfungi, Right, I know there was some afs talk too.  Okay, so let me read up on this code, sounds like there is already a plan in play for better caching.15:31
*** dannywilson has joined #openstack-infra15:32
fungii hope to spend some time before the meeting today figuring out where i left off with the bindep integration patch series15:32
openstackgerritJames E. Blair proposed openstack-infra/system-config: Create dir on eavesdrop for ical publishing  https://review.openstack.org/18564415:32
jeblairfungi: ^15:32
*** nelsnelson has joined #openstack-infra15:32
*** mattfarina has joined #openstack-infra15:33
greghaynesclarkb: did you figure out the glean + dib failues we were seeing?15:33
*** markvoelker has joined #openstack-infra15:34
*** aswadr has quit IRC15:34
clarkbgreghaynes: yes mordred had fixed them but not released the fix15:34
clarkbgreghaynes: so grumble grumble15:34
*** mattfarina has quit IRC15:36
electricalclarkb: i created some yaml/xml test files that are failing. will see if i can track down the issue for this. Want me to share the test files somewhere so others can take a peek as well?15:36
clarkbhopefully the change to log to the console log will make debugging this much better if you don't already know that glean is broken15:36
clarkbelectrical: sure? I proably won't have time to look at them soon but others may15:36
clarkb(thinking about zaro in particular)15:37
electricalokay :-)15:37
pabelangerfungi, cool. When mordred was talking about it, the light bulbs in my brain started to turn on.15:37
*** mattfarina has joined #openstack-infra15:37
pabelangerfungi, There was also some possible discussion about removing the package installs from devstack too, well the package naming within devstack15:37
*** fhubik_afk is now known as fhubik15:38
*** zz_dimtruck is now known as dimtruck15:38
pabelangerwhich was news to me15:38
*** _nadya_ has quit IRC15:38
*** dizquierdo has quit IRC15:39
pabelangerjeblair, if you didn't see, I created the governance review for grafyaml: https://review.openstack.org/#/c/185446/15:40
*** jistr has quit IRC15:40
fungipabelanger: yeah, once we're using bindep to list the packages to cache and install for non-devstack jobs, i think devstack could be fairly easily converted to the same model for consistency and manageability15:41
*** puranamr has joined #openstack-infra15:41
pabelangerfungi, Ya, that is pretty hot actually.15:41
*** mohitsharma has joined #openstack-infra15:41
*** imcsk8 has quit IRC15:41
*** gyee has joined #openstack-infra15:42
*** yamamoto has quit IRC15:42
openstackgerritMerged openstack-infra/nodepool: Use shade for image uploads  https://review.openstack.org/16863315:42
jeblairpabelanger: have you made a project-config change to create it?15:43
*** e0ne is now known as e0ne_15:43
pabelangerjeblair, I did: https://review.openstack.org/#/c/182045/15:43
pabelangerI linked the depend on in the other direction15:44
jeblairpabelanger: cool; you can actually reverse the dependencies for that since it's a simple git repo add to an existing openstack project (infra)15:44
pabelangerjeblair, Roger, doing now15:44
*** leopoldj has quit IRC15:45
*** ujuc has joined #openstack-infra15:45
*** krtaylor has joined #openstack-infra15:46
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add grafyaml into openstack-infra  https://review.openstack.org/18204515:46
*** e0ne_ is now known as e0ne15:46
*** mwhagedorn has joined #openstack-infra15:47
*** freerunner has quit IRC15:48
clarkbHunner: I am going to push up a fix for the linting happening on precise as soon as I can double check it15:48
*** ddieterly has quit IRC15:49
*** derekh_ has joined #openstack-infra15:50
*** amitgandhinz has joined #openstack-infra15:51
*** tonytan4ever has joined #openstack-infra15:52
*** ujuc has quit IRC15:52
*** tonytan4ever has quit IRC15:53
*** eharney has joined #openstack-infra15:53
*** tonytan4ever has joined #openstack-infra15:53
*** derekh has quit IRC15:53
rbradforI'd like to add a feature request to optimize 7 mins (of 43 mins) from the devstack installation portion of check-functional-dsvm-magnum.  Could somebody advise which is the right StoryBoard project to add this to?15:53
*** hrubi has quit IRC15:53
greghaynesclarkb: actually, looks like glean is released at the latest commit15:53
clarkbgreghaynes: ya he fixed it on thursday iirc15:54
jeblairrbradfor: what git repo would the change go in, or if you are unsure, can you briefly describe the change?15:54
*** afazekas_ has quit IRC15:54
greghaynesclarkb: yea, so I think the nodepool stuff is good to go for images whenever we want?15:54
greghayneser, nodepool+shade15:54
clarkbgreghaynes: yup change just merged15:54
clarkbgreghaynes: need to restarted nodepool, then update ubuntu-trusty to use config-drive in rax15:54
*** hrubi has joined #openstack-infra15:55
pabelangerBecause I'm not sure, do project being added into stackforge need some governance approval now?15:55
*** achanda has joined #openstack-infra15:56
Hunnerclarkb: Hi15:56
electricalHunner: hiya bud15:56
*** achanda_ has quit IRC15:56
openstackgerritMerged openstack-infra/devstack-gate: Revert "only force off KEYSTONE_USE_WSGI up until kilo"  https://review.openstack.org/18507315:56
*** freerunner has joined #openstack-infra15:56
Hunnerclarkb: Okay. I was planning on getting back to the beaker/gemfile stuff today15:56
clarkbHunner: hello, so looks like we run those jobs on both precise and trusty, so it passes on trusty and fails on precise. I am fixing it to only run on trusty for consistency. Then you changes are there to be actually correct15:56
*** amitgandhinz has quit IRC15:56
rbradforjetblair: it takes almost 7 mins to retrieve a 735M image, specifically  wget --progress=dot:giga -c https://fedorapeople.org/groups/magnum/fedora-21-atomic-3.qcow2 -O /opt/stack/new/devstack/files/fedora-21-atomic-3.qcow2   Ideally having a RS specific cache of this would help.  It's not something that was discussed in Portland as a first pass of mirrored repos.15:56
clarkbgreghaynes: ^15:57
clarkbI think the tl;dr on that is there was a suggestion to fix it, and it was shot down because "special filesystem sauce"15:57
*** erw has joined #openstack-infra15:58
*** claudiub has quit IRC15:58
*** jlanoux has quit IRC15:59
*** ociuhandu has joined #openstack-infra15:59
greghaynesYea, I went and reverse engineered it a bit, its a pretty funky fs layout but wasnt anything that crazy15:59
jd__fungi: could you git branch -d 1.0 in openstack/gnocchi? It was pushed by mistake a few weeks ago15:59
*** juggler has quit IRC15:59
*** dtantsur is now known as dtantsur|afk16:00
*** juggler has joined #openstack-infra16:00
*** marun has joined #openstack-infra16:00
*** yfried has quit IRC16:00
*** ddieterly has joined #openstack-infra16:00
mordredoh! zomg16:01
mordred15:42:40 openstackgerrit | Merged openstack-infra/nodepool: Use shade for image uploads  https://review.openstack.org/16863316:01
* mordred watches for carnage16:01
*** sarob has joined #openstack-infra16:01
clarkbmordred: well I actually tested it so it should just work (granted did not test the rax specifics as that is not testable with devstack)16:01
clarkbI will be restarting nodepool as soon as I get caffeine16:01
mordredI mean, it should work - I've tested uploading with shade to all of the clouds16:01
mordredgreghaynes: did you see the dib patch for better glean debugging?16:02
greghaynesmordred: yep, +2'd16:02
*** rwsu has joined #openstack-infra16:02
jeblairrbradfor: there is a facility in devstack to enumerate images that should be cached on our nodes; listing that image there should cause it to be locally cached when the run starts.  it's probably a one-liner to fix.  bugs would be in https://bugs.launchpad.net/devstack16:02
*** Bjoern__ is now known as BjoernT16:02
electricalzaro: https://gist.github.com/electrical/ff9d0b55544cbb58b82e <-- Created some tests regarding the defaults handing. except 001 they all fail.16:02
greghaynesmordred: actually, looks like it merged16:03
mordredclarkb: https://review.openstack.org/#/c/185130/ <-- that merged, so when that releases, we should have a better time debugging16:03
greghaynesclarkb: maybe we should cut a dib release too before doing this, will pull in the glean debugging16:03
clarkbjeblair: well sort of16:03
*** e0ne is now known as e0ne_16:04
*** fhubik has quit IRC16:04
clarkbjeblair: there was a thread where they were asked to do the sane thing of not requiring 800MB images and the response was no its a totally custom filesystem so go away16:04
*** soren_ has joined #openstack-infra16:04
rbradforjetblair: thanks, I'll take a look.  I suspect you don't want to cache this size file in the daily image that is built correct?  I was thinking more of a local wget not connecting to a slow fedora download site.16:04
openstackgerritJulien Danjou proposed openstack/requirements: Add sqlalchemy-utils to requirements  https://review.openstack.org/18565216:05
*** claudiub has joined #openstack-infra16:05
openstackgerritClark Boylan proposed openstack-infra/project-config: Run puppet-check-jobs on bare-trusty only.  https://review.openstack.org/18565316:05
clarkbHunner: ^16:05
*** e0ne_ is now known as e0ne16:06
*** koolhead17 has quit IRC16:06
*** uberjay has quit IRC16:07
*** rguillebert has joined #openstack-infra16:07
openstackgerritRyan Petrello proposed openstack-infra/project-config: Remove pecan+storyboard tests.  https://review.openstack.org/18565416:07
*** teran has quit IRC16:08
mordredrbradfor, jeblair: hrm. we normally do want to pre-cache all the images - otoh, a 735M fedora atomic image is a bit extreme and is used only for that one job. might be worth thinking more deeply about a more general solution to caching for thigns like that ... but for now I'd say caching it in the devstack image is probably fine, no?16:08
*** krtaylor has quit IRC16:08
jeblairclarkb, rbradfor: tbh, i'm not sure why anyone think's it's okay for magnum to require a file hosted on 'people.fedora.org'.16:08
mordredjeblair: taht too16:08
clarkbjeblair: we don't16:08
jeblairer, fedorapeople.org16:08
fungijd__: i've now deleted the 1.0 branch which was formerly at commit 936a9c516:08
ryanpetrellomordred jeblair clarkb anybody mind a simple review to remove storyboard tests for pecan ^16:08
clarkbthere was a ml thread, greghaynes suggested they do a saner thing and was shot down16:08
*** teran has joined #openstack-infra16:08
openstackgerritMerged openstack-infra/system-config: Create dir on eavesdrop for ical publishing  https://review.openstack.org/18564416:08
*** amitgandhinz has joined #openstack-infra16:09
ryanpetrelloit's blocking me from passing pecan tests for an upcoming release16:09
clarkbryanpetrello: looking16:09
jeblairryanpetrello: if not in your commit message, at least in a review comment, could you indicate why? :)16:09
ryanpetrellowill do16:09
clarkband also does this mean that the new release will break storyboard?16:09
*** alexpilotti has joined #openstack-infra16:10
mordredgood question16:10
*** uberjay has joined #openstack-infra16:10
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483016:10
clarkbsupposedly the atomic filesytem setup is super fancy and thus custom made minimal images like those built by dib are not useable16:11
clarkbiirc greghaynes looked at it and its just an pair of lvm volumes for / and not / ?16:11
*** ajmiller has joined #openstack-infra16:12
openstackgerritRyan Petrello proposed openstack-infra/project-config: Remove pecan+storyboard tests.  https://review.openstack.org/18565416:12
clarkbin any case I don't think that the answer is as simple as just cache another giant image because this image shouldn't be that much more special than the one used by heat and is already cached16:12
ryanpetrellojeblair: done16:12
ryanpetrellosorry mordred, just undid your +216:12
ryanpetrelloclarkb: it shouldn't16:13
ryanpetrelloit's mostly just that I'm removing it from the tox.ini16:13
jeblairryanpetrello: what kind of failure are you seeing?16:13
*** amitgandhinz has quit IRC16:13
ryanpetrelloI'm updating pecan's tox.ini to with kilo tags16:13
ryanpetrelloand figured I'd also remove storyboard16:13
ryanpetrelloso there isn't a definition in pecan's tox.ini anymore, basically16:13
*** zz_jgrimmn is now known as jgrimm16:13
jeblairryanpetrello: oh, i thought you said tests were failing16:14
rbradforjetblair, clarkb, morded Hence my opening question, where could I pose the request to get more intelligent minds to think of a practical way to solve this. I have my own 2 cents but not experienced on the ideal infra ways of doing things.16:14
*** amitgandhinz has joined #openstack-infra16:14
pleia2good morning16:14
ryanpetrellosorry, I was unclear16:14
greghaynesclarkb: yea, IIRC its two partitions, one ext4 and one lvm, on the lvm two partitions one xfs and one btrfs... or something like that16:14
ryanpetrellojeblair: the tests should still be working just fine, I just figured it didn't make sense to gate pecan on this project if it's going away16:14
taronpleia2: morning o/16:14
greghaynesanyhow - dib has a phase dedicated to you doing whatever partition magic you want16:14
greghaynesso its not really a dib limitation16:14
fungiif we're dropping the pecan test for storyboard compat, i suppose that means we should make sure we update storyboard to pin to the last release of pecan so we don't break our storyboard server16:15
ryanpetrellojeblair: by "the tests are failing", I mean "I can't get a +1, because zuul's running this job, and the latest change I'm working on removes the tox.ini for storyboard definition in pecan"16:15
ryanpetrelloare you all continuing to use it internally?16:15
jeblairryanpetrello: we're still using storyboard, so it's still maintained, for now at least.  so we would be concerned with and act upon any actual breakage.  but if you want to drop it in anticipation of it no longer being supported at all, that's fine.16:15
ryanpetrellooh, okay, I'm fine to continue supporting it for the time being16:15
ryanpetrelloI was under the impression from discussions I've heard and the mailing list that it's basically being abandoned16:15
ryanpetrellobut if you all are still using it for some time, I'm happy to continue to run tests for it (assuming they keep passing :) )16:16
mordredryanpetrello: not yet - it's still infra's production tracker16:16
ryanpetrelloI'll do this later after you all officially discontinue it, then16:16
mordredjeblair: could I get an opinion from you on 179485?16:17
mordredjeblair: I'd like to either land it or abandon it16:17
*** lnr has quit IRC16:17
*** lnr has joined #openstack-infra16:17
*** lnr has quit IRC16:17
mordredjeblair: it's an opinion/judgement call thing16:17
*** lnr has joined #openstack-infra16:17
*** lnr has quit IRC16:17
*** lnr has joined #openstack-infra16:18
*** lnr has quit IRC16:18
jeblairmordred: +216:18
*** lnr has joined #openstack-infra16:18
*** lnr has quit IRC16:18
mordredjeblair: awesome. thanks16:18
*** lnr has joined #openstack-infra16:18
*** lnr has quit IRC16:18
openstackgerritafazekas proposed openstack-infra/elastic-recheck: Remove the EOFError query  https://review.openstack.org/18474616:18
*** unicell has quit IRC16:19
*** lnr has joined #openstack-infra16:19
*** lnr has quit IRC16:19
*** lnr has joined #openstack-infra16:19
*** annegentle has joined #openstack-infra16:20
*** soren_ has quit IRC16:21
*** ayoung has quit IRC16:22
*** ociuhandu has quit IRC16:24
*** ashleighfarnham has joined #openstack-infra16:24
*** lnr has quit IRC16:26
*** lnr has joined #openstack-infra16:27
*** lnr has quit IRC16:27
*** lnr has joined #openstack-infra16:27
*** lnr has quit IRC16:27
*** lnr has joined #openstack-infra16:28
*** lnr has quit IRC16:28
*** lnr has joined #openstack-infra16:28
*** lnr has quit IRC16:28
*** lnr has joined #openstack-infra16:29
*** lnr has quit IRC16:29
*** lnr has joined #openstack-infra16:29
*** lnr has quit IRC16:29
*** lnr has joined #openstack-infra16:29
*** lnr has quit IRC16:29
*** lnr has joined #openstack-infra16:30
*** lnr has quit IRC16:30
*** lnr has joined #openstack-infra16:30
*** lnr has quit IRC16:30
*** lnr has joined #openstack-infra16:30
*** lnr has quit IRC16:30
*** tiswanso has quit IRC16:30
*** vahidh has quit IRC16:30
*** redixin has quit IRC16:30
*** lnr has joined #openstack-infra16:30
*** lnr has quit IRC16:31
*** tiswanso has joined #openstack-infra16:31
*** lnr has joined #openstack-infra16:31
zxiiroIs there any JJB reviewers available to review a patch? it only needs one more +2 https://review.openstack.org/#/c/182116/16:31
*** rbradfor has quit IRC16:34
*** aduarte has joined #openstack-infra16:34
*** sputnik13 has joined #openstack-infra16:35
*** ayoung has joined #openstack-infra16:35
*** miqui has joined #openstack-infra16:35
*** otter768 has joined #openstack-infra16:35
*** lnr has quit IRC16:35
*** lnr has joined #openstack-infra16:36
*** lnr has quit IRC16:36
*** asselin has joined #openstack-infra16:36
*** lnr has joined #openstack-infra16:36
*** lnr has quit IRC16:36
*** lnr has joined #openstack-infra16:36
*** lnr has quit IRC16:36
*** lnr has joined #openstack-infra16:37
*** lnr has quit IRC16:37
*** lnr has joined #openstack-infra16:37
*** ildikov has quit IRC16:37
*** claudiub has quit IRC16:38
*** _nadya_ has joined #openstack-infra16:38
clarkbhrm rbradfor left...16:39
clarkbI have tea now and was going to suggest finding out what magnum needs from the image then hopefully making that overlap with what heat is using16:39
clarkbwhich I don't think is a feature request for infra but a write up requirements for magnum task16:39
*** otter768 has quit IRC16:40
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483016:40
*** davideagnello has joined #openstack-infra16:40
electricalMorning zaro16:42
*** rbradfor has joined #openstack-infra16:42
*** rbradfor_ has joined #openstack-infra16:43
asselinhi zaro16:43
electricalzaro: found a weird bug i can reproduce with some tests. concerns the handling of the defaults parts.16:44
* anteaya also waves a flag at zaro's arrival16:45
*** e0ne has quit IRC16:45
*** unicell has joined #openstack-infra16:45
zaroelectrical: thanks.  just push those up to gerrit with 'WIP: ..' in commit message and i'll take a look.16:46
electricalzaro: ah okay, will do.16:46
*** _nadya_ has quit IRC16:46
*** lnr has quit IRC16:46
mordredclarkb: jeblair has +2d 179485 - do you want to express an opinion on it?16:47
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483016:47
*** lnr has joined #openstack-infra16:47
*** achanda has quit IRC16:48
openstackgerritRichard Pijnenburg proposed openstack-infra/jenkins-job-builder: [WIP] Adding tests surrounding handling of defaults values  https://review.openstack.org/18566116:48
clarkbmordred: not really, I am thinking that glean should document its set of behavior somewhere though16:48
electricalzaro: ^^16:48
clarkbeitehr way is fine (append/truncate) but it should be clear to users which is used16:48
zaroelectrical: just in case you haven't noticed, we have a policy to approve changes.  a change requires two +2 reviews before it is approved.  sorry these things can take a while :(16:49
electricalzaro: no worries. i know it takes a while and i'm patient. Just trying to make sure that changes that are out there are not forgotten ;-)16:49
openstackgerritJames E. Blair proposed openstack-infra/system-config: Update team structure and add council  https://review.openstack.org/18281116:49
mordredclarkb: ++16:50
electricalzaro: i gotta run to an Elastic meetup. but i'll be online on my phone if you have questions.16:51
clarkbI am waiting for the current set of nodepool dib builds to complete before restartin (just so we don't lose what is in progress)16:51
clarkbI also have a change just about ready to go for uploading ubuntu-trusty to rax16:52
*** Swami has joined #openstack-infra16:52
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483016:52
*** Swami_ has joined #openstack-infra16:52
*** gokrokve has quit IRC16:53
*** hdd has joined #openstack-infra16:55
*** lnr has quit IRC16:56
*** lnr has joined #openstack-infra16:57
*** dkehn has quit IRC16:58
zaroelectrical: for concat_defaults002.yaml looks like there are multiple default definitions?  is that a regression? i'm not sure jjb ever supported that.16:59
*** dkehnx has quit IRC16:59
*** dkehn has joined #openstack-infra16:59
*** dkehn_ has joined #openstack-infra17:00
*** markus_z has quit IRC17:00
jeblairfungi: retriggering irc-meetings post job now that puppet is in place17:01
clarkb2015-05-26 17:00:25,482 INFO nodepool.image.build.centos-6: grubby fatal error: unable to find a suitable template17:01
*** mtanino has quit IRC17:02
clarkbI don't think that causes the builds to break and may be intentional as the grub stuff is supposed to start very late in the dib process, but throwing it out there is anyone knows more17:02
*** ihrachyshka has quit IRC17:03
*** tiswanso has quit IRC17:03
fungiclarkb: i want to say we see the same in the devstack-centos7 builds? it looks like the template we're putting in place may be missing magic replacement macros grubby expects to find17:03
mordredI'm pretty sure I don't want to know anything about grubby or macros17:03
electricalzaro: will have to check. Currently in a taxi :-)17:03
*** devvesa has quit IRC17:04
*** rkukura has joined #openstack-infra17:04
jeblairfungi: oh! the upload job did not put things where i was expecting them to17:04
*** tiswanso has joined #openstack-infra17:04
fungijeblair: hrm... i wonder if we neglect to flatten the upload since it's scp17:05
jeblairfungi: http://paste.openstack.org/show/238146/17:05
jeblairfungi: i will move those files around manually then see about changing the job17:05
electricalzaro: in 002 I would expect that the myproject defaults are applied to all job templates.17:05
fungijeblair: aha! we expected that to rename them, not create directories to stuff them in17:05
clarkbhrm I need to run the etherpad lite tests17:05
*** lnr has quit IRC17:06
clarkbI wonder how slow this will be on my fitlet17:06
*** lnr has joined #openstack-infra17:06
*** harlowja has joined #openstack-infra17:07
jeblair"irc-meetigs.ical"  jeez17:07
* mordred loves meetigs17:08
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483017:08
mordredbtw ... sc68cal ^^ you rock for adding that. infra thanks you17:08
*** akscram has joined #openstack-infra17:08
openstackgerritJames E. Blair proposed openstack-infra/system-config: Fix typo in irc-meetings  https://review.openstack.org/18566917:08
sc68calmordred: thanks - anteaya has been a huge help17:09
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update SSH key for fungi  https://review.openstack.org/18567017:09
*** annegentle has quit IRC17:09
clarkbsc68cal: though see comment on the d-g change, it shouldn't be necessary17:09
jeblairunecessary changes are the best kind17:10
*** yamahata has joined #openstack-infra17:10
mordredjeblair: unless they are mandatory17:10
clarkbhrm github needs an easier way to fetch a pull request17:11
mordredclarkb: there is a cantrip17:11
*** tonytan4ever has quit IRC17:11
*** markvan has quit IRC17:11
mordredclarkb: git fetch origin pull/{ID}/head:{BRANCHNAME}17:11
*** markvan has joined #openstack-infra17:12
*** markvan has quit IRC17:12
*** derekh_ has quit IRC17:12
* sc68cal starts grepping for DEVSTACK_LOCAL_CONFIG17:14
*** markvan has joined #openstack-infra17:14
sc68calclarkb: so something like  DEVSTACK_LOCAL_CONFIG="Q_AGENT=linuxbridge" ?17:15
*** ihrachyshka has joined #openstack-infra17:15
clarkbin the job config then you can abandon the d-g change17:15
sc68calclarkb: ok, works for me. Over in this chunk? https://review.openstack.org/#/c/184830/5/jenkins/jobs/devstack-gate.yaml17:15
clarkbsc68cal: ya you can replace line 536 with it17:16
sc68calclarkb: excellent17:16
*** mohitsharma has quit IRC17:18
*** dkranz has quit IRC17:19
mordredmmm. Q_AGENT17:19
*** jsavak has joined #openstack-infra17:19
sc68calquantum will live forever in our hearts17:20
mordredsc68cal: I thought that was just heartburn17:20
sc68calmordred: haha :)17:21
openstackgerritJames E. Blair proposed openstack-infra/project-config: Reconfigure irc-meetings output  https://review.openstack.org/18567717:21
waynrelectrical: i see what you're talking about now, i never ran into that problem because I never use anything other than global defaults17:21
openstackgerritJames E. Blair proposed openstack-infra/irc-meetings: In tox, write output to subdir  https://review.openstack.org/18567817:22
*** mmedvede_ has quit IRC17:22
jeblairfungi: https://review.openstack.org/185677 and https://review.openstack.org/185678 rejigger things around in a way that i think will work17:22
waynri have a global defaults file that is about 600 lines long that is intended to primarily set default values for variables17:22
waynralthough about 80% of that file is whitespace and comments that document the use case of each variable17:22
waynrif JJB supported a way to combine multiple sets of defaults i would probably use more than just global defaults17:23
*** rguillebert has quit IRC17:25
*** davideagnello has quit IRC17:26
*** mohitsharma has joined #openstack-infra17:28
*** davideagnello has joined #openstack-infra17:29
openstackgerritPaul Belanger proposed openstack-infra/system-config: WIP: Jenkins slave for package builds  https://review.openstack.org/18568017:29
pabelangerfungi, v1 for packaging build environments.  Decided to just use puppet to create initial mock chroots17:30
pabelangerbut, open to suggestion17:31
*** yamamoto has joined #openstack-infra17:31
*** achanda has joined #openstack-infra17:33
*** annegentle has joined #openstack-infra17:33
clarkbpabelanger: I thought zigo's debian package existed for that?17:33
clarkbI may have misread the tehrapd17:33
*** mohitsharma has quit IRC17:34
*** HeOS has quit IRC17:34
pabelangerclarkb, Yes, talking to zigo he has specific scripts in pkg-openstacktools (nameing?) that bootstraps an local development environment for openstack packaging.  I need to checkout how they work, but we could pull them into the bootstraping stage for nodepool to cache some things.17:35
electricalwaynr: hehe. Maybe I have to redesign it but would be nice to have it working.17:35
*** yamamoto has quit IRC17:35
zaroelectrical: ahh, yes that is a point of confusion for users of JJB.  I believe this change is meant to address some of that confusion: https://review.openstack.org/#/c/179459/17:35
pabelangerThe other side, is to also allow developers to run their tool chains once the jenkins slave is launched by nodepool, which I see people doing that too17:36
nibalizerpabelanger: sure thing17:36
nibalizerlet me know how I can help17:36
mordredlook it's a nibalizer17:36
electricalzaro: yeah indeed. Besides that there is still something in the defaults handling that feels weird. If I specify in defaults namespace puppet but not in global it fails.17:37
nibalizerohai mordred17:37
pabelangerclarkb, basically, if we infra, could do some initial creating of the chroots (via puppet or nodepool), it should help packagers speed up their builds.  Thats my thought process17:37
clarkbpabelanger: isn't that cost very low already?17:38
*** ihrachyshka has quit IRC17:38
clarkbwe don't do anything like that for dib and they haven't said they need it17:38
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update SSH key for fungi  https://review.openstack.org/18567017:39
zaroelectrical: yes, i agree that concat_defaults004.yaml17:39
zaro is a bug17:39
pabelangerclarkb, unsure to be honest. I think it is part of the line that needs to be figured out.  For me, I don't see much over head to bootstrap chroots via the infra method (puppet or nodepool).  However, people like zigo already have the scripts they have created to do the bootstrapping.  So, I could go either way.  If we provided chroots for people to use great, but if people want to do their own tooling for chroots, great too.17:40
openstackgerritSean M. Collins proposed openstack-infra/project-config: [WIP] Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483017:40
Clintclarkb: would it be better to use per-target images than chroots?17:40
pabelangerhonestly, it was just easy to get a mock chroot to build in puppet, that is why I added.  It it required 10 hours of work, I likely wouldn't do the effort17:41
clarkbpabelanger: I don't want to get in the business of telling distro packagers how to run their tools17:41
clarkbpabelanger: if we can aid in the performance of those tools then we should think about these things otehrwise I think ts on the packaging to bootstrap their own envs17:41
fungiClint: perhaps, though we may still end up with a handful of things preinstalled which wouldn't be in, say, a pbuilder chroot17:41
*** matbu is now known as matbu_afk17:41
clarkbClint: we are trying very hard to get away from per target images in general17:41
Clintfungi: right17:42
clarkbthey are very costly and I know more about all the distros now than I ever wanted to know17:42
*** HenryG has quit IRC17:42
fungiso might still end up with dirty builds (highly unlikely though as we're working to actively minimize the current volume of preinstalled things and just pre-cache it all instead)17:42
*** alexpilotti has quit IRC17:42
fungibut yes, i'd like to avoid additional nodepool images solely because someone wants to build packages for that particular platform (distro, release, processor architecture, et cetera)17:43
*** mmedvede has joined #openstack-infra17:43
pabelangerI didn't see specific nodepool images for each distro.  I just seen a single packaging image, which people then installed the require packages for.  EG: mock, sbuild, etc17:44
*** mtanino has joined #openstack-infra17:44
pabelangersome people I talked to even wanted to use containers for their builds17:44
pabelangeragain, like clarkb said, I wouldn't want infra to decided what that tool chain looked like17:45
pabelangerjust the place for builds to happen, a la the infra way17:45
*** abregman has quit IRC17:45
pabelangerdisposable slaves17:45
mordredpabelanger: so ...17:45
mordredpabelanger: I think that on the one hand providing a place for people to do whatever is awesome17:45
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Add template support for the path.data  https://review.openstack.org/18517717:46
mordredpabelanger: otoh - infra DOES need a packaging toolchain that can produce packages that infra can publish and consume - and that does want to be opinionated and to work the same way across everything17:46
*** HenryG has joined #openstack-infra17:46
*** igorbelikov has joined #openstack-infra17:46
mordredpabelanger: so - it would be super awesome if the results of the distro folks moving things into infra could also result ina  set of reusable things that infra could leverage17:47
Kiallmordred: Just.. Please, not another package manager! Pick one and use it :P17:47
mordredKiall: we have both centos and ubuntu nodes right now - we need to acount for both17:47
Kiallala http://xkcd.com/927/ ;)17:47
pabelangermordred, Right.  So, that is my primary goal too, rolling packages that infra can consume.17:47
pabelangerdistro packaging is 2nd, nice to have17:47
clarkbalso its good to not encode too much into the test env so that developers can more easily reproduce locally17:48
clarkbthats sort of a balancing act though and there are no hard answers to how to deal with it17:48
pabelangerIf we, infra, could get the toolchain in place for our packages.  I _could_ see an easy jump for disto people to also consume that environment.17:48
*** SumitNaiksatam has joined #openstack-infra17:49
*** Guest6609 has quit IRC17:49
dstufftsoftware is terrible, trying to distribute software is extra terrible. Adjust your desires until you stop wanting to do that, then you'll be happier17:49
mordreddstufft: what about eating food? can I do that instead?17:49
dstufftmordred: yes17:50
dstufftthat's a better use of time17:50
*** cloudm2 has joined #openstack-infra17:50
Clintnow i want food17:50
*** ihrachyshka has joined #openstack-infra17:50
dstufftI'm not responsible when we all have heart conditions though17:50
mordredClint: I have just realized that I do not have any of it17:50
Clintthat can be solved17:50
anteayasalad is food17:50
anteayaand good for your heart17:50
dstufftanteaya: I'm not convinced salad is human food tbh17:51
anteayafair enough17:51
dstufftmy wife assures me it is17:51
Clinti've made salad dressing with bacon fat on more than one occasion17:51
dstufftI remain unconvinced17:51
anteayaI got a carrot pencil sharpener17:51
anteayaI'm keen to try it out17:51
anteayafrilly salads17:52
dstufftI'm assuming the goal isn't to have sharp carrots, but to have carrot shavings?17:52
anteayaI was going for the shavings part17:52
anteayabut sharp carrots will also result if if stop at a certain point17:52
pabelangernibalizer, if I understood timrc, you might have some real world example dashboards for grafana?17:52
pabelangerusing statsd from zuul and nodepool17:53
timrcpabelanger, I've only produced some dashboards tracking nodepool task execution times.17:53
pabelangertimrc, able to share the json for it?17:53
timrcpabelanger, http://grafana.busywait.com/17:53
timrcpabelanger, You should be able to extract it from there.17:54
*** claudiub has joined #openstack-infra17:54
*** doug-fish has quit IRC17:54
*** ashleighfarnham has quit IRC17:54
*** arxcruz has joined #openstack-infra17:55
*** doug-fish has joined #openstack-infra17:55
pabelangertimrc, great! Exporting now, to see if grafyaml has coverage for them17:56
timrcpabelanger, Cool.  The graph that depicts the average spark line between the min and max fill will be a good a test.17:56
*** baoli_ has quit IRC17:57
timrcpabelanger, e.g. http://grafana.busywait.com/dashboard/db/nodepool-performance-rackspace?panelId=2&fullscreen17:57
*** BjoernT has left #openstack-infra17:57
*** baoli has joined #openstack-infra17:58
*** annegentle has quit IRC17:58
*** annegentle has joined #openstack-infra17:59
*** doug-fish has quit IRC17:59
pabelangertimrc, Yup, will start with that one18:00
*** HenryG has quit IRC18:01
*** HenryG has joined #openstack-infra18:01
*** tiswanso has quit IRC18:02
openstackgerritSean M. Collins proposed openstack-infra/project-config: Introduce an experimental job for Linux Bridge  https://review.openstack.org/18483018:02
*** Kennan has quit IRC18:02
*** tiswanso has joined #openstack-infra18:03
*** cody-somerville has quit IRC18:03
*** Kennan has joined #openstack-infra18:03
*** bdemers has quit IRC18:04
*** cody-somerville has joined #openstack-infra18:04
*** ivar-lazzaro has joined #openstack-infra18:05
electricalzaro: glad you agree. Thought I was going nuts :)18:05
*** harlowja_ has joined #openstack-infra18:06
*** harlowja has quit IRC18:06
* SpamapS decides its easier to just finish his obsession with pysaml2 than try to set it down and ignore the itch18:07
greghaynesSpamapS: mordred so next step in hp1 is getting our inventory playbooks hitting all the nodes were supposed to have. A few of them are in error deleting or similar state, so I think we might need to do some nova db surgery unless youall have better ideas18:07
SpamapSso close to py3k18:07
*** mtanino is now known as mtanino_away18:08
*** doug-fish has joined #openstack-infra18:08
*** dguerri`away is now known as dguerri18:08
SpamapSgreghaynes: reset-state is sanctioned db surgery. ;)18:08
*** davideagnello has quit IRC18:08
*** davideag_ has joined #openstack-infra18:08
greghaynesYea, I tried that with a few and no luck18:08
SpamapSgreghaynes: reset-state, restart nova-compute18:09
mordredgreghaynes: perhaps ironic commands?18:09
clarkbmriedem: fwiw you can't just cap tox on stable branches18:09
greghaynesmordred: Ironic doesnt want to do anything about the node until the instance uuid from nova is unassociated with the node :(18:09
mordredlike, what does it matter what nova thinks of them right now? if the machine is powered off, it needs booted, no?18:09
mordredgreghaynes: ah18:09
mriedemclarkb: yeah i see that now18:09
greghaynesI was considering just nulling that field though18:09
mordredworks for me18:10
SpamapSmordred: I agree 100%. Worth hitting it with the hammer a few different ways before lighting it on fire though.18:10
mriedemclarkb: i have a fix for the tox thing anyway18:10
mordredSpamapS: ++18:10
greghaynesyes, hammers saught18:10
clarkbmriedem: that would be preferable, otherwise we have to cap it everywhere then explain to everyone what version of tox they need to run the tests18:10
mriedemclarkb: the horizon jshint tests are a bit wonky since they use nodeenv18:11
mriedemsome weird virtualenv stuff going on there, but i know that passenv = * fixes it, but i'd like to scope that down to what's actually needed18:11
*** tonytan4ever has joined #openstack-infra18:11
*** dkranz has joined #openstack-infra18:12
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add pleia2 to statusbot_auth_nicks  https://review.openstack.org/17496218:14
fungiclarkb: starting to stare closer at centos-6 dib in hpcloud... we have images created successfully daily but none uploaded for a couple weeks. i'18:14
fungim going to try manually uploading18:14
*** maishsk_afk has joined #openstack-infra18:14
clarkbfungi: ok I haven't restarted the server yet as there are a couple dib build sstill going18:14
fungidevstacl-precise seems to be in a similar situation there18:15
fungier, devstacl-precise18:15
fungigah. can't type18:15
fungier, devstack-trusty too18:15
fungialso ubuntu-trusty18:15
greghaynesall the images?18:15
* fungi is starting to think that dib-image-upload just hasn't been happening18:15
jroll"devstacl is an open-source bash program to provision access lists for your development cloud"18:15
* greghaynes hopes not related to shade18:15
clarkboh hrm18:16
clarkbgreghaynes: well its not running the latest code yet18:16
fungijroll: try again. it's our new common lisp variant!18:16
clarkbgreghaynes: but entirely possible an earlier change did it, but its tested against devstack so that would surprise me18:16
greghaynesyea, but we did the os-client-config change a bit over a week ago18:16
jrollfungi: hehehe18:16
mordredgreghaynes: which one?18:16
greghaynesmordred: switch to using shade for clients - which is switch to using os-client-config and friends for config18:17
mordredgreghaynes: oh - the one that gets the glance client18:17
mordredgreghaynes: good point18:17
clarkbit does work with devstack at least, but possible not with hpcloud18:17
clarkba manual upload should tell us and fungi is running one of those18:17
*** freerunner has quit IRC18:17
mordredos_client_config defaults image_api_version to v1 - so that should still work as before with nodepool18:18
*** ihrachyshka has quit IRC18:18
mordredand we haven't moved to using hp as a named cloud - I believe our nodepool config is still the "pass in all values explcitly"18:18
greghaynesIt should be, yes18:19
*** maishsk_afk has quit IRC18:19
fungiyeah, i've got `sudo -H -u nodepool nodepool image-upload hpcloud-b1 devstack-trusty` going right now18:19
clarkbI am going to dig around in the logs18:19
mordredthere are errors in the logs18:20
mordredHTTPUnauthorized: Authentication required (HTTP 401)18:20
mordredthat tells me something is messed up in the config :(18:20
clarkbmordred: where in the log? /var/log/nodepool/debug.log doesn't seem to have anything like that according to grep18:21
mordredclarkb: debug.log.2015-05-20_1618:21
mordredfungi: oh! that's a whole different thing18:22
mordredSpamapS: ping ^^18:22
*** sputnik13 has quit IRC18:22
mordredSpamapS: please look at fungi's paste18:22
*** maishsk_afk has joined #openstack-infra18:22
funginote that's from a manual nodepool image-upload run18:23
mordredok. it seems to have gotten "None" for image_name18:23
SpamapSmordred: from a clientlib? lovely.18:24
mordredSpamapS: no - from the nodepool command line client18:25
*** gokrokve has joined #openstack-infra18:25
*** sputnik13 has joined #openstack-infra18:25
mordredfungi: mind if I poke?18:25
fungimordred: feel free. i'm not doing anything since the exception18:26
SpamapSmordred: ahhh18:26
clarkbalso worth noting that grepping for the logs around where nodepool does uploads returns nothing18:26
waynrzaro: https://review.openstack.org/#/c/179459/ isn't the same thing as what electrical is talking about18:27
anteayasc68cal: and we should get mestery to review 184830 too18:27
fungiclarkb: yeah, from the logs i'm getting the feeling this might be something like "scheduled build completes but upload is never triggered"18:27
mesteryandreaf: Will look18:27
clarkbfungi: ya18:27
anteayamestery: thanks18:28
fungiclarkb: i have a feeling the traceback i'm getting is specific to the cli subcommand case18:28
mordredyah - I'm investigating that theory right now18:28
mordredif it pans out, I'll turn to the scheduled case18:28
fungiclarkb: or is it possible this is like that time where build was failing but switching to ready rather than delete?18:29
clarkbfungi: according to the build logs they are building properly18:29
fungiyeah, i'm not seeing any obvious aborts18:29
*** salv-orlando has quit IRC18:29
*** sputnik13 has quit IRC18:31
*** davideag_ has quit IRC18:31
russellbwhat does groups: in project-config/gerrit/projects.yaml do?18:31
clarkbhrm I think I may see the issue18:31
clarkbrussellb: it is a way of saying all these projects use the same bug tracking group18:32
*** achanda has quit IRC18:32
russellbclarkb: perfect, thanks!18:32
mordredclarkb: yah? is your issue you see that shade on the system is old?18:32
SpamapSseems like that shouldn't ever return "None"18:32
armaxthanks clarkb18:32
clarkbrussellb: with LP that means the same project with storyboard it means these projects have a super group that collects all of them together18:32
clarkbmordred: no thats no the problem I am seeing18:32
mordredclarkb: k18:32
*** dprince has quit IRC18:32
mordredclarkb: I have not yet investigated that problem18:33
clarkbthat check there doesn't seem to upload if the image is old18:33
clarkbfungi: ^18:33
*** davideagnello has joined #openstack-infra18:33
*** erhudy has joined #openstack-infra18:34
clarkbexcept the code has been that way since february?18:34
clarkbbut I think we don't run the upload code because of that18:35
*** mohitsharma has joined #openstack-infra18:35
fungipip list says shade 0.6.118:35
fungiwhich is the latest release18:35
fungifrom ~1.5 weeks ago18:35
SpamapSmordred: pretty sure that paste from fungi was not the image name from create_image coming back None18:36
*** otter768 has joined #openstack-infra18:36
mordredSpamapS: ok - so here's where I'm at18:36
SpamapSmordred: that is the cloud name18:36
*** sgolovatiuk has joined #openstack-infra18:37
mordredcreate_image gets a valid name into the name field18:37
*** gyee has quit IRC18:37
clarkbya I take that back, its a different code path for the cron uploads18:37
mordredit is validly passing it to self.get_image(name)18:37
clarkbthe check for missing is purely to get actually missing (and not just old) images uploaded18:37
SpamapSmordred: _that_ is what is None18:37
*** krtaylor has joined #openstack-infra18:37
mordredSpamapS: yes. that is None18:37
clarkbthe checks at https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n1932 would have to be wrong18:38
asselinhello, I'm seeting this issue now that I apt-get update on my ci box. seems to be related to vhd-util. any ideas? I can open the link in my browser...?? http://paste.openstack.org/show/238273/18:38
mordredSpamapS: OH - because we pass in direct values, the cloud object has no name18:38
SpamapShttps://git.openstack.org/cgit/openstack-infra/shade/tree/shade/__init__.py#n236 is the only place that is set AFAICT18:38
SpamapSmordred: correct18:38
*** rkukura has quit IRC18:38
SpamapSso, we can be defensive and cast that to str18:38
SpamapSAND nodepool should name it.18:38
mordredwe should do both things18:38
SpamapSwill handle the shade defense (and a test)18:39
mordredor - if it's none, we can exclude it from the key generation18:39
mordredeither way18:39
asselinhmmm......I see i'm using precise and there's no foloder for that....18:39
mordredI'll get a nodepool patch18:39
clarkbfungi: so we should be able to wait for the disk images to finish building18:39
anteayaasselin: it looks like a connection problem trying to connect to ppa.launchpad.net18:39
*** sputnik13 has joined #openstack-infra18:39
clarkbfungi: then we should hit the code I just linked, which is where the error should happen18:39
*** mohitsharma has quit IRC18:39
zarowaynr: i haven't reviewed that one carefully but from the commit message it seems like it's the same issue?  could you please elaborate on change https://review.openstack.org/#/c/185661/18:39
asselinis 'vhd-util' supported on precise? easy to do, or disable?18:40
anteayaasselin: I don't think it got that far, it says 404 not found18:40
anteayaso either the path has an error or the package has moved18:40
*** angela-s has joined #openstack-infra18:40
clarkbmordred: wait, what does nodepool need to do?18:40
asselinanteaya, b/c there's nothign there: http://ppa.launchpad.net/openstack-ci-core/vhd-util/ubuntu/dists/18:40
clarkbmordred: the consumer should not be aware of caching, that should be transparent18:41
mordredclarkb: there is a VERY strange edge condition - the consumer may not need to do anything18:41
anteayaasselin: that would be the problem18:41
mordredclarkb: but whats happening right now is that nodepool is not passing a required parameter18:41
mordredclarkb: how it's bypassing that it's a required param is what I'm looking at right now18:41
*** otter768 has quit IRC18:41
asselinmordred, I see you're the maintainer?18:41
mordredaha. I see it.18:41
mordredasselin: I am the maintainer of that package18:42
*** ildikov has joined #openstack-infra18:42
mordredasselin: one sec - let me finish this18:42
fungiasselin: yeah, i think it's not set to build a precise package18:42
clarkbmordred: looks like you overloaded cloud and name18:42
clarkband I am guessing nodepool defaults cloud to None which gets passed as a positional arg properly18:43
mordredSpamapS, greghaynes: it's actually that os-client-config does not set self.name, and that's what the openstack_cloud function uses18:43
jeblairasselin, anteaya: http://docs.openstack.org/infra/system-config/nodepool.html#vhd-util18:43
clarkbthen later breaks the caching18:43
mordredclarkb: nope18:43
*** maishsk_afk has quit IRC18:43
mordredclarkb: nodepool is doing the right thing. shade needs the defensive patch SpamapS is making - and I need to make an os-client-config patch real quick18:43
clarkbself.name = cloud18:43
SpamapSmordred: yeah, I think there's probably room for a unit test for the micro-level problem, and a functional test for the general scenario that we have18:43
clarkbif cloud == None then you lose18:43
mordredyes. that is correct18:43
asselinjeblair, I see....ok will disable. thanks18:44
jeblairclarkb: sounds like a tshirt18:44
anteayajeblair: thanks18:44
clarkbfungi: I think debug.log doesn't have the errors yet because the log file rolled isnce the last time we tried uplaoding18:44
fungioh, got it18:44
clarkbfungi: once the image builds complete we should fire off all the uploads and see if this breaks the same as the manual upload (I expect it to)18:44
mordredyes. it will18:45
mordreddon't bother18:45
mordredpatch incoming18:45
*** davideagnello has quit IRC18:45
fungiwell, it's not a matter of bothering18:45
funginodepoold is going to be doing this shortly regardless18:46
clarkbmordred: also I think I have confirmed my story18:46
*** HeOS has joined #openstack-infra18:46
fungias part of its scheduled build process18:46
mordredclarkb: what was your story?18:46
openstackgerritJames E. Blair proposed openstack-infra/project-config: Reconfigure irc-meetings output  https://review.openstack.org/18567718:46
clarkbmordred: nodepool sets cloud to None, that is passed as a kwarg to the Cloud __init__18:46
mordredno. that's not what happens18:46
clarkbmordred: self.name becomes None, then we lose18:46
clarkbmordred: how does that not happen?18:46
mordredalthough it's close18:46
clarkbwe explicitly set it to None18:46
mordredbecause there is a check for if it's None18:46
mordredand if it's None, it does not get appended to kwargs18:47
mordredthe bug is in os-client-config18:47
clarkboh the check is in provider manager...18:47
clarkbwe have way too many onion layers here18:47
mordredyah. but unpeeling them will take some time18:48
openstackgerritJames E. Blair proposed stackforge/gertty: Add ability to review multiple changes at once  https://review.openstack.org/18569518:48
openstackgerritJames E. Blair proposed stackforge/gertty: Fix updating flags on threaded changes  https://review.openstack.org/18569618:48
openstackgerritJames E. Blair proposed stackforge/gertty: Allow reviewing one change in change list  https://review.openstack.org/18569718:48
openstackgerritJames E. Blair proposed stackforge/gertty: Advance cursor on change list toggle  https://review.openstack.org/18569818:48
jeblairplane dump18:48
greghaynesclarkb: so AIUI nodepool doesnt retry dib uploads if the upload fails, right?18:48
clarkbgreghaynes: correct18:48
fungicode airdrop18:48
clarkbgreghaynes: it will try again the next day18:48
jeblairfungi: better18:49
clarkb(or on whatever interval you have it set to build images)18:49
greghaynescurious what the machinery is that youall are talking about that will re-kick off uploads?18:49
greghaynesis that just a new build?18:49
clarkbgreghaynes: yes18:49
greghaynesah, ok18:49
clarkbso we have nodepool, os-client-config, and shade all setting config and they don't agree on how to do it18:49
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Make caching work when cloud name is None  https://review.openstack.org/18569918:49
clarkbfix I am guessing based on what mordred says is to update os-client-config18:50
mordredclarkb: yes. I agree - there are patches coming to align that18:50
*** davideagnello has joined #openstack-infra18:50
*** maishsk_afk has joined #openstack-infra18:50
SpamapSdefensive shade patch done18:51
mordredclarkb, SpamapS, dtroyer: https://review.openstack.org/18570118:51
*** e0ne has joined #openstack-infra18:53
mordred(adding test)18:53
*** ashleighfarnham has joined #openstack-infra18:53
clarkbfungi: we are probably ~1 hour away from seeing hte error in production18:54
clarkbfungi: I don't know when these image builds got so slow but thats another thing we should look into18:54
SpamapSmordred: you really should start doing things the other way around. :)18:54
clarkbit used to take ~15 minutes with a warm cache to build these images18:54
SpamapSvery comforting to have a test that breaks, and then code that fixes :)18:54
pleia2wrt to a translations devstack instance, what's the nova driver thing to fake launch a fake vm?18:55
clarkbfungi: but random guess is that the cinder volume IO is much slower than the locally attached ephemeral disk18:55
greghaynesmordred: you got pep8'd18:55
fungiooh, could be18:55
mordredSpamapS: good point. I ran the new test without the patch and confirmed that it breaks18:55
clarkbpleia2: instead of configuring it to qemu/kvm you tell it to load a "fake" driver that pretends to nova that a vm booted18:55
mordredclarkb, fungi: ++18:55
pleia2clarkb: ok, so just "fake" then18:56
clarkbpleia2: let me find an example of a test job that uses it18:56
pleia2clarkb: thanks18:56
*** sigmavirus24 is now known as sigmavirus24_awa18:56
mordredSpamapS, dtroyer: updated18:56
clarkbpleia2: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n22418:56
clarkbpleia2: I think devstack-gate sets it up when the LARGE_OPS var is >018:57
pleia2clarkb: cool18:57
clarkbpleia2: yup https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n32618:57
pleia2great, thanks18:57
mordredclarkb: you can +A https://review.openstack.org/#/c/185699/ ?18:58
*** maishsk_afk is now known as maishsk18:59
*** tiswanso has quit IRC19:00
clarkbmordred: ya I am trying to grok the test19:00
clarkbI guess the idea is getting [] back and not an exception is what we want19:00
clarkbshouldn't we be asserting the value being cached instead?19:01
greghaynesI think its a 'dont explode' test19:01
mordredclarkb: yes. the test will throw if self.cloud.name is None19:01
clarkbgreghaynes: mordred right I get that, but what we need is a "did None get stringified properly" test19:01
*** tiswanso has joined #openstack-infra19:01
clarkbwhich can be a get against the cache assertion?19:02
jeblairit's meetin time19:02
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Move networking-l2gw project under openstack namespace  https://review.openstack.org/18570519:02
openstackgerritMerged openstack-infra/tripleo-ci: Update delorean URL  https://review.openstack.org/18464019:03
clarkbSpamapS: see comment on https://review.openstack.org/#/c/185699/19:09
*** mpaolino has joined #openstack-infra19:10
SpamapSclarkb: ty, it's more complicated than self._cache.get('None'), but not much.19:11
SpamapSclarkb: and worth the time to add, thanks.19:11
*** mpaolino has quit IRC19:11
*** mtanino_away is now known as mtanino19:12
*** koolhead17 has joined #openstack-infra19:14
*** annegentle has quit IRC19:15
*** nelsnelson has quit IRC19:15
*** nelsnelson has joined #openstack-infra19:16
*** salv-orlando has joined #openstack-infra19:17
*** soren_ has joined #openstack-infra19:19
SpamapStesttools.matchers._impl.MismatchError: <unprintable MismatchError object>19:20
SpamapSwell thats a new one on me19:20
*** panda has quit IRC19:21
mordredSpamapS: I'd like to blame lifeless for that one19:22
*** panda has joined #openstack-infra19:22
SpamapSoh and tox -r -epy34 ends in misery on shade19:23
SpamapS    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr<1.0,>=0.5.21')19:23
mordredSpamapS: how are we passing the gate?19:23
*** annegentle has joined #openstack-infra19:23
*** banix has quit IRC19:23
SpamapS    Download error on https://pypi.python.org/simple/pbr/: [Errno -2] Name or service not known -- Some packages may not be found!19:23
mordredoh. well19:23
SpamapSmy wifi is sketchy today maybe? :-P19:24
mordredthat's a whole other thing19:24
*** mpavone has joined #openstack-infra19:24
*** teran has quit IRC19:25
*** rkukura has joined #openstack-infra19:29
*** hdd has quit IRC19:30
cody-somervilleSpamapS: I sometimes wonder if pip implements its own TCP/IP stack and has poor packet re-transmit support.19:30
clarkbit uses requests19:31
openstackgerritMerged openstack-infra/project-config: Switch on docs and publish jobs for magnum  https://review.openstack.org/18299819:31
SpamapSclarkb: is that supposed to be better? :)19:31
clarkbSpamapS: no :)19:31
clarkbor at least I would be much happier about it if the whole urllib3 warning stuff wasn't so bad19:32
*** SotK_ is now known as SotK19:33
cody-somervilleWhat did upstream do to fix the pip 7 trusted-host stuff btw?19:34
clarkbcody-somerville: set trusted host config19:35
cody-somervilleWe did the same but it turns out people do things like run things locally or in docker images.19:35
clarkbI think by definition you can't fix those things for them and the have to edit pip.conf themselves19:36
cody-somervilleDoes pip still try https before flailing?19:36
*** mohitsharma has joined #openstack-infra19:36
clarkbno, it uses the url you have given it, even if talking to upstream ppi19:37
clarkb*upstream pypi19:37
*** jamespage_ has joined #openstack-infra19:38
*** patrickeast_ has joined #openstack-infra19:40
*** maishsk has quit IRC19:40
*** ParsectiX has joined #openstack-infra19:40
mordredcody-somerville: people upstream do not consume our mirrors19:40
*** mohitsharma has quit IRC19:41
mordredcody-somerville: we do not run mirrors for public consumption - we tell people to point to pypi.python.org19:41
fungi(we hope)19:41
mordredif they do - it's not our job to fix them19:41
mordredcody-somerville: so - you have a different usecase here than we do19:41
fungibut yes, we reserve the right to break anyone using those mirrors, including us19:41
mordredcody-somerville: you probably want to get SSL certs for your mirrors19:42
fungihowever, hosting a bandersnatch mirror via https and adding a trusted cert for it seems like an effective workaround19:42
fungiyeah, that19:42
mordredcody-somerville: you may want to get a proper SSL cert and not an HP one - as most people will not have the HP CA installed in their docker containers19:42
mordredand although they could- the cantrip for supporting that is more complex than just setting a pip.conf setting19:43
mordredcody-somerville: you could also get HP to just fix their horribly broken corporate network19:43
asselincody-somerville, this is what we do: http://paste.openstack.org/show/238392/19:43
*** patrickeast__ has joined #openstack-infra19:45
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Make caching work when cloud name is None  https://review.openstack.org/18569919:45
clarkbSpamapS: why did the test need to be so complicated?19:46
*** rlandy has quit IRC19:46
clarkbSpamapS: all we needed to do was assert the correct cache region existed if I understand the code properly19:46
*** bhunter71 has quit IRC19:47
*** koolhead17 has quit IRC19:47
*** patrickeast_ has quit IRC19:48
SpamapSclarkb: there is only one region. :-P19:48
SpamapSclarkb: I so wish it was that simple.19:48
clarkbright so assert it exists with the correct name19:48
SpamapSThere is only one, and it is not named after the cloud.19:48
*** Swanson has quit IRC19:48
clarkbthen what is the function that was fixed doing?19:48
SpamapSgenerating keys19:48
*** panda has quit IRC19:48
clarkbok then all we need to do is check that the "None" key exists?19:49
clarkbI am confused why we need to make any images at all19:49
SpamapSno, it prefixes all keys with the cloud name19:49
clarkbok maybe I should not dig into this19:49
clarkb(isn't that what different regions exist for?)19:49
SpamapSso the idea is to keep caching config simpler than cloud config. :)19:49
SpamapSwhich complicates the other end, but that seems like a win.19:50
*** maishsk_afk has joined #openstack-infra19:50
SpamapSif you want a region per cloud, that would be neat, but this gives one cache to bind them all.19:50
clarkbbut there is no point to one cache for all clouds19:50
SpamapSgo fix it19:50
clarkbbecause we are already namespacing them in a horrible way19:50
*** Sukhdev has joined #openstack-infra19:50
SpamapShave fun. :)19:50
mordredclarkb: so - yeah - this is just the way it works right now19:51
SpamapSI don't think it's broken, but you're welcome to report broken things.19:51
clarkbSpamapS: no I don't think it is broken either, I think it is highly confusing and makes no sense19:51
clarkbbut the code likely works19:51
*** yamamoto has joined #openstack-infra19:51
SpamapSIt also isn't such a nonsensical thing to say the cache is for _the client_ not for _the cloud_19:51
SpamapSthere is only one client.19:51
mordredright. that's the thing it's there for19:51
clarkbthere is a 1:1 mapping between cloud and shade client19:52
*** patrickeast__ has quit IRC19:52
mordredalso - I believe there was a reaon why using cloud name and cache region did not work when we first looked at it19:52
clarkbat least thats a object not class member19:52
mordredoh - no - so - I remember the details now19:52
mordredthe dogpile cache "region" just does what our region code does, but it doesn't have enough information in it to segment appropriately19:53
mordredit's not realy a data concept19:53
mordredit's just an appended string into the cache key19:53
mordredbut since we have to construct cache keys _anyway_19:53
clarkbmordred: sure but now anything accessing the cache has to have special built code around it19:53
clarkbmordred: rather than just talking dogpile19:54
mordredthat is the case regardless19:54
clarkbwhich leads to confusion19:54
clarkband bugs19:54
clarkb(eg this one)19:54
mordredwe don't have that custome code because of the keys19:54
SpamapSYes, that is always going to be the case because dogpile is really dumb about key generation.19:54
mordredwe have the custom code because of how dogpile likes to be instantiated and live19:54
SpamapSintentionally IIRC19:54
*** panda has joined #openstack-infra19:54
clarkbthis particular bug is a direct descendant of this design decision19:54
mordredit's not a design decision19:54
clarkband its undocumented and makes no sense so we will have more of them19:54
mordredit is 100% necessary19:54
clarkbmordred: ok you haven't explained that at all19:54
clarkbmordred: in theor dogpile handles keys for you19:55
clarkbat least on a region basis?19:55
SpamapSclarkb: read dogpile's default key generator. It will not work for us. You will _always_ need the key generator from shade.19:55
clarkbanyways, my point is the code here is confusing enough to have led to this bug, this makes me question the decision here, it may be necessary and thats fine but wow19:55
*** e0ne has quit IRC19:55
mordredwell, I think the takeaway is that we could probably use a comment explaining why we have one19:56
*** mpavone has quit IRC19:56
SpamapSIIRC they'd like to fix that but it's relatively insane-unreadable-python to do it with a metaclass and some other magic.19:56
mordredso that when we go read the code we don't provoke clarkb (or others) to have a WTF, doesn't dogpile do this for you?19:56
SpamapSmordred: docstring for the key generator would do it I think.19:57
mordredSpamapS: yah19:57
mordredand would be a good idea in general19:57
mordredthat said - I think we've grown big enough in this channel that we may need to step back away from the combative manner of poking at code. it puts people into a defensive mode and is not particularly pleasant19:57
ParsectiXclarkb: Hello. I have made all steps in Dev guide. How I can start contributing :)19:57
mordredI think it worked well when there were 4 of us19:58
mordredbut there's like a billion active people up in here now19:58
openstackgerritGeorge Peristerakis proposed openstack-infra/system-config: Moved the server class out from the logstash and logstash_worker class  https://review.openstack.org/18505319:59
*** marcusvrn has quit IRC19:59
timrcfungi, Was good to meet ya :)20:00
*** e0ne has joined #openstack-infra20:00
fungitimrc: likewise!20:00
clarkbParsectiX: once you have your account stuff setup next step is to find a bug/feature/etc to tackle and push up20:01
SpamapSmordred: btw this has forced me to go back and read dogpile.cache's docs again and it reminds me that we probably also need to use the sha1 key mangler.20:01
clarkbParsectiX: I would star with something you are interest in or have some familiarity with20:02
rbradfor_clarkb, why this specific image I do not know, it's sourced from https://git.openstack.org/cgit/openstack/magnum/tree/devstack/plugin.sh#n25   If I have a list of cached images I can ask the question why that cannot be used.20:02
SpamapSmordred: otherwise people will 'splode lots on memcached bits20:02
mordredSpamapS: k20:02
yolandaasselin, btw, about nodepool work for puppet-openstackci, i started job. But i'd prefer to start moving functionality bits to puppet-nodepool prior to the move to openstackci.  As some bits, as configuring the nodepool.yaml, in a flexible way, are a bit complex20:02
mordredSpamapS: docs ... definitely needed20:02
asselinyolanda, +120:02
SpamapSmordred: also are we planning on running nodepool with some caching turned on?20:02
ParsectiXclarkb: Okay nice.20:02
yolandai'd prefer to do incremental steps for that, or the review for the nodepool module is going to suck a bit20:03
mordredSpamapS: not right now - currently nodepool manages local caches itself20:03
mordredSpamapS: and since it uses a mix of shade and non-shade calls, I think it needs to20:03
clarkbrbradfor_: I think that is what we need to sort out, let me show you which one we currently cache20:03
SpamapSmordred: all in-memory?20:04
asselinyolanda, yeah, agree to do one way or the other...I prefer to avoid mixing the two approaches.20:04
mordredSpamapS: once the nodepool-dib use case is landed and solid, I thin kwe can start to look at that20:04
mordredSpamapS: yah - but it only caches 2 things- flavor list and image list20:04
SpamapSmordred: just thinking about purging.. the memory backends for dogpile don't free memory IIRC20:04
*** radez is now known as radez_g0n320:04
mordredSpamapS: right. it doesn't so anything fancy like that20:04
mordredSpamapS: it is a dict with values in it20:05
*** ayoung has quit IRC20:05
*** erikmwilson_ has joined #openstack-infra20:05
mordredSpamapS: if we do start to turn on dogpile caching for nodepool, I actually think the dbm backend might be nice - that way the cache will persist across restarts and we won't get a herd of nova flavor-list like we do now20:05
SpamapSmordred: ok thats exactly what MemoryBackend is for dogpile.cache20:06
*** rbradfor has left #openstack-infra20:06
SpamapSfine if you plan to just keep overwriting the same few keys. Not so good for a large breadth of things.20:06
SpamapSI'm actually surprised dogpile.cache can't short-circuit entirely and even skip key generation with the NullBackend.20:06
openstackgerritMerged openstack-infra/project-config: Reconfigure irc-meetings output  https://review.openstack.org/18567720:07
yolandaasselin , yes, so i'll move the  functionality i can to puppet-nodepool, then the refactor will be simpler20:07
clarkbrbradfor_: https://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/image_list.sh is the script we run to get the list of images devstack would like us to cache20:07
SpamapSwe could actually noop things in the decorator if we detect nullbackend. That's not a terrible idea if we see CPU problems.20:07
asselinyolanda, ok, please use the same gerrit topic. thank you20:07
clarkbrbradfor_: it looks at stackrc to get a list of images. One of which is a fedora image for heat tests20:07
*** yamahata has quit IRC20:07
*** timcline has quit IRC20:08
clarkbrbradfor_: the HEAT_FETCHED_TEST_IMAGE looks like20:08
* SpamapS steps away from the premature optimization trap and goes back to making things harder faster better stronger.20:08
*** koolhead17 has joined #openstack-infra20:09
clarkbrbradfor_: in theory you can just run that script and have it tell you what images it is caching20:10
clarkbrbradfor_: I am working to confirm that now20:10
openstackgerritJoshua Harlow proposed openstack/requirements: Kazoo 2.1 has a bug which is breaking tooz gate  https://review.openstack.org/18509520:10
*** mtanino has quit IRC20:11
clarkbthats interesting it doesn't list the fedora image when run against master, maybe that is from an older branch?20:11
clarkbrbradfor_: confirmed, stable/juno lists the fedora image20:12
*** aduarte has quit IRC20:13
*** ZZelle_ has joined #openstack-infra20:13
*** prad has quit IRC20:13
rbradfor_clarkb, i am lost with what you mean by stable/juno20:14
*** mtanino has joined #openstack-infra20:14
clarkbrbradfor_: the stable/juno branch of devstack emits the fedora image when you run that script20:14
*** samueldmq has quit IRC20:15
*** stevebaker has quit IRC20:17
*** stevebaker has joined #openstack-infra20:17
*** stevebaker has quit IRC20:17
*** stevebaker has joined #openstack-infra20:17
*** ayoung has joined #openstack-infra20:18
*** jamespage_ has quit IRC20:18
*** sigmavirus24_awa is now known as sigmavirus2420:18
*** stevebaker has quit IRC20:18
*** stevebaker has joined #openstack-infra20:18
fungiso i guess heat has moved their tests which used that to a functional testing job or something?20:18
fungithat might start to get ugly once juno reaches eol20:19
fungior sooner if the image they used for juno devstack jobs gets replaced by a newer one in the functional jobs20:20
clarkbfungi: ya, its likely we need to do something here to better accomodate them20:20
*** freerunner has joined #openstack-infra20:20
clarkband hopefully we can accomodate the current set of folks that have expressed a need for fedora images20:20
*** tiswanso has quit IRC20:22
fungioh, looks like they're building and uploading a http://tarballs.openstack.org/heat-test-image/fedora-heat-test-image.qcow220:22
fungiso they followed in the trove agent tests' footsteps i guess20:22
clarkbbut we don't cache it (unlike the trove agent image)20:22
fungiso that still needs solving20:23
*** sgolovatiuk has left #openstack-infra20:23
*** stevebaker has quit IRC20:23
*** stevebaker has joined #openstack-infra20:23
clarkbso really I think this ends up boiling down to "what do heat and magnum both need?" "build that image or pull one off the shelf" "cache it in nodepool images"20:24
*** nikeshm has joined #openstack-infra20:24
*** stevebaker has quit IRC20:24
nikeshmpatrickeast : hi20:24
*** stevebaker has joined #openstack-infra20:24
*** teran has joined #openstack-infra20:25
nikeshmpatrickeast: i am able to passthrough FC HBA in vms manually20:25
*** maishsk_afk has quit IRC20:25
asselinhi nikeshm, want to chat in cinder channel?20:26
clarkbmordred: I have +2'd SpamapS' latest patchset to fix caching, but could use your rereview20:26
mordredclarkb: awesome- will do after TC meeting20:26
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added npm-run jobs to refstack.  https://review.openstack.org/18573120:26
clarkbmordred: thanks, and i guess it will need a new release too20:26
*** radez_g0n3 is now known as radez20:26
mordredclarkb: yah. so will os-client-config20:26
nikeshmpatrickeast: by your scripts but have some doubts in scripts20:26
*** timcline has joined #openstack-infra20:26
nikeshmasselin: ok20:27
openstackgerritDavide Guerri proposed openstack-infra/shade: Add port resource methods  https://review.openstack.org/17996820:27
mordredI eman, strickly the shade patch will fix it - but the os-client-config is where the fix REALLY should be - as from shade's point of view clouds should always have names20:27
*** stevebaker has quit IRC20:27
*** julim has quit IRC20:27
*** stevebaker has joined #openstack-infra20:27
*** ayoung has quit IRC20:27
clarkbsure, but I can't affect change in that repo directly :)20:27
*** gyee has joined #openstack-infra20:28
mordredclarkb: we could fix that ...20:28
*** samueldmq has joined #openstack-infra20:34
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Enable NeutronL3HA for the HA job  https://review.openstack.org/18546120:36
*** otter768 has joined #openstack-infra20:37
*** yamamoto has quit IRC20:38
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add docs about using subunit2sql-graph  https://review.openstack.org/18369620:38
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add docs about using subunit2sql-graph  https://review.openstack.org/18369620:38
*** prad has joined #openstack-infra20:38
*** yamahata has joined #openstack-infra20:41
*** otter768 has quit IRC20:42
*** ayoung has joined #openstack-infra20:42
*** yamamoto has joined #openstack-infra20:43
clarkbmordred: did the os-client-config change release recently? because this was working in my devstack env that doesn't set cloud: et all iirc20:43
clarkbya no cloud in nodepool.yaml using os-c-c 1.0.020:44
clarkbwhich is supposedly the latest release20:44
*** bhunter71 has joined #openstack-infra20:45
mordredclarkb: if you have a nodepool.yaml, you get a cloud name20:46
mordredclarkb: this only gets triggered if you use no external config at all and do everythign 100% programatically20:46
clarkbmordred: then why does it happen for prod nodepool?20:46
mordredbeause we have not yet landed the clouds.yaml patch for nodepool20:46
*** peristeri has quit IRC20:46
clarkbthat should be fine I don't have a clouds.yaml either20:46
mordredthat is a good question20:47
*** hdd has joined #openstack-infra20:47
mordredclarkb: self.provider.cloud is definitely None in prod20:48
clarkband definitely None in my test instance I am checking versions on nodepool.o.o too20:48
greghaynesmordred: I dont think we pass cloud to shade if you have nodepool.yaml20:48
clarkbthey look the same. shade==0.6.1 and os-c-c==1.0.020:48
greghaynesWe check if you have cloud set, if you dont then we dont pass that kwarg20:49
mordredok. *facedesk*20:49
mordrednope. this is all the way back to clarkb's complaint from ages ago - the mixed use of cloud to mean name of cloud and cloud to mean name of cloud provider20:49
mordredI mean - the other thing IS a bug and it's good that we fixed it20:50
clarkbyes I agree its a bug should be fixed, but not understanding why we are only partially affected (eg only in prod and not in test instance)20:50
mordredagree - and now looking further20:50
*** stevebaker has quit IRC20:50
*** stevebaker has joined #openstack-infra20:50
openstackgerritJames E. Blair proposed stackforge/gertty: Advance cursor on change list toggle  https://review.openstack.org/18569820:52
openstackgerritJames E. Blair proposed stackforge/gertty: Allow reviewing one change in change list  https://review.openstack.org/18569720:52
openstackgerritJames E. Blair proposed stackforge/gertty: Fix updating flags on threaded changes  https://review.openstack.org/18569620:52
openstackgerritJames E. Blair proposed stackforge/gertty: Make "limit" a noop in queries  https://review.openstack.org/17267020:52
openstackgerritJames E. Blair proposed stackforge/gertty: Fix searching by reviewer account id  https://review.openstack.org/17266920:52
openstackgerritJames E. Blair proposed stackforge/gertty: Batch sync change by commit tasks  https://review.openstack.org/17269120:52
openstackgerritJames E. Blair proposed stackforge/gertty: Fix age searching  https://review.openstack.org/17266820:52
openstackgerritJames E. Blair proposed stackforge/gertty: Fix refresh on project and change lists  https://review.openstack.org/18388720:52
openstackgerritJames E. Blair proposed stackforge/gertty: Fix diff display of deleted empty files  https://review.openstack.org/18054620:52
openstackgerritJames E. Blair proposed stackforge/gertty: Add database pruning  https://review.openstack.org/17998120:52
openstackgerritJames E. Blair proposed stackforge/gertty: Attach comments to files  https://review.openstack.org/17998020:52
openstackgerritJames E. Blair proposed stackforge/gertty: Add files table  https://review.openstack.org/17997920:52
openstackgerritJames E. Blair proposed stackforge/gertty: Add ability to review multiple changes at once  https://review.openstack.org/18569520:52
clarkbbut I need lunch so back in a bit to dig in20:52
jeblair(minor changes + rebase)20:52
*** stevebaker has quit IRC20:52
*** stevebaker has joined #openstack-infra20:52
*** ParsectiX has quit IRC20:53
*** jgrimm is now known as zz_jgrimm20:53
openstackgerritMerged openstack-infra/shade: Make caching work when cloud name is None  https://review.openstack.org/18569920:53
clarkbmordred: fwiw I was testing with your change to use shade for image uploads but I don't think that would affect shades cache at all20:53
greghaynesI also tested against devstack and no clouds.yaml or cloud: in my provider config20:54
*** ParsectiX has joined #openstack-infra20:54
greghaynesso theres something else wierd going on20:54
*** aysyanne has quit IRC20:56
*** rguillebert has joined #openstack-infra20:57
mordredok. I must clearly still have jetlag20:58
lifelessmordred: SpamapS: looks like a bug in that particular matcher.20:59
clarkbmordred: greghaynes because food is apparently not that important the unauthorized 401 error is what I see in the logs20:59
fungiSpamapS: mordred: clarkb: now that 185699 has merged we need a 0.6.2 tagged, yeah?21:00
clarkbmordred: greghaynes so I am guessing that we actually have two issues21:00
clarkbthe first is the one fungi hit and we just fixed (need a release)21:00
clarkband the second is whatever causes the 401 errors21:00
*** bswartz has quit IRC21:01
*** dannywilson has quit IRC21:01
greghaynesseems plausible21:01
mordredI foudn that21:01
*** gokrokve has quit IRC21:01
clarkbmordred: ya21:02
mordredthe thing we fixed was a total red herring21:02
*** gokrokve has joined #openstack-infra21:02
mordredpatch coming21:02
clarkbmordred: well sort of, it breaks our ability to fix things manually21:02
clarkbmordred: so good to fix both :)21:02
mordredwell, yeah21:02
clarkbmordred: but I am not sure I grok why the unauthorized error happen21:02
mordredno - I mean, it was a good thing to fix, because yeah21:02
mordredI do21:02
*** dannywilson has joined #openstack-infra21:02
greghaynessuch suspense21:02
mordredget_one_cloud() is broken - the code path of not passing a named cloud in is completely farscically wrong - I have NO IDEA how your testing worked21:02
greghaynesmordred: wah21:03
greghaynesmordred: it was totally broken, I thought I fixed it21:03
jeblairmordred, tchaypo, lifeless, clarkb, fungi, *: does anyone else want to test gertty change 179980 before i land it? it's a major db migration, so if you haven't run it, you may want make a backup of .gertty.db.21:03
*** erikmwilson_ has quit IRC21:03
mordredgreghaynes: nope. now- once I put up the next patch - I REALLY want to understand what's _working_ in your testing21:03
greghaynesmordred: oh! I bet im passing in a cloud21:03
lifelessrussellb: commented on https://review.openstack.org/#/c/182746/21:03
mordredbecause figuring out how we got a false positive is important21:03
jeblairtchaypo: (i think you tested it, right?  you found a bug i fixed in 180546 iirc)21:03
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Use setuptools extras for graphing requirements  https://review.openstack.org/18427821:04
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add docs about using subunit2sql-graph  https://review.openstack.org/18369621:04
clarkbmordred: is the node I tested on21:04
*** patrickeast_ has joined #openstack-infra21:04
greghaynesmordred: hrm, maybe not21:04
mordredclarkb: awesome. I'll poke there next21:04
lifelessjeblair: thanks for asking, but I'm ENOTHINK at the moment - I'll cope, or worst case just start over.21:04
mordredclarkb: I mean, I want to fix this - but I do not want to consider this fixed for us until we understand the false-positive21:04
*** HenryG has quit IRC21:04
clarkbmordred: you can login then run sudo -u clarkb -H venv/bin/nodepoold -c ./nodepool.yaml -d21:04
mordredbcause this should have hilariously broken21:04
clarkbmordred: and that will run nodepool for you as me in my homedir21:04
jeblairlifeless: ack, thx21:04
* tchaypo checks21:04
mordredclarkb: should be able to trigger this with nodepool image-upload even21:05
greghaynesmordred: oh, yep, cloud: fake-cloud21:05
*** jsavak has quit IRC21:05
mordredgreghaynes: that'll do it21:05
*** patrickeast has quit IRC21:05
*** patrickeast_ is now known as patrickeast21:05
mordredgreghaynes: did you have fake-cloud defined somewhere?21:05
*** baoli has quit IRC21:05
russellblifeless: thanks!21:05
clarkbjeblair: I can upgrade after lunch21:05
fungijeblair: it's running for me now. i'll let you know once it completes21:05
mordredgreghaynes: actually - let me make patch - it'll be easier to talk about the problem once you see the break21:05
greghaynesmordred: I dont think so. Just in the fixtures/node_osc.yaml21:06
tchaypoI’m currently running patchset 1 of 183887, which seems to include that21:06
greghayneswhich is the config the test uses21:06
tchaypoif this was the big long migration, I was happy with that21:06
*** rossella_s has quit IRC21:06
jeblairclarkb, fungi: ack, thanks.21:06
fungidoes seem to take a few minutes21:06
fungimust be doing lots of awesome21:06
jeblairtchaypo: yep.  no substantial changes in the rebase.21:06
fungiahh, it's indexing files for the search21:06
greghayneshrm, but theres also a node.yaml test which doesnt have cloud: defined. so no idea how that one passes21:07
mordredoh - ok - it's not as broken as I thought21:07
*** rossella_s has joined #openstack-infra21:07
* mordred has several test cases to write coming up ...21:07
clarkbit gives you a little progress monitor, will eat while that runs21:07
*** HenryG has joined #openstack-infra21:07
clarkbmordred: greghaynes I definitely do not have a cloud: anything in my test nodepool.yaml21:07
greghaynesclarkb: yes, I didnt either21:08
jeblairclarkb: yeah, it's great meal-time entertainment21:08
greghayneshrm, maybe I should try and load up a nodepool.yaml config to hit hpcloud21:09
*** dahoo has joined #openstack-infra21:10
SpamapSlifeless: unfortunately I haven't been able to reproduce, so it may have been a bug with incompatible old versions or something.21:11
clarkbgreghaynes oh ya it could be bad config?21:11
clarkband somehow its inferred?21:11
greghaynesclarkb: yea, all the logic for getting config defaults and creating clients is what changed so thats my suspicion21:12
openstackgerritMerged openstack-infra/subunit2sql: Use setuptools extras for graphing requirements  https://review.openstack.org/18427821:12
*** ldnunes has quit IRC21:14
*** anthonyper has quit IRC21:15
*** mattfarina has quit IRC21:15
*** anthonyper has joined #openstack-infra21:16
*** ayoung has quit IRC21:17
fungijeblair: bad news. it just now died in a KeyError fire21:18
*** rossella_s has quit IRC21:19
fungijeblair: http://paste.openstack.org/show/238511/21:19
*** ayoung has joined #openstack-infra21:19
*** rossella_s has joined #openstack-infra21:20
fungijeblair: want a copy of my ~/.gertty.db file?21:20
clarkbmine is still going fwiw21:21
*** ildikov has quit IRC21:21
fungia copy of my original ~/.gertty.db from prior to the upgrade, i mean21:21
*** weshay has quit IRC21:21
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova/oslo.vmware bug 1459021  https://review.openstack.org/18574121:22
openstackbug 1459021 in OpenStack Compute (nova) "nova vmware unit tests failing with oslo.vmware 0.13.0" [Critical,Confirmed] https://launchpad.net/bugs/145902121:22
jeblairfungi: yes please, that would be great.21:23
samueldmqhi, could I have a couple of eyes on #179661 ?21:25
samueldmqanteaya had +2ed it, but it needed a rebase21:25
*** mohitsharma has joined #openstack-infra21:25
*** ParsectiX has quit IRC21:26
*** ParsectiX has joined #openstack-infra21:26
*** mohitsharma has quit IRC21:29
tchaypoI’m waiting for our tests to start using pip7.0.121:30
tchaypoI’m guessing that will come in the next image update?21:30
*** timcline has quit IRC21:30
asselinmordred, vhd-util should support precise? or should find a way to exclude from: https://git.openstack.org/cgit/openstack-infra/puppet-diskimage_builder/tree/manifests/init.pp#n27?21:30
*** tonytan4ever has quit IRC21:31
mordredasselin: I don't have much of an opinion on precise - I dont' personally really want to support it myself - but I'm not opposed to it existing21:31
asselinalso...how long does -infra plan to support precise?21:31
mordredwe already don't support building images on precise21:32
asselinok, then I'll plan to upgrade21:32
mordredit will make you happier21:32
fungijeblair: http://fungi.yuggoth.org/tmp/gertty_db.gz (11mib)21:33
*** ildikov has joined #openstack-infra21:33
mordredit seems that the generator returned by glance.images.list() ... is a proxy object21:35
mordredand iterating over it results in api calls21:35
*** tonytan4ever has joined #openstack-infra21:37
clarkbmordred: greghaynes anything else I can do to help debug this?21:38
mordredclarkb: no - I'm down a crazy rathole right now21:39
mordredthe thing I thought was the problem was not21:39
mordredbut I'm learning ALL SORTS of new evil21:39
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add project-team-guide  https://review.openstack.org/18574621:39
mordredclarkb: so - when I come up from air - I expect to know many more things21:39
clarkbmordred: it almost looks to me like we aren't giving glanceclietn a keystoneclient21:39
mordredclarkb: and I expect you will not like any of them21:39
jeblairfungi: download complete, thanks.21:39
openstackgerritMatt Riedemann proposed openstack/requirements: Block oslo.vmware 0.13.0 due to a backwards incompatible change  https://review.openstack.org/18574821:40
clarkbbut its making a request and either we dn't have a token or it is invalid21:40
mriedemblacklist is fun! ^21:40
greghaynesat this point id break once the clients are mafe in nodepool and check their config properties21:40
*** claudiub has quit IRC21:40
clarkbmordred: greghaynes so random thought, it takes us ~6 hours to build all the images could our token have expired by the time we try uploading?21:40
mordredI'm actually testing a very small python script that does the same thing21:40
mordredand I'm having issues21:40
mordredso - that's good21:40
*** Swami_ has quit IRC21:41
greghaynesclarkb: I think we upload one at a time...21:41
greghaynesIf I red the dib code correctly21:41
nibalizerfor new projects on project-config, use-storyboard: true or no?21:41
mordredclarkb: maybe21:41
clarkbgreghaynes: we don't. we build all of them then upload all of them21:41
clarkbgreghaynes: let me link you the code21:41
mordredclarkb: but I think we create a new OpenStackCloud object at the start of the upload thing, no?21:41
*** Swami has quit IRC21:42
clarkbgreghaynes: https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n193021:42
clarkbmordred: we start a new DiskImageUpdater thread yes21:42
greghaynesthe clients are created when the providers are IIRC21:43
clarkbgreghaynes: the fun bit is we just changed this code but aren't running it yet :/21:44
clarkbgreghaynes: but I think you are correct21:44
*** mtanino has quit IRC21:44
*** jsavak has joined #openstack-infra21:44
clarkbthe provider makes a shade openstack cloud client thing, and subclients are made at that point21:44
openstackgerritMerged openstack-infra/elastic-recheck: Add query for nova/oslo.vmware bug 1459021  https://review.openstack.org/18574121:45
openstackbug 1459021 in oslo.vmware "nova vmware unit tests failing with oslo.vmware 0.13.0" [Undecided,In progress] https://launchpad.net/bugs/1459021 - Assigned to Matt Riedemann (mriedem)21:45
*** pc_m has quit IRC21:45
SpamapSzigo: https://github.com/rohe/pysaml2/pull/22421:46
SpamapSzigo:  =================================== 9 failed, 440 passed, 16 error in 27.40 seconds ====================================21:46
zigoSpamapS: This rox! :)21:46
SpamapSzigo: it's become an obsession... one I need to let go of soon. ;-)21:46
zigoSpamapS: I hope you can finish them all. I am guessing that as you get deeper, the difficulty increases, right?21:46
SpamapSzigo: indeed, there are some questions about pickle formats and assumption of utf-8 now.21:47
openstackgerritMerged openstack-dev/pbr: Add kerberos deps to build the kerberos wheel.  https://review.openstack.org/18429421:47
openstackgerritMerged openstack-dev/pbr: Add home-page into sample setup.cfg  https://review.openstack.org/18486621:47
SpamapSzigo: and a few of the failures I think will require SAML deep knowledge21:47
SpamapSzigo: there are still a few shallow problems left, then the last few I am going to ask the core developers to address and then turn on python3 in travis-CI21:47
mordredok. my local testing is now working21:48
mordredclarkb, greghaynes: I have now _successfully_ run the code path in question in nodepool by hand with our fixes added and it works against hpcloud21:48
mordred(skipping any named clouds, invoking methods with the same params, etc)21:49
clarkbmordred: these are fixes that have not been pushed yet?21:49
mordredSO - if we see the auth error, I'm going vote on token expiration21:49
clarkbmordred: or fixes that went into the 1.0.0 release?21:49
mordredclarkb: just the os-client-config one that needs to land and release21:49
clarkbmordred: for None cloud?21:49
mordredclarkb: I thought there was a more complex break - but the None was the main one21:49
fungii wonder whether we should upload after each build rather than all at the end?21:50
clarkbmordred: kk21:50
openstackgerritClint Adams proposed openstack-infra/zuul-packaging: WIP - update zuul packaging for zuul 2.0.0  https://review.openstack.org/18575221:50
clarkbfungi: well the clients are made once21:50
mordredclarkb: I have verified however21:50
*** gokrokve has quit IRC21:50
fungiahh, right, at the beginning. not recreated at upload time21:50
mordredthat glanceclient does an api call for each image in list21:50
clarkbmordred: nice21:50
mordredso it's TOTALLY skipping out rate limiting21:50
mordredjeblair: ^^21:50
mordredjeblair: you'll enjoy that21:50
*** Sukhdev has quit IRC21:50
clarkbmordred: and its basically impossible to rate limit without major hax21:50
fungiyeah, that seems like a hole in someone's design21:50
mordredglanceclient.images.list() does not, in fact, talk to teh cloud21:51
*** ParsectiX has quit IRC21:51
mordredit only creates a generator which does as you iterate over it21:51
*** Sukhdev has joined #openstack-infra21:51
fungiprobably worth opening a bug against glanceclient about that21:51
clarkbmordred: ok so getting back to upload problems with auth, do you expect the noew code which we just merged but are not running yet to fix it. eg using the sahde interface instead of client directly?21:51
*** ParsectiX has joined #openstack-infra21:51
clarkbmordred: because I can go ahead and restart nodepool to get that in, but won't bother if not expected to work21:52
* clarkb reads shade now21:52
mordredwell - so we can test by running a nodepool image-upload by hand21:52
mordredwhich will trigger the glance client code via shade and will fail if it's wrong21:52
mordredwhich I think is an easier way than restarting nodepool to make sure it'll work21:53
mordredalso - if that works and nodepool itself gets 401s, then we know to chase token expiration21:53
clarkbmordred: except that os-c-c/shade bug fixes are still outstanding and that will always make a new token just prior to uploading I think21:54
clarkbmordred: but we can at least confirm it explodes less21:54
*** jklare has quit IRC21:54
*** mriedem is now known as mriedem_away21:54
*** waynr has quit IRC21:54
greghaynesSo the theory is our image builds are now taking longer and therefore tokens expire?21:54
clarkbgreghaynes: ya21:54
clarkband instead of reauthing we fail21:54
clarkbI think ideally $client would deal with this transparently21:55
mordredOH - wait a sec - I may have yelled prematurely21:55
mordredjeblair, clarkb: there is only one call to the API - it just happens when you're not expecting it to21:55
*** jklare has joined #openstack-infra21:55
mordredso - it's fixable on our side by changing what we run in a task21:55
clarkbmordred: you mean for the list thing21:56
clarkbmordred: want to make a shade release for the cache fix?21:56
*** salv-orlando has quit IRC21:56
mordredclarkb: well, yeah - we should release and install new shade/occ21:56
mordredyes I do21:56
clarkbI can make sure that gets installed on nodepool.o.o then try the manual image uploads21:56
*** ddieterly has quit IRC21:57
*** jsavak has quit IRC21:57
*** alexsyip has joined #openstack-infra21:58
*** jsavak has joined #openstack-infra21:58
jeblairfungi: can you "pip freeze|grep SQL"?21:58
clarkbjeblair: gertty update, about halfway through the migration21:58
fungijeblair: SQLAlchemy==1.0.421:59
openstackgerritSpencer Krum proposed openstack-infra/project-config: Add beaker-localhost and beaker-nodepool hypervisors  https://review.openstack.org/18575421:59
*** salv-orlando has joined #openstack-infra21:59
nibalizercrinkle: ^^21:59
*** dahoo has quit IRC21:59
vipulmordred: where would i find playbooks that utilize the os_ modules.. just want some examples21:59
clarkbnibalizer: project-config doesn't really have any puppet manifest to apply fwiw21:59
jeblairclarkb: i cleverly arranged the patch series so that change expiration happens _after_ the migration, so it will delete like 80% of the work it's doing right now.21:59
fungirbradfor_: you're being summoned in #openstack-meeting21:59
*** gokrokve has joined #openstack-infra21:59
fungirbradfor_: cross-project meeting21:59
fungithough it's about to end now21:59
clarkbjeblair: need to fuel my space heater somehow22:00
mordredvipul: gimme a sec- brain bleeding on a different task ...22:00
nibalizerclarkb: my change adds some repos for our (to be written) beaker plugins to make beaker less janky22:00
jeblairfungi: oh i'm 0.9.9.  let me see if that's related.22:00
vipulmordred: no worries.. get back to me when you get a break22:00
clarkbnibalizer: oh thats a project add, I read title as add the yaml file that tells beaker how to do things22:00
jeblairfungi: yep. 0.9.9 works, 1.0.4 breaks22:01
fungijeblair: yeah, i 'pip install -U .' for maximum chaos22:01
clarkbjeblair: fungi I did a pip install -U so have latest sqlalchemy22:01
*** dannywilson has quit IRC22:01
clarkbI guess mine will break shortly then22:01
crinklenibalizer: why would we need both localhost and nodepool?22:01
jeblairclarkb: i would not be surprised, but if you don't mind letting it finish for extra data, would be nice.22:01
clarkbjeblair: will do22:01
alexsyipHi, I’m tryihng to setup a tempest job to run in jenkins, but I have a question.  Could someone spare a minute to help me out?22:02
*** Sukhdev has quit IRC22:02
clarkbalexsyip: usually best to just ask22:02
crinklenibalizer: I'm not convinced we need another hypervisor, there seem to be settings internal to beaker that make it a little more complex22:03
alexsyipclarkb: ok.  I have this congress.yaml as a jenkins job: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/congress.yaml22:03
crinklenibalizer: working on that now22:03
*** changbl has quit IRC22:03
alexsyipwhen I run this job, there’s an error saying that the congress/contrib/tempest directory does not exist: http://logs.openstack.org/81/179181/4/experimental/gate-congress-dsvm-api/26311bf/console.html.gz22:04
*** dannywilson has joined #openstack-infra22:04
*** Sukhdev has joined #openstack-infra22:04
alexsyipI assume that’s because devstack does not fetch the congress directory by default.22:04
*** freerunner has quit IRC22:04
clarkbalexsyip: I think there are two problems there, the first is the one you mention, the jenkins workspace doesn't have congress cloned to it22:05
alexsyipI have a change to set ENABLED_SERVICES=congress,tempest here: https://review.openstack.org/#/c/180726/1/jenkins/jobs/congress.yaml22:05
*** gokrokve has quit IRC22:05
alexsyipDoes this look like the right way to fix this?22:05
clarkbalexsyip: the other is going to be that devstack-gate clones to /opt/stack/new/congress not to the workspace22:05
clarkbalexsyip: likely the correct fix is to set it up like a devstack plugin so that all of the cloning is done for you22:06
*** gokrokve has joined #openstack-infra22:06
clarkbalexsyip: then you can copy files as appropriate before having tempest run. You can see how the glusterfs plugin is done22:06
alexsyipIs there a description I can read about that?22:07
openstackgerritMonty Taylor proposed openstack-infra/shade: Make sure glance image list actually runs in Tasks  https://review.openstack.org/18575622:07
mordredclarkb, jeblair, greghaynes, SpamapS: ^^22:07
clarkbalexsyip: yes devstack has a documentation page on it http://docs.openstack.org/developer/devstack/plugins.html22:07
alexsyipwhere can I find the glusterfs example?22:07
*** amitgandhinz has quit IRC22:07
mordredthat's not needed this instant, but is needed overall22:07
clarkbalexsyip: in the same dir as your congress job22:07
mordredI'm going to release the current patch22:08
clarkbalexsyip: I am not sure of the exact file bit grep should tell you22:08
clarkbmordred: thanks22:08
alexsyipclarkb: ok thanks, I”ll check it out.22:08
mordredclarkb: btw - I ran this: http://paste.openstack.org/show/238572/22:09
mordredclarkb: except with the right password22:09
mordredclarkb: since that's what nodepool is doing22:09
mordredclarkb: to verify that I got an image list back22:09
mordredclarkb: and similarly tested that it breaks without the patch, and that it throws a 401 if it does not like my username/password22:10
greghaynesclarkb: any idea what changed to make image builds slower?22:10
clarkbgreghaynes: the disk volume22:10
fungii can try swapping it to an ssd22:10
clarkbgreghaynes: I am fairly certain the local ephemeral drive is significantly faster for IO than a cinder volume over the network22:11
fungias i discovered today, we have rather a lot of ssd quota available22:11
greghaynesclarkb: where its stored to or where the cace is or both?22:11
clarkbgreghaynes: both22:11
clarkbfungi: its probably worht a shot22:11
clarkbmordred: your shade patch will map a single task to many api calls so not perfect for rate limiting but slightly better than the old case22:11
fungii'll go ahead and create/attach one while i'm thinking about it and get the rsync going22:11
mordredclarkb: nope - it actually just winds up being one api call22:12
clarkbgreghaynes: shouldn't help since dib syncs at several locations22:12
mordredclarkb: because BONGHITS22:12
mordredclarkb: it creates a generator that yields, so the first time you iterate on the generator is when the API call gets triggered22:12
greghaynesclarkb: yeaaaa :/22:12
*** ParsectiX has quit IRC22:12
clarkbmordred: gotcha22:13
*** annegentle has quit IRC22:13
mordredclarkb: I patched urllib3 locally so that I could watch when it was actually opening a socket :)22:13
mordredbecause, you know, that should be necessary22:14
*** kutija has quit IRC22:14
mordredclarkb: ok. 0.6.2 tag has been pushed22:14
mordredfor shade22:14
clarkbI see the release jobs are queued22:14
*** enikanorov_ has joined #openstack-infra22:15
mordredalso - there was another image upload patch outstanding ... although it does not affect HP22:15
mordredbut we would have wanted to release before adding rax to the mix anyway22:16
mordredoh - or - wait - no - it affects zero clouds that we use22:16
mordredit's a fix for v2 + PUT22:16
clarkbwhich no one uses not even devstack22:16
mordredaka - all non HP and non RAX clouds22:17
*** Gomeler has joined #openstack-infra22:17
mordredor at least some of them - I added it because I ran in to it testing other clouds :(22:17
*** claudiub has joined #openstack-infra22:17
clarkbhttps://pypi.python.org/pypi/shade/  that shows 0.6.2 now22:18
clarkbI am going to pip install shade==0.6.2 now22:18
clarkbon nodepool.o.o22:18
*** hdd has quit IRC22:19
mordredclarkb: actually, I'm queued up to run the image upload command from before22:19
mordredif you want22:19
mordredor - sudo -H -u nodepool nodepool image-upload hpcloud-b1 devstack-trusty is what I ran back then22:19
clarkbalready running it, thanks22:19
* mordred crosses fingers22:20
clarkbit hasn't immediately thrown an error22:20
clarkbso thats progress22:20
SpamapSseconds of uptime!!22:21
mordredwell - it takes a few minutes - because it has to compute md5 and sha1 of the images22:21
SpamapSthats _plural_ baby22:21
fungiclarkb: jeblair: mordred: i have a pvmove of the sata pv to the ssd pv going on now for nodepool22:21
mordredso don't get excited YES22:21
*** doug-fish has left #openstack-infra22:21
mordredfungi: woot22:21
jeblairfungi: er, can you tl;dr that?22:22
fungiclarkb: jeblair: mordred: in a few hours i should be able to vgreduce the sata out and delete it22:22
jeblairsorry, i got sqlalchemied.22:22
clarkbmordred: fungi assuming the manual upload works I can restart nodepool22:22
clarkbthen tomorrow we will see how it does22:22
fungijeblair: yep, done it many times in years gone by, for example replacing one disk array with another live without taking the server offline22:22
jeblairfungi: oh, sorry, i mean, i missed why you're doing that22:22
fungier, i misread22:23
fungidib performance has apparently gotten slow since we moved the cache from an ephemeral volume to cinder22:23
*** jklare has quit IRC22:23
clarkbjeblair: our per image build time has gone from ~15 minutes to >1hour leading to about 6 hours to build all our dib images each day. We think this is related to the switch from local epehemeral disk to cinder volume22:23
clarkb(those times assume a warm dib cache)22:23
fungiand this is easy to try22:24
jeblairgot it.  ++22:24
fungias in, i can leave it running while i go cook dinner, then check in on it in a while ;)22:24
fungi2% complete now22:24
mordredmmm. dinner22:24
jeblairfungi, clarkb: i have fixed gertty, i'm just inserting the fix into the stack now22:24
fungii realize i should have started it in a screen session. oh well22:25
clarkbjeblair: cool, still waiting for it to fail here22:25
clarkbabout 3/4 of the way done according to my little counter22:25
*** waynr has joined #openstack-infra22:25
mordredclarkb: you havea  little counter?22:25
clarkbfungi: iirc pvmoves are resilient to the process going away and can be restarted22:25
clarkbmordred: ya its telling me how far along it has gotten22:25
fungiclarkb: yep, but i won't bother interrupting it22:25
clarkb7550/10098 ish22:25
*** dims has quit IRC22:25
*** shardy has quit IRC22:27
mordredclarkb: if we are seeing a stale token issue when we run in the daemon - I'm going to have to yell at morganfainberg ... I mean learn more about how to detect and retry that22:27
*** gordc has quit IRC22:27
clarkbmordred: see I thought keystone clients session stuff was supposed to handle this for us which probably means glance client isn't using that stuff?22:27
clarkbjamielennox: ^ is probably the person to ask22:27
*** dims has joined #openstack-infra22:28
*** jklare has joined #openstack-infra22:28
clarkbmordred: image upload done manually worked22:28
morganfainbergclarkb: iirc glanceclient is the odd one out22:28
clarkbmorganfainberg: awesome22:28
morganfainbergclarkb: it has some oddities that make it harder to do the session thing22:28
morganfainbergclarkb: jamielennox was talking about this at one point22:28
*** nelsnelson has quit IRC22:28
clarkbmordred: fungi so I am going to go ahead and restart nodepool.o.o now22:28
clarkber not the host just the service22:29
morganfainbergclarkb: it means we have more work to do :( and my sincerest apology for not having it all done sooner. *sorry* :(22:29
morganfainbergclarkb: cause this does suck as a user to have to deal with22:29
clarkbmorganfainberg: the thing that is weird to me is we all have to auth with keystone22:29
clarkbmorganfainberg: so how hard can it be to all do that the same way in the same language as part of the asme project22:29
*** akshai has quit IRC22:29
clarkbargle bargle22:29
mordredmorganfainberg: so - once it's on session it'll be good?22:29
morganfainbergclarkb: keystoneauth - we are making a push that way for *this* exact reason22:30
clarkbmordred: aiui yes22:30
morganfainbergmordred: yah that is my understanding22:30
clarkbmorganfainberg: sure but keystoneclient was the thing before that22:30
*** camunoz has joined #openstack-infra22:30
clarkbmorganfainberg: so we can't just say having a common lib is going to fix it22:30
mordredclarkb: yah. well, glance and ironic (/me glares at devananda) are the two holdouts22:30
mordredclarkb: well, no - it won't fix it22:30
morganfainbergclarkb: ksc was the way we got most things done. KSA is fixing some issues with KSC as we finish it all up22:30
mordredclarkb: but the other clients hadn't moved all the way to ksc.Session yet22:30
mordredmost notably glanceclient22:30
morganfainbergclarkb: mostly a speed/load lots of cruft issue22:30
clarkbmordred: I know, my point is that saying KSA doesn't fix the problem which is we suck at actually implementing the things22:30
mordredclarkb: totally agree22:31
clarkbmordred: because KSC/KSA isn't aprt of the problem here22:31
mordredthe problem here is finishing porting the client libs to teh solution we actually ahve22:31
morganfainbergand i know glanceclient is on the short list.22:31
mordrednot the KSC/KSA thing22:31
fungiclarkb: sounds good22:31
morganfainbergso we can get it fixed.22:31
mordredmorganfainberg: btw - while we're yelling and screaming22:31
morganfainbergi know it was a real issue in the internal workings.22:31
*** hdd has joined #openstack-infra22:31
*** tonytan4ever has quit IRC22:31
mordredmorganfainberg: I would like to say something _nice_ for a change22:31
clarkbnodepool service has been restarted22:31
*** miqui has quit IRC22:32
morganfainbergmordred: oh crap! *braces for nice words*22:32
mordredI tried to use python to connect to a google API over the weekend using oauth22:32
openstackgerritJames E. Blair proposed stackforge/gertty: Do not enqueue duplicate tasks  https://review.openstack.org/17265022:32
openstackgerritJames E. Blair proposed stackforge/gertty: Support regexes in search  https://review.openstack.org/17264922:32
openstackgerritJames E. Blair proposed stackforge/gertty: Add database pruning  https://review.openstack.org/17998122:32
openstackgerritJames E. Blair proposed stackforge/gertty: Attach comments to files  https://review.openstack.org/17998022:32
mordredkeystone is awesome22:32
openstackgerritJames E. Blair proposed stackforge/gertty: Add files table  https://review.openstack.org/17997922:32
openstackgerritJames E. Blair proposed stackforge/gertty: Add ability to review multiple changes at once  https://review.openstack.org/18569522:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix searching for message  https://review.openstack.org/17266722:32
openstackgerritJames E. Blair proposed stackforge/gertty: Advance cursor on change list toggle  https://review.openstack.org/18569822:32
openstackgerritJames E. Blair proposed stackforge/gertty: Allow reviewing one change in change list  https://review.openstack.org/18569722:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix updating flags on threaded changes  https://review.openstack.org/18569622:32
openstackgerritJames E. Blair proposed stackforge/gertty: Make "limit" a noop in queries  https://review.openstack.org/17267022:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix searching by reviewer account id  https://review.openstack.org/17266922:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix age searching  https://review.openstack.org/17266822:32
openstackgerritJames E. Blair proposed stackforge/gertty: Batch sync change by commit tasks  https://review.openstack.org/17269122:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix refresh on project and change lists  https://review.openstack.org/18388722:32
openstackgerritJames E. Blair proposed stackforge/gertty: Fix diff display of deleted empty files  https://review.openstack.org/18054622:32
openstackgerritJames E. Blair proposed stackforge/gertty: Add support for SQLAlchemy 1.0.4  https://review.openstack.org/18576122:32
jeblairmordred: that's what i have to say about that22:32
openstackgerritClark Boylan proposed openstack-infra/system-config: Add ubuntu-trusty images to rax  https://review.openstack.org/18576222:32
morganfainbergmordred: that... feels like damnation through faint praise :P but I'll still take it!22:32
mordredEVERYTHING about the google api experience was the worst thing ever22:32
jrollmordred: lol. been there, google oauth is the worst.22:32
morganfainbergmordred: that would be my experience with Oauth.22:32
clarkbmordred: fungi greghaynes ^ depending on how adventurous we are that is the next piece in this22:32
morganfainbergmordred: just in general22:32
mordredclarkb: I see no issues with landing that - it won't break existing thigns :)22:33
jeblairclarkb, fungi: ^ should fix it for 1.0.4.  i increased the requirement but also made it work with both 0.9.9 and 1.0.4 to try to avoid breaking people during schema migrations22:33
devanandamordred: I feel like I'm missing some context about KSC and python-ironicclient22:33
clarkbmordred: correct, my vote is to land it then pay attention tomorrow22:33
mordredmorganfainberg: yah. it wasn't really even google - it was "how the heck is someone supposed to _USE_ this oauth crap"22:33
morganfainbergclarkb: so let me chase down jamielennox here sooon and talk about glanceclient fixes and the move to KSA down the line should be good.22:33
mordredclarkb: ++22:33
devanandai also haven't read all of backscroll22:33
mordreddevananda: keystone Session22:33
mordreddevananda: you're still not on it22:34
jrollmorganfainberg: the bonus with google is there's a whole bunch of google-centric oauth libraries that barely work, so it takes a couple hours of rage before you even get to the "let's do normal oauth" rage22:34
mordreddevananda: you want to me22:34
mordreddevananda: you want to be22:34
devanandaoh. yes. I hate our clients lack of session handling22:34
mordreddevananda: yes22:34
mordreddevananda: you do not disagree22:34
clarkbdevananda: which means if tokens expire *I* have to get a new one22:34
morganfainbergjroll: oh so take the hell that is oauth and make it more... oauth-y? :P22:34
fungijeblair: thanks. i'll try it again shortly22:34
clarkbjeblair: still waiting for mine to finish22:34
jrollmorganfainberg: something like that, it's been years :P22:34
mordredjroll: it's gotten worse then if it's been years22:34
*** radez is now known as radez_g0n322:34
morganfainbergmordred: seriously, OIDC - its the least icky of the non-SAML things...22:35
jrollmordred: oh, goody22:35
mordredjroll: NOW they have multiple different api suites that barely work, docs that talk about old auth, and docs that say don't use that use the new oauth2 - except with no libraries really updated to use it22:35
morganfainbergbut it's still a pile of ick22:35
morganfainbergas a developer22:35
devanandamrda: this might be a thing you have thoughts about / time to fix22:35
morganfainbergas a user... it's cool "click" oh lookie i'm logged in22:35
*** lifeless has quit IRC22:35
clarkbmordred: it would also be awesome if your token expired was a different error than lol auth failed22:35
zigoSpamapS: The urlparse issues are trivial to fix.22:35
devanandamrda: since you're already implementing some caching in our client. want to make it use keystone session too? (or does this request make you want to throw cats at me?)22:35
fungijeblair: migration is running again with a fresh checkout of 17998022:36
zigoSpamapS: Should I do it on top of your patches, or are you doing it?22:36
SpamapSzigo: Yes I'm working through things in smaller chunks because fixing those produces more not-easy errors. :)22:36
clarkbmordred: because then I as the user could more easily test that condition and get a new token, whereas the current implementation probably means we will naively assume expired token, make a new one, try then if still failing raise22:36
jamielennoxmorganfainberg: i'm in and out but the glanceclient review is https://review.openstack.org/#/c/141994/ they've been slow in reviewing it but i chased down a few people at summit to move it along22:36
*** annegentle has joined #openstack-infra22:36
jrollmordred: yeah, that sounds like the pain I was hitting, maybe they were google api libraries that didn't work with the oauth version the server was using22:36
morganfainbergclarkb: i'd be ok with making that kind of change - can i use HTTP 402?22:36
mordredclarkb: ++22:36
jamielennoxmorganfainberg: i don't really want to wait for ksa, i'll get it merged like that and it can consume ksa like everyone else will22:36
morganfainbergclarkb: pay us and we'll let you back in? :P22:36
morganfainbergjamielennox: wasn't advocating waiting on glanceclient22:36
mordredclarkb: well, thing is - many things do late binding - so if I detected a token fail HEAVEN HELP ME to fix it22:36
*** mwhagedorn has quit IRC22:36
*** krtaylor has quit IRC22:37
morganfainbergjamielennox: was saying we should get them over to session *then* the move to KSA is easier22:37
mordredbecause the move to ksa will be the same as the other moves to ksa22:37
mrdadevananda: will read this conversation over and take a look22:37
morganfainbergjamielennox: getting them on session is more important than KSA.22:37
zigoSpamapS: I've tried adding python3 support in oslo.messaging, though I get issues with eventlet.monkey_patch(), do you know what should be done for that?22:37
SpamapSzigo: Victor is working on that22:37
clarkbmorganfainberg: 408?22:37
zigoAh, cool.22:37
SpamapSzigo: it already passes most tests22:37
clarkbmorganfainberg: thats probbaly overloading things a bit22:38
jamielennoxyep, ok so it's in progress and i've got a few people on the hook to get it done22:38
morganfainbergclarkb: nope, that implies no-change in the request needed22:38
morganfainbergclarkb: we need to change the request in this case, new authorization22:38
*** otter768 has joined #openstack-infra22:38
clarkbmorganfainberg: glance could tell me 408 and I think that would be ok22:38
morganfainbergcould a token be considered a pre-condition?22:38
clarkbmorganfainberg: then go get a new token, and make same request back to glance again22:39
morganfainberg412 Precondition Failed22:39
clarkbanyways I don't care what the error is just trying to point out that this is all very unhelpful to users22:39
morganfainbergclarkb: from a strict security perspective... timeout token is no different than invalid token22:39
clarkbmorganfainberg: iirc preconditions are specific headers?22:39
*** Gomeler has left #openstack-infra22:39
clarkbmorganfainberg: yes but it doesn't tell your user how to fix the problem22:39
morganfainbergclarkb: "The precondition given in one or more of the request-header fields evaluated to false when it was tested on the server."22:40
clarkbmorganfainberg: 99% of your users will immediately go check username and password22:40
SpamapSzigo: in fact recently I think it was just qpid failing.22:40
clarkbmorganfainberg: then swear up and down they are correct (because they are), then give up and drink until liver failure22:40
zigoSpamapS: As far as I know, the plan is to just disable tests in Py3 + qpid.22:40
zigoWhich is fine to me ...22:40
SpamapSzigo: thats already done, they're just skipped22:40
morganfainbergclarkb: like i said, strictly from a security perspective. - i'd like to look to see if we can find some prior art on how to handle this.22:41
zigoSo, what's the issue then?22:41
*** ayoung has quit IRC22:41
zigolisting tests?22:41
morganfainbergclarkb: and see if we can follow said accepted prior art22:41
SpamapSzigo: some zmq stuff and ??? ask Victor22:41
samueldmq418 I'm a teapot lol22:41
zigoAh ok.22:41
morganfainbergclarkb: if we are making a change.22:41
morganfainbergbut i'm not really opposed to giving better feedback where we can reasonably do so.22:41
mordredmorganfainberg: I may have convinced ayoung that it's ok for us to return unauthorized instead of api not found for api endpoints taht are disabled by policy22:41
mordredmorganfainberg: because even though strict security would blanch at that from a probing perspective22:41
morganfainbergmordred: disabled by policy should be 40322:42
mordreddisagree 100%22:42
morganfainbergbut not 40422:42
morganfainbergdefinitely not 40422:42
mordredoh! yes22:42
mordredagree 100%22:42
mordredI believe right now they 40422:42
morganfainbergif it's a policy, a 403 indicates your request [and authorization] cannot access the resource22:42
clarkbdoes KSC/KSA provide a method to check if the token is expired?22:42
dstufftit should be a random code every request22:43
mordreddstufft: ++22:43
clarkbwe could just brute force it and check expiry prior to every request22:43
morganfainbergclarkb: jamielennox ^ - i think we have a method for that.22:43
clarkbmordred: ^22:43
*** otter768 has quit IRC22:43
morganfainbergclarkb: or a "near expired" check22:43
mordredclarkb: yah. I think we can do that22:43
* mordred goes to look22:43
jamielennoxwe do that22:43
jamielennoxi think it's like within 2 minutes of expiry22:43
clarkbjamielennox: ya but now we need to do it again because glanceclient doesn't ksc/ksa session22:43
mordredand we're hitting app failures because of long-lived processes22:43
morganfainbergmordred: i really really hope we aren't 404ing policy disables. i might need to break some fingers if we are.22:43
samueldmqmorganfainberg, looks like there is a "440 Login Timeout", but as a microsoft standard though22:44
mordredand we need to work around it in the short term22:44
morganfainbergsamueldmq: euuuuuwwww.22:44
openstackgerritMerged openstack-infra/irc-meetings: In tox, write output to subdir  https://review.openstack.org/18567822:44
morganfainbergsamueldmq: :P22:44
clarkbmordred: so that would be my vote for the shade image_create method (and probably others) to check the token, manually make a new one if near expiry then do upload22:44
samueldmqmorganfainberg, shhhh, I didnt really say that :p22:44
pleia2morning jhesketh22:44
openstackgerritJohn L. Villalovos proposed openstack/requirements: Add simplegeneric to requirements  https://review.openstack.org/18577022:44
mordredclarkb: ++22:44
clarkbmordred: now to figure out how to do that in shade22:44
mordredclarkb: well, it probably wants to go into def _glance_client22:45
morganfainbergjamielennox: let me know if i need to chase any glance folks down so we can get that in ASAP22:45
jeblairjhesketh: if you can give 182811 a once-over and make sure the new rev lgty, i would appreciate it22:45
mordredclarkb: since that will get invoked to get the glance_client object anyway22:45
morganfainbergjamielennox: i'm in a chase people down and get things fixed soon for our users mood.22:45
clarkbmordred: you can't do it then as that is too late22:45
clarkbmordred: you have to do it prior to the request22:45
mordredclarkb: it is prior to the request22:45
clarkbmordred: let me rephrase, you have to check it immediately prior to the request22:46
mordredI grok22:46
morganfainbergis this a case where glanceclient is spinning on something long enough to break due to timeout?22:46
mordredclarkb: self.glance_client is a property that gets the glance client object when you request it22:46
jokke_sorry just noticed ... what's up with glance?22:46
clarkbmordred: anything in the middle opens you to breaking either because of the task manager or other22:46
jamielennoxi was putting stuart mclaren on the hook (i ca'nt remember his irc) so if you want to go and poke glance people that would be useful22:46
morganfainbergjokke_: need to get keystoneclient session fix/support merged into glance client22:46
jamielennoxwhat consumer are we discussing?22:46
clarkbjamielennox: shade22:47
mordredclarkb: ok. so it COULD break if the taskmanager kept the task in the queue long enough22:47
jokke_nikhil is online and I'm here ... will have a look22:47
morganfainbergjokke_: https://review.openstack.org/#/c/141994/22:47
mordreddo we have the keystone api thing for checking a token handy?22:48
clarkbmorganfainberg: yes I think what happens is we get glanceclient, then for image in images: client.upload() but since uploads are slow because lots of data eventually the token expires22:48
clarkbmorganfainberg: and instaed of making a new token we fail22:48
*** ddieterly has joined #openstack-infra22:48
jamielennoxexpiry during the operation shouldn't matter22:48
clarkbmordred: no I haven't found it yet was reading shade first to figure out where to put this22:48
clarkbjamielennox: we have a ton of them22:49
mordredclarkb: I think for now let's solve it in glance_Client since the only race there is super-long task queue22:49
morganfainbergclarkb: ok but it's not fundamentally an issue with the operation it's a subsequent request?22:49
clarkbjamielennox: so the first one is fine22:49
clarkbmorganfainberg: I think so22:49
mordredclarkb: we can re-fetch the glance_client each time, which would retrigger the token age check22:49
morganfainbergclarkb: cool22:49
jamielennoxyea, ok, so if we consumed session then it would refresh the token automatically22:49
mordredclarkb: then expect glanceclient ksc.Session to solve it for realses22:49
*** arxcruz has quit IRC22:49
mordredjamielennox: yah - we're talking a fix for us by tomorrow22:49
jamielennoxi'll refresh that patch today to address comments but if people want to put some pressure on the glance-core team to make it a priority that's useful22:49
mordredit can be hacky22:49
mordredwhile we wait on your patch to land and get released22:50
morganfainbergjamielennox: you have jokke_ ready to look at it22:50
jamielennoxok, the hacky fix is to fetch a new token and create a client with it before operation22:50
*** ddieterly has quit IRC22:50
jamielennoxif you do Session(auth=plugin)22:50
*** davideagnello has quit IRC22:50
morganfainbergjamielennox: so i think we can address that with the refresh.22:50
*** prad has quit IRC22:50
jamielennoxthen do session.get_token()22:50
jamielennoxthat token will always be valid for the next 2 minutes22:50
jamielennoxso if you do that prior to every op it should solve the immediate problem22:51
*** davideagnello has joined #openstack-infra22:51
jokke_jamielennox: https://review.openstack.org/#/c/141994 < that one?22:51
jokke_oh thanks morganfainberg22:52
jamielennoxget_token will only re-auth if required as well, it will reuse if possible22:52
jamielennoxjokke_: yep22:52
clarkbI probably don't want to know why image upload is really an image create then an image update22:52
* clarkb goes back to fixing shade22:52
openstackgerritMerged openstack-infra/system-config: Fix typo in irc-meetings  https://review.openstack.org/18566922:53
morganfainbergclarkb: 1st rule of asking questions... if you don't want to know the answer...........22:53
*** e0ne has quit IRC22:53
clarkbjamielennox: by reuse you mean it its at no cost because it will return the same token if it isn't near expiry?22:54
morganfainbergclarkb: correct22:55
clarkbcool that simplifies shade a little22:55
*** dimtruck is now known as zz_dimtruck22:55
*** bswartz has joined #openstack-infra22:55
jokke_yeah so looking the reviews, there is quite a bit of concerns to address still before that's gonna land.22:55
mordredhang on22:55
mordredclarkb: how does that simplify anything?22:55
clarkbmordred: because now shade doesn't ahve to check22:56
jokke_jamielennox: btw thanks for addressing the spec and marking the bp to the commit message22:56
mordredclarkb: and no - you do NOT want to know about the create/update sequence22:56
clarkbmordred: instead shades auth_token property is just return ksc.session.get_token()22:56
mordredclarkb: ah - instead of the caching it does22:56
openstackgerritMerged openstack-infra/shade: Make sure glance image list actually runs in Tasks  https://review.openstack.org/18575622:56
clarkbso I am working on that patch now22:56
mordredclarkb: we'll need to ALSO not cache the glance client object I think22:57
*** dguerri is now known as dguerri`away22:57
mordredclarkb: so that each use of self.glance_client retriggers the auth_token22:57
fungijeblair: gertty starts for me after migration now22:57
jeblairfungi: \o/22:57
mordredclarkb: of course, given the cost of glance operations, that's probably fine22:57
clarkbmordred: correct22:58
mordredclarkb: cool - then we can move back to sanity when jamielennox's patch lands and glanceclient releases22:58
*** stevemar has quit IRC22:58
fungijeblair: and i can successfully search for changes by changed filename... bonus!22:58
mordredalthough I think we should never cache the auth_token22:58
morganfainbergyay sanity will return.22:58
clarkbmordred: I need to figure out what OperatorCloud is doing with the auth token though as its special22:58
mordredsince ksc.Session seems to be doing a good job of that already22:58
jamielennoxafaik after glance it's only swift of the "major" projects that is lacking session22:59
mordredclarkb: it's related to ironic22:59
mordredclarkb: they have a no-keystone mode of operation22:59
clarkbmordred: because ironic doesn't always do keystone22:59
jamielennoxmordred: know any oyhers?22:59
mordredclarkb: which we represent with a auth_type of None22:59
clarkbmordred: gotcha22:59
mordredjamielennox: and ironic22:59
jamielennoxmordred: a no-op plugin to ignore keystone auth will work just fine with Sessions23:00
mordredjamielennox: does one of those exist?23:00
jamielennoxthere's token_endpoint23:00
mordredjamielennox: if not, perhaps we should make one of those as part of moving python-ironicclient to ksc.Session23:00
jamielennoxbecause you will still need to specify the ironic endpoint23:00
jheskethjeblair: reviewed 182811 with a few comments23:00
sigmavirus24is there a way in gertty to determine which changes are hidden? I'm trying to figure out why the dashboard says I have 2 unreviewed changes in a project that shows no changes and I'm guessing the reviews are hidden because of vim memory muscle23:01
jamielennoxi can't remember if there is an issue if you pass an empty/None token to token_endpoint23:01
clarkbsigmavirus24: iirc its because by default you don't see WIP changes23:02
clarkbsigmavirus24: 'L' may list them all23:02
sigmavirus24clarkb: Ah, that might be it. Thanks!23:02
morganfainbergclarkb, mordred: i've been having chats w/ devananda about ironic and non-keystone modes.23:02
devanandamorganfainberg: ++23:03
mordredmorganfainberg: funny - I just pinged you about that in their channel23:03
morganfainbergmordred: my laptop lost wifis again... cannot wait for the x1c tomorrow.23:03
morganfainbergmordred: so missed the ping initially23:03
mordredmorganfainberg: JEALOUS23:03
*** soren_ has quit IRC23:04
morganfainbergmordred: i next-day ordered it because i wanted it before the trip to vegas.23:04
*** bknudson has quit IRC23:05
*** ajmiller_ has joined #openstack-infra23:05
fungisigmavirus24: the context help in the change list view confirms l is "Toggle whether only unreviewed or all changes are displayed"23:05
sigmavirus24fungi: That only shows changes that are already marked as reviewed though. I'm investigating23:06
clarkbmordred: once tox is done running I should be able to push up these changes23:06
*** ZZelle_ has quit IRC23:07
*** jsavak has quit IRC23:07
fungisigmavirus24: if i switch an unreviewed change to hidden with the k yet then l redisplays it23:08
openstackgerritJames E. Blair proposed openstack-infra/system-config: Update team structure and add council  https://review.openstack.org/18281123:08
jeblairjhesketh: ^23:08
fungier, s/yet/key/23:08
sigmavirus24fungi: ah, so it must be WIP changes that are being hidden by my config23:08
sigmavirus24Thanks for the help fungi and clarkb23:09
*** ParsectiX has joined #openstack-infra23:09
*** ajmiller has quit IRC23:09
*** chlong has joined #openstack-infra23:09
*** annegentle has quit IRC23:11
clarkbmordred: this will also make our auth_token stuff more correct as the session we pass to eg novaclient may make a new token without us knowing23:11
clarkbmordred: also we do pass the session to glanceclient but I guess it isn't using it to its full potential23:12
mordredclarkb: ++23:12
mordredclarkb: it pretty much ignores it AIUI23:12
openstackgerritClark Boylan proposed openstack-infra/shade: Don't cache keystone tokens as KSC does it for us  https://review.openstack.org/18578123:12
openstackgerritClark Boylan proposed openstack-infra/shade: Always refresh glanceclient for tokens validity  https://review.openstack.org/18578223:12
clarkbmordred: ^ that should do it23:12
*** Sukhdev has quit IRC23:13
mordredclarkb: woot23:13
*** mohitsharma has joined #openstack-infra23:13
clarkbI am trying to think about how we can test that, but by deferring to KSC I think we are relying on their tests instead23:14
*** chlong has quit IRC23:15
*** markvoelker has quit IRC23:15
mordredclarkb: I agree23:15
mordredclarkb: +2 from me on both23:16
mordredclarkb: perhaps jeblair or jhesketh or pleia2 or fungi will be friendly and give us a +A ...23:16
pleia2I'll have a look23:16
fungii can be convinced to take a break from evening and review it23:16
fungievening is not a gerund. eveninging23:16
Clintit is one, it's just not the one you want23:17
* fungi successfully gerunds a noun which already ends in ing23:17
fungiClint: fair point. it was a gerund which got nouned in common parlance ages ago23:17
*** annegentle has joined #openstack-infra23:17
*** mohitsharma has quit IRC23:18
openstackgerritJoshua Harlow proposed openstack/requirements: Add 'fasteners' package  https://review.openstack.org/16366623:18
mordredfungi: oh - wait - really?23:18
*** yamamoto has quit IRC23:18
jokke_jamielennox: just read your spec ... makes way more sense than on the first read of the patch ... hopefully we get that moving23:19
mordredfungi: evening is actually originally related to the verb "to even" ?23:19
fungimordred: according to the etymologies i find23:19
jheskethjeblair: thanks, lgtm23:20
clarkbI don't even23:20
*** Sukhdev has joined #openstack-infra23:20
Clintbecause the light and the dark become even23:20
jheskethmordred: looking23:20
clarkbharlowja_: I have to question why oslo.* exists if not to be the central location for these utilities (re fasteners above)23:20
clarkbharlowja_: what is the problem with having it live in oslo.concurrency?23:21
harlowja_sooo i think the general idea is that oslo.* libraries just provide openstack specific config, tweaks, ..., while general libraries provide the actual functionality23:22
clarkbjeblair: I may be overthinking it but does the council voting thing need a quorum set?23:22
harlowja_for ex, providing oslo.config common usage (which oslo.concurrency does), providing common decorators (which oslo.concurrency does)23:22
mordredclarkb: I vote overthinking, but we may not have reached quorum23:22
clarkbharlowja_: really? I was always under the impression that oslo was the general functionality lib (granted they grow out of what openstack has been doing)23:22
harlowja_the world is bigger than just openstack imho23:22
clarkbharlowja_: ya but the world should just use oslo that is why oslo exists23:23
pleia2fungi: were you looking at these clark patches too? (held off +A just in case you're in the middle of reviewing)23:23
harlowja_clarkb the world isn't openstack though23:23
fungipleia2: yep, jhesketh approved one and i approved the other23:23
clarkbharlowja_: at least that was the intention way back when markmc made oslo23:23
fungiso we'll all be to blame when it breaks even worse ;)23:23
pleia2oh, hah, I didn't refresh quick enough :)23:23
clarkbharlowja_: yes but oslo isn't supposed to be openstack specific23:23
harlowja_thats pretty much what all the oslo.* libraries are23:23
jeblairclarkb: let's see how it works without over-specifying first.  i think that as long as we have notice, people will have ample time to vote on specs.23:23
clarkbanyways I don't oslo so don't really have a fight here, its just very odd to see oslo be less generic23:23
harlowja_sure, https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Choosing_a_Name sorta states some of this23:25
clarkbmordred: where is the glance v2+PUT shade fix? I can review that so it gets in too before the next release23:25
fungiharlowja_: clarkb: i vaguely recall at least at one point that the idea was that oslo.* libraries were for openstack-specific shared/common functionality and that non-openstack-specific libraries were encouraged to not live in the oslo namespace23:25
harlowja_ya thats my understanding to23:25
mordredclarkb: landed already23:25
harlowja_its a balance in reality23:25
mordredclarkb: it's the the release we just made23:25
fungibut it's always possible the groupthink has moved on since the last time i updated23:25
clarkbfungi: hrm I really thought that the reason markmc picked a name like oslo was specifically so that we wouldn't assume it was all openstack specific23:25
clarkbfungi: otherwise we would've called it openstack-concurrency23:25
mordredclarkb: sorry - when I mentioned it earlier - it was from looking at the list of patches taht were being released23:26
clarkbmordred: gotcha23:26
mordredclarkb: I thnk we were trying to be carefuller about the name back then23:26
mordredthe openstack name23:26
mordredlike, "openstack-concurrency" or "openstack-db" might confusel people23:26
harlowja_clarkb well oslo as a prefix i think has started to mean things like 'oslo.db' provides openstack specifics ontop of sqlalchemy23:27
harlowja_maybe thats not what it was intented to be, but seems how thats how it has turned out23:27
*** gokrokve has quit IRC23:27
harlowja_*imho thats for the better, let other libraries do the work, not recreate the world23:27
clarkbharlowja_: except in this case its a noop23:28
clarkbyou are just taking code from concurrency and putting it in fasteners23:28
clarkbwhether it lives in one or the other is functionally equivalent from a recreate the world perspective23:28
clarkbexcept that oslo.concurrency already exists and provides the functionality23:28
*** annegentle has quit IRC23:29
*** hdd has quit IRC23:30
*** wenlock1 has quit IRC23:30
harlowja_sure, but then all consuming libraries must bring in https://github.com/openstack/oslo.concurrency/blob/master/requirements.txt (which then makes consuming projects that much more openstack only)23:30
*** hemna is now known as hemnafk23:31
*** wenlock has quit IRC23:31
harlowja_and not all libraries configure themselves via oslo.config, so it just makes the dependency tree bigger, all required to just have a interprocess lock...23:32
clarkbharlowja_: probably the only problematic deps there are iso8601 and oslo.confg?23:32
harlowja_right, which goes back to the oslo.concurrency not doing alot of its own stuff (providing the oslo.config 'glue' *only*), but using other libraries that are generalized23:33
*** annegentle has joined #openstack-infra23:33
clarkbor just providing the functionality without those deps23:33
harlowja_sounds like the fasteners package ;)23:33
* clarkb wonders what oslo.config is needed for concurrency helpers23:33
harlowja_https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L41 and a lot of that23:34
clarkbharlowja_: yes, I think my questions is why do we need two23:34
clarkbharlowja_: not that the idea of having simple functioanlity on its own is bad23:34
clarkbmerely that if oslo isn't that then what it is23:34
clarkber what is it23:34
harlowja_well the general libraries in the community imho are the place where we should be using stuff from, providing openstack stuff ontop as needed (config, blah blah utility function)23:35
harlowja_*the providing ontop is where oslo comes to be useful (at least the oslo.* libraries)23:35
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add ubuntu-trusty images to rax  https://review.openstack.org/18576223:35
*** sigmavirus24 is now known as sigmavirus24_awa23:35
fungiclarkb: added logging config update ^23:35
clarkbfungi: gah I missed that, thanks23:36
clarkbfungi: except23:36
clarkbfungi: that shouldn't need a logging update?23:36
clarkbfungi: well its the same image so there is no new logging config23:36
clarkbya I think our logging config generator is too greedy, we only need one logger per dib image23:37
fungioh... this should be set to dib-type logs23:37
clarkbthen its provider + image for snapshots23:37
mordredI really would love to do something different with that logging config23:38
clarkbfungi: so while it shouldn't hurt we may haev a small bug we can clean up here23:38
fungiyeah, i'm checking that now23:38
mordredI'm not going to23:38
harlowja_clarkb now maybe this fasteners thing is a progression of moving to more general things (that aren't prefixed with oslo.*) that are useable by the whole python world (and then oslo shrinks); imho thats a fine way to go23:39
clarkbharlowja_: doesn't that imply either a bug in the underlying lib or a bug in how we are using the underlying lib?23:39
clarkbharlowja_: we shouldn't have to wrap every lib before using it,23:39
mordredharlowja_: I agree that its' a fine way to go23:39
*** rguillebert has quit IRC23:39
mordredI also agree with clarkb that for some of these things we should consider need to wrap them as a flaw upstream that should be fixed if possible23:39
harlowja_flaw downstream imho ;)23:40
harlowja_*in our usage23:40
mordredwell, in this case I'd think the things we're wrapping are upstream from us23:40
mordredbut - let's just say "a flaw in the thing we're wrapping"23:40
harlowja_in some cases, yes, i think23:40
harlowja_but not all23:40
harlowja_oslo.db, now u could say, jeez, why isn't that just all in sqlalchemy (maybe minus the common config stuff)23:41
clarkbharlowja_: yes exactly23:41
harlowja_so then whats left over, the config stuff (only?)23:41
clarkbharlowja_: wheres oslo.config is harder because ConfigParse lives in python stdlin23:41
mordredharlowja_: also, zzzeek is here23:41
harlowja_which is fine with me :-23:41
harlowja_so then maybe oslo is just a bunch of config stuffs23:41
clarkbjeblair: I did eventually hit the keyerror23:41
harlowja_mordred right, i've asked him about that before, maybe his ideas have changed23:41
jeblairclarkb: yay!23:41
clarkbfungi: do I need to start from my db backup or can I rerun with the output form the keyerror?23:41
mordredI mean, we have olso.config largely because termie stuck us originally with gconf because EFORMERGOOGLEEMPLOYEE23:42
jeblairmordred: well, not actually here.23:42
fungiclarkb: that's what i did anyway23:42
mordredjeblair: indeed23:42
clarkbfungi: you started from backup?23:42
fungiclarkb: yep23:42
clarkbfungi: thanks23:42
*** nelsnelson has joined #openstack-infra23:42
*** jamielennox is now known as jamielennox|away23:42
*** hichihara has joined #openstack-infra23:42
fungiclarkb: i didn't trust the partial migration23:42
harlowja_mordred so in my ideal world, oslo becomes nothing more than config wrappers ontop of libraries (as we see fit)23:42
mordredharlowja_: I'm down with that23:42
harlowja_and if we want to stop using oslo.config and its gconf like stuff, thats much easier23:42
clarkbjeblair: would it help you more if I grabbed the bottom of that stack or stuck to 179980 for now?23:43
harlowja_dims and others i'm sure have different thoughts on this23:43
*** tim_o has quit IRC23:43
harlowja_but thats my thinking, and u have to either split off the oslo.* libraries into there general part (and the not-general part) and in a proressive manner make the oslo.* libraries smaller (which seems to be happening)23:44
*** ZZelle_ has joined #openstack-infra23:44
harlowja_^ or u have to just start with the libraries/stuff being generic in the first place (Tooz, taskflow, a couple others)23:44
fungiclarkb: i think there might be a bug in nodepool_log_config.py23:44
*** chlong has joined #openstack-infra23:44
openstackgerritJames E. Blair proposed openstack-infra/project-config: Alter governance repo voting rules  https://review.openstack.org/18578523:45
clarkbfungi: iirc it does try to do the correct thing for dib but I think thats only on the positive side of adding dib.imagename loggers, it likely doesn't remove image loggers from the normal setup if they have a diskimage property23:45
*** annegentle has quit IRC23:45
jeblairttx: see https://review.openstack.org/18578523:45
clarkbfungi: I can take a look shortly if it helps but want to get gertty sorted first23:45
harlowja_i think sileht is working on making kombu (or pika now?) be more directly used in oslo.messaging (so this shrinks what oslo.messaging does, making upstream kombu do it better or other...)23:45
harlowja_*same concept applies there23:45
clarkbjeblair: eg use 185698 or 17998023:46
fungiclarkb: yeah, it looks like it falls back to prepending "dib"23:46
harlowja_whether all of this was the intention of oslo from the start, or not, meh, things evolve23:46
jeblairclarkb: sorry?23:46
clarkbfungi: we do want that, but we don't need the loggers that were added in the latest patchset I think23:46
mordredjeblair: https://review.openstack.org/#/c/164870/ you have a procedural +2 on that pending merge of a dependent change. that change was replaced by two changes, both merged23:46
clarkbjeblair: for testing gertty, is it more valuable to just test 179980? or should I use the end of that stack23:46
clarkbharlowja_: yup evolving is fine. I think what I have issue with is using two libs when clearly one should be sufficient23:47
fungiclarkb: ahh, yep. so it's just adding extra handlers which we don't need23:47
clarkbharlowja_: either you handle concurrency or you don't we don't need two things as a default to handle it23:47
jeblairclarkb: the end of the stack is fine23:47
harlowja_well its already not sufficent imho, seeing https://review.openstack.org/#/c/185249/ and https://review.openstack.org/#/c/185784/ already exist (which this removes)23:48
fungiclarkb: i'm seeing if i can figure out a good way for it not to add the non-dib handlers for images which aren't using snapshot23:48
jeblairmordred: +223:48
clarkbharlowja_: lets back up, from a functional perspective if tooz and taskflow can consume fasteners directly then why couldn't the rest of openstack do so23:49
harlowja_i think this all really goes back to oslo.* libraries provide the openstack config, the openstack special sauce functions, and use other libraries to provide the generalized code (so that others in the bigger world can use this same general libraries if they want)23:49
clarkbharlowja_: and if the rest of openstack can, then could we just kill oslo.concurrency or merge oslo.concurrency and fasterners23:49
harlowja_harlowja_ lack of that openstack config, openstack special sauce functions that aren't general23:49
harlowja_*oops, clarkb ^23:49
*** yamamoto has joined #openstack-infra23:50
clarkbharlowja_: openstack isn't that special23:50
clarkbharlowja_: if its useful it should go in the lib23:50
*** annegentle has joined #openstack-infra23:50
jrollI tend to think oslo_concurrency should wrap tooz, and provide a common API for "get a lock" that can be configured to use different things23:50
jroll(or merge with tooz)23:50
jrollthat's what we really need imo23:50
clarkband if it really is special then it shouldn't go in a lib23:50
harlowja_jroll so that goes back to oslo.concurrency being a config wrapper23:51
*** ashleighfarnham has quit IRC23:51
harlowja_clarkb possibly, maybe there never should have been oslo.concurrency, and it should have just stayed special in the incubator23:51
*** marun has quit IRC23:51
jrollharlowja_: right, which isn't terribad23:51
*** salv-orlando has quit IRC23:51
jrollhow much usage outside of openstack does oslo.config really get?23:51
harlowja_unsure of the numbers23:52
harlowja_but from the download count its one of the most downloaded23:52
harlowja_384519 downloads in the last month23:52
harlowja_*but how much of that is openstack CI idk23:52
clarkbharlowja_: only because openstack uses it23:52
harlowja_right, i can't differentiate23:52
clarkbharlowja_: very little should be us since we have our per region mirrors in use23:53
jrollright, because every openstack/stackforge project uses it :P23:53
clarkbI think in general oslo.config likely isn't consumed in many places because it does weird things to config23:53
*** dannywilson has quit IRC23:53
harlowja_yes, i agree with u clarkb23:53
harlowja_its a different mechanism than what python ships with23:53
harlowja_so therefore imho isn't likely highly consumed23:53
harlowja_*outside of openstack stuffs23:53
*** lifeless has joined #openstack-infra23:54
openstackgerritMerged openstack-infra/system-config: Add ubuntu-trusty images to rax  https://review.openstack.org/18576223:54
*** Sukhdev has quit IRC23:54
clarkbmordred: looks like shade changes will merge soon, were you planning on another release?23:54
clarkbI can get things prepped on nodepoo.o.o if so23:55
harlowja_sooo anyway, at the current time i think the oslo prefixed libraries are trying to do less (and just be more config stuffs), this may help that (and its a totally valid question as to if oslo.concurrency is useful after this, and i'm not sure either)23:56
openstackgerritMerged openstack-infra/shade: Don't cache keystone tokens as KSC does it for us  https://review.openstack.org/18578123:57
openstackgerritMerged openstack-infra/shade: Always refresh glanceclient for tokens validity  https://review.openstack.org/18578223:57
harlowja_maybe the longer term plan is that oslo.concurrency really does shrink down (more than it already has in https://review.openstack.org/#/c/185291/23:57
harlowja_but time will tell23:57
harlowja_*doing  said shrink-down(s) in a way that doesn't kill the world obviously23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!