Monday, 2016-04-04

*** salv-orlando has joined #openstack-infra00:02
*** asettle has quit IRC00:03
*** zz_dimtruck is now known as dimtruck00:04
*** baoli has joined #openstack-infra00:05
*** asettle has joined #openstack-infra00:06
*** baoli has quit IRC00:10
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update contributing resources doc benefits section
fungijhesketh: ^ any chance you could give that a once-over? about to reply to someone at a resource donor about that00:13
jheskethfungi: of course, looking00:13
fungijeblair: ^ you too, if you happen to be around (since you were the original author, so might have omitted those things intentionally)00:14
* fungi won't promise it's exciting weekend reading00:15
jheskethfungi: +200:16
*** asettle has quit IRC00:18
*** achanda has quit IRC00:18
*** asettle has joined #openstack-infra00:19
*** asettle has quit IRC00:23
*** asettle has joined #openstack-infra00:24
*** aeng has quit IRC00:24
*** darrenc is now known as darrenc_afk00:26
*** asettle has quit IRC00:27
*** dimtruck is now known as zz_dimtruck00:27
*** salv-orlando has quit IRC00:28
*** asettle has joined #openstack-infra00:28
*** sflanigan has joined #openstack-infra00:28
*** sflanigan has joined #openstack-infra00:28
*** sdake has joined #openstack-infra00:31
*** sdake_ has quit IRC00:34
*** thorst has joined #openstack-infra00:37
*** thorst has quit IRC00:45
*** markvoelker has joined #openstack-infra00:47
*** EricGonczer_ has joined #openstack-infra00:49
*** aeng has joined #openstack-infra00:51
*** darrenc_afk is now known as darrenc00:53
*** achanda has joined #openstack-infra00:57
*** thorst has joined #openstack-infra00:58
*** hichihara has joined #openstack-infra00:58
*** nadya has joined #openstack-infra00:59
*** thorst has quit IRC01:00
*** tphummel has quit IRC01:02
*** nadya has quit IRC01:03
*** tphummel has joined #openstack-infra01:04
*** aeng has quit IRC01:06
*** achanda has quit IRC01:07
*** mriedem has quit IRC01:11
*** sdake_ has joined #openstack-infra01:16
*** sdake has quit IRC01:18
*** aeng has joined #openstack-infra01:19
*** Goneri has quit IRC01:20
*** mylu has quit IRC01:21
*** salv-orlando has joined #openstack-infra01:27
*** mylu has joined #openstack-infra01:29
*** mylu has quit IRC01:30
*** mylu has joined #openstack-infra01:32
*** EricGonczer_ has quit IRC01:37
*** asettle has quit IRC01:41
*** asettle has joined #openstack-infra01:41
*** thorst has joined #openstack-infra01:44
*** salv-orlando has quit IRC01:44
*** asettle has quit IRC01:45
*** asettle has joined #openstack-infra01:45
*** baoli has joined #openstack-infra01:46
*** bapalm has joined #openstack-infra01:51
*** sdake has joined #openstack-infra01:53
*** thorst has quit IRC01:53
*** thorst has joined #openstack-infra01:53
*** sdake_ has quit IRC01:54
*** thorst has quit IRC02:02
*** mylu has quit IRC02:02
*** asettle has quit IRC02:02
*** asettle has joined #openstack-infra02:02
*** mylu has joined #openstack-infra02:03
*** sdake has quit IRC02:03
*** ramishra_ has joined #openstack-infra02:04
*** ramishra_ has quit IRC02:05
*** kengo_sakai has joined #openstack-infra02:05
*** mylu has quit IRC02:05
*** sdake has joined #openstack-infra02:06
*** mylu has joined #openstack-infra02:06
*** achanda has joined #openstack-infra02:07
*** ramishra has quit IRC02:09
*** ramishra has joined #openstack-infra02:09
*** ramishra has quit IRC02:12
*** achanda has quit IRC02:13
*** gildub has joined #openstack-infra02:14
openstackgerritMerged openstack-infra/subunit2sql: Update TODO.rst to point the StoryBoard
*** pcrews_ has quit IRC02:26
*** pcrews has joined #openstack-infra02:26
*** yamamoto has quit IRC02:27
*** roxanaghe has joined #openstack-infra02:27
*** baoli has quit IRC02:28
*** denisra has quit IRC02:28
*** denisra has joined #openstack-infra02:29
*** roxanaghe has quit IRC02:31
*** yamamoto has joined #openstack-infra02:31
*** sridhar_ram has joined #openstack-infra02:35
*** sridhar_ram1 has joined #openstack-infra02:38
*** sridhar_ram has quit IRC02:40
*** links has joined #openstack-infra02:46
*** tphummel has quit IRC02:46
*** tphummel has joined #openstack-infra02:46
*** yamamoto has quit IRC02:50
*** thorst has joined #openstack-infra02:59
*** nadya has joined #openstack-infra02:59
*** kengo_sakai has quit IRC03:00
*** kengo_sakai has joined #openstack-infra03:00
*** Qiming has quit IRC03:00
*** kengo_sakai has quit IRC03:00
*** kengo_sakai has joined #openstack-infra03:01
*** salv-orlando has joined #openstack-infra03:03
*** nadya has quit IRC03:04
*** thorst has quit IRC03:07
*** achanda has joined #openstack-infra03:09
*** salv-orlando has quit IRC03:13
*** markvoelker has quit IRC03:14
*** achanda has quit IRC03:16
*** [1]Thelo has joined #openstack-infra03:20
*** salv-orlando has joined #openstack-infra03:22
*** Thelo has quit IRC03:22
*** [1]Thelo is now known as Thelo03:22
openstackgerritKATO Tomoyuki proposed openstack-infra/project-config: Enable release notes translation for python projects
*** jamielennox is now known as jamielennox|away03:27
*** salv-orlando has quit IRC03:27
*** sdake_ has joined #openstack-infra03:29
*** fedexo has joined #openstack-infra03:30
*** sdake has quit IRC03:32
*** jamielennox|away is now known as jamielennox03:37
*** abregman has quit IRC03:40
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Register connections when testing configuration
*** nwkarsten has joined #openstack-infra03:53
*** asettle has quit IRC03:54
*** asettle has joined #openstack-infra03:54
*** mylu has quit IRC03:56
*** nwkarsten has quit IRC03:57
*** yamamoto has joined #openstack-infra04:01
*** asettle has quit IRC04:02
*** asettle has joined #openstack-infra04:02
*** akshai_ has joined #openstack-infra04:03
*** thorst has joined #openstack-infra04:05
*** mylu has joined #openstack-infra04:05
*** akshai has quit IRC04:06
*** roxanaghe has joined #openstack-infra04:08
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3
*** sdake_ is now known as sdake04:12
*** thorst has quit IRC04:12
*** roxanaghe has quit IRC04:12
*** achanda has joined #openstack-infra04:13
*** markvoelker has joined #openstack-infra04:15
*** nadya has joined #openstack-infra04:15
*** achanda has quit IRC04:18
*** nadya has quit IRC04:20
*** markvoelker has quit IRC04:21
*** maishsk has quit IRC04:23
*** mylu_ has joined #openstack-infra04:25
*** mylu has quit IRC04:27
*** maishsk has joined #openstack-infra04:35
*** sridhar_ram1 has quit IRC04:36
*** asselin__ has quit IRC04:37
*** netsin has quit IRC04:38
*** maishsk has quit IRC04:39
*** baoli has joined #openstack-infra04:40
*** sshnaidm has quit IRC04:41
*** tphummel has quit IRC04:41
*** salv-orlando has joined #openstack-infra04:42
*** baoli has quit IRC04:45
*** salv-orlando has quit IRC04:45
*** elo has quit IRC04:45
*** achanda has joined #openstack-infra04:46
*** sshnaidm has joined #openstack-infra04:49
*** elo has joined #openstack-infra04:51
*** abregman has joined #openstack-infra04:54
*** akshai_ has quit IRC04:58
*** kengo_sa_ has joined #openstack-infra04:59
*** maishsk has joined #openstack-infra04:59
*** thorst has joined #openstack-infra05:10
*** amotoki has joined #openstack-infra05:11
*** asettle has quit IRC05:12
*** asettle has joined #openstack-infra05:13
*** thorst has quit IRC05:17
*** amotoki has quit IRC05:18
*** mylu_ has quit IRC05:19
*** oanson has joined #openstack-infra05:23
*** akaszuba has joined #openstack-infra05:34
*** netsin has joined #openstack-infra05:34
*** amotoki has joined #openstack-infra05:37
*** jcoufal has joined #openstack-infra05:38
*** nadya has joined #openstack-infra05:40
*** cody-somerville has quit IRC05:41
openstackgerritMasayuki Igawa proposed openstack/os-testr: Remove openstack-common.conf
*** maishsk_ has joined #openstack-infra05:48
*** maishsk has quit IRC05:49
*** maishsk_ is now known as maishsk05:49
*** kengo_sa_ has quit IRC05:52
*** asettle has quit IRC05:55
*** roxanaghe has joined #openstack-infra05:56
*** asettle has joined #openstack-infra05:57
*** asettle has quit IRC05:57
*** roxanaghe has quit IRC06:01
*** nadya has quit IRC06:05
*** asettle has joined #openstack-infra06:10
openstackgerritMerged openstack-infra/project-config: Remove unused magnum-ui mitaka translation job
*** salv-orlando has joined #openstack-infra06:13
*** asettle has quit IRC06:14
*** thorst has joined #openstack-infra06:15
*** netsin has quit IRC06:15
*** salv-orlando has quit IRC06:16
*** asettle has joined #openstack-infra06:16
*** jtomasek has joined #openstack-infra06:20
*** ihrachys has joined #openstack-infra06:21
*** ihrachys has quit IRC06:21
*** thorst has quit IRC06:21
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add version option for CLI
*** salv-orlando has joined #openstack-infra06:24
*** daemontool has joined #openstack-infra06:24
*** salv-orlando has quit IRC06:28
*** scheuran has joined #openstack-infra06:28
*** flepied has quit IRC06:37
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** achanda has quit IRC06:48
*** sdake has quit IRC06:48
*** sdake has joined #openstack-infra06:50
yolandagood morning06:52
*** sdake_ has joined #openstack-infra06:53
*** maishsk has quit IRC06:54
*** kengo_sa_ has joined #openstack-infra06:55
*** sdake has quit IRC06:56
*** yamahata has quit IRC06:58
*** nmagnezi has joined #openstack-infra06:58
*** sdake has joined #openstack-infra06:59
*** sdake_ has quit IRC07:00
*** ihrachys has joined #openstack-infra07:03
*** achanda has joined #openstack-infra07:03
*** kengo_sa_ has quit IRC07:03
*** nadya has joined #openstack-infra07:03
*** mikelk has joined #openstack-infra07:04
*** yamamoto has quit IRC07:04
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add version option for ostestr and subunit-trace
*** yamahata has joined #openstack-infra07:05
*** yamamoto has joined #openstack-infra07:06
*** fedexo has quit IRC07:08
*** nadya has quit IRC07:08
*** asettle has quit IRC07:08
yolanda#status notice Gerrit is going to be restarted due to bad performance07:12
openstackstatusyolanda: sending notice07:12
*** flepied has joined #openstack-infra07:12
-openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance07:14
openstackstatusyolanda: finished sending notice07:15
*** arxcruz has joined #openstack-infra07:16
*** salv-orlando has joined #openstack-infra07:17
openstackgerritMerged openstack-infra/system-config: Update contributing resources doc benefits section
*** thorst has joined #openstack-infra07:19
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** achanda has quit IRC07:24
*** achanda has joined #openstack-infra07:24
*** jlanoux has joined #openstack-infra07:26
*** yamahata has quit IRC07:27
*** thorst has quit IRC07:27
*** tojuvone_ is now known as tojuvone07:28
*** yamamoto has quit IRC07:29
*** yamamoto has joined #openstack-infra07:30
*** |-paul-| has joined #openstack-infra07:30
*** achanda has quit IRC07:32
*** fhubik has joined #openstack-infra07:33
*** vincentll has joined #openstack-infra07:34
openstackgerritMerged openstack-infra/system-config: Add Zuul restart documentation
*** Qiming has joined #openstack-infra07:34
*** nelsnels_ has joined #openstack-infra07:34
*** davideag_ has joined #openstack-infra07:34
*** nelsnelson has quit IRC07:34
*** maishsk has joined #openstack-infra07:36
*** davideagnello has quit IRC07:37
*** eliqiao has quit IRC07:37
*** charz has quit IRC07:37
*** mrunge has quit IRC07:37
*** eliqiao has joined #openstack-infra07:39
*** adreznec has quit IRC07:39
*** taron has quit IRC07:39
*** nibalizer has quit IRC07:39
*** nibalizer has joined #openstack-infra07:40
*** Jeffrey4l__ has quit IRC07:40
*** bswartz has quit IRC07:40
*** kevinshort has quit IRC07:40
*** kevinshort has joined #openstack-infra07:40
*** grassass has quit IRC07:40
*** sflanigan has quit IRC07:40
*** sflanigan has joined #openstack-infra07:40
*** sflanigan has joined #openstack-infra07:40
*** adreznec has joined #openstack-infra07:41
openstackgerritSteve Martinelli proposed openstack/requirements: bump keystoneauth to 2.5.0
*** Jeffrey4l__ has joined #openstack-infra07:43
*** bigjools has quit IRC07:43
*** rcernin has joined #openstack-infra07:43
*** grassass has joined #openstack-infra07:43
*** charz has joined #openstack-infra07:43
*** bigjools has joined #openstack-infra07:43
*** bigjools has quit IRC07:43
*** bigjools has joined #openstack-infra07:43
*** taron has joined #openstack-infra07:44
*** kzaitsev_mb has joined #openstack-infra07:44
*** mrmartin has joined #openstack-infra07:45
*** ifarkas has joined #openstack-infra07:47
*** isaacb has joined #openstack-infra07:48
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add experimental job to build atomic images
*** zeih has joined #openstack-infra07:52
*** salv-orlando has quit IRC07:53
*** sdake has quit IRC07:53
*** fawadkhaliq has joined #openstack-infra07:54
*** maishsk has quit IRC07:54
*** fhubik has quit IRC07:56
*** salv-orlando has joined #openstack-infra07:56
*** fhubik has joined #openstack-infra07:56
*** kengo_sa_ has joined #openstack-infra08:00
*** jamielennox is now known as jamielennox|away08:01
*** gildub has quit IRC08:02
*** yaume has joined #openstack-infra08:02
*** e0ne has joined #openstack-infra08:02
*** kengo_sa_ has quit IRC08:05
*** jed56 has joined #openstack-infra08:06
mrmartinyolanda, morning08:07
*** shashank_hegde has quit IRC08:08
*** pblaho has joined #openstack-infra08:08
mrmartinmay i ask you to switch the DNS records of groups.o.o to the new instance?08:09
yolandasure, everything is fine?08:09
mrmartinthe puppet run is ok now, I've finished with the backup / restore08:09
yolandaok will do08:09
mrmartinand I also added the bup ssh keys from old instance08:09
*** fawadkhaliq has quit IRC08:10
mrmartinwe need to check the bup keys for the groups-dev, but we can live without backup there08:10
mrmartinso it is not mission critical08:10
yolandamrmartin, ok, first i'll change the dns entries and then i can help you with the info you need from bup08:11
*** daemontool_ has joined #openstack-infra08:13
*** jaosorior has joined #openstack-infra08:14
mrmartinsure, this one:
*** daemontool has quit IRC08:17
*** hashar has joined #openstack-infra08:17
*** rossella_s has joined #openstack-infra08:18
*** wfoster_ is now known as wfoster08:19
*** jistr has joined #openstack-infra08:19
*** zeih has quit IRC08:21
*** thorst has joined #openstack-infra08:25
yolandamrmartin, so dns updated on, and i also removed old groups-dev08:29
yolandawe can let some time for old groups instance to live, and then delete as well08:30
mrmartinyeah, give a few days for old groups.o.o08:30
mrmartinthe next one will be the openstackid-dev.o.o, and if it works well, then, I think it is better to upgrade the production to trusty after the summit.08:31
*** nadya has joined #openstack-infra08:31
*** thorst has quit IRC08:31
yolandayep, makes sense08:32
*** achanda has joined #openstack-infra08:32
*** zeih has joined #openstack-infra08:33
*** achanda has quit IRC08:36
*** rpodolyaka_afk is now known as rpodolyaka08:36
*** lezbar has joined #openstack-infra08:38
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
*** sfinucan has joined #openstack-infra08:38
*** lezbar has quit IRC08:38
*** lezbar has joined #openstack-infra08:39
*** derekh has joined #openstack-infra08:41
*** sfinucan has quit IRC08:41
*** sflanigan has quit IRC08:44
*** ggillies has quit IRC08:44
*** sflanigan has joined #openstack-infra08:44
*** sflanigan has joined #openstack-infra08:44
*** ggillies has joined #openstack-infra08:45
*** sfinucan has joined #openstack-infra08:48
*** amotoki has quit IRC08:49
*** amotoki has joined #openstack-infra08:49
*** cdent has joined #openstack-infra08:51
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: check if stack-delete supports -y option
*** dtantsur|afk is now known as dtantsur08:51
*** jaosorior has quit IRC08:57
*** esikachev has joined #openstack-infra08:58
*** maishsk has joined #openstack-infra08:59
*** _degorenko|afk is now known as degorenko08:59
*** wznoinsk_ has joined #openstack-infra09:03
*** wznoinsk_ is now known as wznoinsk09:04
*** jaosorior has joined #openstack-infra09:05
*** claudiub has joined #openstack-infra09:06
*** electrofelix has joined #openstack-infra09:06
wznoinskhi infra09:09
wznoinskwhat's the rationale behind always have 8gb of swapfile in devstack gate's vm?09:10
wznoinsk... over configurable/parametrized amount?09:11
*** sputnik13 has joined #openstack-infra09:13
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: fix stack-delete when -y option isn't supported yet
*** sambetts|afk is now known as sambetts09:14
wznoinskI guess the RAM limit imposed at kernel CMD line is why we have it at 8gb but I'd rather have it no/smaller swapfile in my environment - but I'd like to hear your side first09:14
*** skolekonov|away is now known as skolekonov09:15
*** amotoki has quit IRC09:17
*** amotoki has joined #openstack-infra09:18
*** sputnik13 has quit IRC09:19
*** jordanP has joined #openstack-infra09:20
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: fix stack-delete when -y option isn't supported yet
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: ironic-inspector: make the discovery job voting
dtantsuraarefiev, ^^^09:27
*** exploreshaifali has joined #openstack-infra09:28
jheskethAJaeger: please see my replies to and
openstackgerritJulien Danjou proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config
*** thorst has joined #openstack-infra09:30
*** roxanaghe has joined #openstack-infra09:32
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: refactor stack-delete -y
*** zeih has quit IRC09:33
AJaegerjhesketh: why do it this way?09:36
*** roxanaghe has quit IRC09:36
AJaegerYou can publish directly to the subdirectory...09:37
*** thorst has quit IRC09:37
jheskethAJaeger: the election site to begin with is very basic (ie just the rendered sphinx) but there may be scope in the future to change it.. this will make switching over to its own vhost easier09:39
jheskethAJaeger: it also means if the governance repo somehow creates an election folder/file it won't be overwritten09:39
jheskethAJaeger: it just seemed neater when I was doing it, but I agree it isn't necessary09:39
*** tosky has joined #openstack-infra09:39
openstackgerritSam Betts proposed openstack-infra/project-config: Add post job for building and uploading tinyipa
*** links has quit IRC09:53
*** sfinucan has quit IRC09:53
*** sfinucan has joined #openstack-infra09:54
*** links has joined #openstack-infra09:54
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: ironic-inspector: make the discovery job voting
*** ociuhandu has quit IRC09:59
*** vgridnev has joined #openstack-infra09:59
AJaegerjhesketh: Ok, understood now what you do. Let's see whether fungi and jeblair have any preferences for 292676 and 292675, I'll remove my -1.09:59
jheskethAJaeger: cool, thanks10:00
*** maishsk has quit IRC10:00
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Only reload layout on HUP
*** SotK has quit IRC10:01
*** amrith is now known as _amrith_10:02
*** maishsk has joined #openstack-infra10:02
jheskethjeblair: I believe ^ may help with zuul's current memory issue. I haven't got enough data to say with certainty, but I believe it's a useful fix either way.10:02
*** yamamoto has quit IRC10:03
*** SotK has joined #openstack-infra10:03
*** yamamoto has joined #openstack-infra10:04
*** achanda has joined #openstack-infra10:05
*** achanda has quit IRC10:11
*** sdague has joined #openstack-infra10:12
*** zeih has joined #openstack-infra10:12
*** thorst has joined #openstack-infra10:14
*** yamamoto has quit IRC10:16
openstackgerritMarton Kiss proposed openstack-infra/system-config: Upgrade groups.o.o to trusty
*** yamamoto has joined #openstack-infra10:17
*** thorst has quit IRC10:21
*** yamamoto has quit IRC10:26
*** zeih has quit IRC10:27
*** salv-orl_ has joined #openstack-infra10:30
*** salv-orlando has quit IRC10:32
openstackgerritMerged openstack-infra/project-config: Add bareon-image-elements project
rm_workanyone here happen to be familiar with running ansible playbooks from within python? not relevant to channel, just curious since I figure probably a lot of people here use ansible and I'm having trouble finding docs (or confirming whether anyone actually DOES this, maybe I am crazy)10:34
*** mattt has left #openstack-infra10:35
*** mikelk has quit IRC10:38
*** ociuhandu has joined #openstack-infra10:40
*** vgridnev has quit IRC10:41
*** vgridnev has joined #openstack-infra10:42
*** jlanoux has quit IRC10:46
odyssey4merm_work pabelanger or mordred may have tried - I vaguely recall seeing some chatter about it some time ago... doing it for v1.9x was not a happy place... maybe with v2.x it's better10:47
*** openstackgerrit has quit IRC10:48
*** openstackgerrit has joined #openstack-infra10:48
rm_workhmm yeah there was like one or two guides total for 1.x10:48
rm_workI'm trying with 2.x and it seems a bit ugly right now, but i figure it's because I don't know what i'm doing :P and there are ZERO examples, else my google-fu is weak10:49
rm_workthanks odyssey4me, i'll poke at them later mayhaps10:49
odyssey4merm_work yeah, Ansible's still a young project and it's primarily catering for CLI consumers at this point10:51
rm_workI'm actually wondering if it's realistic to propose it for inclusion in global-reqs :/10:51
rm_workwe'd like to hook an ansible playbook as part of our taskflow pipeline10:52
*** zeih has joined #openstack-infra10:54
*** exploreshaifali has quit IRC10:55
*** nmagnezi has quit IRC10:57
openstackgerritMerged openstack-infra/project-config: Request for new repo - python-k8sclient
*** swat30 has quit IRC10:58
*** acruz has joined #openstack-infra11:01
*** maishsk has quit IRC11:01
*** hichihara has quit IRC11:02
*** fhubik has quit IRC11:02
*** arxcruz has quit IRC11:02
rossella_sAJaeger, hi Adrenas11:09
*** nmagnezi has joined #openstack-infra11:10
*** krotscheck_vaca is now known as krotscheck11:13
*** yamamoto has joined #openstack-infra11:14
*** maishsk has joined #openstack-infra11:14
openstackgerritRossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard
*** yamamoto_ has joined #openstack-infra11:15
*** maishsk has quit IRC11:15
*** chem has joined #openstack-infra11:18
*** yamamoto has quit IRC11:19
*** roxanaghe has joined #openstack-infra11:20
*** exploreshaifali has joined #openstack-infra11:24
*** roxanaghe has quit IRC11:25
openstackgerritJulien Danjou proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config
*** ldnunes has joined #openstack-infra11:28
*** ldnunes has quit IRC11:32
*** ldnunes has joined #openstack-infra11:33
*** weshay has quit IRC11:33
*** sdake has joined #openstack-infra11:34
*** zz_dimtruck is now known as dimtruck11:34
*** sdake_ has joined #openstack-infra11:36
*** gordc has joined #openstack-infra11:36
*** jtomasek_ has joined #openstack-infra11:36
*** electrofelix has quit IRC11:37
*** sdake has quit IRC11:38
*** claudiub has quit IRC11:39
*** thorst has joined #openstack-infra11:41
*** jaypipes has joined #openstack-infra11:42
*** baoli has joined #openstack-infra11:42
*** _amrith_ is now known as amrith11:44
*** baoli_ has joined #openstack-infra11:44
*** jlanoux has joined #openstack-infra11:45
*** claudiub has joined #openstack-infra11:45
*** fhubik has joined #openstack-infra11:47
*** baoli has quit IRC11:48
*** jtomasek_ has quit IRC11:50
*** maishsk has joined #openstack-infra11:51
*** Hal has joined #openstack-infra11:52
*** Hal is now known as Guest4850311:53
*** marcusvrn_ has joined #openstack-infra11:56
*** cdent has left #openstack-infra11:56
*** jordanP has quit IRC12:00
*** zeih has quit IRC12:02
dimsAJaeger : sdague : is initial import of a repo from github a separate step after the repo is created? (python-k8sclient seems empty)12:02
*** HeOS has quit IRC12:03
*** HeOS has joined #openstack-infra12:03
*** amrith is now known as _amrith_12:05
*** keedya has joined #openstack-infra12:06
*** EricGonczer_ has joined #openstack-infra12:07
sdaguedims: it should import12:07
*** kengo_sakai has quit IRC12:08
*** nmagnezi_ has joined #openstack-infra12:08
*** achanda has joined #openstack-infra12:09
*** esikachev has quit IRC12:09
dimssdague : ack, thanks, i'll see if i can find logs why it failed12:09
*** kengo_sakai has joined #openstack-infra12:09
*** weshay has joined #openstack-infra12:11
*** dprince has joined #openstack-infra12:11
*** nmagnezi has quit IRC12:12
*** zeih has joined #openstack-infra12:13
*** erlon has joined #openstack-infra12:13
*** kengo_sakai has quit IRC12:13
*** achanda has quit IRC12:14
*** keedya has quit IRC12:14
*** keedya has joined #openstack-infra12:14
*** markvoelker has joined #openstack-infra12:15
*** _amrith_ is now known as amrith12:16
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repos for fuel plugins
*** eharney has quit IRC12:21
toskyis there a reason why there are two accounts with my name, same email, and so people can't add me to reviews? (ltoscano at
*** Goneri has joined #openstack-infra12:21
toskyI remember it started happening some time ago after some outage with the launchpad authentication12:21
*** pradk has quit IRC12:21
toskyso I guess the real question is: can it be fixed?12:22
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repos for fuel plugins
*** edmondsw has joined #openstack-infra12:24
*** rfolco has joined #openstack-infra12:25
*** EricGonczer_ has quit IRC12:26
*** maishsk has quit IRC12:27
*** pradk has joined #openstack-infra12:28
vgridnevAJaeger, what do you think about tosky question? I'm trying to add him to the group, but I failed12:28
openstackgerritZara proposed openstack-infra/storyboard-webclient: Remove redundant 'code'
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in
*** sdake_ has quit IRC12:30
*** maishsk has joined #openstack-infra12:31
*** salv-orl_ has quit IRC12:33
*** sdake has joined #openstack-infra12:34
*** salv-orlando has joined #openstack-infra12:34
*** rodrigods has joined #openstack-infra12:36
rodrigodshi... can someone give some feedback about this patch ?12:37
rodrigodsthe change that adds the tempest plugin in keystone was merged already12:37
*** rlandy has joined #openstack-infra12:38
*** salv-orlando has quit IRC12:38
marcusvrn_patrickeast: hi! do you know how to get the lost queue from CI? I mean, when CI machine get disconnected/turned off, all builds that were in the queue are lost... There is some tool to retrieve those builds or will I have to develop it?12:38
*** bswartz has joined #openstack-infra12:40
*** xyang1 has joined #openstack-infra12:40
*** kgiusti has joined #openstack-infra12:41
openstackgerritRichard Theis proposed openstack/requirements: Bump openstacksdk version to 0.8.4
AJaegerdims: it should import, if this failed, ask one of the infra-roots, please12:46
dimsAJaeger : It took a while, looks like it imported fine. thanks AJaeger  and sdague12:47
AJaegervgridnev, tosky : Please talk with one of the admin, I don't have those permissions. This can happen if you change your launchpad email address AFAIK12:47
toskyAJaeger: I never did that - who are the admins?12:47
AJaegerdims: changes to project-config in general need until next puppet run to take effect - so always wait an hour ;)12:48
AJaegertosky: infra-root will address them here;)12:48
dimsAJaeger : ah i see. thanks12:48
*** zul has quit IRC12:48
dimsAJaeger : wasn't trying to rush it. just trying to understand the flow :)12:48
*** zul has joined #openstack-infra12:50
AJaegerteam, I have a few errands to run today, won't be online further today.12:50
*** kzaitsev_mb has quit IRC12:52
*** kzaitsev_mb has joined #openstack-infra12:52
*** amrith is now known as _amrith_12:55
dgonzalezhi sdague: if you have time, could you have a look at I'm trying to set the user/project domain id for devstack in openrc, but some grenade tests keep failing and i can't figure out why12:57
*** _amrith_ is now known as amrith12:57
trashSpamapS: remains broken, how do we continue with that? :)13:00
sdaguedgonzalez: is the failure13:01
openstackgerritRodion Promyshlennikov proposed openstack/requirements: Bump alembic version to 0.8.4
dgonzalezsdague: yes, but i do not understand how the two env variables i added to the openrc can cause this failure... any idea?13:03
sdaguedgonzalez: I don't atm, it appears to be changing how openstack client is communicating with services13:04
sdagueit is probably worth asking a keystone or osc person13:04
odyssey4mehmm, it would seem that zuul is dropping events? We had a few in the queue and they seem to have gotten lost. rechecks and rebases aren't making a difference.13:05
dgonzalezsdague: ok thanks, i will ask them13:06
sdagueodyssey4me: do you have specific reviews? it's helpful for people tracking them down13:08
*** roxanaghe has joined #openstack-infra13:08
*** Guest48503 has quit IRC13:09
*** Hal has joined #openstack-infra13:10
*** Hal is now known as Guest1262313:10
*** salv-orlando has joined #openstack-infra13:11
*** smarcet has joined #openstack-infra13:11
amrithhello, what is the current plan for opening newton for tagging releases? I would like to restore for newton and get a new trove client tagged and on pypi. Thx!13:12
*** dtantsur is now known as dtantsur|brb13:12
*** roxanaghe has quit IRC13:13
*** sfinucan_ has joined #openstack-infra13:14
*** jaosorior has quit IRC13:14
*** jaosorior has joined #openstack-infra13:14
smarcetfungi: morning, we are good to move trusty to production whenever u want13:15
*** matt-borland has joined #openstack-infra13:15
*** nwkarsten has joined #openstack-infra13:16
*** sfinucan has quit IRC13:17
*** jaosorior has quit IRC13:17
*** jaosorior has joined #openstack-infra13:18
openstackgerritThomas Morin proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs
*** [1]Thelo has joined #openstack-infra13:20
*** nelsnels_ has quit IRC13:21
*** _ari_ has joined #openstack-infra13:21
*** links has quit IRC13:21
*** akshai has joined #openstack-infra13:21
odyssey4mesdague ah, it appears that either my browser was acting up - or everything's back to normal13:22
*** Thelo has quit IRC13:22
*** [1]Thelo is now known as Thelo13:22
pabelangerinfra-root: I've just deleted 9292911 in nodepool, as it was hung for 6 hours trying to upload to swift13:23
*** sfinucan__ has joined #openstack-infra13:23
*** tiswanso has joined #openstack-infra13:23
*** mtanino has joined #openstack-infra13:24
openstackgerritRussell Bryant proposed openstack/requirements: Add ovs with Python 3 support.
openstackgerritRussell Bryant proposed openstack/requirements: Update ovs to 2.5.0.
*** amitgandhinz has joined #openstack-infra13:25
*** sfinucan_ has quit IRC13:25
Shrewstrash: should be the fix13:27
*** dalgaaf has joined #openstack-infra13:27
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix consent screen OAUTH2.0 flow * updated css on ui consent screen to better mobile UX * added autocomplete off to login form
*** jsavak has joined #openstack-infra13:28
trashShrews: Ah13:29
*** fhubik has quit IRC13:30
*** esker has quit IRC13:30
trashShrews: thanks, seems to work with 1.7.0 now13:35
EmilienMsdague: I have the same problem as odyssey4me13:35
EmilienMsdague: 30078613:35
*** bizarrodan is now known as dansmith13:35
EmilienMwe tried to +2 +A again, recheck, etc. Patch is not going in any queue13:35
*** woodster_ has joined #openstack-infra13:36
EmilienMand I tried 2 browsers13:37
*** mikelk has joined #openstack-infra13:38
openstackgerritGraham Hayes proposed openstack-infra/project-config: Move tox-bandit to check as non voting
EmilienMeven if I push a new patchset, nothing goes in zuul13:39
*** |-paul-| has quit IRC13:39
*** ddieterly has joined #openstack-infra13:41
*** ddieterly has quit IRC13:42
*** mriedem has joined #openstack-infra13:43
*** ddieterly has joined #openstack-infra13:45
*** pradk_ has joined #openstack-infra13:45
*** pradk has quit IRC13:45
*** ddieterly has quit IRC13:46
*** pradk_ is now known as pradk13:46
*** ddieterly has joined #openstack-infra13:46
*** eharney has joined #openstack-infra13:48
AJaegerEmilienM: is there another patch that has a depends-on on that one - and does this some leat to a cyclic dependency? That would explain it....13:48
AJaegerEmilienM: otherwise an admin needs to check zuul debug logs...13:49
EmilienMAJaeger: depends on
dansmithI see the same behavior on new patches I have pushed13:52
mriedemdid something blow up with openstack/requirements on stable/liberty recently?13:53
*** mrunge has joined #openstack-infra13:53
*** ddieterly is now known as ddieterly[away]13:54
*** yamamoto_ has quit IRC13:56
*** yamamoto has joined #openstack-infra13:56
*** eharney has quit IRC13:57
YorikSarHello, #openstack-infra.13:57
YorikSarI have a CR that is being ignored by Zuul.13:57
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
YorikSarNo new patchsets appear on Zuul status page.13:57
*** ddieterly[away] is now known as ddieterly13:57
YorikSarThe CR is
*** aysyd has joined #openstack-infra13:58
mattymoI'm having issues getting to merge. It has wf+1 for several hours, gate jobs succeeded. I tried recheck and wf-0,wf+1, but no luck13:58
*** dtantsur|brb is now known as dtantsur13:58
mattymosimilar to the esteemed YorikSar's issue13:58
YorikSarHave you tried publishing new patchset?13:58
*** aysyd has quit IRC13:59
fungimrmartin: openstackid-dev is already running trusty (after some additional patches), and smarcet has given the go-ahead for us to update production as soon as we're ready13:59
mattymoYorikSar, not yet13:59
*** aysyd has joined #openstack-infra13:59
*** eharney has joined #openstack-infra14:00
odyssey4meyeah, ok - so here's one that just got ignored - it has no deps on any other changes14:00
fungiwznoinsk: i think the swap configuration in devstack-gate is probably something we'd be open to letting jobs adjust on their own (in theory they already can, just swapoff, make different device, swapon again, but could in theory make the setup function reusable and parameterized instead)14:00
mattymoodyssey4me, at least I'm not alone14:01
*** isaacb has quit IRC14:01
YorikSarYeah, odyssey4me, welcome to the club :)14:01
*** fzhadaev has joined #openstack-infra14:01
mrmartinfungi: good to know that14:01
*** dimtruck is now known as zz_dimtruck14:02
*** ddieterly is now known as ddieterly[away]14:03
*** nelsnelson has joined #openstack-infra14:05
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
yolandainfra-root, i just received a confirmation that all infra-cloud servers in east/west have been setup properly, and we are waiting for network team to finish the config14:06
fungidims: i'll look into python-k8sclient shortly if nobody has yet14:07
fungidims: oh, good, i guess it worked out after all14:07
jpmaxmanmrmartin fungi yes we have a ux improvement for the login screen that is holding up the app being published - so we kind of need this to move super quick14:07
jpmaxmanApple tends to take their time, but the quicker we can turn it around to them the better14:08
dimsfungi : y, i was not sure where/when the imports would happen, it finally did :)14:08
*** abregman has quit IRC14:08
dimsfungi : thanks! looks like the first review has the CI run jobs as well. So i am good (thanks to sdague and AJaeger as well)14:08
EmilienMis someone from infra-root investigating why zuul isn't working well at this time?14:08
*** sigmavirus24_awa is now known as sigmavirus2414:09
*** knikolla has joined #openstack-infra14:09
*** burgerk has joined #openstack-infra14:09
fungitosky: yes, chances are you have more than one gerrit account with different launchpad openids. i'll look in a bit to see what their respective states are and try to work out a good way to condense them/deactivate one14:09
*** aysyd has quit IRC14:10
toskyfungi: thanks14:10
*** xarses has quit IRC14:10
fungiEmilienM: i'm in a meeting and also just finished catching up on scrollback. will try to look into it as quickly as i can14:10
*** jamesmcarthur has joined #openstack-infra14:10
*** aysyd has joined #openstack-infra14:10
EmilienMcool, thanks14:10
*** ddieterly[away] is now known as ddieterly14:11
fungidoesn't appear to be a problem with stuck stream-events worker threads on gerrit at least14:11
*** achanda has joined #openstack-infra14:12
yolandai see entries in BUILD in nodepool log14:12
fungii see it adding/updating changes14:12
fungii'll check into one of the specific examples14:12
yolandafungi, i can help with tosky's problem14:12
fungithanks yolanda14:13
toskythanks both :)14:13
*** doug-fish has joined #openstack-infra14:15
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
*** ajmiller has joined #openstack-infra14:16
*** achanda has quit IRC14:17
*** oanson has quit IRC14:17
*** xyang1 has quit IRC14:17
*** xyang1_ has joined #openstack-infra14:17
fungiEmilienM: on your 300786 example, the last thing zuul knew about that change is that it reported job results at 13:13 but it doesn't even seem to have seen the event resulting from its own comment, so something does seem to be amiss here14:18
*** krtaylor has quit IRC14:18
openstackgerritThomas Herve proposed openstack-infra/irc-meetings: Move Heat meeting
*** regXboi has joined #openstack-infra14:20
fungiEmilienM: now it saw a new workflow +1 event for 300786 at 14:18:3614:20
EmilienMright but nothing in gate gueue14:21
odyssey4mefungi it would seem that some of the jobs don't show in the zuul status page, but later post results14:21
odyssey4meso it might have nothing to do with the event queue and everything to do with whatever posts data to the status page14:22
fungiwait, this is a verify +1 not a workflow +114:22
*** rbrndt_ has joined #openstack-infra14:22
*** esker has joined #openstack-infra14:22
*** claudiub has quit IRC14:22
fungitimeline seems way off14:23
*** jamesmcarthur has quit IRC14:23
fungi2016-04-04 14:18:36,495 DEBUG zuul.DependentPipelineManager: Event <TriggerEvent comment-added openstack/puppet-tempest master 300786,1 Verified:1> for change <Change 0x7ff142750d50 300786,1> matched <EventFilter types: comment-added ignore_deletes: True event_approvals: verified:1 username_filters: jenkins> in pipeline <DependentPipelineManager gate>14:23
*** esker has quit IRC14:23
*** mrmartin has quit IRC14:24
*** jamesmcarthur has joined #openstack-infra14:24
fungii think that's the verified +1 from the comment it left at 13:1314:24
*** esker has joined #openstack-infra14:24
*** hieulq has joined #openstack-infra14:24
openstackgerritPradeep Kilambi proposed openstack-infra/tripleo-ci: Include Gnocchi endpoints in enable-tls
*** mrmartin has joined #openstack-infra14:25
fungii'm finding tracebacks in the debug log which are infinite loops between _getChange() and _updateChange()14:26
*** edmondsw has quit IRC14:27
openstackgerritNikolas Hermanns proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs
*** isaacb has joined #openstack-infra14:29
fungiRuntimeError: maximum recursion depth exceeded14:29
*** exploreshaifali has quit IRC14:30
*** zz_dimtruck is now known as dimtruck14:30
yolandatosky, can you verify if your account issue is fixed now?14:30
toskyyolanda: let me check14:30
toskyvgridnev: ^^14:30
*** achanda has joined #openstack-infra14:31
EmilienMfungi: same for new patches, I just sent 301162 and nothing in Zuul14:31
vgridnevyolanda, looks like issue is fixed14:31
*** kushal has quit IRC14:31
vgridnevtosky, ^^14:32
fungiEmilienM: yeah, i think zuul is getting them but is waaaaay backed up on the event stream because of these recursive calls14:32
*** nadya has quit IRC14:32
toskyyolanda: thanks!14:32
odyssey4meEmilienM fungi yeah, same with - nothing14:32
fungiwhich would explain the lengthy delays between log actual events in gerrit and log messages14:32
*** piet has joined #openstack-infra14:32
*** jamesmcarthur has quit IRC14:33
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
*** Jeffrey4l__ has quit IRC14:33
jeblairfungi: perhaps a bug introduced in the temporary foreign projects revert14:34
fungithis is still the same build of zuul with "Dependencies from undefined projects" reverted, so there might be a bug in that revert14:34
fungijeblair: indeed!14:34
*** Thelo has quit IRC14:34
*** amotoki has quit IRC14:35
fungipossibly exposed by yolanda's gerrit restart around 07:15z? but the exceptions aren't showing up until after 10:30z14:36
*** tonytan4ever has joined #openstack-infra14:36
fungifirst change to hit it was 30047514:37
*** esikachev has joined #openstack-infra14:37
fungiwhich has a depends-on loop14:38
fungiso maybe that's the trigger14:38
*** andymaier has joined #openstack-infra14:38
*** kushal has joined #openstack-infra14:38
*** achanda has quit IRC14:38
fungijeblair: what are your thoughts at this point, reupgrade zuul to normal master branch tip and perform a restart?14:39
jeblairfungi: without a manual dequeue command (wasn't someone writing that?) i don't think this instance can recover.14:39
*** Thelo has joined #openstack-infra14:39
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
fungijeblair: on a positive note, memory usage seems to have been far more stable with the revert in
jeblairfungi: yeah.  let me see if i can spot the error quickly before we restart14:40
fungiokay, holding pattern14:40
jeblairwow, that's actually a complex cycle14:42
jeblairthere are 4 different _updateChange calls14:42
jeblairfungi: oh, i think this is all happening before a change is being enqueued so this might be recoverable -- try breaking the dependency cycle in gerrit by modifying the commit messages as needed.  but remember the structure in case we need it for debugging later.14:47
*** andymaier has quit IRC14:47
odyssey4mejeblair so one particular change is holding up the queue somehow?14:48
*** burgerk has quit IRC14:49
fungijeblair: yeah, i'll give that a shot14:50
*** burgerk has joined #openstack-infra14:50
*** asselin has joined #openstack-infra14:50
*** maishsk has quit IRC14:50
fungifor the record, 300475 in fuel-library has a depends-on to 299838 in fuel-noop-fixtures which has a depends-on back to 30047514:51
fungii'm going to remove the depends-on header in the 300475 commit message14:51
*** annegentle has joined #openstack-infra14:53
fungiokay, that particular dependency loop should be broken now14:56
*** eharney has quit IRC14:56
*** mixos has joined #openstack-infra14:59
fungihaven't seen another "maximum recursion depth exceeded" in the log now since 14:53:50z15:00
*** burgerk has quit IRC15:00
*** rhochmuth has joined #openstack-infra15:01
*** mriedem is now known as mriedem_meeting15:01
*** mrmartin has quit IRC15:01
*** edmondsw has joined #openstack-infra15:02
*** krtaylor has joined #openstack-infra15:02
fungiodd that other dependency cycles aren't triggering this. for example, we're still logging this one zuul skipped intentionally: Dependency cycle detected: 300475 in [u'301107', u'300475', u'299838']15:02
*** mrmartin has joined #openstack-infra15:03
jeblairodyssey4me: yes, due to a bug15:03
jeblairfungi: yeah, i think this may be a legit zuul bug exposed by a complex dependency graph.15:03
odyssey4mejeblair ah, thanks for figuring it out - it looks like we're seeing a ton of jobs jump into the queue now15:03
fungiand possibly unrelated to the revert15:03
jeblairfungi: yes, at this point i think it's likely unrelated to the revert15:03
*** sigmavirus24 is now known as sigmavirus24_awa15:04
*** mrmartin has quit IRC15:04
*** sigmavirus24_awa is now known as sigmavirus2415:04
*** EricGonczer_ has joined #openstack-infra15:04
*** jsavak has quit IRC15:04
*** mriedem_meeting is now known as mriedem15:05
YorikSarfungi, jeblair: Thanks!15:06
YorikSarNow we have lots of changes in Zuul, but at least they should go through eventually.15:07
*** cody-somerville has joined #openstack-infra15:07
*** cody-somerville has quit IRC15:07
*** cody-somerville has joined #openstack-infra15:07
*** jsavak has joined #openstack-infra15:08
YorikSarI guess it would be great if CRs didn't affect each other in Zuul...15:08
openstackgerritAdam Reznechek proposed openstack-infra/system-config: Add openstack-powervm to meetbot and statusbot channels
fungiYorikSar: well, normally zuul silently skips testing on changes which are involved in dependency loops... this particular one seemed not to be detected by the normal dependency detection though15:09
*** ddieterly is now known as ddieterly[away]15:10
*** acruz has quit IRC15:10
YorikSarfungi: I mean like make failure domain for such issues very small.15:10
fungiYorikSar: i have no idea what that means in this context15:10
fungigiven that the reason zuul exists is to be able to test changes together, they need to affect each other15:11
YorikSarfungi: For example, handle every CR in a separate (green) thread so that if it fails, it doesn't affect other CRs.15:11
*** eharney has joined #openstack-infra15:12
*** isaacb has quit IRC15:13
*** ddieterly[away] is now known as ddieterly15:13
anteayaYorikSar: that is the description of the check queue15:15
anteayahowever zuul needs to have time to get to them15:15
anteayaand when looping it is sort of stuck15:15
anteayawhich I believe is what happened here15:15
*** jamesmcarthur has joined #openstack-infra15:16
*** swat30 has joined #openstack-infra15:16
pabelangeryolanda: excellent news15:17
mriedemis vexxhost == osic-cloud?15:18
mordredmriedem: nope15:18
mordredmriedem: vexxhost is a public cloud based in canada15:18
fungimriedem: no, vexxhost is a public cloud provider. osic is the "openstack innovation center" (a joing venture between intel and rackspace)15:19
mordredmriedem: osic is the OpenStack Innovaction Center cloud run by rackspace/intel15:19
rcarrillocruzwe don't have auro, do we?15:19
rcarrillocruzspeaking of canadians15:19
mordredrcarrillocruz: we do not have an account on auro15:19
anteayarodrigods: vexxhost doesn't have a grafana dashboard yet?
mordredrcarrillocruz: although I have an auro account and do test that shade/occ works against it15:19
mriedemok, trying to write a logstash query for the multitude of vexxhost job timeouts15:19
*** yamahata has joined #openstack-infra15:19
mriedembut not seeing vexxhost showing up as a node_provider15:19
anteayamriedem: logstash doesn't see vexxhost?15:20
mriedemit does for build_node15:20
mriedemnode_provider is apparently ymq-2115:20
pabelangeralso see
anteayamriedem: that is vexxhost:
openstackgerritMerged openstack-infra/openstackid: Fix consent screen OAUTH2.0 flow * updated css on ui consent screen to better mobile UX * added autocomplete off to login form
mriedemyeah, just wasn't obvious15:21
anteayapabelanger: I don't see vexxhost show up in the main view:
anteayamriedem: I agree, ymq-1 isn't obviously vexxhost15:22
*** vgridnev has quit IRC15:22
*** vgridnev has joined #openstack-infra15:22
mtreinishmriedem: it's the region name15:22
mtreinishI only knew that because my personal stuff is in the same region15:23
pabelangeranteaya: the default list of dashboard is 10, so it is a truncated list now15:23
anteayamtreinish: if you look at node_provider for the rest of them they are hostname-region15:23
*** smarcet has quit IRC15:24
anteayapabelanger: ah, is there pagination?15:24
mriedemmtreinish: ok, that's fine. i'll just use ymq-1. point is, 79% of build timeouts are on vexxhost15:24
*** vgridnev has quit IRC15:24
mriedemsdague: ^15:24
*** mixos has quit IRC15:24
pabelangeranteaya: not on the default. However, is is a dashboard which we can customize.  Somebody just needs to propose a new default page15:25
*** Thelo has quit IRC15:25
*** bkeller` has joined #openstack-infra15:25
hasharianw: thank you for the blog post  . Quite informative, kudos!15:25
*** mikelk has quit IRC15:25
*** Thelo has joined #openstack-infra15:26
*** akaszuba has quit IRC15:26
*** ddieterly is now known as ddieterly[away]15:26
fungijeblair: wondering if you get a moment whether you could quickly review 300912 (even though it's merged now)? i should have left a comment on it that i was going to get your input as the original author of that section, just making sure you don't object and any of that hadn't been omitted intentionally15:26
anteayapabelanger: can we prioritize the nodepool pages on the index page for the time being?15:26
*** vgridnev has joined #openstack-infra15:26
sdaguemriedem: is there a question?15:26
fungijeblair: i'm planning to link to that page when replying to the internap marketing thread15:26
*** vgridnev has quit IRC15:26
*** mylu has joined #openstack-infra15:26
mriedemsdague: no15:26
mriedemi'm going to put up an e-r query for that specifically though15:27
*** mixos has joined #openstack-infra15:27
mriedembecause i'm tired of digging into logs to just see build timeouts on vexxhost15:27
sdaguemriedem: yep, that's fair15:27
fungimriedem: we're still seeing slow performance on vexxhost workers? mnaser seemed to think he had worked out and solved the cause, but i guess if we can collect hostnames of more examples that will help him narrow it down further15:27
pabelangeranteaya: it requires somebody to add a new grafyaml template for it, so about the same amount of work to update the page15:27
sdagueis there a query up for that15:27
*** tbachman has joined #openstack-infra15:28
mordreda logstash query will certainly help to provide hostnames15:28
* fungi is concerned mriedem may be rickrolling him15:28
*** ddieterly[away] is now known as ddieterly15:28
*** abregman has joined #openstack-infra15:29
mordredfungi: mriedem is always rickrolling you, even when he's not15:29
anteayamriedem: I see 60 hits in the last 2 days15:30
fungiindeed, extending that query to the past week, it's still pretty constant15:30
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for build timeouts on vexxhost bug 1565850
*** ddieterly is now known as ddieterly[away]15:31
openstackbug 1565850 in OpenStack-Gate "79% of build timeouts are on vexxhost ymq-1 nodes" [Undecided,New]
mriedemyeah 376 in 7 days,15:31
mriedemcheck and gate and voting jobs15:31
anteayaI can see your point15:31
mriedemthe pie chart on node providers makes it pretty obvious15:31
anteayaI believed you before, but now I can see it15:31
mriedemonce you remove node_provider from teh query15:31
anteayapie chart?15:32
fungiremoving the provider-specific filter from that query, it's still mostly vexxhost (the ovh hits are less likely performance related and merely because ovh provides an order of magnitude more quota)15:32
mriedemdo a trend on node_provider15:32
mriedemanteaya: sorry, remove node_provider from the query, then click on node_provider on the left, do terms > pie15:32
*** nadya has joined #openstack-infra15:32
mriedemover 7 days15:33
anteayamriedem: I will try that, thank you15:33
fungifor only being around 5% of our aggregate quota, the quantity we're seeing time out there are definitely disproportionately high15:33
mordredalso, I _am_ confused as to why node_provider for vexxhost is showing up at ymq-1 and not vexxhost-ymq-115:34
mordredI wonder if somewhere we're splitting on - and doing it wrong15:34
fungimordred: likely a regex issue due to the extra - in it15:34
sdaguemordred: yes15:34
*** jsavak has quit IRC15:35
sdagueno other cloud uses '-' in region15:35
sdaguethe vexxhost entries have 2 dashes15:35
anteayamriedem: look at that! a pie chart15:35
*** jsavak has joined #openstack-infra15:35
openstackgerritSean Dague proposed openstack-infra/project-config: rename vexxhost to match existing conventions
jeblairyeah, this is a bug in the logstash processor15:35
sdagueI kind of wonder if that will help15:35
sdagueor if it uses deeper things15:36
jeblairit hardcodes a mere convention.  it splits on '-' and assumes a certain structure15:36
sdaguesure, but there is no reason we can't enforce that convention, right?15:36
jeblairsdague: no, other than we didn't.15:36
*** fedexo has joined #openstack-infra15:37
jeblair(this has been discussed before, and the possibility this happened was raised, and yet it happened)15:37
jeblair(sorry, possibility it could happen)15:37
*** nadya has quit IRC15:37
*** mixos has quit IRC15:37
mordredwell, I mean, also we can't enforce anything about vexxhost's region name having a - in it15:37
mordredbut we can fix the log processor15:37
jeblairyeah, i'd really prefer a robust fix15:38
sdagueI guess the question is, are we exposing nodepools ideas of naming, or the providers ideas of naming in the final result?15:38
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
mordredsdague: the providers15:39
sdaguemordred: ah, bummer15:39
fungii think nodepool's idea of naming exposes the provider's ideas of naming15:39
jeblairand the region name is actually 'ca-ymq-1'15:39
sdaguetoo bad we don't have an abstraction there15:39
jeblairi don't want one15:39
*** Swami has joined #openstack-infra15:40
mordredyah. I'd really like this to explicitly be {cloud_name}-{region_name} and not something else15:40
jeblairthe cloud is 'vexxhost' and the region is 'ca-ymq-1'.  and when we debug nodepool issues, i don't want a translation table.15:40
mordredotherwise tracking things down gets harder15:40
mtreinishjeblair: oh, that thing I added to try and extract the provider. I don't know if we thought about a double -15:40
mtreinishI know I only tested it with the existing providers at the time15:40
*** catherineD has left #openstack-infra15:40
rcarrillocruzmordred: played with ansible-openstack-cloud-launcher yet? I expect you to break it so I can fix missing things :-)15:40
mordredrcarrillocruz: not yet - it's on my list15:41
fungiit _does_ however mean that a human needs to involve judgment since we're encoding a separator which can also appear in the fields themselves without escaping, so makes it not easily machine-parsed15:41
*** jistr has quit IRC15:41
mordredso - how about this15:41
mordredhow about we track 3 fields15:41
jeblairfungi: the *problem* is that we are *parsing* it.15:41
mordrednodepool_provider - nodepool_cloud and nodepool_region_name15:41
fungijeblair: yep, that i agree with15:41
mordredthey are three different pieces of information and all of them should be directly available15:42
*** salv-orlando has quit IRC15:42
*** mixos has joined #openstack-infra15:42
rcarrillocruzi'll try to write up an example geared to openstack infra conventions15:42
mordredand clearly all of them are useful15:42
mordredrcarrillocruz: it was immediately clear to me how it would map to infra in my brain - so good job15:42
fungii don't think they're actually independently useful at the logstash level15:42
fungiat least for now, the most regions we have at a single provider is 315:43
rcarrillocruzindeed, as a matter of fact i started the example with 'infracloud' and 'rackspace', and with projects 'openstack-ci'/'openstack-jenkins'15:43
fungiit's easier to just treat the provider and region together as a single identifier and be done with it15:43
*** Sukhdev has joined #openstack-infra15:43
rcarrillocruzbut decided to change it to typical ops/qa/rnd scenarios for the blogpost15:43
mordredfungi: I could see that too15:43
sdagueprovider + region is usually the correct error bucket15:43
mordredfungi: but we actually already write all three pieces of information to the node15:43
sdaguethings go wrong in a region most of them15:44
jeblairsdague: cloud+region == provider15:44
sdaguethe time15:44
mordredso it still goes back to - there is no need to parse something15:44
clarkbyou have to parse the hostname, this was always a best effort thing to begin with15:44
jeblairfungi: cloud+region == provider15:44
*** sputnik13 has joined #openstack-infra15:44
clarkbI am not sure what the issue is15:44
fungijeblair: yep, that's what i meant15:44
mtreinishmordred: there is because the log processor doesn't get the provider as a field over zmq15:44
*** shashank_hegde has joined #openstack-infra15:44
mtreinishjust hostname15:44
clarkbthe information provided is accurate15:44
fungicloud+region not provider+region15:44
mtreinishthat's why we added the parsing15:45
clarkbmtreinish: right15:45
jeblairyes, if this were easy, it would have been done :)15:45
jeblair(it will be easy in zuulv3)15:45
jeblairbut at the moment, we don't have a good way to get that metadata from nodepool to zuul15:45
jeblairit is disintermediated by jenkins at the moment15:45
sdaguewell, the regex can be fixed to be made a bit better at least, mtreinish you have a pointer to where it is?15:46
*** tbachman has quit IRC15:46
*** apoorvad has joined #openstack-infra15:46
clarkbthe problem is you can have an aritrary number of -s in the hostname too15:46
*** bpokorny has joined #openstack-infra15:46
*** jaosorior has quit IRC15:46
clarkbwe may have since cleaned up any with 2 or more though15:46
fungitrue. dns is pretty limited in available characters to provide separators, so no way to tell where parts of the node type end and parts of the provider name begin15:47
sdagueregardless, we should really pull vexxhost out of use. It's still randomly killing things by being super slow15:47
jeblairright, we can flip it so the potential failure is in the other direction15:47
*** jaosorior has joined #openstack-infra15:47
*** harlowja_at_home has joined #openstack-infra15:47
*** MarkAtwood has joined #openstack-infra15:47
jeblairwe also have a proposal on the table to make hostnames meaningless15:47
clarkbwe do?15:47
*** jaosorior has quit IRC15:48
*** mixos has quit IRC15:48
*** david-lyle_ has quit IRC15:48
fungisdague: agreed. i'll put together the list of impacted nodes for mnaser and we can lower max-servers to 0 there for now. still need to retry the traceroute bit so we can provide more feedback to rackspace and hopefully turn ord back on though15:48
*** jaosorior has joined #openstack-infra15:48
*** david-lyle has joined #openstack-infra15:48
*** sputnik13 has quit IRC15:48
clarkbfungi: you can add traceroute to the package listing that i updated for uuidgen and friends15:48
jeblairclarkb: yeah, there was talk of that with the nova launch count optimization.  i don't think we're going to head in that direction.  just saying there may in general be good reasons to avoid 'hostname' as a data communication channel.15:48
*** amotoki has joined #openstack-infra15:48
clarkbfungi: you will have to separate add it for bare-* if that isn't already done15:49
mordredalso, we could do  ubuntu-trusty-2-node-9298435.bluebox-sjc1 instead of ubuntu-trusty-2-node-bluebox-sjc1-929843515:49
fungijeblair: that likely means i should reconsider part of 30091215:49
jeblairfungi: well, it's correct now.15:49
mordredor even ubuntu-trusty-2-node-9298435.sjc1.bluebox to match the mirror pattern15:49
jeblairfungi: and like i said, i don't think it's going to change any time soon.15:50
fungiclarkb: yep, i was leaning toward adding it to the infra-package-needs element15:50
*** greghaynes has quit IRC15:50
fungijeblair: okay. in that case if you're not concerned about any changes i made in 300912 i'll go ahead and fire off my reply15:51
jeblairmaybe someone should look into whether we can attach more metadata to jenkins slaves.15:51
jeblairfungi: lgtm15:51
*** ddieterly[away] is now known as ddieterly15:51
jeblairfungi: i think the spirit ("developers can/will know jobs are running on your cloud") will persist beyond the particular implementation of encoding that in hostnames anyway.15:52
fungijeblair: i agree, though that metadata may surface in different places in the future15:52
*** tosky has quit IRC15:52
*** Qiming has quit IRC15:52
fungii still expect it to hold true aside from the current implementation detail of it being in hostnames15:52
*** mixos has joined #openstack-infra15:53
*** piet has quit IRC15:53
*** tbachman has joined #openstack-infra15:54
jeblairokay, i have a way to fix this problem correctly15:54
clarkbwe also need to be careful about hostname limits, mordreds suggestion should neatly deal with that15:54
*** zeih has quit IRC15:54
*** Guest12623 has quit IRC15:54
jeblairby correctly, i mean, "not by parsing hostname"15:55
*** vincentll has quit IRC15:55
jeblairit will involve a nodepool change to pass some extra metadata to jenkins and possibly a zmq-plugin to extract that metadata15:56
jeblairis anyone interested in working on it?15:56
*** tbachman has quit IRC15:56
openstackgerritMonty Taylor proposed openstack-infra/project-config: Use . between node and provider info in hostnames
jeblairno?  all we're going to do is just parse hostnames?15:57
mordredjeblair: I think doing the zmq/nodepool work is an excellent idea15:57
mordredI do not want to offer to work on it right now becaue I'm a little bit slammed15:57
jeblairme too.15:57
openstackgerritSean Dague proposed openstack-infra/puppet-log_processor: a better heuristic for figuring out node provider
jeblairmore parsing.15:58
sdaguehow about that heuristic change15:58
openstackgerritAnita Kuno proposed openstack-infra/project-config: Turn off vexxhost in nodepool
jeblairsdague, mtreinish: could i convince one of you to spend just a little time in nodepool so we can pass the metadata via a channel other than hostname?15:59
sdaguejeblair: that's not really in the cards for me atm given the contexts I need to keep in my head. I can propose the narrow fixes, but the broader stuff is going to have to be left to someone that has nodepool in their head.16:00
mtreinishjeblair: I doubt I'll have the bandwidth for that16:00
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental centos-7 beaker job for testing
*** mixos has quit IRC16:01
*** hieulq has quit IRC16:01
*** mixos has joined #openstack-infra16:02
*** greghaynes has joined #openstack-infra16:02
pabelangerneed to step out for 30mins, but if project-config-core would mind reviewing ^. Adds experimental job for beaker on centos-716:02
mnaserfungi please let me know if you have a list16:02
mnaseror just one16:02
*** mixos has quit IRC16:02
sdaguemnaser: is currently running16:02
sdaguebut will be of the same fail class16:02
sdagueif fungi can translate that to uuid, you could see it in action16:03
mnaseri can do that16:03
anteayamorning zaro16:03
fungimnaser: want me to hold that node so it's not deleted when the job completes?16:03
openstackgerritMerged openstack-infra/project-config: Move tox-bandit to check as non voting
jeblairthis is the sort of low-hanging-fruit project that i really wish people would work on rather than tab alignment16:04
*** kzaitsev_mb has quit IRC16:04
mordredjeblair: ++16:04
*** kushal has quit IRC16:04
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
jeblairit is absolutely something that anyone in this channel can work on16:04
anteayamnaser: we are discussing not having nodes spin up in vexxhost for a bit until this is addressed:
*** ddieterly is now known as ddieterly[away]16:05
anteayajeblair: I was going to try to get the puppet-mediawiki module up on a vm16:05
anteayajeblair: but I can do this first if I am given some guidance16:05
anteayaif you think I can do the work16:05
mnaserfungi: yes please, and can i get access to it?16:05
mnaserthe vm is using ~46% cpu usage16:05
anteayaotherwise I'll default to my first plan16:05
mnaserthe hypervisor is running at 0% iowait, 40% idle cpu16:06
fungimnaser: i've held it now. what's your ssh key again?16:06
*** kushal has joined #openstack-infra16:06
openstackgerritThomas Herve proposed openstack-infra/irc-meetings: Move Heat meeting
jeblairand i find it very frustrating that we would add a known-broken feature like this without folks willing to step up to fix it16:06
*** tiswanso has quit IRC16:06
*** e0ne has quit IRC16:07
* anteaya defaults to the wiki thing16:07
fungimnaser: looks like the current process eating an entire cpu is running "nova-manage db sync"16:07
mnaserim getting quite suspicious this is a kernel issue16:07
mnasersingle-threaded process16:07
*** ddieterly[away] is now known as ddieterly16:07
*** ddieterly is now known as ddieterly[away]16:08
*** ddieterly[away] is now known as ddieterly16:08
mnaserthat actually explains a lot16:08
*** kushal has quit IRC16:08
fungimnaser: ssh jenkins@
mnaserso libvirt applies limits per vcpu instead of entire instance, so single threaded processes become slower instead of bursting up into an entire cpu16:09
*** mixos has joined #openstack-infra16:09
fungiyeah, now apache2 is eating an entire cpu on that worker16:10
fungithat was short-lived though16:10
mnaseri'm fairly certain i have an idea of where the problem is now16:10
mnaserwell, at least our initial resolution fixed it but in turn limited things a bit more than they should be, for single threaded work16:11
fungimnaser: still want a list of instances where we saw similar performance issues in the past half hour?16:11
fungier, past 24 hours i mean16:11
mnaserno, this should be enough to help us find the problem16:11
*** claudiub has joined #openstack-infra16:11
mordredjeblair: I'm totally willing to eventually step up and fix it - it'll just realistically be a few weeks until I'm likely to be able to do it - so I was putting up hacky-workaround related patches to help alleviate the current situation and give space for getting the proper fix done16:11
*** scheuran has quit IRC16:11
fungimnaser: okay, thanks for digging into it!16:11
clarkbI am still not sure I understand what the issue is16:12
clarkbthe data in elasticserach is accurate16:12
clarkbyou can query on it reliably16:12
*** tiswanso has joined #openstack-infra16:12
mordredclarkb: yes. it's just not obvious16:12
clarkband I think people might be more willing to volunteer if the individuals that wrote the existing code aren't made to feel like they shouldn't have bothered16:12
fungiclarkb: i think the complaint is that the "node_provider" parameter doesn't reflect reality16:13
mordredclarkb: that you need to search on ymq-1 to get vexxhost16:13
clarkbfungi: yes I agree it can be better16:13
*** EricGonczer_ has quit IRC16:14
openstackgerritGraham Hayes proposed openstack-infra/project-config: Create openstack/designate-tempest repository
clarkbbut the current info is complete and accurate16:14
mordredclarkb: I also agree - the tone on this was entirely wrong, and I apologize for my part in that16:14
openstackgerritMonty Taylor proposed openstack-infra/puppet-log_processor: Update provider regex to deal with provider subdomains
jeblairclarkb: when the original code was written, this problem was raised as a potential issue but ignored.  it has happened, and yet, no one in stepping up to fix it properly.16:15
*** MarkAtwood has quit IRC16:15
mordredthere is the other half of my thing - based on top of sdague's thing - I will not be offended if those patches are not used, but there they are16:15
jeblairi'm going to write the fix to this16:17
*** esikachev has quit IRC16:17
andreafhi - I'm looking for an API spec for
openstackgerritJan Kundr├ít proposed openstack-infra/gear: Bump version requirement for pbr
clarkbandreaf: is the tool used16:19
*** yaume has quit IRC16:19
openstackgerrityolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks
*** ramishra has joined #openstack-infra16:20
andreafclarkb: thanks!16:21
dimsfungi : can you please add me to the new python-k8sclient-core and -release groups?16:21
morganmordred, fu16:21
*** esikachev has joined #openstack-infra16:21
morganmordred, fungi, clarkb: what are the  amounts of resources wanted if a company is donating to infra?16:21
morganCloud resources that is16:22
anteayamorgan: right now we accept public cloud16:22
morganmordred:  I tab fail16:22
*** fedexo has quit IRC16:22
anteayamorgan: the page jeblair linked16:22
morgananteaya: correct. Was looking for the volume16:22
anteayamorgan: 100 vms16:22
anteayaglad to hear you say so16:23
jeblair(which is on that page)16:23
morganjeblair: yep. Looking at it. Thanks! Important any for $reasons$16:23
*** tbachman has joined #openstack-infra16:24
mnaserquestion: do jobs require 8 cores specifically?  if we provided an instance type with 2 vcpus @ 2.93ghz (but fully dedicated to the instance, not shared), would that work?16:24
morganmnaser: ooh yay vexxhost things. ;)16:24
mordredmnaser: the main thing is "can devstack gate jobs complete in the amount of time that is withing acceptable tolerances"16:25
*** mylu has quit IRC16:25
morganmnaser: the jobs are mostly budgeted for speed of devstack/them..what mordred said16:25
clarkband the extra cpus do help when you are running mysql and opentsack and rabbit all together16:25
fungialso number of vcpus is used to determine some job parallelism16:25
mordreddevstack/tempest have ability to do multi-threaded too - but yah, honestly, if it was one REALLY FAST vcpu I don't think that would be a problem16:26
clarkbosic jobs run on 4vcpu though16:26
mnaserso basically the idea to be transparent here is that 8gb instances get 2 physical cores from the system as a limit overall16:26
mnaserso it's 8vcpus, but not going more than 2 physical cores on the system16:26
mnaserthe problem is that libvirt currently limits each core instead of the entire set of cores combined16:26
*** tphummel has joined #openstack-infra16:27
*** mylu has joined #openstack-infra16:27
mnaserthis is why instances that are doing single-threaded work are struggling, because they're getting 50% of a single core16:27
mordredah - so the budgeted amount of real cpu space is going unused if we have one single vcpu pegged16:27
mnaserideally we'd use the global bandwidth settings but that is to come in libvirt 1.3.316:27
mnaserthis would let us limit all 8 cores to 2 physical, therefore one or two cores can burst up to an entire physical core, or the combination of the 8 will combine to 2 in total16:28
*** EricGonczer_ has joined #openstack-infra16:28
mnaserso the alternative is to either: a) present 2 fully dedicated cores to the OS, b) patch nova to manually set limits until that patch lands in libvirt and we move to 1.3.3 .. can't think of much more16:28
clarkbmnaser: is this a relatively new change? the runtimes we had when we first put vexxhost in place were in line with the other clouds16:30
clarkbbetween 50 minutes and an hour iirc16:30
*** sfinucan__ has quit IRC16:31
*** nmagnezi_ has quit IRC16:31
mnaserclarkb: yes, we didn't have any limits initially but the scheduler was sending instances to already-busy compute nodes16:32
mnaserso the foundation was being it's own noisy neighbor as well as a few other tenants that are very cpu hungry16:32
mordredah. yeah. we can definitely be our own noisy neighbor :)16:33
mnaserwe have the utilization based scheduler in place so it does aim to throw instances at least-busy-compute-nodes now instead of just based on free memory16:33
*** ramishra has quit IRC16:33
*** sputnik13 has joined #openstack-infra16:33
mnaserthat helped in sending instances to not-so-busy compute nodes, but the problem is that you can have 3 instances launched, places on the same node because it's not loaded, they start running their tests and now it's overloaded and you can't do much16:34
*** oanson has joined #openstack-infra16:34
*** rossella_s has quit IRC16:34
*** ramishra has joined #openstack-infra16:35
*** rossella_s has joined #openstack-infra16:35
mnaseris there a way i could a test suite on an independent instance (or to try it out on a different flavor/instance_type?)16:37
clarkbmnaser: yes, we have a script written out by each job that you can run16:38
clarkbmnaser: let me get you a link to one16:38
mnaserawesome, thanks clarkb16:38
*** esker has quit IRC16:39
*** harlowja_at_home has quit IRC16:39
*** shashank_hegde has quit IRC16:39
*** jsavak has quit IRC16:40
mnaserclarkb: perfect, let me do some checking16:40
*** jsavak has joined #openstack-infra16:41
clarkbmnaser: the comment header should give you a quick rundown of what is required ot run that16:41
*** EricGonczer_ has quit IRC16:42
*** EricGonczer_ has joined #openstack-infra16:43
*** salv-orlando has joined #openstack-infra16:43
clarkbmnaser: one of thoes files should be written out for every devstack-gate job that is run. I picked a recent one for a change that merged, but you can use that to reproduce any specific change's tests16:44
SpamapStrash: huh?16:45
SpamapStrash: it merged on March 22?16:45
mnaserclarkb: trying it now16:45
*** esikachev has quit IRC16:45
*** ramishra has quit IRC16:46
*** nmagnezi_ has joined #openstack-infra16:47
*** smarcet has joined #openstack-infra16:47
*** ddieterly is now known as ddieterly[away]16:47
mnaserclarkb: trying to run it now on a dedicated 2 core machine and will see time16:47
*** rcernin has quit IRC16:48
clarkbmnaser: note that you don't want to compare the jobs absolute time because you won't have the mirrors and git cache, but you can compare the tempest run time which should be comparable to the other jobs16:48
mnaserdd if=/dev/zero of=/root/swapfile bs=1M count=819216:49
mnaserwhy not replace this with fallocate16:49
mnaserwaaaaay faster and easier on disk io16:49
*** eharney has quit IRC16:49
clarkbmnaser: because centos in its great wisdom uses xfs16:49
clarkbmnaser: it is possible that once we are 100% building our own images we can switch back to fallocate16:50
clarkbsince we can force everything to be ext416:50
*** ifarkas has quit IRC16:50
mordredclarkb: we're still snapshotting centos right?16:50
clarkbmordred: not snapshotting, but in the middle of switching form using upstream base images to the minimal element16:50
clarkbmordred: the upstream base images start with an xfs /16:51
*** tbachman has left #openstack-infra16:51
mordredclarkb: ah. cool16:51
*** esikachev has joined #openstack-infra16:51
clarkbbut dib defaults to ext4 so as long as we are formatting our own disk setup we should be fine16:51
*** ddieterly[away] is now known as ddieterly16:52
*** bkeller` has quit IRC16:52
clarkbmnaser: we had also tried a sparse file loopback mounted but that caused really odd behaviors of instances basically going away16:52
*** salv-orlando has quit IRC16:52
clarkband never coming back (I left one running overnight to see if it would recover)16:52
mnaserthat's strange behaviour16:52
clarkbmnaser: theory was that we were running low on disk and memory and then trying to swap16:53
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Encode nodepool info in jenkins description
openstackgerritJames E. Blair proposed openstack-infra/zmq-event-publisher: Add node description to event data
mnasershould this be logging things somewhere16:53
clarkband since neither end of that could be satisfied the instance was hosed16:53
clarkbmnaser: in /opt/stack/new/16:53
jeblairclarkb: can you take a quick look at 301326 and 301327 and tell me if you are okay with the approach?16:53
clarkbjeblair: sure16:53
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
*** oanson has quit IRC16:54
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch gate-infra-puppet-apply-centos-7 voting
mnaserit's still just cloning things so unsure if there should be any output or not yet during clone16:55
*** eharney has joined #openstack-infra16:55
*** oanson has joined #openstack-infra16:55
clarkbjeblair: that seems fine, we aren't really using that description otherwise and the slave name itself encodes the details that will no longer be in the description16:55
jeblairclarkb: yeah, and we can add more to that dict in the future16:55
clarkbmnaser: during the clones there isn't output, it is logged to hte setup workspace log file16:55
mordredjeblair: love it16:56
pabelangerif anybody has time to review that will help with our devstack-centos7 to centos-7 migration16:56
jeblairclarkb: an alternative approach i considered was -- apparently you can add node-specific environment variables.  i do not know if those would show up in the parameters/env vars that currently end up in the zmq stream.16:56
clarkbmordred: if you are revieiwng nodepool changes should get in before next restart so that we don't have to manually clenaup the db again16:57
*** mylu has quit IRC16:57
jeblairbut if we don't like using the description field, we could see if those would work.16:58
mordredclarkb: looks generally good - any reason to remove that print?16:58
*** exploreshaifali has joined #openstack-infra16:58
mordredjeblair: I think using the description field sounds like the right amount of effort to fix this until we're on v316:58
clarkbmordred: ya it was debugging code that had snuck in >_>16:58
mordredclarkb: so, I'm going to +2 but not +A since we're still rolled-back on nodepool.o.o16:59
*** aysyd has quit IRC16:59
clarkboh I approved jeblair's nodepool chnag etoo, should I remove the approval and just leave the +2/16:59
mordredclarkb: well, we just need to not restart nodepool, or re-revert the git repo state locally17:00
clarkbI will do that17:00
mordredor just remove the +A for now - the occ change we need is in the midst of landing, so we should be out of weird-zone soon17:00
anteayaI have never seen us out of weird-zone17:00
clarkbmordred: ya I removed hte +A and left my +217:00
*** mylu has joined #openstack-infra17:01
mordredanteaya: well, this specific zone of weird where we're running on a reverted nodepool17:03
*** rossella_s has quit IRC17:03
*** oanson has quit IRC17:04
*** rossella_s has joined #openstack-infra17:04
jasondotstarjeblair: just read back... can you take a moment to talk more about this nodepool issue that you were discussing earlier?17:04
jasondotstaror if there is somewhere I can read up on it, I'll that that.17:05
*** tqtran has joined #openstack-infra17:06
anteayamordred: okay this specific zone17:06
cody-somervilleHappy Monday Folks. \o_17:06
*** hashar has quit IRC17:06
*** armax has joined #openstack-infra17:07
*** EricGonczer_ has quit IRC17:07
*** mylu has quit IRC17:07
*** jamesmcarthur has quit IRC17:08
*** mylu has joined #openstack-infra17:08
pabelangerclarkb: +2 on 300678, hit +A by mistake17:09
*** mylu has quit IRC17:10
*** shashank_hegde has joined #openstack-infra17:11
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib
*** esikachev has quit IRC17:13
pabelangerEmilienM: migration from devstack-centos7 to centos-7 dib ^, if you want to review it.  Doubt it will land today, but will take some time to reload JJB before of it17:13
openstackgerritMerged openstack-infra/elastic-recheck: Add query for build timeouts on vexxhost bug 1565850
openstackbug 1565850 in OpenStack-Gate "79% of build timeouts are on vexxhost ymq-1 nodes" [Undecided,New]
*** aysyd has joined #openstack-infra17:15
*** xarses has joined #openstack-infra17:16
*** Sukhdev has quit IRC17:16
*** ddieterly is now known as ddieterly[away]17:17
*** shashank_hegde has quit IRC17:19
*** nadya has joined #openstack-infra17:19
*** amrith is now known as _amrith_17:19
*** trown is now known as trown|lunch17:20
*** claudiub|2 has joined #openstack-infra17:21
*** jlanoux has quit IRC17:22
*** nmagnezi_ has quit IRC17:23
*** amotoki has quit IRC17:23
*** claudiub has quit IRC17:24
*** ihrachys has quit IRC17:24
*** kzaitsev_mb has joined #openstack-infra17:25
*** mrmartin has joined #openstack-infra17:26
*** sridhar_ram has joined #openstack-infra17:26
openstackgerritRossella Sblendido proposed openstack-infra/puppet-reviewday: Run pip install to install the dependencies
openstackgerritRossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard
mnaserclarkb: how much of a speedup do the mirrors give when running devstak17:31
clarkbmnaser: it drops the git clone setup time to about 5 minutes and devstack setup time to about 10 minutes17:31
clarkbmnaser: you can compare the tempest runtime numbers which it outputs and that should be roughly comparable17:32
mnaseroh it will output the tempest runtime at the end?  thats good17:32
clarkbtestr does that for us17:32
clarkbor the testr wrapper17:32
rossella_sclarkb, hi! I've followed your suggestions in , I'd appreciate if you could have another look17:32
clarkbrossella_s: will do17:32
rossella_sclarkb, thanks a lot, I've also created a patch to run 'pip install' in puppet-reviewday...not sure if the code is right17:33
*** tiswanso has quit IRC17:33
dimsclarkb: can you please add me to the new python-k8sclient-core and -release groups?17:33
*** tiswanso has joined #openstack-infra17:33
*** claudiub|2 has quit IRC17:33
*** oanson has joined #openstack-infra17:33
clarkbdims: sure17:34
clarkbdims: done17:34
dimsawesome clarkb thanks17:34
*** claudiub has joined #openstack-infra17:35
clarkbrossella_s: where is the puppet change?17:38
EmilienMpabelanger: ok - in case you're landing it tomorrow / wednesday, note I'll be afk. Your CI contact will be degorenko or mwhahaha17:38
degorenkohey :)17:38
rossella_sclarkb, , I put a the depends-on tag on the dashboard patch17:38
pabelangerEmilienM: ack'17:38
*** mylu has joined #openstack-infra17:40
*** rhochmuth has quit IRC17:41
*** rhochmuth has joined #openstack-infra17:41
*** rhochmuth has quit IRC17:41
*** piet has joined #openstack-infra17:42
clarkbpabelanger: EmilienM so we should hurry up and do ?17:43
EmilienMclarkb: I'm ready17:43
EmilienMclarkb: I was reviewing the zuul/layout17:43
EmilienMbut lgtm17:43
*** shashank_hegde has joined #openstack-infra17:44
*** pvaneck has joined #openstack-infra17:45
clarkbpabelanger: and you are good to watch the solum and other smaller hcanges if I approve  them?17:45
pabelangerclarkb: yes17:46
clarkbeh I can do that too so will go ahead17:46
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback
*** bkeller` has joined #openstack-infra17:46
clarkbpabelanger: do you need first?17:48
pabelangerclarkb: don't need.  That's to test our infra CI17:48
*** oanson has quit IRC17:48
pabelangerCan rebase it out if needed17:48
openstackgerritNikhil Komawar proposed openstack-infra/irc-meetings: Update the Glare team meeting chair
clarkbno rebasing needed just trying ot understand what oes where17:49
pabelangerI just wanted to make sure out puppet-openstackci beaker job worked before moving infra to centos-717:49
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback
pabelangerpretty sure it will17:49
openstackgerritMerged openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback
*** weshay has quit IRC17:51
clarkbrossella_s: reviewed17:52
*** nwkarsten has quit IRC17:53
*** nwkarsten has joined #openstack-infra17:53
openstackgerritMerged openstack-infra/project-config: Move gate-tempest-dsvm-docker-centos7 to centos-7
*** jamesmcarthur has joined #openstack-infra17:56
*** nwkarsten has quit IRC17:57
*** mrmartin has quit IRC17:58
*** angdraug has joined #openstack-infra17:58
pabelangerbefore we land 301206 (turn off vexxhost) did we want to check if we can re-enable rax-ord?17:58
pabelangersince we'll be down 2 regions17:58
clarkbpabelanger: I think fungi wanted to have traceroute working before turning rax-ord back on17:58
*** bpokorny has quit IRC17:59
clarkbpabelanger: I htink its fine to stop using vexxhost now regardless due to the error rate17:59
clarkband mnaser is looking into it on their side so we shouldn't need more data17:59
fungiyep, just about caught up on other obligations and putting that change to add traceroute in now17:59
fungimost of the work is figuring out what the packages for it are called on all our different platforms18:00
clarkbfungi: its 'traceroute' on ubuntu18:00
*** sambetts is now known as sambetts|afk18:00
*** dtantsur is now known as dtantsur|afk18:01
*** yamamoto has quit IRC18:01
*** yamamoto has joined #openstack-infra18:01
fungiclarkb: yep, that one was easy ;)18:01
clarkbfungi: and on fedora too
fungimnaser: do you still need ubuntu-trusty-vexxhost-ca-ymq-1-9298730 held or am i clear to release it for deletion now?18:02
*** flepied has quit IRC18:02
*** rhochmuth has joined #openstack-infra18:02
mnasersorry, no need for it anymore fungi18:02
openstackgerritMerged openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7
fungiclarkb: i'm hoping centos 6 is similar18:02
fungiand precise18:02
fungimnaser: thanks!18:02
clarkbfungi: seems to be for precise
*** doug-fish has quit IRC18:03
clarkbcentos is harder because they don't seem ot have a fancy package search anywhere I can find18:03
*** doug-fish has joined #openstack-infra18:03
*** doug-fish has quit IRC18:03
*** mixos has quit IRC18:04
*** davideag_ has quit IRC18:04
fungiour bare-trusty workers have traceroute6 but not traceroute18:04
fungier, bare-precise18:04
*** jsavak has quit IRC18:04
*** jsavak has joined #openstack-infra18:05
fungioh, also package installation in jobs for precise is broken anyway, because we're pointing it at our package mirrors18:06
*** degorenko is now known as _degorenko|afk18:06
fungiwe should probably not update sources.list for precise18:06
*** e0ne has joined #openstack-infra18:06
fungior else we should start mirroring precise18:06
fungibut given we only use it in rackspace, having it on our distributed mirrors is of questionable value18:07
fungiespecially since we're working on getting rid of it anyway18:07
clarkbI vote for not mirroring precise and just switching on release in the setup script18:07
*** ddieterly[away] is now known as ddieterly18:08
clarkbfungi: and so I think it is universally traceroute, not sure if traceroute6 comes in a different package on different distros though18:09
*** kushal has joined #openstack-infra18:09
fungithat's the part i'm worried about and currently validating18:10
*** jsavak has quit IRC18:10
*** jsavak has joined #openstack-infra18:11
*** salv-orlando has joined #openstack-infra18:11
fungiwe seem to already on centos7 they're in the same package at least18:11
fungigrr. composing two independent thoughts there ;)18:12
fungion centos7 they're in the same package at least18:12
openstackLaunchpad bug 57167 in ubuntu-meta (Ubuntu) "traceroute6 installed by default but not traceroute" [Undecided,Confirmed]18:13
clarkbseems yhat at least in precise that behavior persisted18:13
clarkbbut we should make sure our debootstrap is installing that iputils package18:14
fungilooks like it's "traceroute" everywhere18:14
*** mixos has joined #openstack-infra18:14
pabelangerfungi: should be easy to test with a bindep job. we gate on all our OSes now18:14
pabelangerminus bare-precise18:15
clarkbmaybr I misread the bug but traceroute6 does seem special on ubuntu18:15
clarkbfungi pabelanger ehat I eould do is guard the traceroutes with a which check18:15
clarkbget that in now, then work on getting images updated18:15
clarkbthen remove the guard at some point in future when all images are happy18:16
fungiclarkb: on trusty you don't get traceroute6 until you install the traceroute package, so seems to be limited to <=precise for the weirdness18:16
*** esikachev has joined #openstack-infra18:17
fungiclarkb: agreed, that's what i was planning to fix. only run traceroute commands if they exist (or optionally, just `|| true` on the end, which my original job config change had and then i spent some time testing various traceroute{,6} exit code behaviors and determined it was "unneeded")18:17
fungiif i'd left ||true on the end, wouldn't have broken the world18:18
*** daemontool_ has quit IRC18:20
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add traceroute to default preinstalled packageset
*** samueldmq has joined #openstack-infra18:21
*** e0ne has quit IRC18:22
*** trown|lunch is now known as trown18:23
scottdaHi Infra. We've had a couple +2's on this patch. Could anyone else have a look:
scottdaAdd functional test job for python-brick-cinderclient-ext18:23
dimsclarkb : fungi : is one hour a good time to wait between adding to ACL and seeing the +2 option in a review?18:25
fungiclarkb: the ||true solution is beginning to grow on me. traceroute even if it fails to reach its destination still exits 0, so this is really only guarding against the executable not existing at all. the only down-side i see is possibly misleading noise in the console log about "-bash: traceroute: command not found" et cetera18:25
fungiclarkb: oh, it does exit 2 if you give it an invalid/unresolvable target hostname18:26
*** salv-orlando has quit IRC18:27
fungidims: yeah, should be plenty. but gerrit caches weird state in your browser. on the page with the voting dialogue, try a forced refresh18:27
clarkbdims: it has to do with gerrit caching, it might be quicker if you log out and in?18:27
*** EricGonczer_ has joined #openstack-infra18:27
dimsclarkb : fungi : ack will try18:27
*** mixos has quit IRC18:27
openstackgerritMerged openstack-infra/project-config: Add experimental centos-7 beaker job for testing
*** mixos has joined #openstack-infra18:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib
dimsclarkb : fungi : worked like a charm. thanks18:29
*** angdraug has quit IRC18:29
clarkbfungi: note that won't affect bare-precise18:30
clarkbfungi: but I approved it anyways and we can just not do tracerouts on bare-precise if you || true18:30
*** bpokorny has joined #openstack-infra18:30
*** mixos has quit IRC18:31
*** esikachev has quit IRC18:31
rossella_sclarkb, thanks a lot!18:31
*** salv-orlando has joined #openstack-infra18:32
fungiclarkb: i also think that job macro is probably not used on jobs running on bare-precise18:32
fungiand we don't need to keep it around indefinitely either if we don't want18:32
*** davideagnello has joined #openstack-infra18:33
*** kushal has quit IRC18:35
*** ddieterly is now known as ddieterly[away]18:35
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Traceroute to git.o.o in net-info macro
jrollit appears patches with a depends-on line, that are approved before the dependency is approved, don't get into the gate queue, is that intentional?
clarkbjroll: if they do not share a gate queue in zuul yes18:36
*** mixos has joined #openstack-infra18:36
fungijroll: _if_ they don't share any jobs in common with the dependency, then yes18:36
anteayawell less intentional ad more how the system works18:36
anteayajroll: zuul doesn't have any memory, it only understands now18:36
jrollit seems they do not, in which case, how do I kick these into the gate?18:36
clarkbjroll: reapprove18:37
clarkbif you have an existing approval vote you will have to remove it and add it18:37
clarkbif another individual has an existing approval you cna just add yours18:37
jrollor neither if I am not core :)18:37
fungijroll: basically since you can't integration-test the dependencies together (because of not sharing a job) it's waiting to test in the gate until the stated dependency has completely merged18:37
*** nwkarsten has joined #openstack-infra18:37
jrolldtroyer: sdague: could you please re-approve and ?18:37
jrollfungi: right, though the dep here is completely merged18:38
clarkbmordred: you probably want in before any release happens right?18:38
clarkbit claims to depend on a change that failed to merge18:39
*** Sukhdev has joined #openstack-infra18:39
clarkbbut the change it depends on merged so it may just need rebase18:39
*** mylu has quit IRC18:40
fungijroll: the other challenge (and reason it doesn't auto-enqueue) is because the depends-on in the commit message isn't easily queryable through gerrit, so there's no clean way to make the merger of the dependency trigger enqueuing the dependent18:41
*** mixos has quit IRC18:41
jrollfungi: yeah, makes sense18:41
jrollfungi: though, quoting the entire depends-on line in the search seems to work ok?
fungior at least it used to not be easily queryable. you can now18:42
*** _amrith_ is now known as amrith18:42
jrollmaybe api behaves differently18:42
fungihah, i was just about to post taht link18:42
jrollaha, yeah18:42
fungiwell, similar one anyway18:42
anteayaAJaeger: thanks18:42
anteayaany objection to me turning off vexxhost in nodepool by self-approving 301206?18:42
fungijeblair: but yeah, the message querying in gerrit was pretty terrible until recent releases18:42
fungier, jroll ^18:43
*** salv-orlando has quit IRC18:43
clarkbanteaya: no, I didn't realize that was still not approved18:43
fungisorry jeblair18:43
openstackgerritRossella Sblendido proposed openstack-infra/puppet-reviewday: Run pip install to install the dependencies
anteayaclarkb: you are welcome to approve it you like18:43
jrollfungi: yeah, makes sense18:43
*** mylu has joined #openstack-infra18:43
clarkbanteaya: can do18:43
anteayaclarkb: would make me feel better18:43
*** e0ne has joined #openstack-infra18:43
*** tosky has joined #openstack-infra18:44
rossella_sclarkb, I've just updated according to your suggestions. Thanks a lot again for your help.18:45
*** mylu has quit IRC18:46
*** EricGonczer_ has quit IRC18:48
*** EricGonczer_ has joined #openstack-infra18:48
*** matrohon has joined #openstack-infra18:49
*** nadya has quit IRC18:50
*** mylu has joined #openstack-infra18:52
mordredclarkb: yes18:52
mordredclarkb: I have rechecked it18:53
mordredclarkb: once that lands, I'll put in a release request18:53
clarkbmordred: which of those changes does the find port with router stuff?18:53
*** mylu has quit IRC18:53
clarkbmordred: or did you decide to solve another way?18:53
*** mylu has joined #openstack-infra18:53
clarkbI guess that would be in shade?18:53
*** kzaitsev_mb has quit IRC18:53
mordredclarkb: - but I need to rework it to use the API I just introduced in the occ change18:54
mordred(and to address jeblair review comments)18:54
mordredclarkb: I'm about to get on a plane, so I plan to do that patch rework when I become airborn18:54
*** mylu has quit IRC18:55
*** mixos has joined #openstack-infra18:55
*** ddieterly[away] is now known as ddieterly18:56
*** mixos has quit IRC18:56
anteayamordred: happy planing18:57
*** annegentle has quit IRC18:57
*** angdraug has joined #openstack-infra18:57
*** sigmavirus24 is now known as sigmavirus24_awa18:58
*** sigmavirus24_awa is now known as sigmavirus2418:58
*** bkeller` has quit IRC18:59
*** bkeller` has joined #openstack-infra19:01
openstackgerritMerged openstack-infra/project-config: Turn off vexxhost in nodepool
*** vgridnev has joined #openstack-infra19:02
*** hashar has joined #openstack-infra19:02
*** mylu has joined #openstack-infra19:03
rossella_sclarkb, thanks a lot for reviewing!19:07
*** kzaitsev_mb has joined #openstack-infra19:08
clarkbanteaya: also vexxhost is in grafana don't know if you found it or not19:08
clarkbyou have to use the drop down menu to get it19:09
*** salv-orlando has joined #openstack-infra19:09
anteayaclarkb: drop down menu, thank you19:09
* anteaya looks again19:09
anteayaah there it is, thank you19:09
anteayathat helps me19:10
*** mdenny has quit IRC19:10
anteayauntil either me or someone else can offer a patch to the home page19:10
*** flepied has joined #openstack-infra19:11
*** sdake_ has joined #openstack-infra19:12
*** annegentle has joined #openstack-infra19:13
*** salv-orlando has quit IRC19:14
*** salv-orlando has joined #openstack-infra19:14
*** sdake has quit IRC19:15
*** imcsk8 is now known as imcsk8|lunch19:15
*** salv-orlando has quit IRC19:16
*** netsin has joined #openstack-infra19:16
*** salv-orlando has joined #openstack-infra19:16
*** salv-orlando has quit IRC19:17
*** ociuhandu has quit IRC19:17
*** salv-orlando has joined #openstack-infra19:17
*** EricGonczer_ has quit IRC19:18
clarkbfungi: the traceroute package change is about to merge I can work on new image builds if you like (or we can just wait for tomorrow to build them)19:18
clarkbI guess start with rax-ord19:19
openstackgerritMerged openstack-infra/project-config: Add traceroute to default preinstalled packageset
pabelangerclarkb: actually, looks like we are out of HDD for dibs19:20
clarkbpabelanger: oh fun, not entirely surprising19:20
pabelangerfailures both today and yesterday19:20
*** nadya has joined #openstack-infra19:20
pabelanger /me nods19:20
clarkbfungi: I will +2 the macro update but not approve it since I am about to eat lunch19:20
*** sigmavirus24 is now known as sigmavirus24_awa19:21
toskyclarkb: hi! Mind a question about my quest for merging-with-history, if you remember?19:21
clarkbpabelanger: removing devstack-trusty will help quite a bit there as each image is about 50GB of disk19:21
clarkbtosky: sure19:21
pabelangerclarkb: Oh, I thought we had devstack-trusty all cleaned up19:22
clarkbpabelanger: 6GB qcow2, 10GB raw, 17GB vhd * 219:22
clarkbpabelanger: we did19:22
*** mixos has joined #openstack-infra19:22
toskyclarkb: so, sahara-tests-release (which I'm now part of) has the "create" permission, which I thought was enough, but I didn't find a way to create this new orphan branch remotely19:23
clarkbpabelanger: but that doens't remove trusty stuff from the cache just the disk images, so when we added in xenail the cache got bigger overall19:23
clarkbtosky: you have to use the gerrit ui19:23
toskyclarkb: also for orphan branches?19:23
clarkbtosky: for any brnach19:23
clarkb(you need force push to use git directly and we don't give that out)19:23
toskyclarkb: oh, I see, let me check (I was granted the permissions few hours ago)19:24
*** yamahata has quit IRC19:24
*** mrmartin has joined #openstack-infra19:25
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: New library split from Magnum - python-k8sclient
*** mylu has quit IRC19:28
toskyclarkb: what happens if I don't specify the "Initial Revision:"? Will master be used, as it seems it happened earlier to the PTL trying to do the same?19:30
toskyfrom (which I guess maps to the web ui command) it seems you need a base branch somehow19:30
clarkbyou should be able to use a sha119:31
clarkblike 00000019:31
clarkbor another base commit that makes more sense19:31
*** Hal has joined #openstack-infra19:32
*** Hal is now known as Guest643719:32
toskythe idea of an orphan branch is no parent commit - but let's 0000019:32
clarkbyes the first commit in any git repo has a parent of 00000019:33
*** slogan has joined #openstack-infra19:33
tosky400 invalid revision "000000"19:33
clarkbbah gerrit why do you hate me19:33
*** derekh has quit IRC19:34
*** mylu has joined #openstack-infra19:34
clarkbwill it take a full sha1 of zeros? it might be conservative with the short sha19:34
sloganquick question: How do I contribute code to a project I'm not a part of, git clone, make changes, git add/commit, git review enogut, or do I have to have myself added to a list for that project? I'm a driver on another project, and considering contributing elsewhere, but not sure what the process is - not super clear in the docs.19:35
openstackgerritMerged openstack-infra/project-config: Switch gate-infra-puppet-apply-centos-7 voting
openstackgerritMerged openstack/diskimage-builder: ironic-agent postinst fails on systemd with no iptables
clarkbslogan: you just write and push the code as you have described19:35
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate to gate-openstackci-beaker-centos-7
toskyclarkb: tried 0000000000000000000000000000000000000000, which should be the right size, no luck19:36
fungiclarkb: thanks, i'll go ahead and approve the macro change and then babysit jjb updates since it looks like it will change 490 jobs19:36
sloganclarkb: thanks19:36
clarkbtosky: zaro may have better ideas but sounds like gerrit may not do what we need :(19:37
toskyclarkb: and wouldn't one of the almighty admin do it for me? *puppy eyes*19:37
clarkbtosky may need a gerrit admin yo force push an initial commit :(19:38
toskyclarkb: how do I identify the gerrit admins?19:38
*** matrohon has quit IRC19:38
*** Sukhdev has quit IRC19:38
*** cloudtrainme has joined #openstack-infra19:39
clarkbtosky: there us a gerrit group called administrators which you can lost membership on iirc19:39
clarkbit should be the samr as infra root19:39
toskylost membership?19:39
clarkblist I mean19:40
anteayaI have to go uptown for a meeting, probably be offline the rest of the day19:40
fungihave a good afernoon, anteaya19:40
*** weshay has joined #openstack-infra19:40
*** nikhil_k is now known as nikhil19:41
*** ams_ has joined #openstack-infra19:41
*** piet has quit IRC19:41
toskyclarkb: there is no administrators, but I guess I can check infra-core19:41
clarkbzaro: do you know if gerrit supports creating an orphan branch via the web ui?19:43
*** exploreshaifali has quit IRC19:44
fungiwow, getting massive packet loss on my v6 tunnel suddenly19:44
*** mylu has quit IRC19:44
fungimassive packet loss to the v4 endpoint in he19:46
fungiso that prolly 'splains it19:46
zarosorry, i do not. i think only way is to force push a new branch.19:46
smcginnisIs it possible to get rights to a channel to do things like ban folks?19:47
*** nt has quit IRC19:47
*** mylu has joined #openstack-infra19:48
clarkbpabelanger: "Job gate-puppet-openstackci-puppet-beaker-rspec-devstack-centos7 not defined" from the puppet switch change19:48
*** EricGonczer_ has joined #openstack-infra19:48
clarkbpabelanger: I think the streams crossed between that change and the other change19:48
clarkbsmcginnis: yes we can and do give individuals op perms in the various channels19:49
clarkbI believe the statusbot is managing that? let me dig up the config file19:49
smcginnisclarkb: Cool. Any chance I can get that for #openstack-cinder?19:49
toskyzaro: which is something not allowed, I guess?19:49
*** mylu has quit IRC19:49
fungiclarkb: smcginnis: for individual chanops on specific channels we just add you via chanserv19:50
openstackgerritMerged openstack-infra/project-config: Traceroute to git.o.o in net-info macro
clarkb is the config file though doesn't look like we manage the individual channel opers there, I guess that others have ya ok that19:50
clarkbfungi: is there a reason it isn't done via the config that does it anywyas?19:50
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib
pabelangerclarkb: Ya, that should fix it^19:50
fungiclarkb: precedent i guess? i don't think anyone spent the time to look them all up and add them to the config19:50
pabelangerclarkb: it was a missed puppet job19:51
fungiclarkb: but also that config file might get a bunch more complex19:51
*** jcoufal has quit IRC19:51
smcginnisclarkb, fungi: So it's just a manual admin change for now?19:51
fungigiven just about ever channel has at least one additional operator (the founder) if we didn't create it19:51
fungismcginnis: right, one of us with admin access to the channel just adds you as a channel operator on that channel19:52
fungiclarkb: were you doing that, or shall i?19:52
clarkbfungi: go for it, trying to review centos-7 things19:52
fungion it19:52
smcginnisclarkb, fungi: Thanks!19:53
openstackgerritRoland Hochmuth proposed openstack-infra/project-config: Add openstack/monasca-transform project
clarkblooks like jgriffith is the original founder?19:54
clarkbpabelanger: are there any other jobs that need switching that you are aware of? or is the proposed change set complete?19:54
zarotosky: actually.  i don't think you can do it at all.  i see the create-branch rest endpoint requires a valid revision.19:54
clarkbzaro: 000000 is a valid revsion19:54
clarkbzaro: it is the parent revision of the first commit on a branch19:55
marcusvrn_Hi all! I want to setup a zuul monitoring dashboard (like openstack zuul status) for my CI. Can someone help me?19:55
*** nt has joined #openstack-infra19:55
*** amitgandhinz has quit IRC19:55
fungismcginnis: should be all set now19:55
pabelangerclarkb: kolla is the last big change.  Then a few 1 lines we can lump together19:56
zarotosky, clarkb : so have you tried with ssh or rest endpoint?19:56
clarkbzaro: just the web ui19:56
*** _ari_ has quit IRC19:56
pabelangerclarkb: I was holding off on that since I wasn't sure if they did their mitaka release yet19:57
clarkbpabelanger: that seems reasonable, I think there is mail about that to the list let me catch up19:57
*** jaosorior has quit IRC19:57
smcginnisfungi: Thanks!19:57
pabelangerclarkb: I can prep the change now19:57
*** jsavak has quit IRC19:57
zarotosky: i would try 0000000 with ssh command
clarkbpabelanger: looks like rc3 is recent19:57
toskyzaro: let me try19:58
clarkbpabelanger: but no proper release yet so holding off seems like a good idea for now19:58
*** tonytan4ever has quit IRC19:58
fungiclarkb: any reason to keep the "old" precise-based around?19:59
fungiseeing what i can clean up to free up some quota for us19:59
*** amitgandhinz has joined #openstack-infra19:59
*** mylu has joined #openstack-infra19:59
jgriffithclarkb: yeah, but I think my perms were turned off when for channel at some point... anyway; sounds like smcginnis has it worked out19:59
clarkbfungi: not that I can think of20:00
openstackgerritRoland Hochmuth proposed openstack-infra/project-config: Add openstack/monasca-transform project
*** tqtran is now known as tqtran-afk20:00
fungiclarkb: thanks, BALEETED!20:00
toskyzaro: fatal: invalid revision "0000000" (same with 0000000000000000000000000000000000000000)20:00
*** bkeller` has quit IRC20:00
*** christiana has quit IRC20:02
zarohmm, clarkb : how can we verify that 000000 is a valid revision?20:03
clarkbzaro: look at the first commit in any repo20:03
clarkbzaro: the parent of that ocmmit should be 00000020:03
clarkbzaro: so it isn't a normal revision but it is valid for the purposes of seeding branches20:04
toskybut it's not an existing commit, so maybe git handles this in a special way20:04
toskybut if gerrit looks into the list of objects which are commits, it won't find it20:04
clarkb(because every commit in gerrit must hvae a parent you need to do something)20:04
fungiyeah, for example git show on system-config shows the first commit with a diff between revisions at index 0000000..a908c6e20:05
*** bkeller` has joined #openstack-infra20:05
fungiet cetera20:05
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate kolla jobs to centos-7 dib
fungihuh, though actually git rev-list --parents shows the first commit as its own parent20:06
fungialso git cat-file -p on the initial revision doesn't show a parent20:08
clarkbhuh git log does though20:08
fungiso the 0 commit in git is indeed so very special you can't really get git tools to show you it's there, i guess20:08
clarkbsounds like gerrit just needs to spceial case it too and probably doesn't today20:09
clarkb"create a new branch with no parent"20:09
*** esikachev has joined #openstack-infra20:09
*** jsavak has joined #openstack-infra20:10
*** mrmartin has quit IRC20:10
*** e0ne has quit IRC20:11
toskybut then how are new projects created? Different code?20:11
clarkbyes gerrit has a create project command20:11
clarkbwhich likely special cases this20:11
toskyso back to square one, if gerrit can't do it with its command, could this still be handled by force pushing for people who have the access?20:13
*** Goneri has quit IRC20:13
*** sdake_ is now known as sdake20:14
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Remove group regex on .testr.conf
toskyor should I simply forget about this (well, I could use the first commit of the project, which is empty)20:14
*** vgridnev has quit IRC20:15
clarkbif you point infra root at the code that should be pushed I am sure we can get to it at some point, but I don't think it will be a priority. Can SergeyLukjanov do it (covers both sides of the equation?)20:16
*** sigmavirus24_awa is now known as sigmavirus2420:17
*** shashank_hegde has quit IRC20:18
*** shashank_hegde has joined #openstack-infra20:18
toskymaybe yes - let's see if someone else would like to comment too20:19
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate diskimage-builder tests to centos-7
*** esikachev has quit IRC20:22
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate anvil to centos-7
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Disable by UUID before rebuilding
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Declare redis 2.8.4 on
*** baoli_ has quit IRC20:23
*** ZZelle_ has joined #openstack-infra20:24
pabelangerclarkb: fungi: So, rally is still using bare-precise.  We should find out why and move them to ubuntu-trusty if possible20:24
pabelangerseems like a basic install check20:24
fungimordred: does 301330 work the way i'm trying to do that?20:24
*** akshai has quit IRC20:25
SergeyLukjanovclarkb, tosky yeah, i can handle it as infra-root for sure.20:26
SergeyLukjanovtosky what's the current approach selected?20:26
*** edmondsw has quit IRC20:26
toskySergeyLukjanov: something like:20:26
fungijgriffith: fwiw i see you as having channel operator permissions on #openstack-cinder too, so it should have been working for you20:27
fungijgriffith: /msg chanserv access #openstack-cinder list20:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate rally to centos-7
pabelangerfungi: nibalizer: So, to late for today, but I'd like to see about migrating puppetdb.o.o to ubuntu-trusty this week.  Just making sure you're both cool with it, and it will be safe to have 2 puppetboard.o.o servers online when we start20:28
fungipabelanger: seems fine, i don't see an issue with it20:29
SergeyLukjanovtosky and then create a merge commit from orphan branch to master?20:29
*** ddieterly is now known as ddieterly[away]20:30
toskySergeyLukjanov: first I need to push the commits - but that I should be able to handle, sahara-tests-release has the permissions20:30
clarkbpabelanger: I notice that our raw images are no longer being smaller due to the block freeing20:31
SergeyLukjanovclarkb if you're ok with I'll work with tosky to make it happen20:31
clarkbor wait I have to ocmpare du -hs to ls -lh20:31
SamYapleclarkb: you mean the fstrim stuff?20:31
*** Sukhdev has joined #openstack-infra20:31
clarkbI take that back they are still smaller on disk20:31
SergeyLukjanovtosky I think we can push the branch with all needed code20:31
fungipabelanger: the main challenge right now is around quota. we're mid-replacement on multiple large servers at the moment so memory quota is getting tight20:31
clarkbSamYaple: ya its working I just forgot that ls shows you the apparent size and du shows you disk use20:31
clarkbSamYaple: du is showing 11GB vs 17/18GB20:31
toskySergeyLukjanov: oh, then you can see it from my work github repository20:31
jgriffithfungi: hmm... I was mistaken, thanks!20:31
SergeyLukjanovtosky yup20:31
pabelangerfungi: I don't have to start now, I figured it would be a trivial upgrade to do.20:32
*** Sukhdev has quit IRC20:32
toskySergeyLukjanov: my master branch here: - you can check from the history graph20:32
*** elo has quit IRC20:32
fungipabelanger: also that one's pretty small, so might not pose a challenge20:33
*** elo has joined #openstack-infra20:33
SamYapleclarkb: btw i figured out how to do the multinode with the current infrastructure where the vms get internet access20:33
pabelangerclarkb: how do we fix that?20:33
*** tphummel has quit IRC20:34
toskySergeyLukjanov: but my master branch was outdated, so either I should update it or you can take the content of that tempest-sahara-api and do the merge20:34
toskyyou know :)20:34
*** sdake_ has joined #openstack-infra20:34
*** sdake has quit IRC20:34
SergeyLukjanovso, I think we can push content of tempest-sahara-api branch (just extraction of code from sahara master) to the sahara-tests branch tempest-sahara-api and then do appropriate work on this branch as CR and then create a merge commit to master. what do you think?20:35
openstackgerritSteve Lewis (stevelle) proposed openstack-infra/project-config: Add os_gnocchi role project for OSA
*** ddieterly[away] is now known as ddieterly20:35
*** tqtran-afk is now known as tqtran20:36
openstackgerritPaul Belanger proposed openstack-infra/project-config: Experimental gate-rally-install-ubuntu-trusty job
*** tphummel has joined #openstack-infra20:37
toskySergeyLukjanov: I was thinking about merging the code as is (it does not conflict with the existing code), then move back to the normal working mode and do the normal reviews which adapts it20:37
SergeyLukjanovtosky I agree, it's even better20:37
toskyjust remember that the available tempest-sahara-api is not an orphan branch, so it should be removed first20:38
toskyor another name should be used20:38
SergeyLukjanovtosky, so, push tempest-sahara-api (with just tempest/api/data_processing folder) from your repo to sahara-tests, create merge CR to master, merge it, delete branch tempest-sahara-api20:39
toskySergeyLukjanov: yes20:39
SergeyLukjanovtosky cool, I think it's safe enough20:39
toskyyeah, I think so :)20:39
SergeyLukjanovtosky is tempest-sahara-api branch in your repo up to date?20:40
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build openstack-pkg-tools
clarkbpabelanger: I have freed 63GB on nodepool.o.o:/opt. I see part of the problem is that the vhd conversion requires 2x the image size of free space20:40
SergeyLukjanovtosky and let's switch back to openstack-sahara to not spam infra channel ;)20:40
toskySergeyLukjanov: I don't think there have been other commits, let me quickly check20:40
clarkbpabelanger: we may want to optimize that and always do vhd conversion first so that we don't have a qcow2 floating around taking up the extra disk space20:41
Shrews~/b 720:41
clarkbpabelanger: there was a stale old image that nodepool didn't know about and a failed vhd conversion on the most recent image on disk20:42
pabelangerclarkb: great20:42
*** mylu has quit IRC20:43
jeblairoh wow20:43
jeblairthe summit party is a lead generation engine20:43
jeblair"The sponsors of StackCity Austin will receive all of the info you provide when registering."20:43
fungithat's frightening20:43
jeblair"All information entered into the StackCity Austin event registration page (Eventbrite) will be provided to the event sponsors. You can view a list of StackCity Austin sponsors here: You are not required to attend StackCity Austin. If you do not want your information to be shared with the event sponsors, then please do not register. "20:44
jeblairthat's in the waiver you must complete to register20:44
fungiyep, i certainly won't be registering for that one20:44
*** mylu has joined #openstack-infra20:44
jeblairin unrelated news, my Tuesday, April 26 is free.20:44
jeblairfungi: grab some beer then? :)20:44
fungijeblair: i'm kinda planning to give most of the vendor parties a miss anyway. there are apparently awesome bars/bbq all over the place anyway20:45
clarkbI registered and seem to recall it was optional to do that?20:45
clarkbmaybe I misread it20:45
*** dprince has quit IRC20:46
jeblairclarkb: name, email, title, org, and waiver (including the above statement) are all required fields according to the little red asterisks20:46
*** tphummel has quit IRC20:47
fungiof course, i suppose you could just register with a temporary throwaway address and then delete it as soon as you have your confirmation20:47
clarkbI might also have mixed up various things20:47
clarkbI swear one of them was "share contact info with organizers" and acheck box20:47
jeblairclarkb: maybe they changed it?20:48
jeblair("hey, no one is choosing to share contact information!"  "i know how to fix that!")20:48
clarkbha maybe20:48
fungiand also i can't imagine they're closely vetting titles and whatnot. i usually put something like "sentient lifeform" or "earthling"20:48
clarkbpabelanger: then when we rmeove devstack-centos7 we should free up another 100GB or so20:49
clarkber thats bad math20:50
*** nadya has quit IRC20:50
clarkb60GB orso20:50
*** annegentle has quit IRC20:50
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Revert "[sahara-tests] create and merge, disable tests"
fungibad math, no mantissa!20:51
clarkbfungi: should I start an ubuntu-trusty build to pick up the traceroute changes? this way I/we can omonitor the build and see if 63GB is plenty for it to complete20:51
clarkb(I think it iwll be very very close)20:51
*** sc68cal has joined #openstack-infra20:51
clarkbdue to the vdh conversion overhead20:52
clarkbits 17gb * 3 (raw copy that dib is using as source, vhd intermediate, vhd actual) + qcow 220:52
fungiclarkb: sounds good to me20:52
clarkbit is building20:52
*** hichihara has joined #openstack-infra20:53
*** sc68cal has quit IRC20:53
*** tphummel has joined #openstack-infra20:55
*** tphummel has quit IRC20:56
*** sc68cal has joined #openstack-infra20:57
*** annegentle has joined #openstack-infra20:57
*** ihrachys has joined #openstack-infra20:57
clarkbpabelanger: looks like the newer centos-7 changes are failing20:58
*** akshai has joined #openstack-infra20:58
*** esikachev has joined #openstack-infra20:59
*** tphummel has joined #openstack-infra20:59
clarkbfungi: pabelanger if this fails the other thing we can try doing is deleting our dib cache and rebuilding it from scratch since that cache is not cleaned up at all20:59
clarkbmeaning it just grows and grows and we likely have old stuff in there that can go away20:59
fungiyeah, we can "expire" it, but that will make the next builds probably a good deal slower21:00
clarkbyup but thats mostly fine21:00
*** hashar has quit IRC21:00
*** jsavak has quit IRC21:01
clarkbI am guessing there is old fedora release stuff and precise stuff in there that can just go away21:01
*** kgiusti has left #openstack-infra21:01
clarkbnot sure how hard to do manual surgery on it would be but deleting it entirely should be fine21:01
*** jsavak has joined #openstack-infra21:01
*** Sukhdev has joined #openstack-infra21:02
clarkbya there are precise and fedora 21/22 things I see21:02
clarkband centos621:02
*** esikachev has quit IRC21:02
*** rossella_s has quit IRC21:02
*** vgridnev has joined #openstack-infra21:03
clarkbI am running a du to get a feel for distribution of data in the cache then can attempt surgery on the ase images at least21:03
*** rossella_s has joined #openstack-infra21:03
*** amitgandhinz has quit IRC21:03
clarkbCentOS-6-x86_64-GenericCloud-20141129_01.qcow2 can go away21:03
*** esikachev has joined #openstack-infra21:03
clarkbit is 1.GB21:03
*** asettle has joined #openstack-infra21:04
clarkbpabelanger: Job gate-dib-dsvm-functests-devstack-centos7 not defined I think you didn't rename all of them in the zuul layout file21:04
*** thorst has quit IRC21:04
*** jamesmcarthur has quit IRC21:04
*** thorst has joined #openstack-infra21:05
clarkbmaybe we should gc the git repos in that cache?21:05
nibalizerpabelanger: ya i like it21:06
*** ociuhandu has joined #openstack-infra21:06
fungiclarkb: i wonder if that will make our images smaller too (or do we pull into the cache and then clone from the cache into the image?)21:06
clarkbfungi: I think it does a clone into the image21:06
*** thorst_ has joined #openstack-infra21:07
fungiokay, so image sizes likely won't improve, but yeah might help performance and some space on the server21:07
clarkbits probably worth removing the cache and having it rebuild just to see (and to fliush out things we know we don't need)21:07
clarkbbut the vast majority of our disk use here is the resulting images21:08
*** thiagop has quit IRC21:09
pabelangerclarkb: ya, working on it now21:09
zigoDo I need special rights to be able to edit the -infra agenda at
pleia2sorry for being absent today, sick, and going back to tend to that21:10
pleia2zigo: nope21:10
clarkbpleia2: get better21:10
*** thorst has quit IRC21:10
zigoWhen I click on "save", it doesn't seem to be saving anything.21:10
pleia2zigo: did you fill out the captcha?21:10
pabelangerpleia2: np, rest up21:10
zigoOh !21:10
zigopleia2: Cheers.21:10
zigopleia2: If the captcha was next to the save button, I'd have notice...21:11
*** thorst_ has quit IRC21:11
pleia2zigo: yeah :\21:11
fungizigo: yeah, hoping to upgrade to a newer mediawiki soon and use better captcha plugin and other spam mitigation solutions available at that point21:11
zigoI wasn't complaining, I was trying to find myself an excuse for being lame! :)21:13
*** jamesmcarthur has joined #openstack-infra21:13
*** daemontool has joined #openstack-infra21:13
fungizigo: i didn't think you were complaining, i just wanted to explain the reason for the sub-optimal captcha implementation we're on now21:13
zigofungi: Any chance I may bother you for a quick look?
*** sdake_ has quit IRC21:14
*** dimtruck has quit IRC21:16
*** ldnunes has quit IRC21:16
clarkbfungi: it is interesting how much smaller the apt cache is htan the yum cache21:16
clarkbjessie + precise + trusty + xenial vs fedora 21/22/23 + centos 6/721:16
zigoclarkb: You pre-populate /var/cache/apt with all of that?21:16
*** |-paul-| has joined #openstack-infra21:16
clarkboh doesn't look like jessie is caching21:16
pabelangerclarkb: I'm gate for deleting cache.21:16
clarkbzigo: yes21:16
zigoI'd use approx for doing it.21:16
clarkbzigo: its built into dib so we use whatever they use21:16
*** trown is now known as trown|outtypewww21:16
clarkbI am just comparing the size of that cache against the size of the corresponding yum cache21:16
*** dimtruck has joined #openstack-infra21:16
* zigo will think about implementing an element for approx then.21:16
clarkband ya jesie doesn't seem to be caching21:16
pabelangerYa, yum was 3.7GB when I checked21:17
openstackgerritBoris Pavlovic proposed openstack/requirements: Bump alembic version to 0.8.4
clarkbpabelanger: boris-42 ^ is the person to ask about rally and bare-precise21:19
pabelangerclarkb: Ya, I posted my questions in openstack-rally too21:19
boris-42clarkb: hi21:19
boris-42pabelanger: hi21:19
pabelangerboris-42: see my comments in #openstack-rally for context21:19
nibalizerpabelanger: so you think we just boot a trusty node, run puppet on it, and see where we get?21:20
nibalizeri'm wondering if we specify our puppetdb version or not21:20
boris-42pabelanger: thanks for work21:20
pabelangernibalizer: Ya, I tested it at midcycle at it worked then21:20
boris-42pabelanger: there are no reasons21:20
boris-42pabelanger: we just want to test that installation work on centos and ubuntu21:21
pabelangernibalizer: so, the plan I think is, boot new node, validate puppet run21:21
pabelangernibalizer: then run puppet again, and post to localhost db, check results.. .if they look good, update DNS records for new server21:21
pabelangerboris-42: okay, great. I added an experimental job first, so if that passes I'll change to ubuntu-trusty in a follow up21:22
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate rally to centos-7
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate diskimage-builder tests to centos-7
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate anvil to centos-7
nibalizerpabelanger: yea i think we'll need to pin puppetboard at least21:22
nibalizerpuppetdb can probably float21:22
pabelangernibalizer: Yup, pretty sure I did that... let me check review21:22
*** aysyd has quit IRC21:23
pabelangernibalizer: ^21:23
*** claudiub|2 has joined #openstack-infra21:26
*** jamesmcarthur has quit IRC21:27
*** ddieterly is now known as ddieterly[away]21:28
*** thorst has joined #openstack-infra21:29
*** claudiub has quit IRC21:29
*** ddieterly[away] is now known as ddieterly21:30
openstackgerritGraham Hayes proposed openstack-infra/project-config: Create openstack/designate-tempest repository
*** thorst has quit IRC21:33
*** bpokorny has quit IRC21:35
*** bkeller` has quit IRC21:35
nibalizerpabelanger: i think we should just tag on this
nibalizersince thats the last version that doesn't require 321:36
*** vgridnev has quit IRC21:36
nibalizerpuppetdb 3, i mean21:36
pabelangernibalizer: sure, if you think we are safe21:36
*** jsavak has quit IRC21:37
nibalizerya we are21:37
pabelangerokay, I'll update the review in a bit21:38
*** jsavak has joined #openstack-infra21:38
*** alex_xu has quit IRC21:39
*** eliqiao has quit IRC21:39
*** tiswanso has quit IRC21:41
*** bkeller` has joined #openstack-infra21:42
*** regXboi has quit IRC21:44
*** daemontool has quit IRC21:45
*** imcsk8|lunch is now known as imcsk821:45
*** eliqiao has joined #openstack-infra21:45
*** alex_xu has joined #openstack-infra21:47
*** sputnik13 has quit IRC21:47
pabelangerclarkb: fungi: we should land just to be safe for centos-7 ansible jobs21:47
clarkbpabelanger: is it still easier at this point to add that to fallback rather than just to the repo in question?21:48
clarkbI guess each repo will need to untangle what they need out of the list but it is self testing21:48
*** sputnik13 has joined #openstack-infra21:48
*** nwkarsten has quit IRC21:49
clarkbfungi: ubuntu-trusty image build succeeded21:50
pabelangerclarkb: Ya, the openstack ansible team is fine, same with the roles I've done.  It this is more for kolla I figure. I haven't looked to see what is needed to migrate them to other-requirements.txt21:50
*** weshay has quit IRC21:50
clarkbit seems that 176782 has leaked21:52
openstackgerritMonty Taylor proposed openstack/requirements: Bump upper constraint on os-client-config
clarkbI take that back, there was just lag between my df ing and the files actually deleting21:54
clarkbI will start uploads to osic, internap, bluebox and rax-ord21:54
clarkbthe first 3 because they are qcow2 consumers and the last because that is where we had network issues21:54
*** sdake has joined #openstack-infra21:54
*** mixos has quit IRC21:55
clarkbfungi: this typically takes and hour or two for a rax upload21:55
clarkbfungi: fingers crossed we can enable rax-ord late today or early tomorrow21:56
*** matt-borland has quit IRC21:58
*** mixos has joined #openstack-infra21:59
*** david-lyle_ has joined #openstack-infra22:02
*** david-lyle has quit IRC22:03
*** sdague has quit IRC22:03
*** david-lyle_ is now known as david-lyle22:03
*** jsavak has quit IRC22:04
*** mixos has quit IRC22:05
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Rework migration 1679b5bc102c to not use the db api
clarkbfungi: osic is done so you should start seeing traceroutes there real soon now22:06
*** xyang1_ has quit IRC22:07
*** bkeller` has quit IRC22:07
jeblairfungi: i think i have reproduced the situation from this morning22:09
jeblair(with zuul and dependency loops)22:09
fungiclarkb: pabelanger: is libselinux-python a python library? is it going to drag in python lib dependencies?22:10
clarkbfungi: there is a traceroute22:10
*** eharney has quit IRC22:10
clarkbfungi: oh thats a good question22:10
openstackgerritMerged openstack-infra/project-config: Add libselinux-python for ansible jobs
fungijeblair: wow--what turned out to be the corner case that triggers it separate from a normal loop?22:10
jeblairfungi: the way the loop was constructed (in terms of patchset updates)22:11
jeblairfungi: possibly also the fact that other dependencies were involved22:11
jeblairi don't fully understand it yet :)22:12
jeblairjust managed to add in enough details to get it to break reliably22:12
*** sc68cal has quit IRC22:12
*** esikachev has quit IRC22:13
fungiit was certainly a strange scenario22:13
fungiglad it didn't entail a restart in the end22:14
*** eric_lopez has joined #openstack-infra22:15
*** mixos has joined #openstack-infra22:15
*** elo has quit IRC22:16
*** nelsnelson has quit IRC22:16
fungipabelanger: clarkb: i think this means it's not depending on any other python libs so probably safe-ish
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints
fungithe upstream url listed there for its source tracker goes to a 40422:17
*** mriedem is now known as mriedem_away22:17
pabelangerfungi: Ya, it shouldn't. It is a weird lib that depends on rpm C library22:18
smcginnisI'm seeing check failures in stacking, timing out to
*** smarcet has quit IRC22:18
smcginnisI get a blank page if I try to hit it directly/22:18
smcginnisKnown issue?22:18
fungismcginnis: that's not a browseable url, so getting a blank page is not surprising22:18
fungismcginnis: it's a git clone url22:19
fungismcginnis: have a link to the job log?22:19
pabelangerfor web interface22:19
fungismcginnis: there seem to be network connectivity issues in rackspace we're trying to help hunt down22:19
smcginnisfungi: That's probably it then.22:19
fungijust now in the process of adding some additional diagnostic details to the console logs22:19
*** sigmavirus24 is now known as sigmavirus24_awa22:20
fungiyep, happened in rackspace22:20
smcginnisfungi: Cool, not critical on my end, so I'll just check back later.22:20
fungibut it ran too soon to add traceroute outpur22:20
fungiwe've still got some image updates and job config updates in progress which are adding those22:20
*** mixos has quit IRC22:21
fungijob updates for that have completed everywhere except jenkins01, which hit a proxy error after 51 minutes. restarting the update there now22:21
EmilienMmhh, when ubuntu-trusty-rax was updated last time?22:22
fungiEmilienM: it's in the process of updating now i think--what are you seeing?22:22
*** Goneri has joined #openstack-infra22:23
EmilienMfungi: this patch for ubuntu (puppet) jobs is not in the rax image (but it's in some other cloud providers, ie bluebox)22:23
clarkbEmilienM: there are 3 rax images22:24
clarkbbecause we hvae 3 regions22:24
clarkbI think they are all roughly 3 days old which is when we ran out of disk22:24
fungior rather, 3 rackspace regions to which we try daily to upload consistent images into22:24
*** asettle has quit IRC22:24
clarkbI have all 3 rax region uploads for ubuntu-trusty in progress now22:25
EmilienMthat's a good news! thanks clarkb22:25
clarkbosic and bluebox are done22:25
EmilienMyeah, no failure to report on them22:25
fungiclarkb: in good news, the traceroute job addition this time isn't making jobs bomb out even though it is logging the lamentable lack of the tools22:26
clarkband it is working where the tools exist22:26
fungiso i think we're good there once image uploads complete22:26
clarkbfungi: see link above for osic example22:26
fungionce ord has a fresh image, we should probably crank up the max-servers there to something small like 10 and then see what we get22:27
*** tpsilva has quit IRC22:27
fungithough as smcginnis points out, we're seeing it happen within dfw too22:27
*** cloudtrainme has quit IRC22:27
clarkbinternap just finished22:27
*** annegentle has quit IRC22:27
clarkbleaving rax and ovh22:27
fungijust not as often as it did in ord (maybe because we booted more instances in ord? but it seems like the error rate there was disproportionately high still even then)22:27
clarkbhave we seen any of these failures in not rax regions? if not maybe its a wan problem for rax?22:28
*** gordc has quit IRC22:29
fungiclarkb: we have not, afaik. and since we should in theory be doing v6 from vexxhost to git.o.o that probably means it's not a v6-specific issue22:30
*** knikolla has quit IRC22:30
*** andymaier has joined #openstack-infra22:30
fungiclarkb: e-r entry is
fungihrm, in the last 10 days it's been 18 hits in rax-dfw, 4 in bluebox-sjc1 and 1 in ovh-gra122:32
*** tmorin has joined #openstack-infra22:32
*** andymaier has quit IRC22:32
fungiclarkb: given how many more nodes we boot in ovh-gra1 than rax-dfw, i have a feeling this is primarily an intra-rackspace connectivity issue, yes22:33
fungiespecially silly given that rax-dfw _is_ where we also host the git farm22:33
tmorinI'm having a hard time understanding why a devstack vm job fails, some help would be welcome22:33
*** mixos has joined #openstack-infra22:33
dougwigtmorin: link?22:34
tmorinconsole.html :
tmorindougwig: ^^22:34
*** dimtruck is now known as zz_dimtruck22:35
tmorinbut devstacklog.txt seems to happily end:22:35
*** EricGonczer_ has quit IRC22:35
*** ddieterly is now known as ddieterly[away]22:35
*** sdake has quit IRC22:36
clarkb I think that returned 222:36
*** asettle has joined #openstack-infra22:36
dougwigtmorin: 2016-04-04 22:30:41.799 | Waiting for bagpipe-bgp to start...22:36
dougwigtmorin: that's the last thing in devstacklog.txt, which is the script that likely failed.  did that return non-zero?22:37
*** markvoelker has quit IRC22:37
tmorinclarkb: I wouldn't expect it to fails without talking more / saying why22:37
tmorinclarkb: but that sounds plausible, yes22:37
clarkbtmorin: I am guessing the wait just timed out and it returned 2 check your init scrpt or whatever is making the service start22:37
tmorinclarkb: would output the summary lines at then end even if something fails ?22:38
clarkbI have no iea22:38
clarkbI would have to read the code22:38
clarkbsince I have to sign like 20 something pages of shit22:39
clarkbgah ww22:39
dougwig5 minute runtime, i doubt it's timeout.  but somethign is returning 2.22:39
clarkbhouse buying for you :)22:39
tmorinit is supposed to  "die xxx" if it fails22:39
tmorincan't see why it would silently fail22:39
clarkbtmorin: huh22:39
clarkbtmorin: is it supposed to do anything after that though?22:39
clarkbseems odd it wouldn't continue if it failed there22:40
tmorinclarkb, yes, it's supposed to run something else22:40
*** Guest6437 has quit IRC22:40
clarkbseems like it is failing at that start service portion22:40
clarkbif it isn't continuing to do the additional steps22:40
dougwigtmorin: i'd put an exit 0 at the end of that script.22:41
tmorindougwig: hmm, I wouldn't think that necessary, but I'll try22:41
tmorinclarkb: the other service is not enabled in this gate config22:42
clarkbtmorin: bash will exit with the exit status of the last command run22:42
fungiokay, jjb updates completed successfully on all jenkins masters now22:42
clarkbtmorin: oh so it may actually be done there22:42
dougwigtmorin: i'd have to read deeper to make sure $xtrace was always 0, otherwise.22:42
*** ybathia has joined #openstack-infra22:43
*** asettle has quit IRC22:43
tmorindougwig, clarkb: ok, you've helped enough to let me investigate more, that helps; I'll investigate  -- many thanks22:44
*** rebase has joined #openstack-infra22:44
*** david-lyle has quit IRC22:44
zarocrinkle: would you be able to help with puppet change?22:45
clarkbtmorin: dougwig wait_for_service may not actually exit with a return code that is useable in that way22:45
clarkbthe last command run in wait_for_service is time_stop which errors if the timer wasn't started but doesn't reflect the service waiting timeout return code22:46
zarocrinkle: why is this failing?
fungijeblair: not sure if you saw the "[neutron] -2'ing all patches on every gate breakage" thread starting at but it starts to delve into discussion of "if only we had some sort of procedural blocking mechanism that wasn't code-review -2..."22:46
*** mixos has quit IRC22:47
clarkbfungi: rax-iad is done but I guess it hasn't exhibited this issue?22:47
*** david-lyle has joined #openstack-infra22:47
*** |-paul-| has quit IRC22:47
zarocrinkle: download object is here:
clarkbovh-bhs1 failed and has been restarted, ovh-gra1 is in progress22:47
fungiclarkb: 18 times for the past 10 days in the gate. it wasn't as rampant as rax-ord22:47
crinklezaro: you want .empty? not .empty22:47
zarocrinkle: i want to check that it's not empty22:48
tmorinclarkb, dougwig: ok, I'll investigate, I'll have a look at how it's done elsewhere  -- there must be a subtle difference in what we do22:48
crinklezaro: right, the method to check that is called ".empty?" with a question mark
dougwigfungi: it just hasn't been a big deal in practice, though.22:49
* tmorin gtg eat somewhere...22:49
clarkbtmorin: to debug you might want to just put a manual curl in there before the die to see the state of things before it bails out22:49
fungidougwig: sure, it's just a topic which has come up before22:49
dougwigfungi: gotcha.22:49
tmorinclarkb: good idea, I think I'll add a lot of echo foo, echo bar, here and there22:49
fungidougwig: so provides a new datapoint for another way people might want to use procedural blocking22:49
tmorinclarkb: :)22:50
tmorinbye guys !22:50
*** tmorin has left #openstack-infra22:50
*** nwkarsten has joined #openstack-infra22:50
zarocrinkle: ohh wat?  thanks!22:51
crinklezaro: yep np :)22:51
fungii will admit i too had trouble parsing the answer as originally worded22:51
clarkbfungi: animaniacs are on netflix now22:52
fungiclarkb: ooh, if i netflixed... then again i should probably just buy the series on disc anyway22:53
fungiit really was that good22:53
clarkbfungi: and I think amazon prime has pinky and the brain22:53
fungialso excellent, yes22:54
*** nwkarsten has quit IRC22:55
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols
pabelangerclarkb: fungi: indeed, watch s01e01 this morning22:56
*** mylu has quit IRC22:58
*** ddieterly[away] is now known as ddieterly22:59
*** ddieterly has quit IRC23:00
*** jamesmcarthur has joined #openstack-infra23:03
*** ams_ has quit IRC23:03
*** rbrndt_ has quit IRC23:04
*** amrith is now known as _amrith_23:04
clarkbfungi: rax-dfw new image is ready23:04
clarkbord failed so retrying that one23:05
clarkbhrm that didn't seem to remove the record from the database23:06
*** apoorvad has quit IRC23:06
clarkbthis might be a bug in the manual upload command not catching the exception and removing the db entry23:07
*** apoorvad has joined #openstack-infra23:07
clarkbI will look into that23:07
*** apoorvad_ has joined #openstack-infra23:08
*** apoorvad has quit IRC23:08
openstackgerritKhai Do proposed openstack-infra/system-config: Make git protocol point to git.o.o
*** alex_xu has quit IRC23:09
clarkbhrm I want to say there was a fix for this already23:10
*** alex_xu has joined #openstack-infra23:11
*** ihrachys has quit IRC23:11
openstackgerritMerged openstack/requirements: Blacklist cryptography 1.3.0
fungiokay, this is a weird openstackci-beaker job output
*** asettle has joined #openstack-infra23:14
fungithe change that reported on went nowhere near openstackci or any modules it relies on23:14
jeblair2016-04-04 20:44:53.579 |   Error: /Stage[main]/Openstackci::Logserver/Package[keyring]/ensure: change from 8.7 to latest failed: Could not get latest version: Timeout while contacting Timeout::Error23:15
jeblairfungi: ^23:15
jeblairi wish i understood what was going on there enough to figure out why it's not using our local mirror23:16
jeblairor rather, how to fix it so that it will23:16
clarkbovh-bhs1 is done23:16
*** akshai has quit IRC23:17
*** chlong has joined #openstack-infra23:18
*** asettle has quit IRC23:19
fungijeblair: ohhh23:19
fungii missed that error message sandwiched in there, thanks23:19
*** mylu has joined #openstack-infra23:19
fungistrange that it was ultimately complaining about non-working default values for parameters23:20
crinklejeblair: i tried to look at that a while ago, found that puppet isn't doing anything that changes /etc/pip.conf ( but I don't know what else might be getting in the way23:21
*** [1]Thelo has joined #openstack-infra23:21
*** salv-orlando has quit IRC23:21
*** ybathia has quit IRC23:22
*** Thelo has quit IRC23:23
*** [1]Thelo is now known as Thelo23:23
*** jamesmcarthur has quit IRC23:24
*** rebase_ has joined #openstack-infra23:25
jeblaircrinkle: i wonder if we're overwriting it with our own puppet manifests?23:26
*** mixos has joined #openstack-infra23:26
crinklejeblair: i don't think so, you can see at the end of this puppet run it's the same
*** rebase has quit IRC23:27
jeblaircrinkle: oh, did we stick a cat on there for this?23:28
crinklejeblair: I did just to test it out
jeblaircrinkle: this is getting interesting now :)23:29
*** rebase_ has quit IRC23:29
crinklejeblair: i think maybe someone should hold a node and tcpdump while running puppet23:30
clarkbnodepools upload image command is calling the onFailed() handler I see that due to the log23:30
jeblaircrinkle: are there any other user accounts involved in this job?  (a per-user pip.conf is the only other thing i can think of right now)23:30
crinklejeblair: because when i tried it locally it seemed to be talking to the mirror23:30
clarkbbut for whatever reason it isn't deleting the snap image properly. more digging needed23:30
crinklejeblair: hmm i don't know23:30
*** mylu has quit IRC23:31
crinklejeblair: i think beaker runs as root23:31
*** mixos has quit IRC23:31
*** mylu has joined #openstack-infra23:31
*** abregman has quit IRC23:31
clarkband it seems to only break in the command line, we have tests for the nodepool internals and they work23:32
*** mixos has joined #openstack-infra23:32
jeblaircrinkle, fungi: i'm guessing we renamed gate-openstackci-beaker-trusty-dsvm to gate-openstackci-beaker-ubuntu-trusty ?23:33
*** mixos has quit IRC23:33
*** ddieterly has joined #openstack-infra23:33
fungijeblair: yes, i believe so23:34
jeblaircrinkle: have an ssh key handy?23:34
fungior rather, since the jobs were named for the node type, we renamed and also moved the job23:34
*** rfolco has quit IRC23:34
clarkbI see the issue, it is a threading problem.23:34
fungiit was previously running on devstack-trusty nodes, now it runs on ubuntu-trusty nodes23:34
*** mixos has joined #openstack-infra23:35
jeblaircrinkle: jenkins@
* crinkle goes adventuring23:36
jeblaircrinkle: can you sudo?23:36
crinklejeblair: nope it asks for a password23:37
*** markvoelker has joined #openstack-infra23:37
jeblaircrinkle: that should be fixed (you may need to login again)23:38
*** xarses has quit IRC23:38
jeblair(i wonder why a job that does things as root removes sudo permissions from jenkins)23:38
*** dalgaaf has quit IRC23:38
crinklejeblair: looks like my ssh key was removed23:39
jeblairoh, it ssh's as root23:39
*** mixos has quit IRC23:40
jeblaircrinkle: that's interesting...23:40
jeblaircrinkle: re-added23:40
crinklejeblair: okay it works now23:41
openstackgerritClark Boylan proposed openstack-infra/nodepool: Synchronize image delete with handle completion
clarkbI think that fixes the issue23:41
jeblairmaybe that means that puppet ran on the host and puppeted the jenkins homedir again23:42
clarkbfungi: is there a change proposed to reenable rax-ord yet? The image isn't uploaded but we could WIP that change until it is23:43
*** Qiming has joined #openstack-infra23:44
anteayaI can offer that23:44
*** _amrith_ is now known as amrith23:44
anteayait would just be a revert of my patch to turn it off23:45
*** keedya has quit IRC23:45
*** mylu has quit IRC23:45
clarkbanteaya: fungi suggested we start with a smaller max server count like 1023:45
clarkband kick the tires on it first23:45
*** mylu has joined #openstack-infra23:45
fungiyep, go for it anteaya--thanks!23:46
openstackgerritAnita Kuno proposed openstack-infra/project-config: Revert "Turn off rax-ord"
fungiand yes, we should start small because the failure rate there was high23:46
anteayaah sorry well I can offer a max-server of 1023:46
anteayamakes sense, sorry wasn't following along23:46
fungino problem, i know you were out and about for a while23:47
*** mylu has quit IRC23:47
openstackgerritMerged openstack/requirements: Add microversion-parse to requirements
clarkbpabelanger: were did we ned up on the xenial mirror?23:48
*** mylu has joined #openstack-infra23:49
clarkbpabelanger: I see that it apperas to exist now, what was the fix?23:49
openstackgerritAnita Kuno proposed openstack-infra/project-config: Turn rax-ord on slowly, max to 10
*** ddieterly is now known as ddieterly[away]23:49
anteayaI abandoned the other one23:50
*** nwkarsten has joined #openstack-infra23:51
*** akshai has joined #openstack-infra23:52
*** bdemers has quit IRC23:54
*** asettle has joined #openstack-infra23:54
*** Qiming has quit IRC23:55
*** mylu has quit IRC23:55
*** bdemers has joined #openstack-infra23:56
*** nwkarsten has quit IRC23:56
*** thorst has joined #openstack-infra23:56
openstackgerritPaul Van Eck proposed openstack-infra/puppet-refstack: Add db backup cron job
clarkbin theory this second ord upload will go faster because the swift portion isnt needed due to shades hashing23:56
clarkbbut it is still going23:56
*** akshai_ has joined #openstack-infra23:58
*** mylu has joined #openstack-infra23:58
*** rhochmuth has quit IRC23:58
*** angdraug has quit IRC23:58
*** ybathia has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!