Friday, 2016-06-24

*** Sukhdev has quit IRC00:00
*** Sukhdev has joined #openstack-infra00:03
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api
*** rajinir has quit IRC00:04
*** salv-orl_ has quit IRC00:06
*** sflanigan has joined #openstack-infra00:08
fungii missed the ansible-test-patching excitement. sorry, got dragged off to do evening errands/chores00:09
fungilooks like it was more fun in here00:10
*** rcernin has quit IRC00:12
*** raunak has joined #openstack-infra00:13
*** rdx_matt has joined #openstack-infra00:16
pabelangerShrews: jeblair: looks like we have a failure already:
*** bhavik has joined #openstack-infra00:19
mmedvedewznoinsk: _alastor_ : I got ci-watch instance running at . Not much data there yet to be useful00:21
openstackgerritIan Wienand proposed openstack/diskimage-builder: Handle locales install on Fedora 24
openstackgerritSpencer Krum proposed openstack-infra/project-config: Move vinz jobs to javascript from python
*** jamesmcarthur has joined #openstack-infra00:24
*** thorst_away has joined #openstack-infra00:24
clarkbnibalizer: re ^ there is a separate vinz-ui or something project that is the javascript one00:25
clarkbthe streams didn't cross did they?00:25
nibalizeri mean both repos are empty00:26
nibalizerso there is a vinz, vinz-client, and puppet-vinz00:26
*** mriedem1 has joined #openstack-infra00:26
clarkbnibalizer: ya vinz-client is the one I am thinking of, should alrady be configured to run js test things00:26
nibalizerall it has is something called gate-vinz-webclient-js-draft00:27
nibalizeri think this was architected at some point... and now that architecture is lost00:28
openstackgerritIan Wienand proposed openstack/diskimage-builder: Handle locales install on Fedora 24
nibalizerso my thought was dump things in 'vinz' until we know where we want to go00:28
*** mriedem has quit IRC00:28
*** jamesmcarthur has quit IRC00:29
openstackgerritMerged openstack-infra/project-config: Move gate-storyboard-js-integration to experimental
*** kzaitsev_mb has quit IRC00:29
jeblairnibalizer: i think the arch was 'vinz is a web service that talks to gerrit, vinz-ui is a javascript client'.  basically, exactly like storyboard.  i think that all predates gerrit being its own web ui, so we may not need 2 components now.  if we only need a js thingy, then i see no reason it shouldn't be called 'vinz'.  :)00:31
jeblaircorrection: 'predates gerrit being its own rest server'00:32
jeblairpabelanger, Shrews, mordred: here is the key line from that log:00:32
jeblair2016-06-24 00:16:03,321 p=26980 u=zuul |  fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time; {'failed': True, 'module_stderr': u'', 'module_stdout': u'', 'msg': 'MODULE FAILURE', 'invocation': {'module_name': u'async_status'}, 'parsed': False}"}00:32
nibalizerjeblair: yea my current production architecture idea is we use apache to take over what happens when you hit and ship some js to the user, which then hits the gerrit api and makes go a code review service00:33
jeblairthat's the content of async_result for debugging00:33
jeblairnibalizer: you said "go"  ;)00:34
jeblairnibalizer: that sounds nice -- the authentication bit is fuzzy for me00:36
jeblairthat might be something to not kick too far down the road00:36
nibalizerwell to start, you auth with the same way you auth now, then head over to urls starting with /vinz00:36
nibalizerbut really i think you can just hit /login and it gets-it-done00:37
nibalizeryep, thats exactly true00:37
nibalizeronce you're on a page ... you're authed essentially00:37
openstackgerritMerged openstack-infra/storyboard: Allow users to be filtered by openid and email
clarkbya basically the way the current UI uses rest and auth I think00:37
openstackgerritSpencer Krum proposed openstack-infra/project-config: Move vinz jobs to javascript from python
clarkbvinz should be able to piggy back on that00:37
*** vhosakot has quit IRC00:38
jeblairi have to forage for food now00:38
*** daemontool has quit IRC00:38
nibalizeryah clark has it right00:41
funginibalizer: at your recommendation, i've added hurgleburgler to vinz-core00:42
fungiphschwartz: do you have any interest in reviewing vinz-related changes? do you happen to know if klmitch does?00:42
nibalizermy first cut of vinz was cssless and sad, it also was as i've been told 'bad javascript'00:42
nibalizerfungi: i have it on good authority that phschwartz has been a father for approx 12 hours, and is not going to be around for a while00:43
funginibalizer: ooh, i write some really terrible javascript! unfortunately i'm busy at the moment writing terrible python and puppet instead00:43
nibalizerfungi: yes, hurgleburgler added the css and fixed the js00:43
fungiphschwartz: ignore my question. in fact, if you actually _see_ my question, i'll be disappointed in you00:44
openstackgerritSpencer Krum proposed openstack-infra/vinz: Add requirements.txt to flask-dev app
*** mriedem1 is now known as mriedem00:45
*** ffrank has quit IRC00:46
*** rdx_matt has quit IRC00:50
*** lvdongbing has joined #openstack-infra00:51
*** zhurong has joined #openstack-infra00:57
*** dimtruck is now known as zz_dimtruck00:58
*** samP has joined #openstack-infra00:58
jamielennoxhey, have we seen ImportError: No module named packaging.specifiers on requirements check jobs?00:59
*** SumitNaiksatam has joined #openstack-infra01:01
*** Sukhdev has quit IRC01:01
fungijamielennox: yeah, someone was in here a little bit ago mentioning a patch they had proposed to the reqs repo to fix that01:02
*** ayoung has joined #openstack-infra01:02
fungii directed them to the new #openstack-requirements channel where the new requirements team hangs out01:02
jamielennoxfungi: thanks, mostly wanted to check if i needed to dig into it - and seems like no01:03
*** Sukhdev has joined #openstack-infra01:04
*** pvaneck has quit IRC01:08
*** gluytium has joined #openstack-infra01:08
*** julim has joined #openstack-infra01:08
*** salv-orlando has joined #openstack-infra01:08
*** Sukhdev has quit IRC01:08
*** shashank_hegde has quit IRC01:11
*** salv-orlando has quit IRC01:15
*** Jeffrey4l has joined #openstack-infra01:16
*** Apoorva has quit IRC01:17
*** tphummel has quit IRC01:17
samPAJaeger: yolanda: Thank you for adding masakari repo ( Could you please add {"Sampath Priyankara", ""} (myself) to masakari-core and masakari-release groups on gerrit?01:19
samPOr is this the correct pace to make this request? or ML?01:20
*** zul has quit IRC01:22
*** zul_ has joined #openstack-infra01:22
*** flwang has quit IRC01:22
*** sdague has quit IRC01:23
*** kzaitsev_mb has joined #openstack-infra01:26
*** Daisy has joined #openstack-infra01:27
*** docaedo has quit IRC01:29
*** xarses has joined #openstack-infra01:31
*** tqtran has quit IRC01:31
*** vhosakot has joined #openstack-infra01:32
*** gluytium has quit IRC01:33
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api
*** nwkarsten has quit IRC01:38
*** nwkarsten has joined #openstack-infra01:38
*** zz_dimtruck is now known as dimtruck01:39
*** amotoki has joined #openstack-infra01:39
*** kzaitsev_mb has quit IRC01:44
*** nwkarsten has quit IRC01:46
*** rdx_matt has joined #openstack-infra01:47
*** PollyZ has joined #openstack-infra01:47
*** docaedo has joined #openstack-infra01:50
*** yanyanhu has joined #openstack-infra01:50
*** gouthamr_ is now known as gouthamr01:54
*** baoli has joined #openstack-infra02:01
*** nwkarsten has joined #openstack-infra02:02
openstackgerritIan Wienand proposed openstack-infra/project-config: Don't use devuser for zuul-worker
*** sflanigan is now known as sflanigan_afk02:06
*** nwkarsten has quit IRC02:07
*** vhosakot has quit IRC02:07
*** mriedem has quit IRC02:09
*** zhurong has quit IRC02:09
*** woodster_ has quit IRC02:09
*** zhurong has joined #openstack-infra02:09
*** salv-orlando has joined #openstack-infra02:15
*** thorst_away has quit IRC02:16
*** baoli has quit IRC02:16
*** thorst_away has joined #openstack-infra02:17
*** PollyZ has quit IRC02:18
*** rdx_matt has quit IRC02:19
*** sflanigan_afk is now known as sflanigan02:24
*** yamahata has quit IRC02:24
*** thorst_away has quit IRC02:25
*** Na3iL has joined #openstack-infra02:26
*** salv-orlando has quit IRC02:27
*** tqtran has joined #openstack-infra02:29
*** raunak has quit IRC02:30
*** tqtran has quit IRC02:34
*** Daisy has quit IRC02:39
*** Daisy has joined #openstack-infra02:40
*** kzaitsev_mb has joined #openstack-infra02:41
*** kzaitsev_mb has quit IRC02:45
*** vhosakot has joined #openstack-infra02:47
*** Daisy has quit IRC02:48
*** esberglu has quit IRC02:49
*** gyee has quit IRC02:51
*** gouthamr has quit IRC02:55
*** yuanying has quit IRC02:55
*** bswartz has joined #openstack-infra02:56
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api
*** vhosakot has quit IRC02:58
*** ramishra has joined #openstack-infra02:59
*** raunak has joined #openstack-infra03:01
*** ramishra has quit IRC03:01
greghaynesnibalizer: jeblair for vinz auth - as long as you can serve up is at the same domain as gerrit the auth is doable. One way is to Ajax to / and parse out the auth token, I hope gerrit has a better way, though03:03
greghaynesTrying to look in to that is what led me to the xss deal recently ;)03:03
greghaynesAh, seems like nibalizer knows what's up03:04
*** ramishra has joined #openstack-infra03:07
*** tphummel has joined #openstack-infra03:09
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** PollyZ has joined #openstack-infra03:13
*** tphummel has quit IRC03:16
*** ramishra has quit IRC03:16
*** rdx_matt has joined #openstack-infra03:16
*** PollyZ has quit IRC03:21
*** ramishra has joined #openstack-infra03:22
*** rm_work has quit IRC03:22
*** thorst_away has joined #openstack-infra03:23
*** cody-somerville has quit IRC03:23
*** zhurong has quit IRC03:23
*** rm_work has joined #openstack-infra03:25
*** amotoki has quit IRC03:26
*** cody-somerville has joined #openstack-infra03:26
*** cody-somerville has joined #openstack-infra03:26
*** lvdongbing has quit IRC03:27
*** lvdongbing has joined #openstack-infra03:28
*** salv-orlando has joined #openstack-infra03:29
*** thorst_away has quit IRC03:30
*** Na3iL has quit IRC03:31
*** zhurong has joined #openstack-infra03:31
*** zhurong has quit IRC03:33
*** rcernin has joined #openstack-infra03:34
*** ramishra has quit IRC03:37
*** salv-orlando has quit IRC03:37
*** zhurong has joined #openstack-infra03:39
notmorgannibalizer, crinkle, mtreinish: any of you have x1c w/ a WWAN card installed?03:40
notmorgannibalizer, crinkle, mtreinish: trying to determine if linux is causing an issue or the WWAN card/sim slot on mine is bad - I can't get it to register the SIM is inserted.03:40
notmorganShrews: ^ cc03:40
notmorganjeblair: ^03:41
notmorganthough i think Shrews and jeblair aren't running this type of setup.03:41
*** kzaitsev_mb has joined #openstack-infra03:42
*** gongysh has joined #openstack-infra03:42
*** coolsvap is now known as coolsvap_03:43
*** ramishra has joined #openstack-infra03:43
*** amotoki has joined #openstack-infra03:44
*** kzaitsev_mb has quit IRC03:47
*** yamamot__ has joined #openstack-infra03:47
*** rdx_matt has quit IRC03:49
*** ramishra has quit IRC03:49
*** yuanying has joined #openstack-infra03:50
*** ramishra has joined #openstack-infra03:54
openstackgerritYanyan Hu proposed openstack-infra/project-config: Add senlin-tempest-functional test job
*** shashank_hegde has joined #openstack-infra03:55
openstackgerritYanyan Hu proposed openstack-infra/project-config: Add senlin-tempest-functional test job
openstackgerritYanyan Hu proposed openstack-infra/project-config: Add senlin-tempest-functional test job
openstackgerritIan Wienand proposed openstack-infra/project-config: Don't use devuser for zuul-worker
*** amotoki has quit IRC04:00
*** sdake_ has joined #openstack-infra04:00
*** sdake has quit IRC04:02
*** amotoki has joined #openstack-infra04:03
*** Sukhdev has joined #openstack-infra04:07
*** markvoelker has quit IRC04:11
*** amotoki has quit IRC04:18
*** david-lyle has quit IRC04:20
*** david-lyle has joined #openstack-infra04:20
*** sdake_ has quit IRC04:22
*** Qiming has quit IRC04:22
*** Qiming has joined #openstack-infra04:23
*** thorst_away has joined #openstack-infra04:28
*** baoli has joined #openstack-infra04:28
*** sflanigan has quit IRC04:31
*** tqtran has joined #openstack-infra04:31
*** sflanigan has joined #openstack-infra04:33
*** sflanigan has joined #openstack-infra04:33
*** baoli has quit IRC04:33
*** ramishra has quit IRC04:34
*** thorst_away has quit IRC04:35
*** tqtran has quit IRC04:35
*** Daisy has joined #openstack-infra04:36
*** psachin has joined #openstack-infra04:36
*** links has joined #openstack-infra04:37
*** sdake has joined #openstack-infra04:38
*** amotoki has joined #openstack-infra04:38
*** salv-orlando has joined #openstack-infra04:39
*** Daisy has quit IRC04:41
*** ramishra has joined #openstack-infra04:41
*** _Fdaisuke_ has quit IRC04:43
*** kzaitsev_mb has joined #openstack-infra04:43
*** rdx_matt has joined #openstack-infra04:46
*** kzaitsev_mb has quit IRC04:47
*** rdx_matt has quit IRC04:50
*** links has quit IRC04:50
*** links has joined #openstack-infra04:50
*** ramishra has quit IRC04:50
*** coolsvap_ has quit IRC04:51
*** salv-orlando has quit IRC04:51
*** yamamot__ has quit IRC04:53
*** ramishra has joined #openstack-infra04:56
openstackgerritIan Wienand proposed openstack-infra/project-config: Don't use devuser for zuul-worker
*** Daisy has joined #openstack-infra05:06
*** julim has quit IRC05:10
*** markvoelker has joined #openstack-infra05:11
*** shahid__ has joined #openstack-infra05:13
*** _nadya_ has joined #openstack-infra05:13
*** salv-orlando has joined #openstack-infra05:14
*** markvoelker has quit IRC05:16
*** ramishra has quit IRC05:17
*** bhavik has quit IRC05:18
*** Daisy has quit IRC05:18
*** Daisy has joined #openstack-infra05:18
*** ramishra has joined #openstack-infra05:18
*** kzaitsev_mb has joined #openstack-infra05:19
*** zeih has joined #openstack-infra05:20
*** coolsvap has joined #openstack-infra05:21
*** Daisy has quit IRC05:23
*** links has quit IRC05:25
coolsvapany background work happening with gerrit, seems very slow05:26
zarocoolsvap: seems ok to me.  which part is slow for you?05:27
coolsvapzaro, in general, i am the review open time, time to change between files in review05:28
coolsvapits taking more than usual05:28
coolsvapjust added one comment05:29
*** rdx_matt has joined #openstack-infra05:30
*** maishsk_ has joined #openstack-infra05:31
zarocoolsvap: which project?  i just added a comment to and it wasn't slow for me.05:31
coolsvapthe comment in review is not uploaded instead i got server unavailable05:31
coolsvapzaro, ^^05:32
*** sdake has quit IRC05:32
*** maishsk has quit IRC05:33
*** maishsk_ is now known as maishsk05:33
*** thorst_away has joined #openstack-infra05:33
*** sdake has joined #openstack-infra05:34
zarocoolsvap: just added a comment to that change, didn't seem slow either.05:34
zaromaybe your internet connection?05:34
coolsvapzaro, i though the same, restarted everthing, but everything else is working fine mail, vpn n all05:35
coolsvapwill check after some time05:35
*** rcernin has quit IRC05:35
*** rdx_matt has quit IRC05:36
*** links has joined #openstack-infra05:39
*** Sukhdev has quit IRC05:40
*** thorst_away has quit IRC05:40
*** prabu_ has joined #openstack-infra05:42
*** ramishra has quit IRC05:42
*** tlian has quit IRC05:44
*** yamamot__ has joined #openstack-infra05:46
*** ramishra has joined #openstack-infra05:47
*** dimtruck is now known as zz_dimtruck05:48
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Start fleshing out ansible-launcher tests
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Clean up results returned from ansible-launcher
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add jjb and zmq to requirements
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Prepare files for testing zuulv25
*** hashar has joined #openstack-infra06:00
*** YorikSar has quit IRC06:01
*** Daisy has joined #openstack-infra06:01
openstackgerritMerged openstack-infra/irc-meetings: Switch to odd, update chairs
openstackgerritIan Wienand proposed openstack-infra/project-config: Don't use devuser for zuul-worker
*** YorikSar has joined #openstack-infra06:03
*** signed8bit_Zzz is now known as signed8bit06:04
*** signed8bit has quit IRC06:05
*** links has quit IRC06:05
*** Daisy has quit IRC06:06
*** raunak has quit IRC06:08
*** Na3iL has joined #openstack-infra06:09
*** YorikSar has quit IRC06:10
*** Nikolay_St has joined #openstack-infra06:11
*** rcernin has joined #openstack-infra06:12
*** YorikSar has joined #openstack-infra06:12
*** ccamacho has joined #openstack-infra06:12
*** _nadya_ has quit IRC06:12
*** markvoelker has joined #openstack-infra06:13
*** _nadya_ has joined #openstack-infra06:13
*** Na3iL has quit IRC06:13
*** ramishra has quit IRC06:16
*** ramishra has joined #openstack-infra06:17
*** ccamacho has quit IRC06:17
*** ccamacho|Zzz has quit IRC06:17
*** _nadya_ has quit IRC06:17
*** markvoelker has quit IRC06:17
*** pgadiya has joined #openstack-infra06:17
*** kzaitsev_mb has quit IRC06:17
*** linuxmaniac_ has joined #openstack-infra06:17
*** javeriak has joined #openstack-infra06:18
*** linuxmaniac has quit IRC06:18
*** links has joined #openstack-infra06:18
*** aeng has quit IRC06:20
*** ramishra has quit IRC06:22
*** javeriak has quit IRC06:23
*** javeriak has joined #openstack-infra06:23
*** shashank_hegde has quit IRC06:25
*** ramishra has joined #openstack-infra06:27
*** salv-orlando has quit IRC06:28
*** ramishra has quit IRC06:28
*** shashank_hegde has joined #openstack-infra06:29
*** Daisy has joined #openstack-infra06:30
*** ramishra has joined #openstack-infra06:30
*** tqtran has joined #openstack-infra06:32
*** rdx_matt has joined #openstack-infra06:32
*** rdx_matt has quit IRC06:32
*** hashar has quit IRC06:34
*** andreas_s has joined #openstack-infra06:35
*** tqtran has quit IRC06:37
*** links has quit IRC06:37
*** _nadya_ has joined #openstack-infra06:38
*** thorst_away has joined #openstack-infra06:38
*** thorst_away has quit IRC06:45
*** links has joined #openstack-infra06:49
*** sarob has joined #openstack-infra06:50
*** sarob has quit IRC06:55
*** javeriak has quit IRC06:55
*** javeriak has joined #openstack-infra06:55
*** tesseract- has joined #openstack-infra06:57
*** rdx_matt has joined #openstack-infra07:00
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent"
*** pkoniszewski has quit IRC07:03
*** yolanda has joined #openstack-infra07:04
*** javeriak_ has joined #openstack-infra07:06
*** rdx_matt has quit IRC07:07
*** e0ne has joined #openstack-infra07:07
*** maishsk_ has joined #openstack-infra07:08
*** javeriak has quit IRC07:09
*** maishsk has quit IRC07:10
*** maishsk_ is now known as maishsk07:10
*** matrohon has joined #openstack-infra07:11
*** sdake_ has joined #openstack-infra07:12
*** pkoniszewski has joined #openstack-infra07:12
*** _nadya_ has quit IRC07:13
*** ifarkas has joined #openstack-infra07:13
*** markvoelker has joined #openstack-infra07:13
*** sdake has quit IRC07:15
*** markvoelker has quit IRC07:19
*** flepied1 has quit IRC07:25
*** arxcruz has joined #openstack-infra07:26
*** ihrachys has joined #openstack-infra07:28
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job
*** amoralej|off is now known as amoralej07:29
*** esikachev has joined #openstack-infra07:30
*** shardy has joined #openstack-infra07:31
*** jlanoux has joined #openstack-infra07:31
*** Na3iL has joined #openstack-infra07:31
*** yaume has joined #openstack-infra07:34
*** yamahata has joined #openstack-infra07:35
*** shardy_ has joined #openstack-infra07:35
*** salv-orlando has joined #openstack-infra07:36
*** ramishra has quit IRC07:37
*** ramishra has joined #openstack-infra07:38
*** jpich has joined #openstack-infra07:40
*** jlanoux_ has joined #openstack-infra07:41
*** Daisy has quit IRC07:41
yolandagood morning07:41
*** Daisy has joined #openstack-infra07:41
*** thorst_away has joined #openstack-infra07:43
*** jlanoux has quit IRC07:44
*** shardy has quit IRC07:45
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Make gate-tempest-dsvm-neutron-full-ssh voting
*** javeriak has joined #openstack-infra07:46
*** javeria__ has joined #openstack-infra07:47
*** javeriak_ has quit IRC07:47
samPyolanda: good morning, Thank you for adding masakari repo (
*** Na3iL has quit IRC07:48
yolandahi samP , glad to help07:49
*** ihrachys has quit IRC07:50
*** javeriak has quit IRC07:50
samPyolanda: I would like to ask one more help, which is add me to gerrit group masakari-core and masakari-review07:50
*** javeriak has joined #openstack-infra07:50
*** thorst_away has quit IRC07:50
*** ramishra has quit IRC07:50
*** ramishra has joined #openstack-infra07:51
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job
samPyolanda: me (Sampath Priyankara,, what info you need?07:51
yolandai added you07:52
openstackgerritMerged openstack-infra/project-config: start a readme describing the release tools
samPyolanda: excellent, thank you very much.07:52
*** javeria__ has quit IRC07:53
*** markmcd has quit IRC07:55
chemhi people, on the Xenial image is there the dbus.service Unit or should I install it ?07:56
*** YorikSar has quit IRC07:56
*** YorikSar has joined #openstack-infra07:57
*** flepied has joined #openstack-infra07:57
openstackgerritMerged openstack-infra/project-config: Upgrade experimental searchlight ES 2.3.2
*** rcernin has quit IRC07:59
openstackgerritMerged openstack-infra/project-config: Add senlin-tempest-functional test job
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
chemor a simpler question, what is the image used for xenial vm, is it the official one or another ?08:00
*** Daisy_ has joined #openstack-infra08:00
*** sdake_ has quit IRC08:02
*** lucas-afk is now known as lucasagomes08:02
*** dizquierdo has joined #openstack-infra08:03
*** rdx_matt has joined #openstack-infra08:03
*** rdx_matt has quit IRC08:04
*** Daisy has quit IRC08:04
*** rdx_matt has joined #openstack-infra08:04
openstackgerritMerged openstack-infra/project-config: Run identity v3 only job in heat
*** eezhova has joined #openstack-infra08:05
*** _nadya_ has joined #openstack-infra08:07
*** shashank_hegde has quit IRC08:09
*** rdx_matt has quit IRC08:09
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Report job status 'QUEUED' for queued jobs
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names
*** abregman has joined #openstack-infra08:10
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Allow using numeric change number in addition to ID in Depends-On tag
*** sflanigan has quit IRC08:13
*** _degorenko|afk is now known as degorenko08:13
*** arif-ali has joined #openstack-infra08:13
*** dtantsur|afk is now known as dtantsur08:14
*** dmk0202 has joined #openstack-infra08:17
*** daemontool has joined #openstack-infra08:17
*** rossella_s has joined #openstack-infra08:19
*** ramishra has quit IRC08:20
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Add new project Zunclient to openstack repo
*** acoles_ is now known as acoles08:21
*** HeOS has quit IRC08:23
*** markmcd has joined #openstack-infra08:24
*** javeriak has quit IRC08:27
*** jamesmcarthur has joined #openstack-infra08:29
*** shardy_ is now known as shardy_afk08:30
*** binbincong has joined #openstack-infra08:30
*** kzaitsev_mb has joined #openstack-infra08:30
*** ramishra has joined #openstack-infra08:30
wznoinskmmedvede, superslow but content looks ok08:30
*** markmcd has quit IRC08:31
*** electrofelix has joined #openstack-infra08:32
*** jamesmcarthur has quit IRC08:33
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** rdx_matt has joined #openstack-infra08:34
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** matrohon has quit IRC08:38
*** rdx_matt has quit IRC08:39
*** denisra has quit IRC08:41
*** tosky has joined #openstack-infra08:42
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Make gate-tempest-dsvm-neutron-full-ssh voting
*** BobBall_AWOL is now known as BobBall08:46
*** d0ugal has quit IRC08:46
*** derekh has joined #openstack-infra08:46
*** d0ugal has joined #openstack-infra08:46
*** d0ugal has quit IRC08:46
*** d0ugal has joined #openstack-infra08:46
*** BobBall is now known as Guest7426708:46
*** gongysh has quit IRC08:47
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** thorst_away has joined #openstack-infra08:48
*** sarob has joined #openstack-infra08:51
*** kzaitsev_mb has quit IRC08:51
*** dizquierdo has quit IRC08:54
*** yanyanhu has quit IRC08:54
*** thorst_away has quit IRC08:55
*** trash_ is now known as trash08:55
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** sarob has quit IRC08:56
*** Na3iL has joined #openstack-infra08:57
openstackgerritMerged openstack-infra/system-config: Add compute088.useast metadata for better tracking
*** jordanP has joined #openstack-infra08:59
*** jyuso1 has quit IRC09:00
*** markusry has joined #openstack-infra09:00
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** ramishra has quit IRC09:01
*** javeriak has joined #openstack-infra09:02
*** markusry has quit IRC09:02
*** markusry has joined #openstack-infra09:02
*** ramishra has joined #openstack-infra09:03
*** Daisy_ has quit IRC09:04
*** Daisy has joined #openstack-infra09:04
*** javeriak has quit IRC09:06
*** javeriak has joined #openstack-infra09:07
*** Daisy has quit IRC09:09
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** markusry has quit IRC09:09
*** markusry has joined #openstack-infra09:09
*** javeriak has quit IRC09:13
*** javeriak has joined #openstack-infra09:13
*** HeOS has joined #openstack-infra09:16
*** markvoelker has joined #openstack-infra09:16
*** sheeprine has quit IRC09:19
*** sheeprine has joined #openstack-infra09:19
*** javeriak has quit IRC09:19
*** javeriak has joined #openstack-infra09:20
*** markvoelker has quit IRC09:20
*** hashar has joined #openstack-infra09:21
*** aarefiev has quit IRC09:21
*** amotoki has quit IRC09:22
*** tosky has quit IRC09:22
*** amotoki has joined #openstack-infra09:23
*** tosky has joined #openstack-infra09:24
*** markusry has quit IRC09:25
*** shardy_afk is now known as shardy09:27
*** samP has quit IRC09:29
*** markusry has joined #openstack-infra09:30
*** amotoki_ has joined #openstack-infra09:34
*** markusry has quit IRC09:35
*** rdx_matt has joined #openstack-infra09:35
*** markusry has joined #openstack-infra09:35
*** rdx_matt has quit IRC09:35
*** amotoki has quit IRC09:35
*** rdx_matt has joined #openstack-infra09:38
*** markusry has quit IRC09:38
*** sambetts|afk is now known as sambetts09:39
*** markusry has joined #openstack-infra09:39
*** dizquierdo has joined #openstack-infra09:41
*** fawadkhaliq has joined #openstack-infra09:42
*** fawadkhaliq has quit IRC09:42
*** markusry has quit IRC09:43
*** ramishra has quit IRC09:45
derekhso nodepool is now spinning up instances on RH2 \o/ , but they're not being spun up with any network attached, the nodepool config specifies the "private" network, any ideas about whats going wrong
*** kzaitsev_mb has joined #openstack-infra09:49
*** ramishra has joined #openstack-infra09:52
*** thorst_away has joined #openstack-infra09:52
*** markusry has joined #openstack-infra09:54
*** markvoelker has joined #openstack-infra09:54
*** thorst_away has quit IRC10:00
*** zhurong has quit IRC10:06
*** yamahata has quit IRC10:07
*** eezhova has quit IRC10:08
*** sdague has joined #openstack-infra10:09
*** pcaruana has joined #openstack-infra10:09
*** javeriak has quit IRC10:11
*** ramishra has quit IRC10:13
*** ramishra has joined #openstack-infra10:13
*** ccamacho has joined #openstack-infra10:13
*** salv-orlando has quit IRC10:14
*** amotoki_ has quit IRC10:19
*** ccamacho has quit IRC10:20
*** javeriak has joined #openstack-infra10:24
*** claudiub has joined #openstack-infra10:28
*** ramishra has quit IRC10:30
*** ramishra has joined #openstack-infra10:30
*** dtantsur is now known as dtantsur|brb10:32
*** kzaitsev_mb has quit IRC10:34
*** lvdongbing has quit IRC10:35
*** javeriak has quit IRC10:35
*** Guest74267 is now known as BobBall10:36
*** ccamacho has joined #openstack-infra10:36
*** salv-orlando has joined #openstack-infra10:36
openstackgerritBin Hu proposed openstack-infra/project-config: New project openstack/gluon
*** javeriak has joined #openstack-infra10:43
*** gouthamr has joined #openstack-infra10:43
*** eezhova has joined #openstack-infra10:43
*** javeriak has quit IRC10:45
*** javeriak has joined #openstack-infra10:45
*** craige has quit IRC10:46
*** bauzas is now known as bauwser10:46
*** kzaitsev_mb has joined #openstack-infra10:47
openstackgerritMerged openstack-infra/storyboard-webclient: Fix alignment of long-running simple events
*** aarefiev has joined #openstack-infra10:51
*** sarob has joined #openstack-infra10:52
*** craige has joined #openstack-infra10:54
*** acabot has quit IRC10:54
*** sarob has quit IRC10:56
*** thorst_away has joined #openstack-infra10:58
*** salv-orl_ has joined #openstack-infra10:59
*** ramishra has quit IRC11:01
*** salv-orlando has quit IRC11:02
*** ramishra has joined #openstack-infra11:02
*** abregman is now known as abregman|afk11:02
*** kzaitsev_mb has quit IRC11:03
*** thorst_away has quit IRC11:05
*** acabot has joined #openstack-infra11:06
*** thorst_away has joined #openstack-infra11:07
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Updated Worklist API
*** weshay has joined #openstack-infra11:12
*** ramishra has quit IRC11:14
*** thorst_away is now known as thorst_11:15
*** Srinu has quit IRC11:16
*** kzaitsev_mb has joined #openstack-infra11:18
*** cody-somerville has quit IRC11:20
*** shardy has quit IRC11:23
*** azvyagintsev has quit IRC11:23
openstackgerritMerged openstack-infra/infra-ansible: Make services optional
*** shardy has joined #openstack-infra11:24
openstackgerritAlexander Chadin proposed openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate
*** ramishra has joined #openstack-infra11:32
*** rhallisey has joined #openstack-infra11:32
yamamot__can anyone consider merging this?   this is blocking midonet dev badly.11:32
*** dprince has joined #openstack-infra11:33
*** tqtran has joined #openstack-infra11:34
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** ramishra has quit IRC11:36
*** jordanP has quit IRC11:37
*** tqtran has quit IRC11:39
*** javeriak has quit IRC11:40
*** javeriak has joined #openstack-infra11:40
*** alaski is now known as lascii11:41
AJaegeryamamot__: this needs a neutron infra liason +1 first - ask dougwig, armax or mestery for review, please11:42
yamamot__AJaeger: ok thank you11:43
*** andreas_s has quit IRC11:44
*** zhurong has joined #openstack-infra11:44
*** ldnunes has joined #openstack-infra11:45
*** sarob has joined #openstack-infra11:50
*** ddieterly has joined #openstack-infra11:53
*** sarob has quit IRC11:55
*** prabu_ has quit IRC11:57
*** ddieterly has quit IRC11:59
*** psilvad_ has joined #openstack-infra12:00
*** ccamacho is now known as ccamacho|lunch12:00
*** jtomasek_ has joined #openstack-infra12:01
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Add support for building images capable of UEFI
Shrewsnotmorgan: i do not have such a setup12:02
openstackgerrityolanda.robla proposed openstack-infra/puppet-httpd: Update context on vhost for selinux
*** Na3iL has quit IRC12:04
*** rcernin has joined #openstack-infra12:04
openstackgerritJoshua Hesketh proposed openstack-infra/release-tools: Automate retiring old branches
*** rfolco has joined #openstack-infra12:08
*** yamamot__ has quit IRC12:08
*** pradk has joined #openstack-infra12:09
*** ddieterly has joined #openstack-infra12:11
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** liusheng has quit IRC12:13
openstackgerritBin Hu proposed openstack-infra/project-config: New project openstack/gluon
*** jordanP has joined #openstack-infra12:15
asselinrcarrillocruz, let me know when you're around. I'm having trouble find the dependent ansible roles referenced in your patch:
*** dtardivel has joined #openstack-infra12:15
olaphnotmorgan: I have a x250, and have never gotten the wwan to work12:15
*** liusheng has joined #openstack-infra12:16
openstackgerritDaniel Mellado proposed openstack-infra/project-config: Fix simple typo
rcarrillocruzasselin: you need the depends-on and
*** javeriak has quit IRC12:17
asselinrcarrillocruz, I added the depends-on patch. Which ones from that topic query? all of the open ones?12:18
*** matt-borland has joined #openstack-infra12:18
*** tpsilva has joined #openstack-infra12:19
esikachevhey, folks! how you prepare nodes for running system-configs puppets? whitch source.lists?12:20
*** aysyd has joined #openstack-infra12:21
rcarrillocruzasselin: ^12:21
openstackgerrityolanda.robla proposed openstack-infra/puppet-httpd: Update context on vhost for selinux
asselinrcarrillocruz, ok i it's not a patch chain? I'll cherry-pick them and try again.12:23
rcarrillocruzbear in mind that approach is a replacement for the current launch-node, i.e. create just one server with values passed in on the command line12:23
rcarrillocruzasselin: right, not a patch chain, just cherry pick and you should be good12:23
*** chem has quit IRC12:24
asselinok will give it a shot12:24
rcarrillocruzyou can just create a resources.yml and feed to the role with -e "@resources.yml"12:25
rcarrillocruzyou can put whatever resources you want in there12:25
*** ddieterly has quit IRC12:26
openstackgerritAlexander Chadin proposed openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate
*** binbincong has quit IRC12:29
*** kdas_ has joined #openstack-infra12:30
*** jcoufal has joined #openstack-infra12:30
*** kdas_ is now known as kushal12:32
*** binbincong has joined #openstack-infra12:32
*** lucasagomes is now known as lucas-hungry12:32
*** kushal has quit IRC12:32
*** kushal has joined #openstack-infra12:32
*** Na3iL has joined #openstack-infra12:35
*** psachin has quit IRC12:36
*** gordc has joined #openstack-infra12:37
*** psachin has joined #openstack-infra12:38
*** jtomasek_ has quit IRC12:40
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Build wholedisk images capable of secure boot
*** zul_ is now known as zul12:44
*** erlon has joined #openstack-infra12:45
*** jtomasek has quit IRC12:46
*** alex_xu has quit IRC12:46
*** ddieterly has joined #openstack-infra12:46
asselinrcarrillocruz, thanks for explaining -e usage. I got it talking to my cloud. hitting the ssl issue again, but I know how to work around that. thank you!12:47
*** jtomasek has joined #openstack-infra12:47
*** rlandy has joined #openstack-infra12:48
*** alex_xu has joined #openstack-infra12:48
*** chem has joined #openstack-infra12:49
openstackgerritBin Hu proposed openstack-infra/project-config: New project openstack/gluon
*** coolsvap has quit IRC12:52
*** edmondsw has joined #openstack-infra12:55
*** trown|outtypewww is now known as trown12:56
*** yamamoto has joined #openstack-infra12:57
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-ansible: Set hostnames before the common play
*** baoli has joined #openstack-infra12:58
*** links has quit IRC13:00
*** andymaier has joined #openstack-infra13:01
*** dizquierdo has quit IRC13:01
*** ddieterly has quit IRC13:02
*** amitgandhinz has joined #openstack-infra13:02
*** baoli_ has joined #openstack-infra13:02
*** infra-red has joined #openstack-infra13:04
*** infra-red has quit IRC13:05
*** baoli has quit IRC13:05
*** infra-red has joined #openstack-infra13:06
*** binbincong has quit IRC13:09
*** akshai has joined #openstack-infra13:14
*** pgadiya has quit IRC13:14
*** denisra has joined #openstack-infra13:15
*** dtantsur|brb is now known as dtantsur13:15
openstackgerritSean Dague proposed openstack-infra/devstack-gate: DNM: experimenting with local.conf
AJaegerjhesketh: thanks for retirng kilo13:16
*** mdrabe has joined #openstack-infra13:16
jheskethno worries, sorry it took so long13:16
*** amotoki has joined #openstack-infra13:16
mtreinishnotmorgan: I didn't pay for the modem, but it still has the sim slot (and antennas I think) If I wanted to use it I'm pretty sure I'd have to put a cellular modem in first13:18
mtreinishnotmorgan: but I've not actually tried it13:18
*** Na3iL has quit IRC13:18
*** infra-red has quit IRC13:19
*** esberglu has joined #openstack-infra13:19
AJaegerjhesketh: did you also sign up for removing kilo from zuul/layout?13:20
*** sdake has joined #openstack-infra13:21
jheskethAJaeger: nope, but I can13:21
AJaegergreat! Please rember to remove now unused job definition as well - those that *only* run on kilo...13:22
*** esberglu has quit IRC13:24
*** nwkarsten has joined #openstack-infra13:26
*** akshai_ has joined #openstack-infra13:26
*** piet has joined #openstack-infra13:26
*** amoralej is now known as amoralej|lunch13:29
openstackgerritMerged openstack-infra/python-storyboardclient: Updated Worklist API
*** akshai has quit IRC13:30
*** xyang1 has joined #openstack-infra13:30
*** mriedem has joined #openstack-infra13:31
*** dmk0202 has quit IRC13:31
*** rbrndt_ has joined #openstack-infra13:32
*** ddieterly has joined #openstack-infra13:33
*** bin_ has joined #openstack-infra13:33
bin_Hi, I submitted patch for creating a new project "openstack/gluon":
bin_There is a build error: Gerritbot can only handle 120 channels but found 121.
bin_Looks like it cannot handle my IRC channel "#openstack-gluon" that I added in "accessbot/channels.yaml" and "gerritbot/channels.yaml" even if the access is ok "INFO:checkaccess:openstackinfra access ok on #openstack-gluon"13:35
AJaegerbin_: in this case: Do not touch gerritbot file. Accessbot change is fine.13:35
bin_OK. I will remove the added lines in gerritbot file13:36
bin_thank you13:36
*** woodster_ has joined #openstack-infra13:37
openstackgerritMerged openstack-infra/project-config: Fix simple typo
bin_BTW, can I also add "    meetbots:      - open_stack" in accessbot file?13:37
*** dprince has quit IRC13:38
bin_does it mean the meetbots will be added to my channel "#openstack-gluon"?13:38
*** Ravikiran_K has joined #openstack-infra13:38
*** esberglu has joined #openstack-infra13:39
*** dmellado_ is now known as dmellado13:40
*** ddieterly is now known as ddieterly[away]13:40
openstackgerritBin Hu proposed openstack-infra/project-config: New project openstack/gluon
*** Daisy has joined #openstack-infra13:45
*** piet has quit IRC13:46
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Remove kilo jobs
jheskethAJaeger: ^ there's a good chance I've missed something, but it's late and I'm going to bed sorry13:46
*** lucas-hungry is now known as lucasagomes13:46
*** ddieterly[away] is now known as ddieterly13:47
*** Daisy has quit IRC13:49
*** amitgandhinz has quit IRC13:50
*** inc0 has joined #openstack-infra13:51
*** amitgandhinz has joined #openstack-infra13:51
bin_Do I need to specifically add reviewers? or as long as Gerritbot announced it this channel "Bin Hu proposed openstack-infra/project-config: New project openstack/gluon", it is set. and I just need to wait?13:51
*** HeOS_ has quit IRC13:51
SamYaplebin_: you just need to wait, give it a few days/week and if you dont have any reviews yet ask the channel here13:52
*** dprince has joined #openstack-infra13:52
*** hongbin has joined #openstack-infra13:53
bin_Sure, I will just wait. Thank you13:54
*** shahid__ has quit IRC13:56
*** jordanP has quit IRC13:56
*** matrohon has joined #openstack-infra13:59
AJaegerjhesketh: thanks, will review when I'm fully awake (post-lunch trauma right now)14:01
*** afred312 has quit IRC14:05
*** eharney has joined #openstack-infra14:07
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add new db api method to get recent successes by metadata
*** julim has joined #openstack-infra14:16
*** yaume has quit IRC14:19
*** ikhudoshyn has joined #openstack-infra14:21
*** amoralej|lunch is now known as amoralej14:22
*** maishsk has quit IRC14:24
*** tonytan4ever has joined #openstack-infra14:24
*** yamahata has joined #openstack-infra14:27
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add pytest_results.html to test-results publisher
*** bhavik has joined #openstack-infra14:29
*** YorikSar has quit IRC14:30
*** inc0 has quit IRC14:30
*** inc0 has joined #openstack-infra14:31
openstackgerritMark Hamzy proposed openstack/diskimage-builder: dmidecode does not exist for ppc64/ppc64el
*** psachin has quit IRC14:33
Shrewsjeblair: pabelanger: mordred and I have another patch to try:
*** lvdongbing has joined #openstack-infra14:33
Shrewsjeblair: pabelanger: tl;dr, removes another "reading from the same file we are currently writing to" scenario14:34
pabelangerShrews: Ah, nice fined14:35
*** markusry has quit IRC14:36
*** inc0 has quit IRC14:36
Shrewspabelanger: mordred's patch, i just did the grunt work14:36
*** inc0 has joined #openstack-infra14:37
*** mdrabe has quit IRC14:37
*** rossella_s has quit IRC14:37
*** rossella_s has joined #openstack-infra14:38
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Add support for WorklistItems
*** rook has quit IRC14:39
*** YorikSar has joined #openstack-infra14:39
*** dingyichen has quit IRC14:39
*** rook has joined #openstack-infra14:40
*** inc0 has quit IRC14:40
*** gongysh has joined #openstack-infra14:40
*** inc0 has joined #openstack-infra14:41
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Removed whitespace for PEP8 test
openstackgerritBilal proposed openstack-infra/python-storyboardclient: Add support for WorklistItems
*** mdrabe has joined #openstack-infra14:44
*** julim has quit IRC14:44
*** dprince has quit IRC14:44
jeblairShrews, mordred, pabelanger: good morning!  i've been catching up on #ansible-devel14:46
*** eezhova has quit IRC14:46
pabelangerI should do the same, let me get some coffee14:47
*** Hal has joined #openstack-infra14:49
*** Guest25417 has quit IRC14:49
*** dprince has joined #openstack-infra14:49
*** Hal is now known as Guest3846614:50
*** bhavik has quit IRC14:50
*** Guest38466 has quit IRC14:50
*** rook has quit IRC14:50
*** zz_dimtruck is now known as dimtruck14:51
*** eezhova has joined #openstack-infra14:51
*** ajmiller has joined #openstack-infra14:53
*** rook has joined #openstack-infra14:53
*** xenogear has quit IRC14:55
*** bhavik has joined #openstack-infra14:55
*** mrunge has quit IRC14:55
openstackgerritAntoine Cabot proposed openstack-infra/project-config: Add specs entry for Watcher specs
jeblairShrews: that patch lgtm.  i also really want to do something about communicate, but also can't get past the fact that at the end of the day we're going to have 'result' in memory in order to modify it... so... :)14:56
*** infra-red has joined #openstack-infra14:56
*** ddieterly is now known as ddieterly[away]14:58
openstackgerritAntoine Cabot proposed openstack-infra/project-config: Add specs entry for Watcher specs
jeblairShrews: i guess that's more of an ansible-module protocol issue -- like, if it were a multi-line protocol or something (like "result={json}\nstderr=string\nfuture_thing=foo") we wouldn't need to store anything in memory.14:58
tristanCfungi: about your openid provider question, red hat uses saml. google auth is probably a good candidate to act as an openid proxy for saml identity provider such as red hat14:59
*** hurgleburgler has joined #openstack-infra14:59
*** mhickey has joined #openstack-infra15:00
fungitristanC: saml for staff authentication or for community authentication?15:01
tristanCand software-factory uses an apache module called cauth to abstract authentication (supporting local user, openid, ldap, keystone) and it relies on the remote_user header15:01
fungii guess after a very long time of not using red hat i'm unclear on what/where their "community" is15:01
pabelangerderekh: that is the exception we see in nodepool15:02
tristanCfungi: oh well, saml is for @redhat user, e.g. all employees15:02
openstackgerritMerged openstack-infra/project-config: stop merging tags from stable branches into master
fungitristanC: so anyway, the question was more about teh fact that the rdo project's gerrit has the ability to authenticate with github (openid) and it seemed like a strange choice since the fedora project has an openid provider up and running which all fedora project community members presumably have accounts in15:03
*** zhurong has quit IRC15:03
fungibut it's mostly my ignorance. i have no idea if fedora and red hat are at all tightly interrelated, or if red hat is a company with employees who turn fedora into a product15:04
Shrewsjeblair: also, that output is limited to async_status output, so maybe that won't ever get too large?15:04
*** bhavik has quit IRC15:04
tristanCfedora community would use, and rdo picked github because most contributor already exist there... cauth query for user ssh key and inject them in gerrit account15:04
*** jamesmcarthur has joined #openstack-infra15:04
*** Nikolay_St has quit IRC15:05
*** dizquierdo has joined #openstack-infra15:05
fungimy last serious experiences with red hat stopped around red hat linux 7.3 and rhel 2.115:05
*** devananda has joined #openstack-infra15:06
*** acoles is now known as acoles_15:06
fungiand fedora didn't exist back then, hence my lack of clarity on the community relationships15:06
derekhpabelanger: instance on rh2 were going to an error state, I'm currently investigating why, so I've disabled nodepools access to rh2 while I do that15:06
*** tphummel has joined #openstack-infra15:06
derekhpabelanger: I think the problem is that the images are > 5GB and as a result chunked in swift15:07
jeblairShrews: but that would include stdout from, say, a shell module invocation, right?15:07
tristanCfungi: i really don't think red hat intent is to turn fedora into a product, and the community is self hosted15:07
pabelangerfungi: tristanC: to be fair, user ssh keys do exist on fedora FAS: too15:07
jeblairShrews: it won't be too big for us at least, since we're only running zuul_runner15:07
*** vhosakot has joined #openstack-infra15:07
jeblairShrews: more of a general-case (or possibly zuulv3) concern15:07
derekhpabelanger: so I'm trying to find out if we have problems with images when they are split into segments15:08
pabelangerderekh: ya, we cache a lot of things on them15:08
pabelangerderekh: thanks for the heads up15:08
*** rhochmuth has joined #openstack-infra15:08
Shrewsjeblair: i suppose so. good point15:08
tristanCpabelanger: then using fedora openid is probably something we can investigate for rdo15:09
fungitristanC: i meant take the output from the fedora community, stabilize and productize it as a downstream/derivative distribution15:09
fungisort of like what ubuntu does with debian15:09
pabelangerderekh: if you don't get things fixed for the weekend, do you mind setting tripleo-test-cloud-rh2 to -1 in nodepool.yaml? So were not continuing to launch servers15:09
derekhpabelanger: np, once I sort it out I'll re enable15:09
derekhpabelanger: lets set it know15:09
derekhpabelanger: *now15:09
*** e0ne has quit IRC15:10
*** e0ne has joined #openstack-infra15:10
openstackgerritBin Hu proposed openstack-infra/project-config: New project openstack/gluon, which brings a model-driven, extensible framework for L3 Networking Services. When a telecom service provider needs to provides its customers with a new L3 Networking Service, the new service is modeled in a YAML file. Gluon f
pabelangerderekh: great, thanks15:10
fungitristanC: thanks. my main questions were around how much cross-over there is (or isn't) between the fedora developer community and the red hat developer community, or if red hat even has such a thing as a "developer community" vs simply "employees"15:11
derekhpabelanger: so do you want a patch to set "min-ready: -1" ? or are you setting it somewhere else?15:11
*** rajinir has joined #openstack-infra15:11
*** number80 has joined #openstack-infra15:11
pabelangerderekh: yes, a patch please15:11
*** eharney has quit IRC15:12
fungitristanC: like, do interested users contribute patches to rhel, or does red hat tell them to go stick it in fedora and they'll pull from there later?15:12
*** tesseract- has quit IRC15:12
*** matrohon has quit IRC15:13
openstackgerritDerek Higgins proposed openstack-infra/project-config: Disable instances on rh2
derekhpabelanger: ^15:13
*** dansmith is now known as superdan15:13
*** infra-red has quit IRC15:14
*** rook has quit IRC15:14
*** _nadya_ has quit IRC15:16
*** tonytan4ever has quit IRC15:19
*** salv-orl_ has quit IRC15:20
tristanCfungi: i believe things need to happen in fedora first before it gets into rhel.15:21
fungitristanC: i guess that includes rdo? rdo packages are developed in fedora and then adopted by rhel later?15:22
*** bhavik has joined #openstack-infra15:22
fungijust trying to understand how many of the people using the rdo gerrit are fedora community people15:23
tristanCfungi: that is correct, rdo is acting like fedora for the rhel openstack distribution15:23
rajinirNeed some help with publishing console and devstack-logs. Devstack-logs , how do I setup?15:24
fungirajinir: are you asking how to make a webserver?15:24
tristanCand it's likely we used github authentication for simply because it's working out of the box in software-factory15:24
*** rook has joined #openstack-infra15:24
*** arxcruz has quit IRC15:24
tristanCwell that upstream/downstream interaction is the true challenge of having both a community and corporate hat15:25
rajinirfungi: Have a log server and able to copy the logs. But missing the devstack-logs, so I guess, I'm missing some setup intructions15:25
fungitristanC: got it, so that was one of the canned openid authentication options in software factory? it's not configurable to allow you to just specify an openid provider of your choice?15:25
*** d0ugal has quit IRC15:25
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for ceph scheduling bug 1475333
openstackbug 1475333 in devstack "gate-tempest-dsvm-full-ceph fails with Insufficient free space for volume creation on host ubuntu@ceph#ceph (requested / avail): 1/0.0" [High,Confirmed]
tristanCfungi: it's likely id.fedora or google auth will also work out of the box, though from my understanding there are some variable to take into account when changing openid provider.15:26
fungirajinir: if you're using devstack-gate, one of its final steps is to copy the logs into one spot on the job node so that your runner )jenkins or whatever) can retrieve those and publish them somewhere15:26
*** eharney has joined #openstack-infra15:27
fungitristanC: cool. i think that answers my questions. i didn't know if that deployment wasn't using the fedora community idp because of compatibility issues betwene gerrit and ipsilon, or for unrelated reasons15:27
tristanCfor example launchpad doesn't need any action, while github requires you to create an "application"15:27
rajinirfungi: You mean the logs are copied to the node to node where Jenkins is installed or copied to a location on the slave devstack node15:28
fungirajinir: if you're using jenkins, its publishers pull logs off the jenkins slave back to the jenkins master and then push them somewhere else15:28
*** javeriak has joined #openstack-infra15:29
rajinirfungi: May be I'm missing the spot on the jenkins master where they are published  I guessed15:29
fungirajinir: so devstack-gate collects various devstack and tempest and other kinds of logs on the slave into a "logs" subdirectory of the job workspace and we have a jenkins publisher defined which publishes those15:29
*** arxcruz has joined #openstack-infra15:30
tristanCfungi: differences in authentication providers is the reason we developped cauth, so that all software factory services relies on a unified remote_user header mechanism15:30
fungirajinir: the publisher definition we're using is this one
fungitristanC: oh, right, so cauth is taking care of the openid auth anyway, and gerrit doesn't know about openid there15:30
fungitristanC: thanks, my question was actually irrelevant in that case ;)15:31
*** xdcc has joined #openstack-infra15:31
tristanCfungi: exactly, well cauth will call an internal service (called managesf) that will inject user and ssh key per service databases15:31
tristanCfungi: you're welcome, glad i could clarify that question :)15:32
fungirajinir: basically an scp publisher which acts on the contents of the logs directory on the slave after the job completes and uploads it to a server we're maintaining called static.openstack.org15:32
*** ddieterly[away] is now known as ddieterly15:32
rajinirfungi: only console-logs are in the job workspace on the jenkins master. I'll have to check why the logs from the slave node are not published15:33
fungirajinir: oh, no console logs are special. are you getting all your logs _except_ the console logs showing up?15:33
rajinirfungi: only the console logs , devstack-logs are missing. May be I have some wrong logserver settings in the template15:34
rajinirwill check15:34
fungirajinir: console logs are actually stored in memory on the jenkins master, not a file anywhere. we fake copying that by retrieving it from memory with this publisher
fungirajinir: and it depends on a custom feature clarkb hacked into the upstream scp publisher some years ago, so requires a fork of that plugin to get the console log from memory15:35
openstackgerritMerged openstack-infra/project-config: Disable instances on rh2
sdaguefungi: is there a way to make a job fail but capture the logs? my local.conf d-g change is stalling on subnode and I'm 100% not sure why with the telnet console log15:36
fungirajinir: also in fairness i should mention that all of this is (recently) historical, we no longer use jenkins as of a couple weeks ago and are doing basically the same things with ansible and a new daemon in zuul now15:36
*** lvdongbing has quit IRC15:36
*** tqtran has joined #openstack-infra15:36
clarkbfungi: its not a fork anymore but is on unreleased master of plugin iirc15:36
fungisdague: hrm, i could probably ssh into it and kill something to abort the job payload early15:36
rajinirfungi: ok. Thanks for the info. I would like to make this work before we get rif of jenkins. will check the log server setup.15:36
fungirajinir: oh, i'm not suggesting getting rid of jenkins, just mentioning that this is how it worked for us in the past, but i'm not implying that we're doing it the same way any longer15:37
sdagueoh, wait, I might see the issue in the code15:37
rajinirfungi: ok, got it15:37
sdaguestupid missing .15:37
fungisdague: sometimes it's amazing how something so small as a dot can create a bug15:38
*** sdake has quit IRC15:38
*** sdake has joined #openstack-infra15:38
*** sdake has quit IRC15:38
*** sdake has joined #openstack-infra15:38
*** pcaruana has quit IRC15:38
*** gouthamr has quit IRC15:39
mordredjeblair: I also made this:
mordredjeblair: which I think is likely not needed in reality with the async_module patch15:39
mordredbut may be decent belt-and-suspenders15:39
clarkbfungi: rajinir also thwy arw stored on disk but not in html or text. they are encoded base64 or something. One thibg that plugin does is reencode into utf8 and wrap in html15:39
*** gouthamr has joined #openstack-infra15:39
*** rcernin has quit IRC15:39
*** jlanoux_ has quit IRC15:40
*** tqtran has quit IRC15:40
openstackgerritSean Dague proposed openstack-infra/devstack-gate: support local.conf instead of localrc
*** salv-orlando has joined #openstack-infra15:40
sdagueclarkb: ^^^ I think is remarkably close to working15:40
*** roxanaghe has joined #openstack-infra15:41
jeblairmordred: yeah, though, do we want to apply that now, as it might hide what we're tracking down?15:41
sdaguealso, why is d-g still running kilo jobs? :)15:42
*** roxanaghe has quit IRC15:42
mordredjeblair: no - I think we should see if the async_module patches fix it15:42
*** jamesmcarthur has quit IRC15:42
mordredjeblair: then if they do, make sure that the ansible patch doens't break it :)15:42
*** xdcc has left #openstack-infra15:43
jeblairmordred, Shrews, pabelanger: so i will now apply today's patch from Shrews instead of yesterdays patch.  sound good?15:43
*** jamesmcarthur has joined #openstack-infra15:43
pabelangerno objection from me15:43
*** Daisy has joined #openstack-infra15:45
jeblairShrews:   right?15:45
*** ams__ has joined #openstack-infra15:45
Shrewsjeblair: correct15:46
*** pt_15 has quit IRC15:46
*** arxcruz has quit IRC15:46
rajinirclarkb, fungi ok, thx15:46
*** zeih has quit IRC15:46
*** zul has quit IRC15:47
*** ccamacho|lunch is now known as ccamacho|away15:47
fungisdague: i think jhesketh was proposing (or proposed) a change today to remove kilo now that he's deleted the branches15:49
*** e0ne has quit IRC15:49
*** itisha has quit IRC15:50
*** Daisy has quit IRC15:50
*** xarses has quit IRC15:50
*** daemontool_ has joined #openstack-infra15:52
*** baoli has joined #openstack-infra15:53
*** lvdongbing has joined #openstack-infra15:53
*** baoli_ has quit IRC15:53
*** daemontool has quit IRC15:54
*** ddieterly is now known as ddieterly[away]15:55
*** baoli has quit IRC15:55
*** alex_xu has quit IRC15:55
*** baoli has joined #openstack-infra15:55
*** raunak has joined #openstack-infra15:56
*** baoli_ has joined #openstack-infra15:57
*** ddieterly[away] is now known as ddieterly15:58
*** andymaier has quit IRC15:58
*** alex_xu has joined #openstack-infra15:58
jeblairfunny story -- ansible and the core modules repo share a history so you can push one to the other.  whoops.15:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove kilo jobs
*** baoli has quit IRC16:01
clarkbjeblair: as a fast forward? awesome16:01
jeblairclarkb: well, not as a fast-forward -- i can't ff these branches anyway since i'm rewinding a commit16:02
jeblairokay, i think i have them all staged now16:03
*** cody-somerville has joined #openstack-infra16:03
jeblairso i will do the stop/install/start thing16:03
*** vhosakot has quit IRC16:03
*** vhosakot has joined #openstack-infra16:04
*** kushal has quit IRC16:04
asselinrcarrillocruz, so I'm noticing that with the current cloud-launcher, floating ip times out per ansible, but it succeeds after the timeout. Running ansible again and it complains there's already an ip.
*** Nikolay_St has joined #openstack-infra16:05
*** degorenko is now known as _degorenko|afk16:06
*** esikachev has quit IRC16:06
*** gongysh has quit IRC16:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Gerritbot: Remove obsolete kilo conditions
jeblairokay, i've got this in a playbook16:09
jeblairso i'm about to run an ansible command that will end up aborting all running jobs in fairly quick succession16:09
*** esikachev has joined #openstack-infra16:09
mordredjeblair: neat! sledgehammers!16:09
jeblair(and they should all get restarted immediately)16:09
jeblairmordred, pabelanger, clarkb: ^ that look right?16:10
jeblair(it's basically a copy of the graceful restart playbook, but without the graceful, and with the ansible install)16:10
*** piet has joined #openstack-infra16:10
jeblairthe timeout should actually be more like 30 seconds than 3 hours, but hey16:10
*** akshai_ has quit IRC16:11
mordredjeblair: yah. lgtm16:11
pabelangerjeblair: Ya, looks good from what I see16:12
rcarrillocruzasselin: that's not an issue with the role, but an issue with your cloud16:12
rcarrillocruzi had similar issues with other HOS clouds16:12
pabelangerjeblair: poll could be reduced from 60s, but the is a nit16:12
rcarrillocruzcheck on a devstack, you will see it just works16:12
asselinrcarrillocruz, how to work around?16:12
jeblairpabelanger: oh, yeah, i'll bump it down to 516:12
mordredrcarrillocruz: if it fails on HOS but succeeds on devstack, we should see if shade can do a better job of dealing with that perhaps?16:13
rcarrillocruzyeah, iirc it was some issue with the sync between neutron and nova16:13
jeblairoh, i formatted the chdir with shell wrong16:13
rcarrillocruzit would should show in some neutron table the floating ip was there16:13
pabelangerAh, right, I think that need args16:13
rcarrillocruzbut nova didn't have that synced16:13
rcarrillocruzand the call waits16:13
rcarrillocruzthat was a while ago16:14
mordredrcarrillocruz: yah. this is a bug we've seen in older openstack versions for sure16:14
*** bin_ has quit IRC16:14
rcarrillocruzasselin: try reproducing with the corresponding shade call16:14
mordredrcarrillocruz: I've got a patch somewhere to make shade disregard nova's view of floating ips and to use neutron's instead16:14
rcarrillocruzfeel free to open a bug or ping me16:14
rcarrillocruzmordred: right, i even remember pinging you about it at some point about this16:14
mordredsince it doesn't actually matter if nova knows about the FIP at all, as long as neutron has it wired up to NAT to the right thing16:14
rcarrillocruzasselin: i remember i had to circument that but not using os_server auto_ip, but use os_floating_ip, and then reference the IP created to the os_server task16:15
rcarrillocruzbut obviously, that's not in the launcher16:15
*** esikachev has quit IRC16:15
mordredrcarrillocruz: maybe we should get that patch into shape16:16
rcarrillocruzthat workaround i tell you predates the cloud launcher, it was my first baby steps with ansible openstack modules16:16
asselinok yeah...but auto_ip should check if there's already an ip, no? how does it know when to stop?16:16
*** roxanaghe has joined #openstack-infra16:16
rcarrillocruzasselin: the problem is that you have the FIP in two tables16:16
rcarrillocruzand neutron16:16
*** xarses has joined #openstack-infra16:16
rcarrillocruzso you may have the FIP created, yet the sahde call waiting16:16
mordredasselin: yah. it's an openstack issue - and one of the many reasons that clouds that require floating ips are horrible to work with16:17
rcarrillocruzmordred: happy to work that ticket16:17
rcarrillocruzasselin: do this, try to reproduce with a shade call16:17
rcarrillocruzi'll get an account in your HOS clouds16:17
jeblairmordred, pabelanger, Shrews: restarts complete16:17
rcarrillocruzand i'll work it with monty's patch16:17
asselini this is a openstack issue shade doesn't (yet) make it transparent16:17
*** tphummel has quit IRC16:18
*** weshay is now known as weshay_brb16:18
asselinmordred, nice commit message :)16:18
clarkbjeblair: ya that seems fine to me, maybe add it to a tools repo in zuul?16:18
pabelangerjeblair: what do you think about adding an optional argument to graceful, which is the max time we want to wait? If the timeout is reached, we move to stop() regardless if self.builds is None16:19
Shrewsjeblair: where does the patching part come in?16:20
Shrewsis that done manually?16:20
rcarrillocruzasselin: let me know how the reprdouce goes, feel free to open a story on storyboard about it16:20
rcarrillocruzi'll pick up16:20
*** amitgandhinz has quit IRC16:20
*** abregman|afk has quit IRC16:21
*** amitgandhinz has joined #openstack-infra16:21
*** abregman|afk has joined #openstack-infra16:21
jeblairShrews: yeah, i have not playbooked that yet.  it's a bunch of git commands which i have already done.16:21
*** abregman_ has joined #openstack-infra16:21
asselinrcarrillocruz, so shade storyboard?16:21
jeblairShrews: so the /root/ansible repo is already prepped on all the hosts16:21
rcarrillocruzasselin: yep16:21
Shrewsjeblair: ah, ok16:22
jeblairpabelanger: that could be nice.  a sort of *mostly* graceful.16:22
jeblairpabelanger: i tried to add an argument to the command pipe stuff recently, it's not trivial.  :)16:22
*** mriedem has quit IRC16:22
clarkbpabelanger: jeblair we can all it the tripleo-timeout16:22
pabelangerjeblair: Ya, maybe in cases like this, we can buffer 10mins to give jobs that are close to the end time to finish, then abort16:22
*** abregman_ has quit IRC16:23
jeblairclarkb: just what i was thinking16:23
pabelangerclarkb: :D16:23
*** gouthamr has quit IRC16:23
*** abregman has joined #openstack-infra16:23
*** zul has joined #openstack-infra16:23
jeblairpabelanger: if you attempt this -- start with the argparse part of the change -- that's the hardest part :)16:23
*** piet has quit IRC16:24
jeblairoh, and there's the spike visible on the node graph16:24
fungialso sometimes the ironic timeout16:25
*** Apoorva has joined #openstack-infra16:25
*** eharney has quit IRC16:25
*** abregman|afk has quit IRC16:26
openstackgerritMilan Kováčik proposed openstack-infra/project-config: Introduce experimental Inspector grenade job
clarkbjeblair: Shrews' patch didn't fix the job fails? Did we at least get more info from the extra logging?16:27
asselinrcarrillocruz,!/story/2000659. will try to figure out the shade reproducer next.16:27
jeblairclarkb: we did get info from the logging and identified another similar/related potential failure, so we just installed v2 of Shrews patch that corrects that16:27
pabelangerjeblair: sure, I'll see about giving it a try this afternoon16:27
clarkbjeblair: with the finally block?16:28
jeblairdid anyone else's irc just look like this?
clarkbjeblair: no16:28
*** akshai has joined #openstack-infra16:28
clarkbjeblair: try a ^R16:28
jeblairclarkb: and communicate()16:28
*** weshay_brb is now known as weshay16:28
pabelangerno, I got the full line correctly too16:29
jeblairclarkb: i'm in screen; i just switched screens to refresh and it's still like that16:29
pabelangerjeblair: can I delete the currently help nodes in nodepool?16:29
clarkbhuh, I see two separate lines fungi | also sometimes the ironic timeout16:29
jeblaireavesdrop also saw things correctly.  that's very weird.16:29
jeblairpabelanger: please do16:29
*** hashar has quit IRC16:29
pabelangerjeblair: done16:30
fungiweechat didn't do anything weird on my end, but that could be explained by the first comment being mine so different echo16:30
pabelangerI have to admit, I've had to purge 0 leaked nodes in nodepool since switching to zuul-launcher.  It is very nice16:31
*** yamahata has quit IRC16:31
fungii'm guessing it's something to do with some terminals' rendering of the name "Milan Kováčik"16:31
*** javeriak has quit IRC16:31
*** ajmiller has quit IRC16:31
fungimaybe one of those causes the cursor to move up one line?16:31
*** ajmiller has joined #openstack-infra16:32
fungipresumably the č16:32
fungiwhich seems to have been interpreted in your terminal as Ä<up cursor>16:32
*** alex_xu has quit IRC16:32
jeblairfungi: you just made my terminal go wonky16:33
*** ajmiller has quit IRC16:33
fungihah, i can recommend some setting for traditional xterm! ;)16:33
*** ajmiller has joined #openstack-infra16:33
jeblairso i think i did get all the data in the correct order16:33
jeblairthe gerritbot line just ended up doing some cursor movement16:33
*** eezhova has quit IRC16:34
fungii spent a long time playing around with making sure utf-8 encoded japanese rendered correctly over telnet, tmux, ssh, et cetera16:34
jeblairfungi: can you make it go back now?16:34
jeblair [16:34] [jeblair(+Zi)] [3:freenode/#openstack-infra(+cnt)] [Act: 2,4,5,6<up cursor>16:34
jeblairfungi: ^16:34
pabelangerha ha ha16:34
*** dmk0202 has joined #openstack-infra16:34
pabelangerascii injections!16:34
jeblairwhat's weird is i thought i just got this thing working right with utf816:34
asselinrcarrillocruz, mordred fwiw that patch gets past my error16:34
*** jamesmcarthur has quit IRC16:34
jeblairpabelanger: seriously16:34
*** alex_xu has joined #openstack-infra16:35
openstackgerritAlexander Chadin proposed openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate
jeblairEmilienM: we have a new version of the patch intended to fix the ansible problem in place16:36
fungiactually, it looks like i've since been able to undo most of my tweaks to xterm configuration (except for needing to keep eightBitInput disabled)16:36
*** jpich has quit IRC16:36
fungiso really, no clue there16:37
fungibut yes, that is certainly disturbing behavior16:37
fungiwhat terminal emulator are you using? rxvt if memory serves?16:37
clarkbShrews: jeblair the github "change" lgtm. Also thats a fun race, everyone read and write to the same file all at once16:37
*** eharney has joined #openstack-infra16:38
*** jamesmcarthur has joined #openstack-infra16:38
clarkbI am also impressed that the stderr and stdout were going to the same place but we were expecting valid json in the end16:38
fungii guess now i have enough info to be able to engineer ansi animation over irc for rxvt users ;)16:38
*** javeriak has joined #openstack-infra16:39
openstackgerritMilan Kováčik proposed openstack-infra/project-config: Introduce experimental Inspector grenade job
clarkbjust keep getting Milan to push more patches16:40
openstackgerritMilan Kováčik proposed openstack-infra/project-config: Introduce experimental Inspector grenade job
jeblairoh good grief16:40
clarkbjeblair: do you use urxvt-256color?16:41
jeblairclarkb: i use urxvt, but i think the problem is screen16:41
mordredasselin: my shade patch for the FIPs? that's good to know16:41
pabelangerclarkb: almost like you planned that16:41
clarkbright screen needs to be started with -U and compiled with that support in place16:41
asselinmordred, yes
mordredasselin: cool!16:42
*** tphummel has joined #openstack-infra16:42
asselinshade can handle schrodinger's cat16:43
*** ddieterly has quit IRC16:44
*** sarob has joined #openstack-infra16:44
*** rbrndt_ has quit IRC16:44
*** gouthamr has joined #openstack-infra16:44
*** dtantsur is now known as dtantsur|afk16:45
*** derekh has quit IRC16:45
clarkbjust not floating IPs16:46
*** sarob has quit IRC16:47
*** zeih has joined #openstack-infra16:47
*** sarob has joined #openstack-infra16:47
clarkbsemi random probably terrible idea16:47
*** lucasagomes is now known as lucas-afk16:47
Shrewsjeblair: mosh. try it. live it. love it16:48
clarkbwhat if we ran the infracloud control plane somewhere a bit more static, like in a cloud16:48
clarkbthen whenever HP or whoever has hardware we set them up as compute hosts against that control plane16:48
clarkbthen infracloud never goes away, just the compute hosts do16:48
nibalizerherp a derp!16:48
bkeroWhat advantage does that provide? You can quantify how many compute hosts you lack for the services?16:49
nibalizerclarkb: don't we need pretty strong links between controller and compute?16:49
clarkbttx: ^ was it you that was talking to an openstack deployer that does similar16:49
*** Nikolay_St has quit IRC16:49
clarkbnibalizer: I think ttx was saying someone is doing something similar with geographically distant hosting locations16:49
clarkbbkero: the advantage is not needing to reroll all of the services every 2 months16:49
nibalizerclarkb: i think thats a capital idea though! since we'd then be able to keep the control plane (essentially pets) online and updated16:50
clarkbbkero: compute hosts are relatively simple16:50
*** rbrndt_ has joined #openstack-infra16:50
*** acabot has quit IRC16:50
nibalizerthen we just semi dynamically spin up and down computes16:50
bkeroIt sounds like a good ide aif you expect that your compute resources are going to shift out from under you16:50
nibalizerbkero: thats the reality16:50
bkeroThat's been the reality, yeah.16:50
bkeroWill it always be? and if it isn't anymore, does the added complexity make it worth it?16:51
jeblairnibalizer: pets are the ones you euthanize when sick, right?16:51
*** rossella_s has quit IRC16:51
*** zeih has quit IRC16:51
clarkbbkero: I think we expected it to be a one and done situation, then they decided to do it again. So while I can't say it will always be this way all signs point to yes16:51
nibalizerjeblair: haha16:51
nibalizeri do love the jeblair-corollary to the pets-cattle metaphor16:52
bkeroclarkb: then having a lightweight control plane set up somewhere else makes sense16:52
bkeroWhat's the bandwidth between compute and control plane though?16:52
Shrewsjeblair: mordred: looks like maybe no love on the patch?
rcarrillocruzEl mñn16:52
clarkbbkero: the real hurt there is going to be our test images16:52
clarkbsince they are huge and transfering from hosts in eg rax to hardware in HP might not be happyness16:53
bkero11gb over commodity internet for every new test image?16:53
clarkbbkero: ~5GB since we could qcow216:53
clarkbanyways if its something we want to investigate we can take that into account and determine if there are ways of dealing with that16:53
bkeroThat's okay if it's line speed, but I doubt it'll be line speed16:53
nibalizerso for ephmeral, as i understand it, the cinder controller pushes a copy of each glance image on to each compute16:54
nibalizerso that would be sorta painful16:54
mordredShrews: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time; {'failed': True, 'module_stderr': u'', 'module_stdout': u'', 'msg': 'MODULE FAILURE', 'invocation': {'module_name': u'async_status'}, 'parsed': False}"} _EXCELLENT_16:54
jeblairpabelanger: ^16:54
clarkbnibalizer: no cinder is involved16:54
nibalizerbut i think clarks point is a) we could solve that problem and b) less painful than reloading the control plane every X weeks?16:54
nibalizeroh sorry maybe I meant glance?16:55
clarkbnibalizer: qemu/nova/libvirt cache the qcow2 on the hypervisors16:55
jeblairmordred: does that tell you more about the problem?16:55
mordredjeblair: nope16:55
mordredjeblair: it's the same error as before16:55
clarkbnibalizer: so essentially we would have large network bw needs once a day, or we would need to tackle this differneyl16:55
Shrewsare we 100% certain that zl* node has the patch?16:56
Shrewsb/c ugh16:56
clarkbits probably not the sanest idea, and there are likely real challenges involved with it. Just trying to think of ways we can be productive without wondering/waiting for next move16:56
*** Goneri has joined #openstack-infra16:56
jeblairmordred: oh so that was sarcasm :)16:57
*** ajmiller has quit IRC16:57
jeblairShrews: i'll go hunting16:57
*** ajmiller has joined #openstack-infra16:57
nibalizerclarkb: yah16:58
Shrewsjeblair: we've got to devise a better way to test things like this, because this is pretty disruptive and difficult16:58
*** trown is now known as trown|lunch16:58
nibalizerfor the midcycle, did we settle on topics? I don't see topics on
*** salv-orl_ has joined #openstack-infra16:59
nibalizeri heard infracloud and zuulv3, but I don't see it written down16:59
jeblairShrews: i'm open to suggestions.  though, to be honest, i would prefer that happen in the ansible context :)17:00
*** rkukura has quit IRC17:00
anteayanibalizer: there has been no decision that I have seen17:00
*** inc0 has quit IRC17:00
anteayanibalizer: you could write them down as suggestions, then at least they are written down17:00
*** inc0 has joined #openstack-infra17:01
nibalizerok i will create a topics section17:01
*** acabot has joined #openstack-infra17:01
*** rkukura has joined #openstack-infra17:01
*** rkukura has quit IRC17:02
*** bhavik1 has joined #openstack-infra17:02
nibalizerwe shoul try to figure out what are 'accepted' topics though pretty soon so the relevant personnel can get travel authorized17:02
*** akshai has quit IRC17:02
*** salv-orlando has quit IRC17:02
*** akshai has joined #openstack-infra17:02
openstackgerritAlexander Chadin proposed openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate
Shrewsall of the centos jobs succeeded. this is xenial only17:03
Shrewsmaybe we should dig into the differences there17:03
clarkbxenial uses python3 by default, maybe ansible is using that instead of 2?17:04
*** piet has joined #openstack-infra17:04
Shrewsorly? hrm17:04
clarkbShrews: `python` is still python2 though17:04
clarkbso depends on how ansible invokes the remote python processes17:04
Shrewsclarkb: it will use the system python unless you give it an explicit one in the host file17:05
*** bhavik has quit IRC17:05
*** bhavik1 is now known as bhavik17:05
Shrewsansible_python_interpreter or such17:05
anteayanibalizer: I think that is a reasonable position17:05
openstackgerritMilan Kováčik proposed openstack-infra/project-config: Introduce experimental Inspector grenade job
mordrednibalizer: ++17:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove kilo jobs
AJaegerI've just updated jhesketh 's change ^17:07
nibalizerokay created a topics section, with proposed and accepted subsections
*** sdake_ has joined #openstack-infra17:07
nibalizermight be a thing that's easier done on an etherpad17:07
AJaegerproject-config cores, I think this is fine now - but it's a larger change, so quick reviews - instead of endless rebases - would be appreciated for 333910.17:07
*** sdake has quit IRC17:09
*** lvdongbing has quit IRC17:12
AJaegerEmilienM: could you double check the puppet branches, please? There're still some juno and icehouse there that surprised me - see 33391017:13
rcarrillocruzasselin: cool, I will take a look and assign myself17:13
*** stajkowski has quit IRC17:14
*** jed56 has quit IRC17:15
*** dmk0202 has quit IRC17:15
clarkbAJaeger: jhesketh I am reviewing the kilo removal change in project-config now17:15
*** ifarkas has quit IRC17:15
*** dmk0202 has joined #openstack-infra17:16
*** dtardivel has quit IRC17:17
*** yamahata has joined #openstack-infra17:17
clarkbAJaeger: also looks like networking-odl has juno and icehouse rules in zuul's layout file17:17
EmilienMAjaeger: currently on my phone, will look later. I'm on holiday today17:17
*** flepied has quit IRC17:18
openstackgerritMerged openstack-infra/infra-specs: Priority specs cleanup/update for Newton cycle
mordredShrews: Python 2.7.11+ (default, Apr 17 2016, 14:00:29)17:19
*** kzaitsev_mb has quit IRC17:19
mordredShrews: that's the python that shows up with /usr/bin/python on the xenial nodes17:19
notmorganmordred: ++17:20
strigaziAJaeger: hi, magnum's job for the install-guide is expected to published here, right?
*** piet has quit IRC17:20
notmorganmordred: i think latest fedora and the rolling releases are the only ones defaulting to py317:20
Shrewsmordred: jeblair: can we add my pub key to one of the held xenial nodes and let me run a few tests against it?17:20
clarkbmistral has a juno rule too17:20
notmorganmordred: for "/usr/bin/python"17:20
clarkbnotmorgan: fedora shouldn't use py3 in /usr/bin/python either17:21
AJaegerEmilienM: double check it next week then - we cna do more cleanjup as needed...17:21
clarkbnotmorgan: defaulting to python3 means python2 is not installed but `python` should always give you a python2 unless you arch17:21
clarkbor gentto17:21
AJaegerclarkb, thanks for reviewing. There's still a branch in git for networking-odl.17:21
clarkbAJaeger: as long as we don't have to keep old test platforms alive for them its probably fine17:21
Shrewsi'm curious if even a simple async playbook will work correctly17:21
AJaegerclarkb: I hope not ;)17:21
AJaegerclarkb: I'm patching jenkins/jobs/ now for kilo...17:22
notmorganclarkb: it also in some cases depends on the order the packages are installed =/ I ran into this on a VM i had17:22
asselinrcarrillocruz, i'm seeing an issue though with launch node. So on subsequent runs it seems to use a different ssh key instead of the one that's registered17:22
clarkbnotmorgan: python upstreams says `python` is always python2, most distros are at least attempting to honor that17:23
*** SumitNaiksatam has quit IRC17:23
*** electrofelix has quit IRC17:23
notmorganclarkb: silly distros... =/17:23
notmorganclarkb: that don't honor it that is17:23
notmorgannot that do.17:23
notmorgani am guessing i'd feel the same way with perl and perl6 :P if i used perl still.17:24
*** roxanaghe has quit IRC17:24
*** cbader has quit IRC17:24
*** roxanaghe has joined #openstack-infra17:24
rcarrillocruzthat is not an issue, but expected. The key generated in the playbook is created with a random file17:24
*** dprince has quit IRC17:24
rcarrillocruzIf you want idempotency, just use the role with a resources.yaml with the keypair there17:25
mordredShrews: totes. one sec17:25
*** sabari has left #openstack-infra17:25
asselinrcarrillocruz, ok17:25
*** tosky has quit IRC17:26
mordredShrews: is still valid?17:26
*** amitgandhinz has quit IRC17:26
Shrewsmordred: aye17:26
mordredShrews: jenkins@
clarkbAJaeger: jhesketh I have approved that change, will keep an eye on it for fallout if we missed something17:27
*** cody-somerville has quit IRC17:27
*** cody-somerville has joined #openstack-infra17:27
*** cody-somerville has quit IRC17:27
*** cody-somerville has joined #openstack-infra17:27
*** amitgandhinz has joined #openstack-infra17:27
jeblairmordred, Shrews: that node ran
AJaegerthanks, clarkb !17:27
asselinrcarrillocruz, you don't need it inside launch_node?17:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Jenkins/jobs: Kilo cleanup
AJaegerclarkb: and one more ^17:28
AJaegerproject-config cores, 334008 and are the rest for the kilo cleanup17:28
asselinrcarrillocruz, also it's set here: { role: cloud-launcher, generate_keypair: yes }17:28
jeblairmordred, Shrews: i think that might be a legit failure (eg, not ansible)17:28
*** grassass has quit IRC17:29
*** tqtran has joined #openstack-infra17:30
mordredjeblair: oh. piddle17:30
jeblairmordred, Shrews: node 1975977 looks like an ansible failure:
clarkbAJaeger: wow no wonder upgrading openstack is so difficult :)17:30
mordredjeblair: k. I'll stick Shrews' key on that one too17:31
*** shashank_hegde has joined #openstack-infra17:31
rcarrillocruzright, that flag is to generate a    keypair on the fly to use for server creation17:31
clarkbAJaeger: 334008 is a bit scarier but good cleanup too17:31
rcarrillocruzJust dont use it17:31
*** claudiub|2 has joined #openstack-infra17:31
rcarrillocruzAnd create your named keypair on the resources.yml file17:31
*** inc0 has quit IRC17:32
mordredShrews: is the node jeblair mentions above - and you also have access to jenkins@ on it17:32
rcarrillocruzCheck the sample on the role repo17:32
*** grassass has joined #openstack-infra17:32
*** HeOS has quit IRC17:33
AJaegerclarkb: I did this manually, hope I got it right ;)17:33
asselinrcarrillocruz, which sample?17:33
clarkbAJaeger: just more thinking if something depends on old kilo behavior that they shouldn't17:33
clarkbAJaeger: +2 we have to keep moving forward17:34
*** dprince has joined #openstack-infra17:34
rcarrillocruzresources.yml.sample :-)17:34
*** daemontool_ has quit IRC17:34
AJaegerclarkb, thanks.17:35
*** claudiub has quit IRC17:35
rcarrillocruzInfra ansible repo I think it is17:35
*** dmk0202 has quit IRC17:35
AJaegerAny other project-config reviewers for 334008, if possible, please?17:36
*** mriedem has joined #openstack-infra17:36
*** rkukura has joined #openstack-infra17:37
*** roxanaghe has quit IRC17:38
nibalizerAJaeger: on it17:38
*** woodster_ has quit IRC17:39
AJaegerthanks, nibalizer !17:39
*** dmk0202 has joined #openstack-infra17:39
pleia2reviewed, nibalizer can +A if he thinks it looks good17:39
pleia2lots of good cleanup there \o/17:40
pleia2thanks AJaeger17:40
*** cbader has joined #openstack-infra17:40
Shrewsmordred: jeblair: this is not a race17:40
Shrewsthe async file was successfully written 6 minutes AFTER async_wrapper gave up17:41
*** roxanaghe has joined #openstack-infra17:42
AJaegerthanks, pleia217:42
AJaegerif you're in kilo review mode, there's also - for gerritbot...17:43
*** ajmiller has quit IRC17:45
* pleia2 has a look17:45
*** amitgandhinz has quit IRC17:46
pleia2good good17:46
*** amitgandhinz has joined #openstack-infra17:46
*** inc0 has joined #openstack-infra17:47
*** baoli_ has quit IRC17:47
AJaegerthanks, pleia217:48
*** baoli has joined #openstack-infra17:48
*** akshai has quit IRC17:48
*** zeih has joined #openstack-infra17:48
*** akshai has joined #openstack-infra17:49
*** sambetts is now known as sambetts|afk17:49
*** chem is now known as chem|week-end17:50
*** baoli has quit IRC17:50
Shrewsi dunno. i'm stumped. mordred: done with those nodes. i can't glean anything new from them17:51
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop)
notmorganooh hmm. i think i found an issue with the opportunistic testing in nodepool (mysql). seems like it has cases where it didn't cleanup the dynamic users/grants =/17:54
*** zeih has quit IRC17:54
notmorgani should really use a docker container for this instead of local mysql.17:54
mordredShrews, jeblair: at this point, I think I would accept _any_ explanation if there was actually an explanation17:54
mordredthis whole "oops, sometimes it doesn't work" is extra weird17:55
notmorganmordred: ++ i don't like "oops, it doesn't work sometimes"17:55
pabelangerShrews: mordred: jeblair: I'm starting to feel, this condition should be broken into 2 different checks. If finshed: break, thats straightforward. If any other failure, we need to decided abort the poll process early or continue regardless until we reach our poll timeout.17:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove kilo jobs
* AJaeger had to remove an extra ")" to fix the regex ^17:56
notmorganAJaeger: yay REGEX!17:56
*** rbrndt_ has quit IRC17:56
pabelangerpersonally, I feel we should continue until poll has timeout, then error "async didn't complete within requested time'17:57
AJaegernotmorgan: I shouldn't have removed the extra () that jhesketh wisely left in ;(17:57
*** yamamoto has quit IRC17:57
* AJaeger tests locally first17:57
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Jenkins/jobs: Kilo cleanup
pabelangerI think switching to 328649 could give us better control of that process17:59
openstackgerritMerged openstack-infra/project-config: Gerritbot: Remove obsolete kilo conditions
AJaegerclarkb: do you want to review my one ")" removal in 333910  again, please? The job passed locally now18:01
*** Ravikiran_K has quit IRC18:02
clarkbAJaeger: lgtm you cna probably approve since yo uonly changed one character18:02
*** zul has quit IRC18:02
AJaegerclarkb: will do, thanks18:02
*** dizquierdo has quit IRC18:03
*** openstackgerrit has quit IRC18:03
*** openstackgerrit has joined #openstack-infra18:03
*** sdake_ has quit IRC18:04
openstackgerritPaul Belanger proposed openstack-infra/zuul: Switch to async_status for zuul_runner task
*** e0ne has joined #openstack-infra18:05
*** jamesmca_ has joined #openstack-infra18:06
*** inc0 has quit IRC18:06
*** flepied has joined #openstack-infra18:06
AJaegeraccording to our calendar, there's no meeting going on - what about merging (channels have been renamed) and and - all three for system-config18:06
*** flepied1 has joined #openstack-infra18:07
*** inc0 has joined #openstack-infra18:07
jpmaxmanfungi mordred so it has been a few days and the wiki seems to be holding steady.  Do you want me to officially submit the changes to enable recaptcha as a patch?18:07
fungijpmaxman: yes, i've heard no complaints with it so far18:07
*** jamesmc__ has joined #openstack-infra18:07
fungijpmaxman: please do!18:07
jpmaxmanok sounds good - I might keep an eye on it over the weekend just to be sure then do it on Monday if tha's ok18:08
*** ajmiller has joined #openstack-infra18:08
mordredjpmaxman: sounds awesome18:08
pleia2AJaeger: good idea, reviewing now18:09
AJaegerthanks, pleia2 !18:09
jpmaxmangreat I think when everything gets updated to the latest version it will also include the new recaptcha that looks a little nicer.18:09
fungijpmaxman: sounds great. thanks again18:09
*** jamesmcarthur has quit IRC18:09
clarkbare new accounts able to be made now?18:09
fungiclarkb: that's not been turned back on yet, no18:10
*** flepied has quit IRC18:10
fungii expect that's the next step is to see if the spammers creating new accounts (assuming there's no option to enforce captchas on account creation as well) are at all thwarted from making edits with this harder captcha implementation18:11
jpmaxmanby the way - pleia2 on the new server are you using trusty or xenial?18:11
fungijpmaxman: the current plan is to do trusty18:11
jpmaxmanI've seen some suspicious accounts created18:11
*** jamesmca_ has quit IRC18:11
fungiwe're wary of adding xenial servers for heavily-used services this early in its lifecycle18:12
jpmaxmanbut so far none have made any edits or additions18:12
jpmaxmanok sounds good trusty is trusty :)18:12
clarkbjpmaxman: all new accounts should only happen when we flip thr bit temproarily, is there an example you can share that we can check against that with?18:13
dougwigclarkb: i think that's a good idea for the controller/db/rabbit, but i'm more worried about how much pain you're going to have with the neutron router being that far away.18:13
fungijpmaxman: oh, indeed there were quite a few created on wednesday it looks like. was new account creation back on for a while?
dougwigclarkb: which underlay network is at play?  vxlan?  provider hardware?18:14
clarkbdougwig: provider nets18:14
clarkbdougwig: so that shouldnt be an issue18:14
fungiand yeah, it looks like the only one of those created in that timeframe with contribs is toby (who is on the board) editing the release name proposals pages18:15
dougwigclarkb: well, the neutron node still provides DHCP and metadata for those.18:15
jpmaxmanyes people can create new accounts18:15
jpmaxmanthat was part of the test to see if this was working :)18:15
*** eharney has quit IRC18:15
clarkbdougwig: on the compute host though I thought18:16
fungijpmaxman: i still see /srv/mediawiki/Settings.php has "$wgGroupPermissions['*']['createaccount'] = false;" so what's working around that currently?18:16
AJaegerdougwig: the midonet folks asked, could you review their change , please?18:16
dougwigclarkb: no, it comes from a namespace on the neutron node, which needs to be plugged into the same provider net.  but, a local neutron node and distant controller/db still moves a lot of the setup complexity.18:17
jpmaxmanblorg - well actually it looks like my changes reverted18:18
dougwigAJaeger: done18:18
jpmaxmanincluding the recaptcha18:18
jpmaxmanis it still being overwritten with the puppet config18:18
jpmaxmanI guess we still know nothing :(18:18
jpmaxmanas to if it worked or not18:18
fungijpmaxman: hrm, yeah it looks like puppet is getting run there according to the syslog18:19
fungii'm tracking that down now18:19
mordredinfra-root: all of the groups in the generated groups file are empty18:19
*** amitgandhinz has quit IRC18:19
*** amitgandhinz has joined #openstack-infra18:19
jpmaxmanok thanks :)  I'll reapply on Monday and keep an eye on it18:20
openstackgerritPaul Belanger proposed openstack-infra/puppet-mediawiki: Require users to confirm their email address before they can edit
AJaegerdougwig: thanks. Will rebase and approve after the kilo eol-change is in...18:20
jpmaxmanso no super hurry18:20
fungimordred: i guess that means including the emergency/disable groups?18:20
jpmaxmandon't want to make any changes before the weekend hits18:20
mordredfungi: emergency groups are there18:20
pabelangerfungi: we should consider ^ for wiki.o.o too18:20
pabelangerconfirm email address for edit18:20
fungijpmaxman: agreed, sorry about that. digging into it now obviously18:20
mordredfungi: but the larger set of generated groups not so much18:20
openstackgerritMerged openstack-infra/system-config: Rename IRC channel of container service
mordredfungi: I'd lke to run - any concerns with that?18:21
fungimordred: oh, so any of the ones we have set in public git are not getting set now?18:21
mordredfungi: yes. that's correct18:21
openstackgerritPaul Belanger proposed openstack-infra/puppet-mediawiki: Require users to confirm their email address before they can edit
jpmaxmanI created a test account in launchpad and am pretty sure I had to confirm my e-mail before I could login to the wiki18:21
*** trown|lunch is now known as trown18:21
jpmaxmanI can try it again to be sure18:21
pabelangerya, I am not sure if it matters with openid18:22
pabelangerbut since we still will allow mobile users, I think we want that18:22
fungipabelanger: jpmaxman: yeah, i'm pretty sure lp already requires account confirmation, so it's unlikely to be thwarting anyone if we do the same on the wiki18:22
pabelangersince spammers used mobile IIRC18:22
jpmaxmanI think mobile was disabled by impossible captcha - not sure if that's still the case18:22
fungipabelanger: nope, the spammers were using lp until we changed teh captcha to something guessable and then they started using the mobile account creation18:22
fungipabelanger: we previously had mobile account creation "disabled" by making that captcha impossible to answer as jpmaxman mentioned18:23
fungibut since we wanted to add the captcha to edits we needed to make it answerable, which in turn allowed spammers to create accounts on the wiki without going through lp openid18:24
*** e0ne has quit IRC18:24
*** _nadya_ has joined #openstack-infra18:25
*** zul has joined #openstack-infra18:25
*** sarob has quit IRC18:25
*** sarob has joined #openstack-infra18:26
*** rbrndt_ has joined #openstack-infra18:26
*** SumitNaiksatam has joined #openstack-infra18:26
mordredfungi: I have re-run expand groups18:26
fungii have a (very vague) recollection that the impossible captcha workaround was something Ryan_Lane added so that we could enable the mobile view extension even though the openid extension didn't apply to it18:26
mordredand the generated-groups file is now correct18:26
fungimordred: should we have to run that manually under normal circumstances? i thought our was supposed to be doing it automatically18:27
fungii guess we changed something which subtly broke it?18:28
mordredfungi: we shoudl not. also, it was definitely _run_ - but something broke it whenever it was18:28
mordredfungi: unfortunately, I cannot find logs to indicate what broken18:28
mordredfungi: the file had all of the group headings in it - just no content in each grouo18:28
fungioh, ew18:28
openstackgerritMerged openstack-infra/system-config: Add #openstack-fwaas to statusbot/meetbot
openstackgerritMerged openstack-infra/system-config: Add #openstack-pkg to statusbot & meetbot_channels
fungiaside from causing stuff we had in the disabled group in the public groups file to get puppeted anyway, what else could that have broken i wonder?18:29
*** shardy has quit IRC18:29
openstackgerritBin Hu proposed openstack-infra/project-config: New project of a Model-Driven Extensible Framework
fungimordred: was the wiki getting puppeted even though it was in the public disabled list what led you to this discovery, or did you spot something else broken too?18:30
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: move the release tools that need to run on secure nodes
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: networking-midonet: Make ml2 job non-voting
*** kushal has joined #openstack-infra18:31
mordredfungi: yes. it was puppet breaking jpmaxman18:32
fungiokay, so we haven't spotted anything else this impacted (yet). that's good at least18:32
AJaegerI rebased 333097, any project-config core  wants to review and approve to unblock them, please?18:33
* AJaeger wishes everybody a great weekend18:33
mordredfungi: \o/18:34
*** eharney has joined #openstack-infra18:34
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add a "ptl" command to show info about how to contact a PTL
fungihave a great weekend AJaeger18:34
fungii need to step away for a bit and do some yardwork while the weather is still fair18:35
fungibefore my lawn develops a taste for human flesh18:35
anteayaAJaeger: why do you create a new rule for job? why not just have voting: false under the current rule that defines test branches?
anteayafungi: enjoy yardwork18:37
openstackgerritMerged openstack-infra/project-config: Remove kilo jobs
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support
*** kushal has quit IRC18:39
dougwigAJaeger: you cool with the 3 hour timeout on ?  i thought those were getting shrunk.18:39
*** kushal has joined #openstack-infra18:39
openstackgerritMerged openstack-infra/jenkins-job-builder: Add addTestResults to GitHub pull request builder trigger
notmorganmordred, jeblair, ^ this is just a rebase/update of the initial migration script, next step is the auto-run.18:40
notmorganmordred, jeblair: just an FYI.18:40
anteayadougwig: thanks for asking, I'm not okay with a 3 hour timeout18:40
*** openstackstatus has quit IRC18:40
*** openstack has joined #openstack-infra18:42
*** openstackstatus has joined #openstack-infra18:42
*** ChanServ sets mode: +v openstackstatus18:42
notmorganmordred: sorry, yeah . irc lagging for me =/18:42
anteayadougwig: I agree with you, the timeout is off18:42
notmorganmordred: though it was *very* easy to create the initial revision. woo :)18:42
notmorgani like the "automatic" migration script creation18:42
notmorganmordred: so my plan is: on an unmanaged DB, just hook into the sql-a meta.create_all() and stamp it as "latest" - if the db is not under management but has data, just stamp it with "latest", and if it is under management but not latest, perform the upgrade18:44
notmorganmordred: on startup.18:44
notmorganfirst thing there was "unmanaged and empty db" sorry18:44
openstackgerritMerged openstack-infra/jenkins-job-builder: Update config file provider docs to have default options
*** r-mibu has quit IRC18:45
openstackgerritBen Nemec proposed openstack/diskimage-builder: Clean more from ironic-agent ramdisk image
*** r-mibu has joined #openstack-infra18:45
openstackgerritBen Nemec proposed openstack/diskimage-builder: Don't create an image just to delete it
*** claudiub|2 has quit IRC18:45
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci
*** amitgandhinz has quit IRC18:46
*** amitgandhinz has joined #openstack-infra18:46
notmorganmordred: lol you already tell it to upgrade on connection :P18:48
notmorganmordred: in NodeDatabase18:48
notmorganmordred: you did the magic work for that already ^_^18:48
*** salv-orl_ has quit IRC18:49
openstackgerritMerged openstack-infra/jenkins-job-builder: Update sonar plugin
*** zeih has joined #openstack-infra18:50
clarkbya all teh work is done except for conflict resolution and updating to current db state iirc18:50
*** gouthamr has quit IRC18:51
*** amitgandhinz has quit IRC18:51
*** amitgandhinz has joined #openstack-infra18:52
*** hashar has joined #openstack-infra18:52
sdaguecorrupt mirror?18:53
*** kushal has quit IRC18:54
AJaegeranteaya: the change is based on another one that removes the lines you quote18:55
*** zeih has quit IRC18:55
*** ikhudoshyn has quit IRC18:55
*** amotoki has quit IRC18:55
anteayaAJaeger: patchset one looked fine to me18:56
anteayawhere is the change that removes the rule?18:56
AJaegeranteaya: just merged...18:57
AJaegeranteaya: I12adc42629da0089b8520a6f06d3f59d9522fc0218:57
*** yamamoto has joined #openstack-infra18:58
anteayaAJaeger: okay thank you18:59
*** gouthamr has joined #openstack-infra19:00
*** yamamoto has quit IRC19:00
*** ajmiller has quit IRC19:00
*** yamamoto has joined #openstack-infra19:00
*** ajmiller has joined #openstack-infra19:00
*** javeriak has quit IRC19:01
*** rvba has quit IRC19:02
*** csomerville has joined #openstack-infra19:02
*** _nadya_ has quit IRC19:03
*** sdake has joined #openstack-infra19:03
*** sarob has quit IRC19:03
*** cody-somerville has quit IRC19:06
*** Sukhdev has joined #openstack-infra19:06
*** abregman has quit IRC19:08
openstackgerritMerged openstack-infra/project-config: networking-midonet: Make ml2 job non-voting
*** adreznec has quit IRC19:12
*** adreznec has joined #openstack-infra19:13
*** baoli has joined #openstack-infra19:15
*** salv-orlando has joined #openstack-infra19:16
jeblairsdague: when i manually d/l that file i get a correct zip file.  suspect that's pip actually saying "network hiccup; incomplete transmission" but not able to find the right words19:19
rcarrillocruzasselin: you sorted out?19:19
rcarrillocruzwith the cloud launcher, you feed a file which you have your layout of resources19:19
rcarrillocruz*and* you can also keypairs19:19
rcarrillocruzso you can perfectly put there a 'bootstrap' key which you use to create servers19:20
rcarrillocruzand ofc, because ansible modules are idempotent19:20
rcarrillocruzrunning the role against that resources.yml will be too19:20
rcarrillocruznow, the launch-node playbook with the launcher19:20
rcarrillocruzuses the feature of generating a keypair on the fly19:20
rcarrillocruzto have a feature parity with the current launch-node. In the current launch-node, they don't use a 'static' keypair to create servers19:20
rcarrillocruzthey create a keypair on each run19:20
rcarrillocruzwhich is what the 'generate_keypair' flag does, feature parity ^19:21
sdaguejeblair: ok19:21
rcarrillocruzhope it makes sense now, i was off home and was hard to write from my phone19:21
*** ajmiller has quit IRC19:22
*** ajmiller has joined #openstack-infra19:22
*** csomerville has quit IRC19:23
sdaguejeblair: maybe rax ord network issues again?19:25
*** njohnston|afk is now known as njohnston19:25
rcarrillocruzasselin: besides the resources.yml.sample from infra-ansible, you can also check out
sdague - tempest full is a net fail on ansible setup19:25
rcarrillocruzit has the bootstrap key resource i refer about19:26
*** esberglu has quit IRC19:26
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support
notmorganmordred, jeblair, Shrews, nibalizer: ^ alembic migration support. it may need some tweaking still, but pushing it up for eyes.19:28
*** _nadya_ has joined #openstack-infra19:29
*** yamamoto has quit IRC19:29
notmorganmordred, jeblair, Shrews, nibalizer: I am inclined to make this another "hard" break upgrade where dropping the schema is best (or require a direct alembic migration by cli) rather than just assuming the DB is "up-to-date" if it is populated.19:29
mordrednibalizer: as a person on the receiving end of a hard-break upgrade - which of the above do you prefer?19:31
*** Jeffrey4l_ has joined #openstack-infra19:31
*** xarses has quit IRC19:32
mordredone more hard-break to get onto the gravy train of no more hard breaks ... or assume that the current schema is up to date and just stamp it as good and move on19:33
*** jswarren has joined #openstack-infra19:33
notmorgani mean... i could try and get REALLY fancy and handle conditional generated schema upgrades based upon reflection... but it makes my skin crawl19:33
notmorganand i know it's doable...19:33
notmorganjust... please lets not do that :P19:33
nibalizerjswarren: moravec dunno how much scrollback you have19:34
nibalizerbut notmorgan has posted a path
jswarrenI have none, unfortunately19:34
nibalizer to allow database migrations for CDing nodepool, which would help us19:34
*** Jeffrey4l has quit IRC19:34
*** esberglu has joined #openstack-infra19:34
openstackgerritMerged openstack-infra/puppet-ulimit: Fix deprecated warnings
nibalizerso there is some question about the behaviour post-patch, basically if we need to have a hard break19:34
*** ccarmack has joined #openstack-infra19:34
*** _nadya_ has quit IRC19:35
notmorganmordred: the big note here is empty DBs use reflection to create the schema, so if our migration scripts diverge from the actual models... we get in an icky place [but we'd be there anyway] for upgrades19:35
mordredjswarren: tl;dr - two options - assume existing db is up to date, stamp it good so that it's set up for migrations in the future - downside, if your db is _not_ currently up to date this will lead to weird problems in the future19:35
notmorganjswarren: weird, brokeness or not... or who knows what will end up happening19:35
mordredjswarren: or the other option is to make this one more hard break that requires you as a consumer to do a manual task19:35
notmorganit will be weird.19:35
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: have the ptl command parse the liaison wiki page
*** sdake has quit IRC19:36
*** woodster_ has joined #openstack-infra19:36
jswarrenI'm thinking a manual, one-time task would be OK.19:36
*** sdake has joined #openstack-infra19:36
*** amitgandhinz has quit IRC19:36
notmorgansince historically we've said "drop the schema" and let it be recreated -- i think we're better off continuing with that.19:36
notmorganbut like i said... i could do something fancy here if it *really* matters.19:37
mordredcool. that's the safest. and then moving forward after that it'll just magically work19:37
mordrednotmorgan: related - do we need the nodepool-db-manage command? I know it's there, but with the magic auto-upgrade in place, I'm not sure it's a needed thing anymore19:37
notmorganok let me revise this to raise an exception saying OMG NO, if the DB is populated.19:37
*** amitgandhinz has joined #openstack-infra19:37
mordrednotmorgan: ++19:37
notmorganmordred: i figure it never hurts to let people off-line migrate for safety19:37
notmorganmordred: i wouldn't take it away tbh.19:38
* notmorgan thinks other openstack projects should do the auto-upgrade on start if needed [*cough* keystone *cough*]19:38
mordrednotmorgan: keystone maybe ... probably not nova19:38
notmorganbut i know that is not a "good plan"(tm) to spring on people19:38
notmorganmordred: eh, if it has the proper no-downtime-upgrade code -- impact would be minimal19:39
jswarrennotmorgan: is there a risk of orphaned resources if the DB is just wiped out?19:39
mtreinishjeblair: I finally got around to testing your patch and it got past the stacktrace. Now it's working through the ~1400 thing sync queue (kinda slowly, but that's probably just the coffeeshop wifi)19:39
notmorganmordred: ^ defer to you on that one.19:39
notmorgani ... i don't know -- maybe?19:39
*** ajmiller has quit IRC19:39
*** mhickey has quit IRC19:39
mordredjswarren: if they're there, they'll show up in nodepool alien-list ... but i'm going to re-defer to nibalizer19:39
*** baoli has quit IRC19:40
* mordred hands nibalizer the conch19:40
notmorganahhhh alien-list19:40
jswarrenWill we need to manually destroy VMs, I assume images will need to be recreated.19:40
nibalizerim a fan of simple not fancy19:40
mordredso - you could also manually add the alembic migrations table19:40
mordredsince you happen to konw you're up to date19:40
jswarrenOh OK19:41
nibalizeroh yea btw jswarren and moravec are 2 of hte folk i work with at ibm19:41
nibalizerthey are the people who actually had nodepool die on them, my role is more advisory i guess19:41
jswarrenpschwar is our nodepool guy, but he's out at the moment.19:42
mordrednotmorgan: maybe put the "add the migration code if it's not there" logic into nodepool-db-manage? that way an admin who knows what's up can opt-in to running the code to update it19:42
mordredwhich can happen while nodepool is running and then a nodepool restart would be just a normal restart19:42
jswarrenI guess he's phschwartz on here19:42
nibalizeri mean running a nodepool (especially against a derpy cloud) puts you in the position of occasionally having to clean it out, thats just standard operating procedure19:42
mordrednotmorgan: like nodepool-db-manage bootstrap --yes-i-mean-it or something19:43
notmorganmordred: i mean you could just create the table in the DB directly19:43
notmorganmordred: and populate it19:43
nibalizer I don't really have an opinion on how to design it, it makes sense to me that we'd have another breaking change to add alembic support.19:44
pabelangerI've only see tripleo-test-cloud-rh1 require manual deleting recently. I haven't had to do it on our other clouds in a while19:44
nibalizerpabelanger: thats good19:44
notmorgannibalizer: in many cases i wouldn't advocate a breaking change. but right now, we have historically just broken things to add new schema stuff19:44
notmorgannibalizer: this fits the pattern we have adhered to until alembic19:44
notmorgannibalizer: so... i feel better just making this the "last breaking change"19:44
*** csomerville has joined #openstack-infra19:45
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support
nibalizerand jswarren is the operator I came to defend yesterday, so if he's cool with it im cool with it19:45
nibalizerthanks a ton notmorgan and the others who have worked on that patch19:45
sdagueis there a patch up yet to stop doing grenade on liberty?19:46
*** ajmiller has joined #openstack-infra19:46
notmorganmordred: i mean, it is trivial to add "alembic_command.stamp('head')19:46
notmorganmordred: if you feel even vaguely strong about that, i'm happy to do it...but i also like this breaking change because it forces consistency that may have been lost if people did manual sql-updates for say contraint/constraing-names that don't impact runtime but would make future migrations icky19:47
nibalizerjesusaur: you also run a downstream nodepool, opinions?19:48
* jesusaur reads some backscroll19:48
pabelangernotmorgan: ++19:50
*** zeih has joined #openstack-infra19:51
*** ajmiller_ has joined #openstack-infra19:53
*** csomerville has quit IRC19:54
openstackgerritSean Dague proposed openstack-infra/project-config: Don't run grenade on liberty
openstackgerritSean Dague proposed openstack-infra/project-config: remove all reference to partial-ncpu job
sdagueliberty is wedged until the first of those is handled19:54
*** csomerville has joined #openstack-infra19:55
*** zeih has quit IRC19:56
jesusaurnibalizer: notmorgan: i agree that we shouldnt assume that a populated database is up-to-date19:56
*** amitgandhinz has quit IRC19:56
*** ajmiller has quit IRC19:57
jesusaurthis seems like a reasonable path forward19:57
anteayasdague: +219:57
*** Nikolay_St has joined #openstack-infra19:57
*** amitgandhinz has joined #openstack-infra19:57
anteayasdague: do say if you don't get any other takers in the next 15 minutes19:57
*** ajmiller__ has joined #openstack-infra19:58
*** ccarmack has left #openstack-infra19:59
sdagueok, well I'm about to head to beer o'clock. Hopefully if people want liberty working again, they'll notice :)20:01
*** ajmiller_ has quit IRC20:02
anteayasdague: okay cool, have a great weekend20:03
sdagueyou too20:03
nibalizerim gonna be sad when liberty is eol and all these double meanings go away20:03
*** Apoorva has quit IRC20:03
anteayawhat double meanings?20:03
anteayaI'll be glad to see it gone20:04
anteayamaybe next time summit is in Canada we can actually honour Canada when selecting the name20:04
*** Apoorva has joined #openstack-infra20:04
anteayaI hold onto the hope in any case20:05
russellbcanada doesn't like liberty?20:05
mtreinishanteaya: there's no liberty in canada?20:05
russellbmtreinish: i win20:05
*** kzaitsev_mb has joined #openstack-infra20:05
mtreinishI guess there is still a queen :)20:05
mtreinishrussellb: haha20:06
mtreinishyou do20:06
anteayamtreinish: it is an american concept20:06
anteayato the extent is a brand and slogan20:07
clarkbsdague: is gate-greande-dsvm-multinode not a job anymore?20:07
mordredanteaya: well, we did rework the naming process in protest20:08
nibalizerimm having a hard time parsing 334072 tbh20:08
anteayamordred: I saw that and I understand that vancouver is booked for a summit in the future with dates to be determined, hence my hope20:09
* rcarrillocruz loved Vancouver summit, best venue20:10
anteayarcarrillocruz: it really was an amazingly designed building for acoustics20:10
rcarrillocruzbeing able to sit in a table with those views...20:11
anteayabut 8 straight days of sunshine, well, don't expect that to happen twice in a row20:11
anteayayes I do love the mountains20:11
anteayavancouver never gets 8 straight days of sunshine, that weather was extremely unusual20:11
*** ajmiller__ is now known as ajmiller20:11
clarkbsdague: I am trying to figure out why you removed the grenade multinode rules20:12
clarkboh is it because it has the same prefix as the other grenade jobs so the names will match in both cases20:13
*** HeOS has joined #openstack-infra20:15
*** xarses has joined #openstack-infra20:16
fungimordred: jswarren: the alien list is less relevant now that nodepool can automatically clean up leaked instances (identified by the metadata it stashes on them). if you completely wipe out the db, you'll probably have to delete some leaked _images_ but no leaked instances--nodepool should take care of them20:17
clarkband even if you don't delete the db you will likely have leaked images, bceause it turns out we leak images20:18
fungisomebody should really call a plumber20:18
clarkbmake sure they specialize in glance20:18
* nibalizer gets a wrench20:18
clarkbit has to do with the way the glance api works, lack of documentation, and bad assumptions20:19
fungiwith all the leaks we've been dealing with at the house these past couple weeks, i've been starting to joke that buying a house on the water is almost like buying a boat that goes nowhere20:19
clarkbbasically to make an image is a multistep process and if you fail you need to properly unroll any prior successful bits, and none of the clients do that (shade might now though)20:19
clarkbfungi: ouch20:20
*** Nikolay_St has quit IRC20:20
anteayafungi: wow20:20
anteayais part of your house below the water line?20:20
*** vhosakot has quit IRC20:21
funginot quite, but *close*20:21
*** burgerk has quit IRC20:21
anteayaoh my20:21
fungibut these leaks are above-ground. one in the flashing around one of the drain vents up on the roof, a missing patch of roof shingles, a cracked primary condensate tray in the hvac, a leaked supply to one of the toilets, a cracked exterior line out to the dock where we didn't realize one of the valves didn't properly winterize...20:22
fungioh, and the water which was streaming down our chimney in every storm20:22
clarkbsdague: did you want to write the d-g patch to remove the partial ncpu logic?20:22
anteayaoh okay20:22
openstackgerritMerged openstack-infra/project-config: Don't run grenade on liberty
anteayasounds like those things could happen regardless of how close you were to the ocean20:22
anteayastill not fun20:22
fungiso it was sort of _like_ being in a boat. water coming in everywhere20:22
*** akshai has quit IRC20:23
anteayahave you figured out the water down the chimney issue yet?20:23
anteayalike coming down the inside of the chimney?20:23
fungiyeah, we just had them cap off the chimney since we're using propane logs in it20:23
clarkbwe had a largeish amount of rain yesterday and I did not feel like I was in a boat so yay20:23
fungithat put a stop to it20:23
anteayagood to hear it20:24
anteayaclarkb: yay!20:24
anteayaclarkb: how is the house shaping yup?20:24
anteayaany boxless spots yet?20:24
anteayacan you sit in a chair at all?20:24
fungiit does always feel like victory when you have a piece of furniture you can finally sit down on20:25
clarkbanteaya: so many things to do, but we are living in it. The kitchen and living room ar emostly unpacked20:25
clarkbthe dining area has a table in it surrounded by many boxes20:26
clarkbmost of my office is in a closet20:26
*** Daisy has joined #openstack-infra20:26
clarkbbut that is because I sort of want to rearrange how I utilize various pieces of hardware, old desktop will be new server box, old htpc will be new desktop etc20:26
*** sdake has quit IRC20:26
anteayayay unpacked kitchen!20:27
*** amitgandhinz has quit IRC20:27
sc`moving. so fun. i can't wait until i move out of my apartment. have to move a 42u cab out, among other things20:27
anteayaso many things to do, but we are living in it describes anyone living in a house that I've seen20:27
*** amitgandhinz has joined #openstack-infra20:27
*** hashar has quit IRC20:27
rcarrillocruzsc`: i hear ya, i have a 31U cab, and I may eventually move in some months :/20:28
*** sarob has joined #openstack-infra20:28
sc`rcarrillocruz: mine is pure torture. 90s era steel sub microsystems cab :)20:28
sc`not sub20:29
fungisc`: i lugged one around myself for years. finally decided it was no longer owrth the effort to care for (and power, and cool) that much equipment. i was sort of sad to see it go though20:29
rcarrillocruzmine are repurposed old hw crammed on 4U cases :/20:29
*** yamamoto has joined #openstack-infra20:30
*** sarob has quit IRC20:30
sc`fungi: i had a late 80s cab i carried with me across texas. as soon as i moved out of texas, i gave it to someone i knew, fully loaded20:30
clarkbfungi: I am glad to have decided that early on, also I have moved on average once a year for the last 10 years or so20:30
clarkbsuper excited to live in one place for at least 5 years20:30
sc`i like the sun cab just fine, but it may go away if i move more than across town20:30
*** Daisy has quit IRC20:31
sc`or across the bay, as it might be, here20:31
openstackgerritMerged openstack-infra/project-config: remove all reference to partial-ncpu job
clarkbsdague: one suggestion for the localconf change posted20:32
sc`really liking zuul v3 so far. i get my streaming logs back, which is infinitely helpful debugging integration gates20:32
fungisc`: yeah, i gave mine up when we moved out to the beach. to save up before buying we moved into a sub-1ksqft townhouse and were using one of the two bedrooms purely as a storage unit for our stuff. made some pretty drastic choices about what was worth keeping and what i could part with20:35
fungisc`: that's actually zuul v2.5 (really just zuul with added features to allow you to use not-jenkins). but i expect that's a feature which is coming in v3 too20:35
sc`ah. thought i read it was v3 somewhere. my bad. i still like that i get my streaming logs back20:35
*** ajmiller has quit IRC20:36
fungiv3 is still in its own separate feature branch20:36
jeblairmordred, pabelanger, Shrews: what's the sitrep on the thing?20:36
sc`no waiting forever for the failure message if it's already failed20:36
fungikrotscheck: any chance you're already looking into the npm-run job issue mentioned on the -dev ml?20:37
fungii was going to take a peek for obvious causes if not20:37
anteayaclarkb: yay living in the same place for multiple years in a row \o/20:38
*** ajmiller has joined #openstack-infra20:38
fungiclarkb: i'm feeling a similar rush. now that i own a house i'm actually inclined to, like, paint rooms maybe without worrying that i'll have to repaint when my lease is up20:39
*** roxanaghe has quit IRC20:39
Shrewsjeblair: i'm stumped. i think we should try mordred's patch and pabelanger's change (probably separately)20:39
fungifeels kinda like sometimes20:39
jeblairfungi: in america, house owns you20:40
fungitoo true!!!20:40
jeblairShrews: mordred's change is the 'be resilient' change, right?  what's pabelanger's?20:41
*** piet has joined #openstack-infra20:41
fungii hope pabelanger's is called "be vigilant"20:41
clarkbjeblair: the logstash backlog is erratic and we have at least one worker that stopped doing stuff on the 16th with appears related to gear, that look familiar to you at all?20:41
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci
*** yamamoto has quit IRC20:42
clarkbjeblair: looks like underlying network connectivity caused some gear reconnects but then it wait into a wait state that I don't understand20:43
anteayafungi: in a kind of extrapolation that comic also describes any city person that comes to where I live20:43
anteayawhat all this vibrant nature, all these beautiful animals, all this peace and quiet?20:44
anteayalet's destroy that20:44
Shrewsjeblair: mordred's was
jeblairclarkb: i don't see a 'grab_job' packet20:45
*** rvba has joined #openstack-infra20:45
*** rvba has quit IRC20:45
*** rvba has joined #openstack-infra20:45
*** dmk0202 has quit IRC20:46
clarkbnotmorgan: btw exploded spectacularly, not sure if a major issue in keystone unittests but probably worth a look20:46
*** jamesmc__ has quit IRC20:46
clarkbjeblair: I just hosed my browser opening ^ so will be a sec before I can refer back to that paste20:46
clarkbjeblair: I wonder if when the worker goes down on GRAB_JOB if it doesn't resend if servers flap under it?20:48
clarkbmaybe not in all cases?20:48
jeblairpabelanger, Shrews: with 649,  we will emit quite a large number of log lines20:49
jeblairthey'll be uploaded to logs.o.o too20:50
fungijust to refresh my memory on how builders are supposed to work. for a shell builder, unless you set errorexit on a multi-line builder, the builder "fails" if the very last command run exits nonzero, right?20:50
clarkbI am going to restart one of the 4 workers on that host, and see if it persists. Possibly a firewall issue20:50
fungii _think_ that's why the npm-run builder is silently passing stuff which fails `npm lint` but we have an awful lot of multi-line builders which don't set errorexit and might benefit from it if that's the case20:51
*** zeih has joined #openstack-infra20:51
jeblairfungi: they are "#!/bin/bash -x" by default20:52
clarkboh we also need to update these workers to deal with new zuul worker console logs20:52
fungigot it. so no -e20:52
jeblairfungi: i'm pretty sure we matched that to jenkins, but it is ... difficult for me to check that at this point20:52
clarkbjeblair: do the zuul launchers continue to copy logs after the zmq finished event is sent like we did with jenkins console logs? or did that get corrected?20:52
fungijeblair: yeah, that's my recollection from jenkins days, just trying to be sure20:53
jeblairclarkb: zmq completion event will only happen after log copying is complete20:53
clarkbjeblair: woot thanks20:54
fungii guess these npm jobs don't generate subunit streams so i'm not going to find them in the health dashboard. time for graphite20:54
jeblairShrews, pabelanger, mordred: i could get behind adding mordred's patch now20:54
jeblairShrews, pabelanger, mordred: i want someone to say "i have thought about the log explosion issue and i think this is the best thing to do and everything will be fine" before pabelanger's patch though.  :)20:55
*** rfolco has quit IRC20:55
openstackgerritClark Boylan proposed openstack-infra/system-config: Don't retry console log downloads
clarkbjeblair: ^ is the change to take advantage of that new behavior20:55
*** zeih has quit IRC20:56
jeblairclarkb: that is a complicated change20:56
jeblairi mean, for a friday20:56
jeblairis there a good way for me to 'git fetch' this?
clarkbjeblair: git fetch 11bfa671246601c01767914a4f8b28f434fa80a020:58
jeblairoh that works?20:58
Shrewsjeblair: when one of the nodes is held, would it be possible to re-run the same job from the launcher, but by hand and with ANSIBLE_DEBUG=1 turned on?20:58
clarkbit should, you might need to add an https://20:58
fungijeblair: similar to the fetch syntax gerrit suggests for changes... `git fetch refs/changes/18/118818/1` (just substitute the repo url and refname)20:59
jeblairShrews: we can download the ansible directory that the job used and run it from anywhere (including the launcher).  though most jobs (and especially these) are not idempotent, so i expect it to fail quickly20:59
jeblairfungi: yeah, i just didn't think you could fetch a git sha for some reason20:59
fungioh, yeah you definitely can. done it quite a few times21:00
mordredjeblair: here's hoping my bandaid works with the other bandaids21:00
jeblairerror: no such remote ref 11bfa671246601c01767914a4f8b28f434fa80a021:00
mordredand bandaids over the still-unknown-failure21:00
Shrewsjeblair: poo. would be nice if we could capture that somehow21:00
jeblairthat may be why i thought that21:00
fungimight be refs/11bfa671246601c01767914a4f8b28f434fa80a021:00
jeblairnot so much21:00
mordredjeblair: I would do get fetch resilient-async21:01
mordredwhich is the branch it's in21:01
jeblairmordred: neat.  that should work, but i wish i knew how to come up with that on my own21:01
mordredjeblair: I went here:
*** dprince has quit IRC21:01
mordredand then at the bottom of the commit message there is a little branch icon with the name of the branch21:01
jeblairoh there it is, thanks21:02
mordredjeblair: sure! I also find the ui very confusing - I only know this because I went through the same thing a few hours ago grabbing Shrews change21:02
*** sarob has joined #openstack-infra21:02
jeblairokay, i am going to try to write a playbook to rsync this repo onto the launchers21:03
jeblairi think rsync will be way better than a bunch of weird git commands21:03
fungihuh, i could have sworn i've fetched a sha before. this is weird21:04
fungii wonder what i did21:04
anteayaI've seen you fetch a sha21:04
anteayayou have done so before21:04
jeblairi never have21:04
fungidigging in git manpages to try and see21:04
anteayaI would have considered you the kind of person that would21:04
jeblairmind you, i like fungi's world better and would like to live in it21:04
jeblairanteaya: well, in my world, it's not possible21:05
anteayajeblair: ah21:05
fungiyeah, me too. if i could find my world again i'll send you an invite21:05
anteayaha ha ha21:05
*** Daisy has joined #openstack-infra21:05
anteayawear rubber boots21:05
jeblairmaybe you can only fetch git shas one day every 100 years21:05
*** sarob has quit IRC21:07
*** trown is now known as trown|outtypewww21:07
jeblairi *know* there are people in this channel that got that.21:07
*** esberglu has quit IRC21:07
*** sarob_ has joined #openstack-infra21:07
*** sarob_ has quit IRC21:08
*** sarob_ has joined #openstack-infra21:08
mordredjeblair: did lin-manuel write that musical too?21:08
clarkbya pretty sure you can fetch sha1s21:08
clarkbbecause all a ref does is point to a sha121:08
* fungi thought of davey jones21:08
fungiclarkb: now the question is how to make the syntax21:09
jeblairmordred: thank you21:09
*** Daisy has quit IRC21:09
jeblair"fungi lives in brigadoon" is the answer we were looking for21:09
anteayaha ha ha21:09
fungioh, indeed. been forever since i've seen that21:10
*** yamamoto has joined #openstack-infra21:11
*** mriedem has quit IRC21:11
jeblairmordred: i want a playbook that does 1 task on all launchers, then does several more tasks on all the launchers (only after the first task has been done everywhere successfully)21:13
jeblairmordred: what does that look like?21:13
clarkbjeblair: iirc you just add a new top level type section after the first one and specify your hosts again and all that21:14
*** esberglu has joined #openstack-infra21:14
jeblairso the yaml 'file' thing: ---21:14
mordredyou don't need th e---21:14
clarkbyou might not even need that its just another list entry21:14
jeblairno the playbook is a list already21:14
*** esberglu has quit IRC21:14
jeblairso just another element in the top level list21:15
jeblairgot it21:15
mordredeach of the things in the list is a "play"21:15
*** yamamoto has quit IRC21:15
*** piet has quit IRC21:15
*** nwkarsten has quit IRC21:18
*** ldnunes has quit IRC21:18
*** inc0 has quit IRC21:18
*** jbernard has quit IRC21:19
*** nwkarsten has joined #openstack-infra21:19
fungiokay, apparently i'm more insane than i realized. it's not possible without git >=2.5 on the server and uploadpack.allowReachableSHA1InWant set21:19
clarkbclearly we should all upgrade21:19
jeblairwe should upgrade github21:20
clarkbjeblair: yes that, oh wait21:20
jeblairclarkb: to be fair, external parties *have* upgraded github before21:20
*** weshay is now known as weshay_afk21:20
fungiso i'm quite sure i've not interacted with a git 2.5+ server often enough to have been likely to have ever successfully done that thing21:20
jeblairsomething about ruby iirc21:20
*** ajmiller_ has joined #openstack-infra21:21
fungi[new in 2.5] "git upload-pack" that serves "git fetch" can be told to serve commits that are not at the tip of any ref, as long as they are reachable from a ref, with uploadpack.allowReachableSHA1InWant configuration variable.21:22
*** persia has quit IRC21:22
fungiso prior to 2.5, fetch only groks ref tips21:22
*** nwkarsten has quit IRC21:23
fungior rather, the server only knows how to hand them out21:23
fungiif they're ref tips (tags, heads, et cetera)21:23
openstackgerritMerged openstack-infra/system-config: Don't retry console log downloads
*** lascii is now known as alaski21:23
fungiokay, my curiosity is satisfied. the likelihood that i've been hopping between almost identical parallel dimensions increases another notch21:24
*** persia has joined #openstack-infra21:24
clarkbwhat if the sha1 is a ref tip? I guess it would've been in jeblairs case21:25
*** ajmiller has quit IRC21:25
*** aysyd has quit IRC21:25
notmorganhmm... looks like i can't "check" current revision of the db like that... hhmmm21:25
*** ajmiller_ has quit IRC21:27
clarkbwith logstash sorted for the moment /me looks into that keystone derp21:28
*** rlandy has quit IRC21:29
clarkbjeblair: we likely want to put a sanity check into our file uploads, I am up to 175MB and counting on this console log download21:29
clarkbnow over 200MB21:29
clarkbI am impressed with keystones ability to do this21:29
jeblairclarkb: did we have that with jenkins?  i know we put some stuff in the run-tox scripts...21:30
clarkbjeblair: I am not sure we did with jenkins21:30
jeblairi'm going to hard-restart all the zuul launchers with mordred's change now21:30
jeblairwell first21:31
jeblairhow does that playbook look?21:31
jeblairclarkb, mordred: ^21:31
*** ajmiller has joined #openstack-infra21:32
jeblair(i ran the first play separately, and the resulting git repo lgtm21:32
stevemarclarkb: what did we do now? :)21:32
fungistevemar: manufacture very large job console logs?21:33
clarkbjeblair: yes lgtm21:33
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support
clarkbstevemar: be warned its over 500MB now21:33
jeblairhere goes21:33
* clarkb is happy he doesn't have a bw quota to worry about21:33
clarkbstevemar: not sure if it is only that change or if larger scale21:34
clarkbbut in general if your unittests produce that much output its unhappytimes21:34
clarkbstill waiting for my download to finish so I can dig in and see what happened21:34
stevemartheres a lot of
clarkbits that create grant for miles21:35
stevemarFile "keystone/assignment/backends/", line 458, in create_grant domain_id, project_id, inherited_to_projects)21:35
stevemarclarkb: god damn revoke api21:36
* stevemar closed the tab21:36
*** gouthamr has quit IRC21:36
clarkbstevemar: this change does modify that file21:36
jeblairdoesn't python have a 1000-frame stack limit?21:36
jeblairoh, is it one of those for every test?21:36
stevemarjeblair: maybe not *every*21:37
clarkbjeblair: ya the actual error is the stack limit being hit21:37
clarkbso every one of the failed tests writs ~2k lines21:37
stevemarclarkb: slapped a -2 on it, can you kill the job?21:38
jeblairpabelanger, mordred, Shrews, clarkb, EmilienM: i have restarted all the zuul launchers with mordred's ansible change (which we also expect to reduce incidences of the error).  i will delete any held nodes now.21:38
clarkb is the ps that exploded21:38
clarkbstevemar: oh the job is long dead21:38
clarkbstevemar: and the explosion happened on an older ps21:38
stevemarclarkb: ah21:38
stevemaryeah, i see that now, 22 vs 1621:38
clarkbstevemar: what I am trying to figure out is if this is more persistent but probably isn't, so if we can see that latest ps works then maybe we are ok21:39
clarkboh lol I see the bug ya its related to that ps21:39
clarkbthere is no check to stop recursing so its an infinite loop (except for stack limits)21:39
* clarkb looks at latest ps21:39
fungiplaying around with graphite, it looks like the last time gate-horizon-npm-run-lint failed in the check pipeline was late on june 17, and i don't see any obvious changes to their job configuration around that timeframe21:40
clarkbstevemar: ok I thin kwe are fine now, probably do want to have a sanity check somewhere for this though21:40
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support
EmilienMjeblair: thanks for the update21:41
* clarkb wonders why people don't run tox locally first21:41
fungiso around june 18th it seems like that job started passing always even when it should fail21:41
fungiclarkb: or `npm lint` in this case21:42
fungi(it doesn't use tox)21:42
clarkbfungi: ya thinking of the infinite recursion make half a gig of log data case21:42
fungioh, hah21:42
fungiso that's what was going on with the huge keystone log?21:43
fungia loop/recursion?21:43
*** amitgandhinz has quit IRC21:43
fungifun stuff21:43
*** matt-borland has quit IRC21:43
stevemarfungi: we excel in fun21:43
clarkbyup there was a recursive call with no break, it hit stack limits which made massive tracebacks which happened over and over in many tests21:43
clarkbits unfortunate that Turing guy figured out we can't test for this21:43
*** claudiub|2 has joined #openstack-infra21:44
clarkbwe could run a gate-halting on all projects21:44
*** PollyZ has joined #openstack-infra21:44
fungiyeah, someone should really do something about him21:44
stevemarwe should create a test after him, a Turing test maybe21:45
jeblairfungi: june 16 was no-more-jenkins day21:46
fungijeblair: yep, i don't think the npm job situation is related because your message went out late on the 16th and we had failures through late the next day according to graphite21:47
jeblairfungi: so if you are certain the correct behavior manifested between june 16 and june 18, that should exclude zuul-launcher behavior change as a potential cause21:47
clarkbwow its only 2:45pm, this waking up at 3am business has me all confused21:47
jeblair21:47 < clarkb> wow its only 2:45pm, this waking up at 3am business has me all confused21:47
pabelangerjeblair: re: 649. Let me look into our logging issue more. And possibly propose a change21:48
anteayaclarkb: did you wake up at 3am again today?21:48
jeblairthat's some lag :)21:48
clarkbanteaya: no but everything is wonky21:48
fungier, i was misreading the graph, so failures up to nearly noon utc on the 18th i think?
anteayaclarkb: ah21:48
jeblairfungi: agreed21:49
*** thorst_ has quit IRC21:49
fungianyway, i need to go make pizza. this dough isn't gonna toss itself21:49
fungiback in a while21:49
*** thorst has joined #openstack-infra21:50
anteayahappy pizza21:50
*** mdrabe has quit IRC21:50
*** bauwser is now known as bauza21:52
*** bauza is now known as bauzas21:52
*** zeih has joined #openstack-infra21:52
*** gouthamr has joined #openstack-infra21:53
*** gouthamr_ has joined #openstack-infra21:55
*** roxanaghe has joined #openstack-infra21:56
*** zeih has quit IRC21:56
*** roxanaghe has quit IRC21:56
*** gouthamr has quit IRC21:58
*** Apoorva has quit IRC21:58
*** Apoorva has joined #openstack-infra21:58
*** thorst has quit IRC21:58
*** roxanaghe has joined #openstack-infra21:59
clarkbok I think logstash setup is happyness now22:00
anteayayay happyness22:03
*** dimtruck is now known as zz_dimtruck22:05
*** csomerville has quit IRC22:06
anteayazaro: are you around today at all? I'm wondering if you have any thoughts on SotK's last comment on this patch:
anteayazaro: you had said you are way next week, I wonder if you are here today or gone already22:07
*** sdague has quit IRC22:08
*** pradk has quit IRC22:12
*** ajmiller has quit IRC22:13
*** harlowja has quit IRC22:17
*** tpsilva has quit IRC22:18
pabelangerjeblair: it looks like FAILED - RETRYING: TASK: command (1 retries left) loggings lines are removed in the latest version of ansible. So, long term, I don't think it is an issue for us. was the reported issue.22:19
clarkbdoing more digging into the logstash stuff we may want to consider adding the zuul launcher info to the event (or maybe its there and I just have to inject it into ES)22:19
clarkband I should probably stop using type == jenkins :)22:19
anteayaclarkb: meaning zuul launcher id number?22:20
pabelangerjeblair: I'll test in a bit with latest ansible to confirm22:20
clarkbanteaya: the hostname or something like that ya22:20
clarkbalso build_status is now a json dict rather than just SUCCESS or FAILURE22:20
clarkbjeblair: ^ I think that is a slight api change in the zmq events22:20
jeblairclarkb: what's build_status?22:21
clarkbjeblair: was SUCCESS, FAILURE, ABORTED before iirc22:21
jeblairclarkb: oh hrm,22:22
anteayathat sounds reasonable to me22:22
jeblairclarkb: it may end up as a dict22:22
*** rhochmuth has quit IRC22:22
clarkbjeblair: fields["build_status"] = event["build"].get("status", "UNKNOWN") is what we do to get it from zmq22:23
clarkbjeblair: and {"result": "SUCCESS"} is what we get from that22:23
*** ajmiller has joined #openstack-infra22:24
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api
*** rbrndt_ has quit IRC22:26
*** PollyZ_ has joined #openstack-infra22:28
*** harlowja has joined #openstack-infra22:30
*** PollyZ has quit IRC22:32
*** edmondsw has quit IRC22:32
zarofungi: this should have turned off puppet on review-dev correct?
zarofungi: i'm seeing files getting reverted again.  was wondering how to verify that it's off?22:35
zaroanteaya: i'm here22:36
*** PollyZ_ has quit IRC22:36
*** raunak has quit IRC22:37
anteayazaro: ah thank you22:38
anteayazaro: the bit we cant' seem to agree on is the response examples, are you okay if I leave that for now so we can merge the command examples?22:39
anteayazaro: I'd like to get this first section merged then move onto other sections22:39
anteayawe can discuss the response examples later in follow up patches?22:39
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: clean up result handling
clarkbwoo e-r grpahs are all ancient /me looks at that next22:40
jeblairclarkb: ^22:40
jeblairzaro: there was an error in ansible this morning that would have caused that, i believe it has been corrected22:40
clarkbjeblair: danke22:41
*** mriedem has joined #openstack-infra22:41
jeblairclarkb: will the log processor handle  {"build": {"status": null}} ?22:41
clarkbjeblair: I think it will, if that json parses null into None then we will set it to the strin "UNKNOWN"22:42
jeblairclarkb: yes, json will do that22:42
clarkbif it parses it into the string "null" then we will set it to "null:22:42
clarkbcool so that should be fine22:42
fungizaro: yeah, all the servers we disabled ansible/puppet for in git got updated when something caused ansible to not expand our host groups (and "disabled" is one of our groups)22:42
jeblairclarkb: oh wait, but it looks for 'ABORTED' right?22:43
clarkbthats curious we have a bunch of flock processes with hour long waits that are way older than an hour22:43
clarkbjeblair: it will ignore ABORTED ya22:44
jeblairclarkb: we don't send 'ABORTED' anymore -- it'll be None instead in those cases22:44
clarkbok, we can update the workers22:44
jeblairclarkb: (basically, a zuul abort is indistinguishable from an operator-triggered abort)22:44
zaroanteaya: sure, i'm flexible as comment on the change22:44
*** rajinir has quit IRC22:44
jeblair(and they both result in None)22:44
anteayazaro: oh thank you22:45
*** mrhillsman is now known as mrhillsman_22:46
*** yamahata has quit IRC22:47
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Handle zuul launcher aborted jobs
clarkbjeblair: ^ that should do it22:47
*** Goneri has quit IRC22:48
jeblairclarkb: er, actually, i think build_status will be None there22:49
zarowell, that was good to know. thanks.22:49
jeblairclarkb: because .get will succesfully return the value in the dict -- None22:49
clarkbjeblair: oh huh22:50
clarkbright .get only returns if the key doesn't exist22:50
clarkbin this case the key does exist22:50
jeblairclarkb: we could have zuul launcher not return a status instead of saying 'null'22:50
jeblairor we could check for that in the log processor22:50
clarkbthe current thing is fine, and we can just check for the serialized "None"22:50
jeblairit doesn't really matter; the interface here is vague and zuul scheduler handles both22:51
clarkbsince the gearman stuff should serialize it to None22:51
clarkber "None"22:51
*** ajmiller has quit IRC22:51
*** ajmiller has joined #openstack-infra22:52
clarkboh wait no because its json again22:52
*** raunak has joined #openstack-infra22:52
*** ajmiller has quit IRC22:52
jeblairyeah, we should end up with None in both places i think22:52
clarkbya job = gear.Job(b'push-log', json.dumps(output).encode('utf8'))22:53
*** zeih has joined #openstack-infra22:53
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Handle zuul launcher aborted jobs
clarkbI hvae killed the oldest (from june 9th) flocked process on status.o.o hoping that the e-r things will start flowing now22:55
*** gordc has quit IRC22:55
*** thorst has joined #openstack-infra22:57
*** Apoorva has quit IRC22:57
*** zeih has quit IRC22:57
*** rvba has quit IRC22:58
*** salv-orl_ has joined #openstack-infra22:58
*** salv-orlando has quit IRC23:01
*** markvoelker has quit IRC23:01
*** yamahata has joined #openstack-infra23:02
*** markvoelker has joined #openstack-infra23:02
*** thorst has quit IRC23:04
*** ams__ has quit IRC23:04
clarkbjeblair: ok is working agian and you can see the dicts there, but those will filter out over the next 10 days or so once we get these changes in23:08
clarkbjeblair: one neat thing is the lack of jenkins slaves went offline :)23:09
clarkbgit connectivity seems to be our biggest issue in the env according to e-r now23:10
*** amitgandhinz has joined #openstack-infra23:14
anteayaclarkb: do we need more git servers in the farm?23:15
clarkbanteaya: no I think it is the rax ord connectivity issues23:15
*** amitgandhinz has quit IRC23:18
*** nwkarsten has joined #openstack-infra23:21
*** hongbin has quit IRC23:22
*** nwkarsten has quit IRC23:26
*** markvoelker has quit IRC23:26
rm_workHey, can the job timeout for our gate jobs be configured on an individual job basis?23:35
rm_workMy understanding was that it wasn't something we were able to change on our own23:35
rm_workIs there a reason we can't just set our job timeout to, say, 120 minutes?23:36
fungirm_work: yeah, the timeout is set per job or job-template in the jenkins/jobs yaml configuration files23:37
rm_workOK, so there's no reason we couldn't turn it up a bit?23:37
rm_work(like, a political reason, if not a technical one)23:37
*** superdan is now known as dansmith23:38
rm_workAny policy that says "don't use over <X> timeout value"?23:38
fungiwell, there's always the "why are our jobs taking longer? dunno... let's just up the timeout!" factor23:38
rm_workheh, yes23:38
rm_workbut if we KNOW why, and need the timeout higher, it's possible?23:38
rm_workI wonder why I thought we couldn't do that23:39
rm_workI swear we investigated it and were told we couldn't do it23:39
fungilike, you should _understand_ why your jobs take that long and be sure you can't improve your job performance before you consider just letting them run longer23:39
JayFThere's also the "how long do you want to wait for things to check / merge" factor23:40
fungialso if your job is instantiated from a job-template shared by other jobs/projects you may need to copy that template to a slightly different name23:40
JayFpart of why we did the work to stabilize ironic jobs before going with grenade, b/c they already take a while and rechecks hurt23:40
fungiand if your project has these jobs running in a shared gate queue with other projects, having jobs take too long somewhat slows down gating for other projects23:42
rm_workThis is related to "octavia jobs need to spin up nested VMs and there's no vt-x support in the gate"23:42
rm_workwe split into like 5 jobs to run everything in parallel23:42
*** sdague has joined #openstack-infra23:43
rm_workbut even so, some of those are getting precariously close to timeout, and we don't even have many tests in each yet23:43
JayFsimilar to the problems ironic has had, we just had to chip away at it and do some optimizations to make it faster23:43
JayFI'm not familiar with octavia much, but I know using the virtio network driver in qemu and using ipxe for most pxe booting helped save us a little bit of time23:44
JayFseems like that might not be relevant for you all though, doesn't look liek you do much pxe booting23:44
rm_workprobably worth looking into23:44
rm_workno, we do not23:44
rm_workthe virtio maybe?23:45
JayFif you're just testing an lb, is there any way to fake out the servers instead of booting them?23:45
rm_workdoes that increase boot-time?23:45
rm_workthese are our functional tests23:45
*** thingee has quit IRC23:45
rm_workwe need to make sure that the configs are generated and shipped correctly, AND that the agent inside the vm sets them up right, and that haproxy actually starts and loadbalances23:45
JayFAh. Well good luck then, sounds like you have a similar but non-overlapping problem to Ironic.23:45
JayFcould you use something like a container instead?23:45
rm_workWe'd LOVE to do that23:46
JayFThat would likely move a bit faster than a full VM, if that's what you're having to do.23:46
JayFin Ironic we have to spawn the vm because pxe booting and other shenanigans23:46
rm_workthen we'd need support for containers, which is... not there yet23:46
rm_workbut also that wouldn't test our VM driver :/23:46
fungian option to avoid the slows-down-gating-for-other-projects factor (if that's an issue) is to run those jobs _only_ in the check pipeline23:46
*** roxanaghe has quit IRC23:46
jeblairrm_work: yeah, we've had a 60 minute target for a while23:46
rm_workyeah fungi i think that's probably reasonable23:46
fungibut that's usually a bad idea for letting deadlocks slip through, so best avoided if it can be23:47
*** salv-orl_ has quit IRC23:47
rm_workjeblair: i remember the global default was lowered a while back from like 80?23:47
jeblairrm_work: i beleve the *default* is 30 minutes23:47
rm_workI might look at bumping ours back to like 90m though23:47
rm_worklol ok23:47
rm_workit takes 30m just for devstack to fully spin up IME23:48
jeblairwell, that's for all jobs, only about 50% of which are devstack23:48
rm_workI always laugh at the "this could take up to five minutes" line >_<23:48
clarkbrm_work: devstack is down to about 1 minutes now23:48
clarkber 1023:48
*** woodster_ has quit IRC23:49
rm_workwell, we have to build a VM image as part of our stack plugin, which means an extra 10m or so23:49
rm_workand we rely on many projects so they have to install too23:49
fungifor example, if your pep8 job runs longer than the default 30-minute timeout, you might be thankful we don't let that keep going23:49
rm_workour gate installs like ... nova, glance, keystone, neutron, neutron-lbaas, octavia, barbican and maybe a couple other things i'm not remembering from the base23:49
*** adam_g has quit IRC23:50
rm_workheh yeah i think pep8 is much lower ^_^23:50
fungiyou can also trim down what services are started in d-g if that helps at all23:50
*** thingee has joined #openstack-infra23:51
clarkbalso does novas container stuff not work?23:52
clarkbin theory you could have a container flavor and boot those and you wouldnt know the difference from an api perspective23:52
fungias in nova-docker?23:53
rm_workI thought that wasn't ... "official"23:54
*** zeih has joined #openstack-infra23:54
rm_workI think it might do network plugging differently? which affects us23:54
rm_workwe do manual networking stuff23:54
rm_workand the agent is fairly aware of what system it's running inside, so we'd need a slightly different driver for it23:54
clarkbeither nova docker or nova lxc23:54
rm_workwe'd LOVE to have containers working, but it's not a simple undertaking23:55
rm_workit's definitely on our roadmap though23:55
clarkbyou do more than just listen on interface with haproxy and check another service somewhere?23:55
rm_workwe have an agent that runs inside the Amphora23:56
rm_workVM/container/whatever it23:56
rm_work*it is23:56
clarkbya but why does it care about the external networking23:56
clarkbif you have tcp/udp it shpuld be happu no?23:56
* fungi loves the word "amphora"23:56
rm_workwe plug/unplug networks with neutron23:56
rm_workand use namespaces internally23:56
rm_workthe agent facilitates that23:57
rm_workwell, along with changes in our custom image23:57
rm_worki don't know how the images work for containers23:57
rm_workI'm guessing we'd need to build a different type of image for it to work as a container image?23:57
clarkbyou shouldnt need to23:58
funginot necessarily, no23:58
*** sdake has joined #openstack-infra23:58
rm_workso we build an ubuntu 14.04 image with DIB, we could just boot that as a container?23:58
*** zeih has quit IRC23:58
clarkbit will just ignore ypur kernel23:58
fungithe main difference is that the kernel/bootloader in your image is irrelevant in a container, i think?23:58
*** Sukhdev has quit IRC23:58
rm_workinteresting, i wonder how difficult it'd be to test that23:58
rm_workif i can just ... enable the lxc/docker flavor easily and switch the flavor tag we use....23:59
rm_workI *feel* like there's more to it, but23:59
*** SumitNaiksatam has quit IRC23:59
rm_worki'm willing to try it out, for sure23:59
* clarkb wonder why you need to muck with neutron in theory you just need tcp/udp and or other l3 connectivity to lb23:59

Generated by 2.14.0 by Marius Gedminas - find it at!