Wednesday, 2016-08-03

mtreinishfungi: I could have messed it up, IIRC I copied it from puppet-elastic_recheck though00:00
*** rossella_s has quit IRC00:00
fungii think something has probably gone sideways with my ipv6 tunnel which is causing an interactive editor over ssh to puppetmaster to just hang at arbitrary points00:00
pabelangerjeblair: ++00:00
*** rossella_s has joined #openstack-infra00:00
jeblairfungi: want me to type for you?00:00
*** gildub has joined #openstack-infra00:00
mtreinishfungi: yep:
fungiany change in the editor which shifts a significant number of characters pauses for 10+ seconds (sometimes indefinitely)00:00
fungijeblair: i think i've got it now00:01
fungimtreinish: yeah, we're just inconsistent about how we do it across multiple modules and need to standardize at some point00:02
*** sflanigan has quit IRC00:03
fungiokay, so the reason to add the ip addresses along with the hostname is to prevent ssh from adding entries for the ip addresses on additional lines in known_hosts (which puppet will then keep resetting)00:03
jeblairfungi: i'm not seeing the inconsistency00:04
jeblairmaybe i'm missing something00:04
fungijeblair: i could have sworn in some places we templated the ip addresses so we didn't duplicate them all over the place00:04
fungibut also we'd be better off putting one entry in public hiera for this rather than individual copies in group and fqdn hiera files00:05
jeblairfungi: i see review, status, zuul, and zuul-mergers using the hiera key 'gerrit_ssh_rsa_pubkey_contents' which contains only "ssh-rsa <base64...> comment"00:05
jeblairfungi: and their uses in system-config templating into them with copied (and sometimes wrong, but that's beside the point here) values for the ip addresses/hostname00:06
jeblairmodules/openstack_project/manifests/zuul_prod.pp:    known_hosts_content            => ",,2001:4800:7818:102:be76:4eff:fe05:9b12 ${gerrit_ssh_host_key}",00:06
jeblairlike that00:06
fungijeblair: yeah, we have other modules just stuffing all the content from hiera (like the one for the firehose server)00:06
fungiincluding hostname and (now) ip addresses00:07
jeblairfungi: that shouldn't be in the modules00:07
fungiactually the hiera entry when i found it didn't even have the hostname00:07
fungibut expected it00:07
fungijeblair: agreed00:07
fungiwhich is what i meant by inconsistency00:07
*** amitgandhinz has joined #openstack-infra00:08
fungiaccording to mtreinish the elastic_recheck module does the same way, though i haven't looked yet00:08
*** pahuang has joined #openstack-infra00:08
fungilooking to see how to make them consistent with the ones you mentioned00:09
jeblairfungi: the er module expects both a host and a host key parameter, then concatenates them00:10
jeblairunfortunately, it also uses that parameter for other things00:11
jeblairso it will not support an 'ipv4,ipv6' formate00:11
*** mtanino has quit IRC00:11
jeblairi think it would be best to have both puppet-elastic_recheck and puppet-germqtt just take 'known_hosts_content' as a single string like the other modules00:12
*** rlandy has quit IRC00:12
*** amitgandhinz has quit IRC00:12
fungialso it looks like openssh would prefer the format to be more like... []:29418,[2001:4800:7819:103:be76:4eff:fe05:8525]:29418,[]:29418 ssh-rsa AAA...00:12
fungi(port-specific entries)00:12
jeblairi think that's step 1, then step 2 is update system-config to configure those servers just like the others, then step 3 is to move the whole string blob into public hiera to make it easier to deal with00:13
fungiyep, i concur00:13
fungiworking on that now00:13
fungimtreinish: in the meantime, any content coming from the firehose now?00:13
jeblairfungi: so probably add a new argument "known_hosts_content=''" and if it has a value, have it supercede the old behavior, then later drop the old argument00:14
fungiyep, that should be safe00:14
mtreinishfungi: not that at I can see00:15
*** esberglu has quit IRC00:15
pabelangerjeblair: do you have a moment to confirm 349119 was what you were suggesting in your comments from 34911900:16
*** piet has quit IRC00:16
fungimtreinish: the last time i restarted germqtt (~12 minutes ago) the only thing it logged was... No handlers could be found for logger "gerrit.GerritWatcher"00:16
mtreinishfungi: hmm, not seeing anything00:17
mtreinishis there is anything in the mosquitto log?00:17
mtreinishmaybe now it's getting events and not able to write them to mqtt00:18
*** sflanigan has joined #openstack-infra00:18
fungimtreinish: it's logging new connections/closes and saving its in-memory database. no errors about events (or anything about events)00:20
jeblairpabelanger: yes, but see comment00:20
fungimtreinish: though also gerrit show-connections isn't mentioning an active connection from germqtt00:20
pabelangerjeblair: sure, changing00:21
mtreinishfungi: hmm, the connections are probably me constantly reconnecting00:21
mtreinishare they all from a client id with sazabi in the name?00:21
mtreinishfungi: hmm, that probably means germqtt isn't working00:22
mtreinishI really should add better logging to it00:22
fungii'm systemd fumbling trying to find where/how the germqtt service is defined00:22
mtreinishas opposed to the none it has now :)00:22
mtreinishit's in /etc/systemd/system/germqtt.service00:22
mtreinishI had to write it by hand00:22
fungiaha, thanks00:22
fungii've switched to running `/usr/local/bin/germqtt /etc/germqtt.conf --foreground` as the germqtt user to see if i get any better feedback00:23
fungiso far the only log entry is the one about the logger handler00:23
fungiNo handlers could be found for logger "gerrit.GerritWatcher"00:24
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Include subnodes in updateStats function
mtreinishfungi: yeah it has 0 logging right now, so there isn't likely gonna be anything useful unless there is an error message00:25
jeblairpabelanger: +200:25
pabelangerjeblair: great, thanks for the help00:25
fungimtreinish: it looks like it's connecting to gerrit over and over every ~5 seconds, according to tcpdump00:28
fungii'll check gerrit's logs00:29
mtreinishfungi: did doing an ssh to review.o.o with the germqtt user show anything. I was normally able to debug issues doing that00:31
*** thorst has joined #openstack-infra00:31
fungimtreinish: that worked. it's how i was diagnosing the known_hosts stuff00:31
*** xarses has joined #openstack-infra00:32
*** rfolco_ has joined #openstack-infra00:32
*** rfolco_ has quit IRC00:32
fungi`ssh -p 29418 gerrit stream-events` as the germqtt user on firehose01 works fine00:33
fungiand i see gerrit events00:33
*** hichihara has joined #openstack-infra00:33
*** baoli has joined #openstack-infra00:33
mtreinishhmm, I wonder what's up with gerritlib doing it then...00:33
*** kaisers_ has joined #openstack-infra00:34
fungimtreinish: here's the gerrit section from germqtt.conf on the server
jeblairnow might be a good time to add that logging00:34
jeblairlogging.basicConfig when running --foreground might be sufficient00:35
fungimtreinish: looks like the template says $gerrit_key belongs on that last line, and is empty00:35
fungimtreinish: would that cause a problem?00:35
mtreinishfungi: yeah, that needs to point the public key file00:36
mtreinishdid I make a typo?00:36
fungigerrit_key doesn't seem to be a class parameter that i can find00:36
mtreinishfungi: oh, that's probably from an earlier rev when I thought it was gonna copy the file around as a param00:37
mtreinishuntil I looked at how e-r's puppet did it00:37
fungihrm, another bug for you... germqtt doesn't handle sigint very well (or at all as far as i can tell)00:37
mtreinishso we should change that to /home/germqtt/.ssh/id_rsa00:37
mtreinishfungi: it's the gerritlib watcher thread00:38
mtreinishit waits for that to die00:38
fungimtreinish: i wonder, shouldn't gerritlib default to using ~/.ssh/id_rsa anyway?00:38
*** kaisers_ has quit IRC00:38
fungidoes it really need that specified?00:38
*** thorst_ has joined #openstack-infra00:39
mtreinishfungi: probably not, but that requires a germqtt change00:39
mtreinishsince it is a required param right now00:40
*** apetrich has quit IRC00:40
mtreinishfungi: i'm getting events now \o/00:40
fungiokay, looks like we have a live gerrit connection00:40
fungithat's after fixing the config manually00:41
*** thorst has quit IRC00:41
*** thorst_ is now known as thorst00:41
*** karthik__ has quit IRC00:42
openstackgerritMatthew Treinish proposed openstack-infra/puppet-germqtt: Fix path for ssh key file
mtreinishfungi: ^^^00:42
*** apetrich has joined #openstack-infra00:42
mtreinishfungi: although it doesn't look like the websocket port is working00:43
mtreinishoh, actually that's probably a client config issue. Lemme double check00:43
fungii'll work on the ssh known_hosts cleanup across our puppet modules, but probably not tonight at this point00:43
mtreinisherr, no it looks right00:44
openstackgerritBrad Behle proposed openstack-infra/project-config: Make ovn-install-kuryr job voting
fungistill have some non-openstack things i need to get done with what's left of my evening00:44
*** dprince has quit IRC00:44
*** baoli has quit IRC00:45
mtreinishfungi: sure, thanks for the help.00:45
*** sarob has quit IRC00:47
fungimtreinish: strangely, i still see connections being made to gerrit every 5 seconds according to tcpdump00:47
fungiwhich don't seem to be showing up in gerrit's ssh log00:47
mtreinishhmm, that's odd00:47
fungifiltering on just packets with "Flags [S]" there's one to 2001:4800:7819:103:be76:4eff:fe05:8525.29418 every 5 seconds00:48
fungifrom various (increasing) ephemeral local ports00:49
funginetstat shows a dozen or so of them in TIME_WAIT state00:50
mtreinishfungi: hmm, I don't know what could be causing that00:50
fungiyou're still getting events?00:50
*** _sarob has joined #openstack-infra00:50
mtreinishfungi: also looks like the event stream shas stopped00:50
fungioh, maybe this is connected00:51
*** gongysh_ has joined #openstack-infra00:51
fungiyeah, i see no established ssh connection in gerrit's connections list for the germqtt user now00:51
fungiso something caused it to die00:51
fungiaha, i bet puppet reconfigured it00:51
fungithe config is missing its key path again00:52
mtreinishfungi: haha00:52
fungiokay, i'll knock off for now. this should in theory correct itself00:52
fungionce yuor patch merges00:52
*** hichihara has quit IRC00:53
*** hichihara has joined #openstack-infra00:53
mtreinishfungi: heh, hopefully at least :)00:53
*** sarob has joined #openstack-infra00:54
fungijeblair: on the list of inconsistent known_hosts management for gerrit access, proposal.slave has no ~jenkins/.ssh/known_hosts at all00:54
*** _sarob has quit IRC00:55
*** karthik__ has joined #openstack-infra00:56
fungi(nor does because i forgot to add it until now)00:57
fungithis is unfortunate00:57
*** sdake has quit IRC00:57
fungi~jenkins/.ssh/config on proposal.slave has UserKnownHostsFile=/dev/null and StrictHostKeyChecking=no00:58
fungii'll look into this in more detail in the morning00:58
fungisame on signing01.ci00:58
*** sarob has quit IRC00:58
*** sdake has joined #openstack-infra00:59
fungilooks like puppet-jenkins sets it01:00
fungispecifically the jenkins::jenkinsuser class01:00
fungiguess i'll work on that tomorrow too01:00
funginight all01:01
jeblairfungi: you may need to ask <> why that is01:01
jeblairfungi: good luck getting in touch with that guy!01:01
fungigit blame to teh rescue!01:01
fungisomething tells me that e-mail address no work good01:01
jeblairthat commit includes words like "orchestra" "tarmac" "lvm snapshot" and "cobbler"01:02
jeblairthat's like 4 blasts from the past01:02
funginow i'm gonna have nighthmares01:02
*** tphummel has quit IRC01:02
*** tphummel has joined #openstack-infra01:03
fungithankfully all things you had fixed by the time i came onto the scene01:03
fungi(for various definitions of "fixed" anyway)01:03
jeblairto be fair, i *created* "lvm snapshot", but also got rid of it :)01:03
*** ramishra has quit IRC01:04
jeblairfungi: the upshot i see from this is that there is that there is that the reasoning that led to "Remove known_hosts file, add .ssh/config file that ignores known_hosts." from that commit is likely no longer relevant, and i think it should be safe to alter our approach.01:04
fungii'll guessing this was around the same timeframe as the gpg signing key i just removed from jenkinsuser01:05
fungifor the packages in our ppa01:05
*** flyingtt has quit IRC01:05
jeblairindeed :)01:05
jeblairalso, that was a fantastic sentence i just constructed.  i don't think i can top that, so i think i'll go do something else now.01:06
fungiwondering if we have any jobs which have grown a dependence on this configuration now though01:06
fungilike, that ssh to something on localhost as the jenkins user01:06
fungiand are blissfully ignorant of any need to handle host keys as a result01:07
*** karthik__ has quit IRC01:07
fungiso if we just rip it out, we could see some unfortunateness in some jobs, but i can't think of a way at the moment to find and fix those without breaking them01:08
*** tphummel has quit IRC01:08
*** pvaneck has quit IRC01:08
fungior i could just make it conditional and only apply to single-use nodes01:08
fungiso we can do proper host key management on our long-lived nodes01:08
*** amitgandhinz has joined #openstack-infra01:09
fungianyway, topics for tomorrow. really disappearing now. night!01:09
*** zhurong has joined #openstack-infra01:10
*** psilvad has joined #openstack-infra01:10
*** Sukhdev has quit IRC01:12
*** shashank_hegde has quit IRC01:12
*** Apoorva has quit IRC01:12
*** amitgandhinz has quit IRC01:13
*** mriedem has quit IRC01:19
*** karthik__ has joined #openstack-infra01:19
*** akshai has joined #openstack-infra01:23
*** Apoorva has joined #openstack-infra01:23
*** dimtruck is now known as zz_dimtruck01:23
*** tqtran has quit IRC01:23
*** hichihara has quit IRC01:24
*** hichihara has joined #openstack-infra01:24
*** cody-somerville has joined #openstack-infra01:26
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Deleting files surplus to this module.
*** akshai has quit IRC01:35
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** moravec has joined #openstack-infra01:40
*** Apoorva_ has joined #openstack-infra01:40
*** sdake has quit IRC01:44
*** Apoorva has quit IRC01:44
*** Apoorva_ has quit IRC01:44
prometheanfiregate stalled?01:46
*** thorst has quit IRC01:47
*** thorst has joined #openstack-infra01:48
*** baoli has joined #openstack-infra01:49
*** itisha has quit IRC01:50
*** yanyanhu has joined #openstack-infra01:52
*** karthik__ has quit IRC01:53
*** cody-somerville has quit IRC01:55
*** pt_15 has quit IRC01:56
*** thorst has quit IRC01:56
*** zz_dimtruck is now known as dimtruck01:58
*** cody-somerville has joined #openstack-infra01:59
*** shashank_hegde has joined #openstack-infra02:00
*** amitgandhinz has joined #openstack-infra02:09
*** tphummel has joined #openstack-infra02:10
*** psilvad has quit IRC02:11
*** pahuang has quit IRC02:14
*** amitgandhinz has quit IRC02:14
*** gildub has quit IRC02:17
openstackgerritDean Troyer proposed openstack/os-client-config: Fix precedence for pass-in options
*** coolsvap has joined #openstack-infra02:19
*** ramishra has joined #openstack-infra02:21
*** kaisers_ has joined #openstack-infra02:23
*** gouthamr has joined #openstack-infra02:24
*** ramishra has quit IRC02:26
*** gouthamr_ has joined #openstack-infra02:27
*** kaisers_ has quit IRC02:27
*** ramishra has joined #openstack-infra02:28
*** yamahata has quit IRC02:28
*** elarson has quit IRC02:29
*** gouthamr has quit IRC02:30
*** reed_ has joined #openstack-infra02:34
*** reed_ has quit IRC02:36
*** adrian_otto has joined #openstack-infra02:39
*** markvoelker has joined #openstack-infra02:39
*** zhurong has quit IRC02:39
*** zhurong has joined #openstack-infra02:40
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** tphummel has quit IRC02:42
*** tphummel has joined #openstack-infra02:43
*** camunoz has quit IRC02:45
*** sflanigan has quit IRC02:46
*** yuanying has quit IRC02:48
*** piet has joined #openstack-infra02:48
adrian_ottoDoes anyone know how to upload image files to the OpenStack Wiki? I have done it in the past, but now I can't figure out where the file upload feature is. Was it taken away?02:50
loquacitiesadrian_otto: i believe it was removed after the Great Spam Attack02:51
adrian_ottoany chance it could be temporarily re-enabled?02:51
*** thorst has joined #openstack-infra02:54
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** sflanigan has joined #openstack-infra02:58
*** hemanthm|afk is now known as hemanthm02:59
*** piet has quit IRC03:01
*** thorst has quit IRC03:02
*** oanson has joined #openstack-infra03:03
*** Srinu has joined #openstack-infra03:05
lifelessfungi: the stock tox.ini's that Nakato put together all use an environment variable03:07
lifelessfungi: so no change should be needed there - we already down the constraints file locally in python jobs, its just not edited at the moment03:07
openstackgerritSteve Baker proposed openstack-infra/shade: Depend on python-heatclient>=1.0.0
*** docaedo has quit IRC03:10
*** signed8bit_Zzz is now known as signed8bit03:10
*** amitgandhinz has joined #openstack-infra03:10
*** signed8bit has quit IRC03:10
*** oanson has quit IRC03:11
*** amitgandhinz has quit IRC03:15
*** gildub has joined #openstack-infra03:15
*** adrian_otto has quit IRC03:18
*** docaedo has joined #openstack-infra03:24
*** r-mibu has quit IRC03:24
*** r-mibu has joined #openstack-infra03:24
*** shashank_hegde has quit IRC03:24
*** dimtruck is now known as zz_dimtruck03:26
*** zz_dimtruck is now known as dimtruck03:26
*** Nakato has quit IRC03:27
*** gouthamr_ has quit IRC03:29
*** camunoz has joined #openstack-infra03:32
*** Nakato has joined #openstack-infra03:32
*** yamamoto_ has joined #openstack-infra03:33
*** tphummel has quit IRC03:35
*** roxanaghe has joined #openstack-infra03:35
*** dimtruck is now known as zz_dimtruck03:35
openstackgerritMerged openstack/os-client-config: Update citycloud to list new regions
*** EricGonczer_ has quit IRC03:43
*** yuanying has joined #openstack-infra03:47
*** ayoung has quit IRC03:48
*** yuanying has quit IRC03:49
*** yuanying has joined #openstack-infra03:49
*** sflanigan has quit IRC03:49
SrinuHi. In jenkins after completion of Job(whether success/failure) how can we stop not killing the slave.jar process running on the slaves03:49
*** tphummel has joined #openstack-infra03:50
*** sdake has joined #openstack-infra03:50
*** sdake_ has joined #openstack-infra03:54
*** markvoelker has quit IRC03:55
*** sdake has quit IRC03:58
*** thorst has joined #openstack-infra03:59
*** rossella_s has quit IRC04:00
*** rossella_s has joined #openstack-infra04:00
*** tphummel has quit IRC04:01
openstackgerritMerged openstack-infra/puppet-mosquitto: Enable real logging
*** sdake_ has quit IRC04:05
*** sflanigan has joined #openstack-infra04:06
*** sflanigan has joined #openstack-infra04:06
*** thorst has quit IRC04:06
*** zhurong has quit IRC04:07
*** links has joined #openstack-infra04:10
*** amitgandhinz has joined #openstack-infra04:11
*** zhurong has joined #openstack-infra04:12
Srinui am using two slaves for running 2 jobs and when one job finished first while other job not finished yet while deletion of nodepool vm of completed job,2 slave java process is killed Slave JVM has terminated. did any one face this issue04:13
*** Sukhdev has joined #openstack-infra04:13
*** amitgandhinz has quit IRC04:16
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Configure HTTPD and HTTPS certificates
*** amotoki has joined #openstack-infra04:20
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** shashank_hegde has joined #openstack-infra04:20
*** kaisers_ has joined #openstack-infra04:23
*** kaisers_ has quit IRC04:23
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** shashank_hegde has quit IRC04:29
*** markvoelker has joined #openstack-infra04:33
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Add MySQl & Other Configuration Options to Pholio
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add .bindeprc as additional filename
*** markvoelker has quit IRC04:39
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Make python35 jobs voting on congress projects
*** furlongm has quit IRC04:43
*** furlongm_ has joined #openstack-infra04:43
*** gildub has quit IRC04:45
*** shashank_hegde has joined #openstack-infra04:46
*** shashank_hegde has quit IRC04:51
*** psachin has joined #openstack-infra04:54
openstackgerritMerged openstack-infra/zuul: Remove build descriptions
*** rfolco has joined #openstack-infra05:02
*** julim has quit IRC05:02
openstackgerritMerged openstack-infra/zuul: Remove job registration check
openstackgerritMerged openstack-infra/zuul: Re-enable test_jobs_launched
openstackgerritMerged openstack-infra/zuul: Re-enable test_initial_pipeline_gauges
openstackgerritMerged openstack-infra/zuul: Move test_cloner out of its own subdirectory
*** jaosorior has joined #openstack-infra05:03
openstackgerritMerged openstack-infra/zuul: Add utility program to print test fixture configuration
*** pahuang has joined #openstack-infra05:03
openstackgerritMerged openstack-infra/zuul: Use RecordingLaunchServer to run all tests
*** thorst has joined #openstack-infra05:04
*** roxanaghe has quit IRC05:05
*** furlongm_ has quit IRC05:05
*** furlongm_ has joined #openstack-infra05:06
*** yamahata has joined #openstack-infra05:06
*** rfolco has quit IRC05:06
*** ihrachys has joined #openstack-infra05:08
*** karthikp_ has joined #openstack-infra05:09
*** rbuzatu has joined #openstack-infra05:09
*** tphummel has joined #openstack-infra05:11
*** thorst has quit IRC05:11
*** amitgandhinz has joined #openstack-infra05:11
*** e0ne has joined #openstack-infra05:12
*** e0ne has quit IRC05:15
*** amitgandhinz has quit IRC05:16
*** ihrachys has quit IRC05:17
*** yamahata has quit IRC05:19
*** aeng has joined #openstack-infra05:22
*** csomerville has joined #openstack-infra05:24
*** oanson has joined #openstack-infra05:26
*** cody-somerville has quit IRC05:27
*** armax has quit IRC05:28
*** markvoelker has joined #openstack-infra05:28
*** rcernin has joined #openstack-infra05:28
*** markvoelker has quit IRC05:33
*** aeng has quit IRC05:39
*** ihrachys has joined #openstack-infra05:46
*** shashank_hegde has joined #openstack-infra05:50
openstackgerritMerged openstack-infra/system-config: Don't override the mosquitto log path
*** Sukhdev has quit IRC05:55
*** julim has joined #openstack-infra05:57
openstackgerritMerged openstack-infra/system-config: Enable ntpdate service for centos-7
*** ihrachys has quit IRC05:58
*** gildub has joined #openstack-infra05:59
*** sflanigan has quit IRC06:02
*** timl0101 has quit IRC06:06
*** sdake has joined #openstack-infra06:07
*** sdake has quit IRC06:07
*** sdake has joined #openstack-infra06:07
*** timl0101 has joined #openstack-infra06:07
*** bhavik has joined #openstack-infra06:08
*** javeriak has joined #openstack-infra06:09
*** thorst has joined #openstack-infra06:09
*** amitgandhinz has joined #openstack-infra06:12
*** sflanigan has joined #openstack-infra06:15
*** sflanigan has joined #openstack-infra06:15
*** tphummel has quit IRC06:15
*** thorst has quit IRC06:16
*** amitgandhinz has quit IRC06:17
*** mikelk has joined #openstack-infra06:17
*** rfolco has joined #openstack-infra06:18
*** mhickey has joined #openstack-infra06:19
*** tqtran has joined #openstack-infra06:21
*** tphummel has joined #openstack-infra06:22
*** markvoelker has joined #openstack-infra06:22
*** florianf has joined #openstack-infra06:23
*** sdake has quit IRC06:23
*** rfolco has quit IRC06:24
*** tqtran has quit IRC06:25
*** markvoelker has quit IRC06:26
*** flepied has joined #openstack-infra06:27
openstackgerritMerged openstack-infra/puppet-germqtt: Fix path for ssh key file
*** sarob has joined #openstack-infra06:31
*** sarob has quit IRC06:36
*** pgadiya has joined #openstack-infra06:36
yolandagood morning06:40
openstackgerritCsaba Henk proposed openstack-infra/project-config: templatize Manila GlusterFS jobs
openstackgerritCraige McWhirter proposed openstack-infra/infra-specs: Initial commit of the Pholio spec
*** tesseract- has joined #openstack-infra06:45
AJaegergood morning, yolanda !06:46
*** gildub has quit IRC06:49
*** thorst has joined #openstack-infra06:55
*** thorst has quit IRC07:03
*** karthikp_ has quit IRC07:03
*** pahuang has quit IRC07:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add py27, py35 jobs for openstack-doc-tools
openstackgerritMerged openstack-infra/project-config: Add integration test job for Senlin
openstackgerritMerged openstack-infra/project-config: Add tarball publisher to tripleo-ci for images
rcarrillocruzfungi , mtreinish : hiya. perplexed, so changing the log dest allows to bind ports < 1024 ?07:12
rcarrillocruzthat's a veeery odd bug07:12
*** amitgandhinz has joined #openstack-infra07:13
*** ihrachys has joined #openstack-infra07:14
*** jaosorior is now known as jaosorior_brb07:15
*** markvoelker has joined #openstack-infra07:16
*** e0ne has joined #openstack-infra07:16
*** pahuang has joined #openstack-infra07:17
*** berendt has joined #openstack-infra07:17
*** amitgandhinz has quit IRC07:18
*** berendt has quit IRC07:19
*** pcaruana has joined #openstack-infra07:21
*** markvoelker has quit IRC07:21
openstackgerritCsaba Henk proposed openstack-infra/project-config: templatize Manila GlusterFS jobs
*** links has quit IRC07:23
*** Na3iL has joined #openstack-infra07:24
*** jlanoux has joined #openstack-infra07:24
*** jlanoux has quit IRC07:24
*** jlanoux has joined #openstack-infra07:26
*** ekarlso has quit IRC07:29
openstackgerritMerged openstack-infra/system-config: Fix resource references should now be capitalized warnings
*** jpich has joined #openstack-infra07:35
*** links has joined #openstack-infra07:36
*** ekarlso has joined #openstack-infra07:37
*** claudiub has joined #openstack-infra07:37
openstackgerritMerged openstack-infra/project-config: telemetry: Don't enable panko on old branch
*** shashank_hegde has quit IRC07:38
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add .bindeprc as additional filename
*** javeriak has quit IRC07:38
openstackgerritMerged openstack-infra/project-config: Run basic tests in check and gate
*** furlongm_ has quit IRC07:38
openstackgerritMerged openstack-infra/project-config: Update networking-generic-switch-dsvm-tempest job
*** tphummel has quit IRC07:40
*** vincentll has joined #openstack-infra07:40
*** furlongm_ has joined #openstack-infra07:42
openstackgerritMerged openstack-infra/project-config: Add Ironic multinode job
openstackgerrityolanda.robla proposed openstack-infra/puppet-jeepyb: Require packages before installing jeepyb
*** Goneri has joined #openstack-infra07:54
openstackgerritMerged openstack-infra/project-config: Make python35 jobs voting on congress projects
*** Goneri has quit IRC07:56
*** Goneri has joined #openstack-infra07:56
*** daemontool has joined #openstack-infra07:57
*** shardy has joined #openstack-infra07:59
*** jistr is now known as jistr|training07:59
*** zzzeek has quit IRC08:00
*** amotoki has quit IRC08:00
*** thorst has joined #openstack-infra08:01
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add update_server method
*** zzzeek has joined #openstack-infra08:02
*** amotoki has joined #openstack-infra08:04
*** amotoki has quit IRC08:05
*** atom_ has joined #openstack-infra08:06
*** coolsvap is now known as coolsvap_08:08
*** thorst has quit IRC08:08
*** markvoelker has joined #openstack-infra08:10
*** Hal has quit IRC08:12
*** jaosorior_brb has quit IRC08:12
*** jaosorior_brb has joined #openstack-infra08:13
*** amitgandhinz has joined #openstack-infra08:14
*** markvoelker has quit IRC08:14
*** amotoki has joined #openstack-infra08:17
*** Hal has joined #openstack-infra08:17
*** amotoki_ has joined #openstack-infra08:18
*** Ravikiran_K has joined #openstack-infra08:18
*** amitgandhinz has quit IRC08:18
*** amotoki has quit IRC08:21
*** jaosorior_brb is now known as jaosorior08:23
*** tqtran has joined #openstack-infra08:23
*** tqtran has quit IRC08:27
*** baoli has quit IRC08:27
*** amotoki_ has quit IRC08:29
*** vincentll has quit IRC08:31
*** yaume has joined #openstack-infra08:35
*** mouli has joined #openstack-infra08:37
mouliHi All08:37
mouliI am facing the issue in updating the contact information in gerrit08:38
mouliIn browser I tried to update the contact information and it returned with Code-review error cannot store contact information08:38
mouliyour help will be appreciated08:38
AJaegermouli, you need to be a "Foundation member" to be able to contribute08:41
AJaegerThis needs your gerrit preferred e-mail address to match a primary08:41
AJaegere-mail address for a foundation individual member account.08:41
AJaegermouli: If you already followed the instructions at completely and in order, and still get that, see for additional troubleshooting tips.08:42
openstackgerritWitold Bedyk proposed openstack-infra/project-config: Disable services for tempest gate
*** electrofelix has joined #openstack-infra08:43
*** amotoki has joined #openstack-infra08:45
*** vincentll has joined #openstack-infra08:46
*** priteau has joined #openstack-infra08:48
*** jistr|training is now known as jistr08:49
*** mhickey has quit IRC08:51
*** atom_ has quit IRC08:52
*** amotoki has quit IRC08:53
*** jpr has joined #openstack-infra08:53
mouliThanks AJaeger, issue fixed08:53
*** eliqiao has quit IRC08:54
*** ihrachys has quit IRC08:54
*** eliqiao has joined #openstack-infra08:55
*** zakora_ has joined #openstack-infra08:56
*** zakora_ has quit IRC08:56
*** hichihara has quit IRC08:57
*** vincentll has quit IRC08:59
*** binbincong has joined #openstack-infra08:59
*** mikelk has quit IRC09:00
*** amotoki has joined #openstack-infra09:02
*** shardy is now known as shardy_mtg09:02
*** asettle has joined #openstack-infra09:02
*** amitkqed has quit IRC09:04
*** markvoelker has joined #openstack-infra09:04
*** amitkqed has joined #openstack-infra09:04
AJaegerglad to hear, mouli ! Happy contributing!09:05
*** thorst has joined #openstack-infra09:06
*** Srinu has quit IRC09:08
*** markvoelker has quit IRC09:09
*** sambetts|afk is now known as sambetts09:11
*** mikelk has joined #openstack-infra09:11
*** mrmartin has joined #openstack-infra09:11
*** mouli has quit IRC09:11
*** thorst has quit IRC09:13
*** vincentll has joined #openstack-infra09:14
*** amitgandhinz has joined #openstack-infra09:15
*** amitgandhinz has quit IRC09:19
*** null0 has joined #openstack-infra09:23
*** dtantsur|afk is now known as dtantsur09:24
openstackgerritMartin Mágr proposed openstack-infra/tripleo-ci: Add opstools candidate repo
openstackgerrityolanda.robla proposed openstack-infra/puppet-jeepyb: Require packages before installing jeepyb
*** null0 has quit IRC09:33
*** john-davidge has joined #openstack-infra09:36
*** furlongm_ has quit IRC09:37
*** john-dav_ has joined #openstack-infra09:37
*** furlongm_ has joined #openstack-infra09:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add update_server method
*** electrofelix has quit IRC09:40
*** electrofelix has joined #openstack-infra09:40
*** john-davidge has quit IRC09:41
*** yamamoto_ has quit IRC09:41
*** skolekonov has joined #openstack-infra09:42
*** skolekonov has quit IRC09:42
*** evgenyl has quit IRC09:42
*** tlbr has joined #openstack-infra09:43
*** shardy_mtg is now known as shardy09:43
*** rguillebert has quit IRC09:43
*** evgenyl has joined #openstack-infra09:44
*** kambiz has joined #openstack-infra09:44
*** rguillebert has joined #openstack-infra09:44
*** simonmcc has quit IRC09:44
*** sflanigan has quit IRC09:46
*** sflanigan has joined #openstack-infra09:46
*** Qiming has quit IRC09:46
*** simonmcc has joined #openstack-infra09:47
*** hughhalf has quit IRC09:47
*** Qiming has joined #openstack-infra09:50
*** javeriak has joined #openstack-infra09:51
*** hughhalf has joined #openstack-infra09:52
*** yamahata has joined #openstack-infra09:56
*** dirk has quit IRC09:57
*** dirk has joined #openstack-infra09:57
*** Jaison has joined #openstack-infra09:58
*** itsuugo has quit IRC09:58
*** markvoelker has joined #openstack-infra09:58
*** links has quit IRC09:59
*** itsuugo has joined #openstack-infra10:00
*** nmagnezi has joined #openstack-infra10:01
*** markvoelker has quit IRC10:04
*** Goneri has quit IRC10:08
*** _degorenko|afk is now known as degorenko10:10
*** chem has joined #openstack-infra10:10
*** amit213 has quit IRC10:10
*** yamahata has quit IRC10:11
*** thorst has joined #openstack-infra10:11
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth
*** amit213 has joined #openstack-infra10:12
*** john-dav_ is now known as john-davidge10:12
*** Qiming has quit IRC10:12
*** zhurong has quit IRC10:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add common testing repo to OpenStack-Ansible
*** serverascode has quit IRC10:15
*** amitgandhinz has joined #openstack-infra10:15
*** bastelfreak has quit IRC10:16
*** tdasilva has quit IRC10:16
*** serverascode has joined #openstack-infra10:16
*** yamamoto has joined #openstack-infra10:17
*** Qiming has joined #openstack-infra10:17
*** thorst has quit IRC10:18
*** bastelfreak has joined #openstack-infra10:18
*** amitgandhinz has quit IRC10:20
*** rfolco has joined #openstack-infra10:20
*** javeriak has quit IRC10:21
*** yamamoto has quit IRC10:22
*** yamamoto has joined #openstack-infra10:23
*** Goneri has joined #openstack-infra10:24
*** tosky has joined #openstack-infra10:25
*** rfolco has quit IRC10:25
*** asettle has quit IRC10:27
*** asettle has joined #openstack-infra10:27
*** yamamoto has quit IRC10:28
*** amotoki has quit IRC10:30
*** rodrigods has quit IRC10:30
*** rodrigods has joined #openstack-infra10:30
*** mhickey has joined #openstack-infra10:36
*** tdasilva has joined #openstack-infra10:38
*** amotoki has joined #openstack-infra10:39
*** tosky has quit IRC10:39
*** tosky has joined #openstack-infra10:39
*** tosky has joined #openstack-infra10:40
*** amotoki has quit IRC10:41
*** tosky has quit IRC10:41
*** tosky has joined #openstack-infra10:43
*** eliqiao has quit IRC10:47
*** Jaison has quit IRC10:49
*** tosky has quit IRC10:49
*** eliqiao has joined #openstack-infra10:51
*** javeriak has joined #openstack-infra10:51
*** gongysh_ has quit IRC10:52
*** links has joined #openstack-infra10:53
*** yanyanhu has quit IRC10:53
*** tosky has joined #openstack-infra10:54
*** bhavik has quit IRC10:55
*** sdague has joined #openstack-infra10:55
*** yamamoto has joined #openstack-infra10:55
*** Ravikiran_K has quit IRC10:56
rcarrillocruzKiall, mugsie :  needs some love :/10:58
zubchickHi folks, could you please take a look at the patch It had +1 workflow but still have not merged11:09
mugsiercarrillocruz: damn pep8 -_-11:10
AJaegerzubchick: is that change dependent on others?11:11
*** yamamoto has quit IRC11:11
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Reinstall python-requests in bootstrap script
AJaegerzubchick: Click on the depends-On and check that all changes it shows are merged. Your job will not merge before those are in.11:11
*** javeriak has quit IRC11:12
zubchickAJaeger: oh, thanks11:12
*** jpr has quit IRC11:14
*** jpr has joined #openstack-infra11:15
*** thorst has joined #openstack-infra11:16
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Don't assume an eth1
tlbrmordred, pabelanger , hi, please review
*** lucasagomes is now known as lucas-hungry11:21
*** rfolco has joined #openstack-infra11:22
*** thorst has quit IRC11:25
*** rhallisey has joined #openstack-infra11:25
*** rfolco has quit IRC11:26
*** yolanda has quit IRC11:27
*** javeriak has joined #openstack-infra11:29
openstackgerritCsaba Henk proposed openstack-infra/project-config: Add jobs for Manila glusterfs, glusterfs-native drivers with Heketi layout
openstackgerritCsaba Henk proposed openstack-infra/project-config: templatize Manila GlusterFS jobs
*** yolanda has joined #openstack-infra11:31
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Don't assume an eth1
*** thorst has joined #openstack-infra11:41
*** jkilpatr has joined #openstack-infra11:41
*** amitgandhinz has joined #openstack-infra11:44
*** sshnaidm is now known as sshnaidm|afk11:44
*** coolsvap_ is now known as coolsvap11:45
*** dprince has joined #openstack-infra11:45
*** amitgandhinz has quit IRC11:46
*** tpsilva has joined #openstack-infra11:48
*** markvoelker has joined #openstack-infra11:52
*** yamamoto has joined #openstack-infra11:54
*** psilvad has joined #openstack-infra11:55
*** yamamoto has quit IRC11:59
*** rossella_s has quit IRC12:00
*** yamamoto has joined #openstack-infra12:00
*** rossella_s has joined #openstack-infra12:00
*** psilvad has quit IRC12:01
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Make $TRIPLEO_ROOT overrideable
*** psilvad has joined #openstack-infra12:04
*** yamamoto has quit IRC12:05
*** weshay has joined #openstack-infra12:07
*** javeriak has quit IRC12:08
*** acoles_ is now known as acoles12:09
*** psilvad has quit IRC12:10
*** yamamoto has joined #openstack-infra12:11
*** raildo has joined #openstack-infra12:11
*** ccamacho is now known as ccamacho|lunch12:11
*** sigmavirus_away is now known as sigmavirus12:14
*** signed8bit has joined #openstack-infra12:16
*** xyang1 has joined #openstack-infra12:16
*** narayrak has joined #openstack-infra12:17
*** chem has quit IRC12:18
*** ramishra has quit IRC12:18
*** chem has joined #openstack-infra12:18
*** edmondsw has joined #openstack-infra12:19
*** witek has joined #openstack-infra12:22
*** binbincong has quit IRC12:22
witekI would like to disable all services and enable only 4 in jenkins job definition12:24
witekhow should I make it?12:24
*** gordc has joined #openstack-infra12:25
AJaegersdague: can you help witek? I think this needs OVERRIDE_ENABLED_SERVICES for devstack-gate ^12:25
*** rfolco has joined #openstack-infra12:25
*** tqtran has joined #openstack-infra12:25
*** baoli has joined #openstack-infra12:26
vponomaryovwitek: example:
witekgreat, thanks12:26
*** baoli_ has joined #openstack-infra12:27
openstackgerritWitold Bedyk proposed openstack-infra/project-config: Disable services for tempest gate
*** tqtran has quit IRC12:30
*** gouthamr has joined #openstack-infra12:30
*** baoli has quit IRC12:30
*** pt_15 has joined #openstack-infra12:31
AJaegerwitek, LGTM12:32
*** zhurong has joined #openstack-infra12:33
witekthanks Andreas12:34
*** sshnaidm|afk is now known as sshnaidm12:34
HenryGWhere/how can I find periodic job results?12:35
*** daemontool_ has joined #openstack-infra12:35
AJaegerhealthcheck in general - or
*** sdake has joined #openstack-infra12:37
*** daemontool has quit IRC12:37
AJaegerHenryG: One EXample (needs job to use proper reporting):
HenryGthanks AJaeger12:38
AJaegerOr directly
*** narayrak has quit IRC12:41
*** rlandy has joined #openstack-infra12:41
HenryGAJaeger: I see this:
*** lucas-hungry is now known as lucasagomes12:43
*** bhunter71 has quit IRC12:43
HenryG^^ seems to indicate periodic-neutron-py34-with-neutron-lib-master has a problem12:43
*** javeriak has joined #openstack-infra12:45
*** javeriak has quit IRC12:46
*** piet_ has joined #openstack-infra12:46
*** amitgandhinz has joined #openstack-infra12:47
*** julim has quit IRC12:48
*** sarob has joined #openstack-infra12:48
*** yamamoto has quit IRC12:49
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new projects
*** matt-borland has joined #openstack-infra12:51
*** annegentle has joined #openstack-infra12:52
*** narayrak has joined #openstack-infra12:52
*** amitgandhinz has quit IRC12:52
*** sarob has quit IRC12:53
*** tosky has quit IRC12:53
*** yaume has quit IRC12:54
*** narayrak has quit IRC12:54
*** narayrak has joined #openstack-infra12:54
*** tosky has joined #openstack-infra12:55
*** jsavak has joined #openstack-infra12:56
*** signed8bit is now known as signed8bit_Zzz12:56
AJaegerHenryG: health has no data since 1st of August - best to ask mtreinish about that.12:56
HenryGAJaeger: Ah, did not notice the dates, thanks12:57
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend
openstackgerritMonty Taylor proposed openstack-infra/shade: Bump heatclient dependency to 1.0.0
openstackgerritMonty Taylor proposed openstack-infra/shade: Add ability to upload duplicate images
openstackgerritMonty Taylor proposed openstack-infra/shade: Add a 'meta' passthrough parameter for glance images
*** rhallisey has quit IRC13:00
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
*** xarses_ has joined #openstack-infra13:00
*** annegentle has quit IRC13:00
*** burgerk has joined #openstack-infra13:03
*** xarses has quit IRC13:03
*** signed8bit_Zzz is now known as signed8bit13:04
*** piet_ has quit IRC13:05
*** rhallisey has joined #openstack-infra13:05
*** kgiusti has joined #openstack-infra13:06
*** mriedem has joined #openstack-infra13:06
*** ramishra has joined #openstack-infra13:11
*** jcoufal has joined #openstack-infra13:12
*** mdrabe has joined #openstack-infra13:12
openstackgerritMonty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips
*** yamamoto has joined #openstack-infra13:13
*** AJaeger has quit IRC13:15
*** john-davidge has left #openstack-infra13:15
*** zhurong has quit IRC13:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Depend on python-heatclient>=1.0.0
*** zhurong has joined #openstack-infra13:16
*** ccamacho|lunch is now known as ccamacho13:17
openstackgerritMonty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips
openstackgerritMonty Taylor proposed openstack-infra/shade: Go ahead and admit that we return Munch objects
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Enable net-iso for ovb jobs
*** burgerk has quit IRC13:18
*** AJaeger has joined #openstack-infra13:18
*** ihrachys has joined #openstack-infra13:18
*** AJaeger has joined #openstack-infra13:19
*** markvoelker has quit IRC13:19
*** esberglu has joined #openstack-infra13:19
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement tripleo-ci-centos-7-nonha-multinode-upgrades job
*** akscram has quit IRC13:23
*** dkranz has joined #openstack-infra13:24
*** pgadiya has quit IRC13:25
*** akscram has joined #openstack-infra13:25
openstackgerritGabriele Cerami proposed openstack/diskimage-builder: secure_path in sudoers: deal with possible quotes
*** zhurong has quit IRC13:25
*** yamamoto has quit IRC13:26
*** AJaeger has quit IRC13:26
*** zhurong has joined #openstack-infra13:26
*** AJaeger has joined #openstack-infra13:27
*** ihrachys has quit IRC13:29
*** tonytan4ever has joined #openstack-infra13:32
*** mmandel has joined #openstack-infra13:32
*** ttx has quit IRC13:32
*** Ravikiran_K has joined #openstack-infra13:33
*** ttx has joined #openstack-infra13:33
*** ttx has quit IRC13:33
*** ttx has joined #openstack-infra13:34
*** ttx has quit IRC13:35
*** ttx has joined #openstack-infra13:36
*** asselin has joined #openstack-infra13:36
*** _ari_ has joined #openstack-infra13:40
*** thiagop has joined #openstack-infra13:40
*** asettle has quit IRC13:41
*** EricGonczer_ has joined #openstack-infra13:41
openstackgerritMerged openstack-infra/puppet-openstackci: Use project_config to get nodepool.yaml path
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth
*** ldnunes has joined #openstack-infra13:46
*** markvoelker has joined #openstack-infra13:46
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth
openstackgerritJeremy Stanley proposed openstack-infra/bindep: Build universal wheels
robcresswellAJaeger: I updated to use the xenial node; is my entry correct? I'm a little muddled by the config, whether I Should be using the -nv suffix, or defining voting:false etc.13:48
*** amitgandhinz has joined #openstack-infra13:48
*** bhavik has joined #openstack-infra13:48
*** zhurong has quit IRC13:48
*** zhurong has joined #openstack-infra13:49
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Make bindep Python 3.5 unit test jobs vote
*** zz_dimtruck is now known as dimtruck13:50
mtreinishAJaeger: on which pages, it looks like there is data in the DB from today (in general, maybe not for a particular query though)13:51
AJaegerrobcresswell: I commented, the job is now X-ubuntu-xenial, so you need t orename in zuul/layout as well13:51
mtreinishrcarrillocruz: yeah it's a bit weird, I filed it upstream:
*** amitgandhinz has quit IRC13:52
AJaegermtreinish: see discussion with HenryG - and
mtreinishrcarrillocruz: I'm not actually sure it's the bind failing though, it only happens if the user mosquitto drops down to internally doesn't have permissions to create the log file13:53
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth
mtreinishrcarrillocruz: the error reporting is all broken by not being able to write the log file, so that's definitely the first issue to fix13:53
*** eharney has joined #openstack-infra13:54
rcarrillocruzi even forced the conf file to have 'user root', despite the docs in the file saying it defaults to root it was worth checking13:54
rcarrillocruzsince it couldn't bind the port13:54
AJaegerrobcresswell: LGTM now13:54
rcarrillocruzbut didn't help either13:54
openstackgerritMerged openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
mtreinishrcarrillocruz: when I added 'user root' to the conf file that worked for me13:54
rcarrillocruzoh, and also set the cap to bind ports on the mosquitto binary13:54
robcresswellAJaeger: Thanks. Will look to move more to Xenial post Newton release.13:54
rcarrillocruzand same issue, so not an apparmor/selinux problem13:54
*** akshai has joined #openstack-infra13:54
rcarrillocruzreally? 'root' *and* the log file modified?13:55
rcarrillocruzcos i just tweaked the user 'root' on the conf file and didn't help13:55
AJaegerrobcresswell: we're in the process of moving *everything* on master to Xenial...13:55
*** shamail has joined #openstack-infra13:55
mtreinishAJaeger: so that looks like it;s caused by the data source limitation with subunit2sql. That job is failing prior to anything that generates subunit, so there is nothing to use for populating the DB13:55
mtreinishAJaeger: the graphite/grafana data comes straight from zuul so it always knows about failed jobs13:56
AJaegerHenryG: ^13:56
AJaegermtreinish: thanks for checking!13:56
HenryGmtreinish: AJaeger: ack, thanks13:57
mtreinishAJaeger: it's been a bit thing to fix, but the problem is there isn't a queriable data source straight from zuul that openstack-health can look at and correlate to the other data sources13:57
*** burgerk has joined #openstack-infra13:57
*** zhurong has quit IRC13:57
robcresswellAJaeger: Yep, understood, wasn't sure how much of that work was being done by infra vs the project teams moving their own tests (as in that previous patch)13:57
mtreinishjhesketh has been working on a sql reporter for a while to provide that13:57
*** rockyg has joined #openstack-infra13:57
HenryGmtreinish: NP, I found a failed job on logs.o.o to look at13:57
*** asettle has joined #openstack-infra13:58
mtreinishrcarrillocruz: hmm, I'm pretty sure. Let me test it right now13:58
*** zhurong has joined #openstack-infra13:58
*** rbrndt has joined #openstack-infra13:58
AJaegerrobcresswell: help is welcome on getting this done!13:58
rcarrillocruzmtreinish: however is, is the thing at working now? need me anything to check13:58
mtreinishrcarrillocruz: I think we need to restart germqtt on firehose.o.o13:59
mtreinishrcarrillocruz: there were some issues with the puppet-germqtt module but I think the fix made it through the gate overnight13:59
mtreinishoh, nope it's working now13:59
mtreinishat least on the mqtt port13:59
*** krot_sickleave is now known as krotscheck14:00
mtreinishrcarrillocruz: I'm still having trouble using the websocket port. BUt that could just be an issue with the js client I've been using for testing14:00
*** akshai has quit IRC14:00
openstackgerritMerged openstack-infra/tripleo-ci: Don't assume an eth1
*** xarses_ has quit IRC14:01
ttxfungi: when you have a chance, please set owner for the new "*-release-branch" groups to "Release Managers" so that we can set members from there14:01
*** tongli has joined #openstack-infra14:01
mtreinishrcarrillocruz: you can test it too with: 'mosquitto_sub --topic '#' -h'14:01
*** esker has joined #openstack-infra14:01
fungittx: working on it now--will let you know when it's done14:01
ttxfungi: thx!14:01
*** akshai has joined #openstack-infra14:02
*** asettle has quit IRC14:02
*** eharney_ has joined #openstack-infra14:04
mtreinishrcarrillocruz: oh cool, the websocket port looks like it's working too. (it was just a client issue on my end)14:04
*** eharney has quit IRC14:04
mtreinishrcarrillocruz: I think this means all the work items from the spec are complete :)14:04
*** tongli has quit IRC14:04
*** tongli has joined #openstack-infra14:05
mtreinishI still want to get tls ports implemented before we close it out though14:05
mtreinishwe probably don't need it for external random subscribers, but for any internal communication between servers we should force it14:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
mtreinishfungi, rcarrillocruz: if you want to play with the websocket side I used: to test it14:07
rcarrillocruzmtreinish: firehose doesn't have 'user root' on the conf file14:07
rcarrillocruzit has it commented it out14:07
rcarrillocruzjust as it was when i checked it yesterday14:07
rcarrillocruzso i guess what fungi changed about the log file dest fixed somehow the bind port issue14:07
*** bhunter71 has joined #openstack-infra14:07
fungittx: all set. just double-checked and all 20 look like they're updated now14:08
mtreinishrcarrillocruz: yeah we set it back to the default location of /var/log/mosquitto/mosquitto.log which the mosquitto user has write permission on14:08
mtreinishthat unblocked things14:08
fungircarrillocruz: right, for strange reasons it couldn't bind to privileged ports if the log file wasn't writable by the mosquitto user14:09
fungii'm guessing there's still some sort of subtle bug with their privilege separation code14:09
fungircarrillocruz: also the hiera entry for the gerrit public key was all sorts of wrong. it looked like it was formatted as an authorized_keys entry instead of a known_hosts entry14:10
fungircarrillocruz: but on subsequent discussion we're all over the map with how we've implemented configuration management of known_hosts entries si i'm going to try to clean that up across all our modules and make it consistent and centralized14:11
fungigerrit public host key i mean14:11
*** bhavik has quit IRC14:11
*** yaume has joined #openstack-infra14:11
*** ayoung has joined #openstack-infra14:12
AJaegerfungi, corvus, could you put reviewing the constraints/zuul-cloner change on your dance card for today, please? . Other project-config cores are welcome to review as well!14:13
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Sign all tarballs and wheels
rcarrillocruzfungi: i think i copy pasted from zuul hiera yaml file to the firehose yaml straight14:14
fungircarrillocruz: probably, though the zuul servers puppet the contents of that file differently than this module14:15
fungircarrillocruz: our zuul management concatenates the key material with hostname/address data to build the line rather than just using it as the sole content14:16
*** ramishra has quit IRC14:16
*** EricGonczer_ has quit IRC14:16
fungiso for now i've added the hostname and addresses to that hiera key which got it working14:16
*** EricGonczer_ has joined #openstack-infra14:17
fungibut ultimately i want to make this consistent everywhere, and also not in private hiera14:17
*** oanson has quit IRC14:17
fungiit can just be one global value in our public hiera and all our servers that need it get it from the same entry that way14:17
rcarrillocruzyeah, cos having a key name with different formatting is bad14:17
rcarrillocruznot sure if we could standardize something about formatting on our key naming14:18
*** amotoki has joined #openstack-infra14:18
rcarrillocruzotherwise we'll cargo cult and copy paste around14:18
rcarrillocruzor maybe document somewhere how should the keys be formatted, be in the puppet manifest docs or in some docs at infra namespace14:19
rcarrillocruzbrb, dog walk14:19
mtreinishrcarrillocruz: well that's what I did for puppet-germqtt I just copy and pasted it from puppet-elastic_recheck14:20
*** siva_krish has joined #openstack-infra14:21
*** links has quit IRC14:23
*** claudiub has quit IRC14:24
*** dkehn has quit IRC14:24
*** siva_krish has quit IRC14:24
*** jaosorior has quit IRC14:25
*** dkehn_ has quit IRC14:25
pabelangerfungi: are we ready for 350603? I can +3 now14:26
*** siva_krish has joined #openstack-infra14:26
*** yamamoto has joined #openstack-infra14:26
AJaegerpabelanger: will take 3h+ to merge...14:26
pabelangerAJaeger: okay, I'll wait for checks to finish14:28
rcarrillocruzi couldn't know if the key was ok or not as the service was not even up :/14:28
*** ramishra has joined #openstack-infra14:28
AJaegerpabelanger: you can approve now if you like - if it fails, it will not merge ;)14:28
rcarrillocruzanyway, worth specing out the formatting of this stuff at some point14:28
*** yamahata has joined #openstack-infra14:29
AJaegerpabelanger: just wanted to point out that we're not able to see this merge *now* since Zuul has a huge backlog14:29
*** EricGonczer_ has quit IRC14:31
fungipabelanger: AJaeger: yeah, should be fine though i'll also give the release team a heads up14:31
*** ihrachys has joined #openstack-infra14:31
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch tripleo jobs back to rh1
*** dkehn has joined #openstack-infra14:31
*** EricGonczer_ has joined #openstack-infra14:31
pabelangerAJaeger: Nope, that works for me. Lets wait until zuul votes then line up a time to land14:31
*** yamamoto has quit IRC14:33
*** siva_krish has quit IRC14:33
openstackgerritDoug Hellmann proposed openstack-infra/project-config: comment out skip for manual release
*** jlanoux has quit IRC14:37
*** rbuzatu has quit IRC14:37
*** xarses_ has joined #openstack-infra14:38
*** dkehn_ has joined #openstack-infra14:38
*** burgerk has quit IRC14:38
fungipabelanger: AJaeger: i gave the release team a heads up, so feel free to approve whenever14:38
*** caowei has joined #openstack-infra14:39
*** mtanino has joined #openstack-infra14:40
*** amotoki has quit IRC14:40
openstackgerritMerged openstack/os-client-config: Add support for deprecating cloud profiles
pabelangerfungi: AJaeger +314:42
openstackgerritMerged openstack/os-client-config: Add support for listing a cloud as shut down
*** burgerk has joined #openstack-infra14:43
openstackgerritMerged openstack/os-client-config: Refactor fix magic in get_one_cloud()
*** ramishra has quit IRC14:44
*** Swami has joined #openstack-infra14:45
openstackgerritMonty Taylor proposed openstack/os-client-config: Add the new DreamCompute cloud
*** amotoki has joined #openstack-infra14:45
*** toabctl has quit IRC14:46
*** ramishra has joined #openstack-infra14:46
pabelangerAJaeger: we should land 347941 to avoid endless rebasing too14:46
pabelangerxenial-docs jobs14:46
*** toabctl has joined #openstack-infra14:47
*** annegentle has joined #openstack-infra14:47
*** hieulq_ has joined #openstack-infra14:47
openstackgerritDoug Hellmann proposed openstack-infra/project-config: provide a manual override mode
*** amitgandhinz has joined #openstack-infra14:49
pabelangerclarkb: did you hear back from ovh about nodepool capacity?14:49
dhellmannhmm, openstackgerrit doesn't seem to be posting in #openstack-release14:49
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add config support for ZooKeeper servers
AJaegerpabelanger: I wanted to wait for clarkb - but he's on vacation now AFAIK, so let's go forward with it...14:49
pabelangerAJaeger: Ah, I missed the PTO memo :)14:50
AJaegerpabelanger: there's also - which might need a zuul restart for recalculation...14:50
*** ramishra has quit IRC14:50
*** eharney_ is now known as eharney14:51
*** berendt has joined #openstack-infra14:52
*** mdrabe has quit IRC14:53
*** ramishra has joined #openstack-infra14:53
pabelangerAJaeger: okay, lets land them and see how calculation looks14:53
*** rbuzatu has joined #openstack-infra14:53
pabelangerwe restarted zuul ~12 hours ago FYI14:53
*** amitgandhinz has quit IRC14:53
pabelangerAJaeger: I also have which will provide a more realistic reflection of the nodes we are using in graphite / grafana.  Today we are missing sub-nodes in the calculations14:55
*** karthikp_ has joined #openstack-infra14:55
openstackgerritMerged openstack-infra/project-config: update tag-release git command to fall back to master
*** igorn has joined #openstack-infra14:56
*** siva_krish has joined #openstack-infra14:58
*** asettle has joined #openstack-infra14:59
fungidhellmann: odd. it's present in the channel, and it's saying things in here14:59
fungiwhat did you push that it should have echoed in there?14:59
*** pradk_ has joined #openstack-infra14:59
openstackgerritMerged openstack/os-client-config: Fix precedence for pass-in options
*** esker has quit IRC15:00
*** mdrabe has joined #openstack-infra15:01
*** nmagnezi has quit IRC15:02
*** asettle has quit IRC15:03
*** hieulq__ has joined #openstack-infra15:05
*** jlanoux has joined #openstack-infra15:06
*** amotoki has quit IRC15:06
*** hieulq_ has quit IRC15:07
*** roxanaghe has joined #openstack-infra15:07
*** PollyZ has joined #openstack-infra15:07
*** akshai has quit IRC15:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING
*** dprince has quit IRC15:10
*** siva_krish has quit IRC15:12
openstackgerritJeremy Stanley proposed openstack-infra/zuul: Cloner: Update heads and tags from the cache
fungijeblair: ^ that just crossed my mind when talking to bhunter71 about local git caches on our workers15:14
phschwartzIs there a way to get the id of the nodepool node a job is running on? I want to display it in a notification for build failure.15:14
fungiphschwartz: yep, it's a parameter and it's displayed at the start of the job15:14
fungiphschwartz: assuming you mean the nodepool id and not the nova uuid15:15
phschwartzfungi: yes, the nodepool id15:15
*** dimtruck is now known as zz_dimtruck15:15
*** zz_dimtruck is now known as dimtruck15:16
*** julim has joined #openstack-infra15:16
phschwartzfungi: I see it displayed in the jenkins log, just can't find a parameter in the param list that matches it15:16
fungiphschwartz: as an example, we attach the build_node as metadata on elasticsearch documents15:16
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Retry stack-delete in case of failure
fungiphschwartz: so that lines of output can be correlated by build_node15:16
*** dkehn has quit IRC15:17
phschwartzfungi: awesome, let me look at the upstream job and I can go from there15:17
*** akshai has joined #openstack-infra15:18
rcarrillocruzShrews , mordred , mugsie : revert landed15:18
rcarrillocruzi'l recheck shade changes15:18
rcarrillocruzyolanda: ^15:18
yolandaah great15:19
*** priteau has quit IRC15:19
*** dkehn_ has quit IRC15:22
*** amotoki has joined #openstack-infra15:22
fungiphschwartz: skimming i think it's emitted in zeromq events as the "node_name" parameter15:24
fungiphschwartz: that may not be reflected in any zuul/jenkins job parameters though15:25
*** dimtruck is now known as zz_dimtruck15:25
*** caowei has quit IRC15:26
*** dkehn has joined #openstack-infra15:26
*** ihrachys has quit IRC15:29
*** ihrachys has joined #openstack-infra15:30
*** bhavik has joined #openstack-infra15:31
*** vvalderrv has joined #openstack-infra15:31
*** dkehn_ has joined #openstack-infra15:32
*** zhurong has quit IRC15:33
*** tlbr has quit IRC15:36
*** piet has joined #openstack-infra15:39
*** zz_dimtruck is now known as dimtruck15:40
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: WIP: Add foreground mode
openstackgerritArtur Zarzycki proposed openstack-infra/project-config: Add fuel-ccp-ingress repository
mtreinishfungi, rcarrillocruz: did something happen to germqtt? it looks like the stream stopped15:41
*** mikelk has quit IRC15:42
*** vincentll has quit IRC15:42
fungimtreinish: the process is still running, last started 06:43 utc today15:43
*** jpeeler has quit IRC15:43
fungimtreinish: and gerrit claims there's an open connection for the account15:43
fungiopen since 06:43:14 utc15:43
mtreinishfungi: oops, nm it was on my end15:44
rcarrillocruzhaven't touched it15:44
rcarrillocruzah k15:44
mtreinishI was running with a very narrow topic, and forgot it was that terminal15:44
fungimtreinish: cool, was gonna say i also see traffic back and forth for port 2941815:44
mtreinishfungi, rcarrillocruz: if you're curious I've got a gerritbot running with ^^^ over mqtt up on #throwaway-gerritbot-mqtt-test-channel15:47
*** armax has joined #openstack-infra15:47
mtreinishit seems to be working well15:47
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: WIP: Add foreground mode
fungimtreinish: harlowja might be interested in that for the errbot-based rewrite too15:48
mtreinishhmm, maybe not so well, it just missed it's own commits ^^^ :(15:49
fungithough we should almost certainly treat firehose.o.o as a proof of concept for a while still, and not actually make any production services depend on it15:49
mtreinishheh, and now it's stack tracing15:49
fungilike all good software15:49
mtreinishfungi: sure, but I figure it'd be good to have something on the client side to show off15:49
fungimtreinish: absolutely15:50
*** amitgandhinz has joined #openstack-infra15:50
*** ihrachys has quit IRC15:51
*** mdrabe_ has joined #openstack-infra15:51
*** shashank_hegde has joined #openstack-infra15:52
*** PollyZ has quit IRC15:53
*** Sukhdev has joined #openstack-infra15:54
*** mdrabe has quit IRC15:55
*** amitgandhinz has quit IRC15:55
*** PollyZ has joined #openstack-infra15:58
*** akshai has quit IRC15:59
*** ramishra has quit IRC15:59
*** rossella_s has quit IRC16:00
openstackgerritJeremy Stanley proposed openstack-infra/puppet-jenkins: Optionally manage .ssh/known_hosts on slaves
*** rossella_s has joined #openstack-infra16:00
*** annegentle has quit IRC16:01
*** rbrndt has quit IRC16:01
*** annegentle has joined #openstack-infra16:02
*** ddieterly has joined #openstack-infra16:04
*** itisha has joined #openstack-infra16:04
zarofungi: are we waiting for another gerrit restart to approve the its-storyboard plugin?16:06
*** rbrndt has joined #openstack-infra16:06
*** jistr is now known as jistr|biab16:06
zaroi'm wondering how we can puppet it onto review-dev.o.o before that happens?16:07
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: WIP: Add foreground mode
mtreinish^^^ I think that does it now16:07
mtreinishit seems to be working very reliably in the throwaway channel16:07
* mtreinish shouldn't jinx himself16:07
*** psachin has quit IRC16:11
openstackgerritDean Troyer proposed openstack/os-client-config: Add release notes for 1.19.0 release
fungizaro: i don't think we're waiting to approve it, we just need to plan to do a lengthy reindex outage the next time we have a planned gerrit maintenance16:11
*** shashank_hegde has quit IRC16:12
fungizaro: i'll go ahead and review those outstanding changes now16:12
*** adrian_otto has joined #openstack-infra16:13
*** dtantsur is now known as dtantsur|afk16:15
zarofungi: i would suggest _not_ doing a reindex just for change
zarothat's not required to get storyboard gerrit integration work16:16
zaroi just set it to the same topic because it's related, but not necessary16:17
zarothe only requirement for gerrit storyboard integration is a gerrit restart after those changes are approved16:17
zarosorry if that was not clear.16:17
*** Goneri has quit IRC16:18
fungiyeah, i figured it was just the one adding story/task ids to the index that would need a reindex run16:18
*** PollyZ has quit IRC16:18
zaroi think we should approve them now so puppet can put it all on review-dev.o.o. that would allow testing before we do for realz16:19
*** roxanaghe has quit IRC16:20
*** hieulq__ has quit IRC16:21
*** jistr|biab is now known as jistr16:22
jeblairnotmorgan, jhesketh, mordred: sorry, i accidentally self-approved a change without review:
jeblair(it was on the tip of a series that needed re-approval after a rebase)16:25
notmorganjeblair: probably not an issue.16:25
*** ccamacho is now known as ccamacho|out16:26
zaroyay! force gerrit online reindex finally got approved
jeblairzaro: that is wonderful news :)16:27
*** tphummel has joined #openstack-infra16:27
*** tqtran has joined #openstack-infra16:27
*** dansmith has quit IRC16:27
*** Apoorva has joined #openstack-infra16:27
notmorganjeblair: trying to read that patch before coffee has settled...16:28
*** dansmith has joined #openstack-infra16:28
*** dansmith is now known as Guest2098816:28
notmorganjeblair: notmorgan.waitUntilCoffeeSettled()16:28
*** yaume has quit IRC16:28
jeblairnotmorgan: haha :)16:28
notmorganjeblair: at a glance, it looks ok, fwiw16:29
*** Guest20988 is now known as dansmith16:29
*** akshai has joined #openstack-infra16:30
jeblairi think the most objectionable part of it is adding a parameter to zuul.launcher.server.AnsibleLaunchServer.runAnsible() which the method does not actually need, but the test subclass does.16:30
jpmaxmanjust fyi fungi I looked the extensions that are actually registered and I get a much smaller number:
jeblairi really hate that, but it makes the testing very simple and reliable.  i at least left a comment about it.  :)16:30
notmorganjeblair: yay comments16:31
jeblairnotmorgan: what test were you working on re-enabling?  i was thinking of working on testScheduler.test_parallel_changes...16:31
fungijpmaxman: that looks much closer to what the version page reports16:31
*** tqtran has quit IRC16:31
notmorganjeblair: jeblair the tests in test_connection16:31
jpmaxmansome of those are built in16:32
jpmaxmannot sure they get reported16:32
fungijpmaxman: i wonder why some of those don't appear on the version page (or whether they're builtins(16:32
*** akshai_ has joined #openstack-infra16:32
fungiyeah, what i was thinking too16:32
jeblairnotmorgan: cool, so no collision there16:32
notmorganjeblair: basically it was/is a small set of tests, but requires a full config, figured it was a good place to work16:32
fungijpmaxman: had any of that list turned up as upgrade problems so far?16:32
notmorganjeblair: one of them enables with no changes, and the other two are just chasing config changes down.16:32
jpmaxmanI'm running some tests16:33
*** jsavak has quit IRC16:33
jpmaxmanright now the nocaptcha shows16:33
*** dprince has joined #openstack-infra16:33
jpmaxmanbut even if you succesfully verify16:33
jpmaxmanit says you haven't16:33
*** narayrak has quit IRC16:33
jpmaxmanso trying to figure out where the conflict is16:33
jpmaxmangenerally it is looking good16:33
*** jsavak has joined #openstack-infra16:33
openstackgerritMerged openstack-infra/puppet-gerrit: Add configurations for gerrit its plugins
notmorganjeblair: but i wanted to be sure it was fixture-per-class or per-test-file or whatevert before building all the config things.16:33
*** ddieterly is now known as ddieterly[away]16:34
pabelangerfungi: rcarrillocruz: did I hear you say infracloud was back online?16:34
*** sdague has quit IRC16:34
pabelangermordred: notmorgan: do you mind adding 349119 to your list of nodepool reviews?16:35
*** akshai has quit IRC16:35
*** roxanaghe has joined #openstack-infra16:35
fungipabelanger: rcarrillocruz said he received some promising e-mails but hadn't had time to follow up on them due to unanticipated plague16:36
*** Hal has quit IRC16:36
fungii don't recall if there was more recent news since the weekend16:36
*** rcernin has quit IRC16:38
notmorganpabelanger: i don't have an issue with it. lgtm (but I can't +2 it)16:38
fungizaro: all the topic:storyboard-its changes are approved now except your wip one enabling the plugin in production16:38
pabelangerthanks, so it sounds like it might be online for the mid-cycle16:38
notmorganpabelanger: +1'd.16:38
pabelangerStill working on getting travel approval for QAInfraNewtonSprint and wanting to confirm the tasks we'll be working on. Otherwise, I'll work on them remotely16:39
mordredpabelanger: +A16:39
pabelangernotmorgan: mordred: great thanks16:39
openstackgerritMerged openstack-infra/system-config: Configure its-storyboard plugin
*** ddieterly[away] is now known as ddieterly16:40
*** Sukhdev has quit IRC16:40
*** mhickey has quit IRC16:41
openstackgerritMerged openstack-infra/project-config: networking-midonet: Add periodic jobs for stable branches
mtreinishAJaeger: heh, the throwaway channel is pretty boring, it's just my local bot running with: using firehose.o.o16:42
*** ramishra has joined #openstack-infra16:43
*** Teresita-Warrior has joined #openstack-infra16:43
*** sdague has joined #openstack-infra16:43
*** e0ne has quit IRC16:44
openstackgerritMerged openstack-infra/system-config: Add storyboard task to Gerrit tracking id
*** roxanaghe has quit IRC16:45
*** lucasagomes is now known as lucas-dinner16:45
Zarafungi: thanks for reviewing the gerrit-storyboard integration patches!16:46
fungisorry for not getting to them earlier!16:46
jpmaxmanfungi any idea what this one did?
pabelangermordred: jeblair: Is there any plans / schedule for zuulv3 in place for the mid-cycle?16:46
*** roxanaghe has joined #openstack-infra16:47
*** arosen12 has joined #openstack-infra16:47
fungijpmaxman: looking at i can't imagine what we'd have used it for, if anything ever16:47
fungi"Currently the main purpose of this extension is to implement compilation into JavaScript of templates sent the server to the client by ResourceLoader, while the HTML templating language RFC is implemented."16:47
jpmaxmanok well I'm going to kill it and see if anything or anyone complains :D16:48
fungijpmaxman: looks like at one time it was used by the mobilefrontend extension, according to that16:48
*** javeriak has joined #openstack-infra16:48
fungijpmaxman: so presumably a dependency since dropped in newer versions of the extension16:48
*** furlongm_ has quit IRC16:49
*** furlongm_ has joined #openstack-infra16:50
*** hieulq_ has joined #openstack-infra16:51
*** arosen12 has left #openstack-infra16:51
*** amitgandhinz has joined #openstack-infra16:52
jpmaxmanok - well i'd already disabled mobile front end as it was complaining16:52
jpmaxmanmay need to get the newest version of that16:52
*** krotscheck is now known as kro_focused16:53
*** adrian_otto has quit IRC16:53
jeblairpabelanger: i don't think so yet.  we should regroup and see where we are.16:55
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Created DSVM Job for NPM Projects
*** Hal has joined #openstack-infra16:55
*** roxanaghe has quit IRC16:56
*** amitgandhinz has quit IRC16:56
*** adrian_otto has joined #openstack-infra16:56
*** jpr has quit IRC16:57
*** roxanaghe has joined #openstack-infra16:57
pabelangerjeblair: perfect16:57
*** roxanaghe has quit IRC16:57
*** roxanaghe has joined #openstack-infra16:57
*** asettle has joined #openstack-infra17:00
*** pcaruana has quit IRC17:00
*** tqtran has joined #openstack-infra17:00
dhellmannfungi : oh, you know, I think I was looking for the link to to show up there but I forgot it's in the project-config repo now. Is there a way to have the bot look at paths on files, so we could have it announce changes to those scripts in #openstack-release too?17:00
*** tqtran_ has joined #openstack-infra17:01
*** annegentle has quit IRC17:02
fungidhellmann: i don't think so, but would need to dig in the bot source to find out if it has a filter option like that17:02
*** degorenko_ has joined #openstack-infra17:02
*** tesseract- has quit IRC17:03
*** tqtran has quit IRC17:04
zarofungi: thanks!17:05
*** rhedlind has joined #openstack-infra17:06
*** sputnik13 has joined #openstack-infra17:06
*** openstackgerrit_ has joined #openstack-infra17:06
*** nmagnezi has joined #openstack-infra17:07
*** degorenko_ has quit IRC17:08
*** openstackgerrit_ has quit IRC17:08
*** piet has quit IRC17:09
*** jlanoux has quit IRC17:09
mgagnewith grafyaml, is it supposed to support configuration of multiple datasources?17:10
fungipabelanger: ^ ?17:10
mgagnelooks like internal structure supports multiple sources but syntax in config file makes it impossible to configure more17:10
fungimight just be that it was designed with that in mind but we've not tried it so implementation there is actually incomplete?17:10
*** akshai_ has quit IRC17:11
mgagneoh =(17:11
*** karthikp_ has quit IRC17:11
mgagneor it's one datasource per file?17:12
*** coreyob has quit IRC17:12
*** rbuzatu has quit IRC17:12
mgagneI thought it would be datasources: instead of datasource: :O17:12
openstackgerritMerged openstack-infra/shade: Depend on python-heatclient>=1.0.0
*** sarob has joined #openstack-infra17:13
fungimgagne: looks like we only use one anyway
*** jpr1 has joined #openstack-infra17:13
*** coreyob has joined #openstack-infra17:14
mgagneyea, but looking at how dashboard are managed, it looks like it's one dashboard/datasource per file17:14
fungiso basically we're just using it to graph data sourced from our graphite server17:14
mgagneright, I happen to have more than one =)17:14
fungiwe don't currently have any other sources of data backing grafana so it was probably just not a design consideration (but doesn't seem like something we would reject as an enhancement)17:15
*** yamahata has quit IRC17:15
mgagnewill test before complaining more =)17:16
*** piet has joined #openstack-infra17:16
pabelangermgagne: fungi: I cannot remember.  Let me check17:17
pabelangermgagne: fungi: however, it would be easy to add a unit test to validate it17:17
*** PollyZ has joined #openstack-infra17:17
mgagneok, multiple files = multiple datasources17:18
mgagneconfirmed on my side, sorry for the noise :D17:18
*** jsavak has quit IRC17:18
pabelangerya, that sounds right17:18
*** jsavak has joined #openstack-infra17:18
kro_focusedNot to be all whiny and stuff, but how much would the gate improve if we had dib create a devstack image instead of installing it with devstack-gate?17:19
*** eharney has quit IRC17:20
fungikro_focused: poorly since then we wouldn't be able to test devstack-gate17:20
mordredI think it depends, honestly17:20
mordredthe things that make up devstack, nova and friends, would not really be able to use that17:21
mordredbut things that use devstack clouds as a way to test themselves would likely benefit17:21
fungidepends on if kro_focused means preinstall devstack or just prerun the bits of devstack-gate that don't run stack.sh17:21
mordredsince they largely only need to consume the configured endpoint of a cloud17:21
mordredfungi: yah. I mean, I could see using a pre-installed devstack in shade's functional tests, for instance17:22
*** dimtruck is now known as zz_dimtruck17:22
mordredsince shade does not care at all about the installation of opentsack17:22
kro_focusedWell, I'm looking at a 4 hour wait on most jobs right now, and DSVM nodes seem to take the longest.17:22
mordredbut that's also a fringe player in terms of gate usage17:22
jeblairkro_focused, mordred, fungi: yeah, i think it might be worth a shot.17:22
jeblairkro_focused: oh, well, i don't think it's going to radically change *that*17:22
*** Swami has quit IRC17:22
mordredjeblair: have a "yesterday's devstack" image type perhaps?17:22
jeblairmordred: right17:22
jeblairkro_focused: this would perhaps reduce a 10 minute wait in the job to probably a minute or two?17:23
fungimordred: the way to really improve that is probably what we were discussing a while back, something like nodepool that sets up throwaway tenants^Wprojects in an existing openstack deployment and then assigns them to jobs where a worker can make api calls17:23
fungikro_focused: we don't have dsvm nodes17:23
*** Swami has joined #openstack-infra17:23
*** csomerville has quit IRC17:23
kro_focusedfungi: Potato, potato.17:23
fungikro_focused: the dsvm jobs are just using plain old trusty and xenial (and sometimes centos or fedora) nodes used to run other non-dsvm-based jobs17:24
jeblairkro_focused: it would be helpful within jobs that use it, but i don't think it would make a significant impact on the overall system utilization17:24
*** slaweq__ has quit IRC17:24
fungikro_focused: so it's more accurate to say that there are a bunch of dsvm jobs waiting for node assignments, but there are a bunch of jobs in general waiting for node assignments17:24
*** cody-somerville has joined #openstack-infra17:24
*** cody-somerville has joined #openstack-infra17:24
*** ddieterly is now known as ddieterly[away]17:24
fungikro_focused: more likely the jobs you see completing sooner are using ubuntu-trusty and most of the dsvm jobs for master are running on ubuntu-xenial and we might not be building enough of the latter17:25
kro_focusedYa know, this seems like a topic for the cross-project technical goals thread.17:25
openstackgerritMerged openstack-infra/system-config: Install its-storyboard plugin on review-dev.o.o
dhellmannfungi : ok, it's not worth worrying about right now. If we're rewriting those bots we can add it then17:25
*** slaweq_ has joined #openstack-infra17:25
*** hieulq_ has quit IRC17:25
kro_focusedfungi: I don't have enough insight into pool queue usage.17:25
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic: Make ironic-multitenant-network a voting job
*** hieulq_ has joined #openstack-infra17:27
*** e0ne has joined #openstack-infra17:27
fungikro_focused: yeah, i think one thing we're lacking right now is grafana graphs of job nodes by node type17:27
openstackgerritMerged openstack-infra/shade: Go ahead and admit that we return Munch objects
AJaegermtreinish: it was far too noisy for me ;)17:28
*** e0ne has quit IRC17:28
pabelangerfungi: kro_focused: People see value in knowing how many ubuntu-trusty nodes are running / building / deleting?17:29
mordredwoot! shade patch landed17:29
pabelangerfungi: kro_focused: If so, I was thinking about updating to include that info17:29
kro_focusedpabelanger: Well, as a measure of "Here's the number of consumers vs. the provided number of resources"17:30
fungikro_focused: yeah, looks like as a raw statistic we've got ~3x as many trusty nodes as xenial at the moment
kro_focusedpabelanger: Call it "Zuulconomics". More demand for xenial makes the cost of jobs higher ;)17:30
kro_focusedHah. And one lonely precise-node17:31
pabelangerright, but once we migrate more jobs away from ubuntu-trusty, xenial will become dominate17:31
fungikro_focused: nodepool is supposed to be interpreting the work requests in zuul's gearman to determine how many of what node type should be built17:31
mtreinishharlowja: I'm running that gerritbot patch off of firehose.o.o right now on #throwaway-gerritbot-mqtt-test-channel if you're curious17:31
harlowjamtreinish nice17:31
pabelangerkro_focused: fungi: should help with that17:32
*** slaweq_ has quit IRC17:32
*** shashank_hegde has joined #openstack-infra17:33
jeblairfungi: nodepool does do that, however, it has been altered to slightly favor less-used nodes17:34
fungiyep, or else under heavy demand we can end up never building them17:34
*** javeriak has quit IRC17:35
fungiif they represent a smaller ratio than one out of our usable aggregate quota17:35
*** rockyg has quit IRC17:36
pabelangerclarkb has an email out to ovh about upping our resource in ovh-gra1, I should follow up on that17:36
jeblairour current deficit is 3x as many xenial as trusty17:37
jeblair(and yes, that appears to be the inverse of what fungi said our actual is.  i don't know if that's coincidence or the way the allocator math works out)17:37
*** daemontool_ has quit IRC17:38
*** slaweq__ has joined #openstack-infra17:38
jeblairanyway, zuulv3 will be fifo.17:38
*** slaweq__ has quit IRC17:38
bkeroIt might be worth pointing out that afaict the dsvm jobs haven't been switched over yet.
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added documentation draft jobs for nodejs-based projects
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added documentation publish jobs for the JavaScript SDK
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Maintain Gerrit host keys in a global known_hosts
*** jsavak has quit IRC17:40
fungijeblair: mtreinish: rcarrillocruz: ^ as discussed last night, spitball system-config patch first to model the plan before i go implementing it across individual modules (so far only the puppet-jenkins depends-on is proposed)17:40
*** jsavak has joined #openstack-infra17:40
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: WIP: Add foreground mode
mtreinishfungi: cool, so that will just pass the content and the module is expected to write that to the known_hosts file?17:42
*** PollyZ has quit IRC17:43
*** e0ne has joined #openstack-infra17:43
fungimtreinish: that's the idea anyway. it gets rid of a lot of the hackery of composing individual known_hosts entries for starters. and having additional known_hosts entries on a system doesn't hurt even if they're unused17:43
openstackgerritMerged openstack-infra/project-config: Move {name}-tarball jobs over to xenial
*** shardy has quit IRC17:44
fungimtreinish: we could in theory add common entries for other things to the same publig global hiera entry in the future17:44
fungiwhich is why i didn't give it a gerrit-specific name17:44
mtreinishfungi: ah, yeah makes sense17:45
*** piet has quit IRC17:45
fungimtreinish: there were a few ways i considered going about it, but ultimately this seemed like the least complex and easiest to understand/update17:47
kro_focusedIs this a thing we want?
*** mat128 is now known as mat128|afk17:47
* kro_focused is starting to think no, because canonical locations of clouds.yaml17:47
fungikro_focused: is it likely to be something other than /etc/clouds.yaml?17:48
kro_focusedfungi: You tell me ;)17:48
fungier, /etc/openstack/clouds.yaml i mean17:48
* kro_focused wants his jobs not to break if infra decides to change that location.17:48
mordredI don't expect it to ever be anywhere else17:48
fungithis would be less infra deciding and more os-client-config deciding anyway17:49
fungithat's its default expected location afaik17:49
mordredalthough I do not have a problem witih exporting a variable with the canonical location in it17:49
fungii doubt it hurts to add it, though it will likely always remain entirely redundant, and it does slightly add to environment pollution17:50
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic: Make ironic-multitenant-network a voting job
*** amitgandhinz has joined #openstack-infra17:53
*** Na3iL has quit IRC17:53
*** julim has quit IRC17:53
*** yamahata has joined #openstack-infra17:54
*** nmagnezi has quit IRC17:54
*** claudiub has joined #openstack-infra17:54
*** EricGonc_ has joined #openstack-infra17:55
*** Na3iL has joined #openstack-infra17:56
*** julim has joined #openstack-infra17:56
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Splitting current/deprecated integ. test
*** EricGonczer_ has quit IRC17:57
*** amitgandhinz has quit IRC17:57
*** harlowja has quit IRC17:58
openstackgerritMerged openstack-infra/project-config: Sign all tarballs and wheels
*** ddieterly[away] is now known as ddieterly18:00
*** e0ne has quit IRC18:00
*** harlowja has joined #openstack-infra18:00
*** hieulq_ has quit IRC18:02
*** sambetts is now known as sambetts|afk18:02
*** fernnest has joined #openstack-infra18:04
*** piet has joined #openstack-infra18:04
*** rockyg has joined #openstack-infra18:05
*** PollyZ has joined #openstack-infra18:06
*** degorenko is now known as _degorenko|afk18:07
*** asettle has quit IRC18:07
*** tonytan4ever has quit IRC18:08
*** zz_dimtruck is now known as dimtruck18:09
*** tosky has quit IRC18:09
*** rbrndt has quit IRC18:10
*** eeiden_ has joined #openstack-infra18:10
*** cody-somerville has quit IRC18:10
*** cody-somerville has joined #openstack-infra18:11
*** acoles is now known as acoles_18:11
*** skipp has quit IRC18:12
*** stewie925 has quit IRC18:12
*** rbuzatu has joined #openstack-infra18:13
*** eeiden has quit IRC18:13
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
*** rbrndt has joined #openstack-infra18:15
*** pvaneck has joined #openstack-infra18:16
*** ddieterly is now known as ddieterly[away]18:16
*** ddieterly[away] is now known as ddieterly18:17
*** rbuzatu has quit IRC18:17
*** stewie925 has joined #openstack-infra18:17
*** rbrndt has quit IRC18:18
*** yamahata has quit IRC18:18
*** yamahata has joined #openstack-infra18:19
*** adrian_otto has quit IRC18:21
*** claudiub has quit IRC18:21
*** jpich has quit IRC18:22
*** electrofelix has quit IRC18:22
*** HeOS has quit IRC18:26
*** Na3iL has quit IRC18:26
*** rwsu has quit IRC18:27
openstackgerritJames E. Blair proposed openstack-infra/zuul: Correct default value for hold-following-changes
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: Restore support for examining commits in fake builds
openstackgerritJames E. Blair proposed openstack-infra/zuul: Restore test_parallel_changes
openstackgerritJames E. Blair proposed openstack-infra/zuul: Remove unused Inheritable class
bkeroprometheanfire: are these going to be added to a review?
* bkero is doing tab culling and have a diff open18:29
rcarrillocruzfungi: ++, will check in a bit18:29
*** jaosorior has joined #openstack-infra18:29
*** jaosorior has quit IRC18:30
*** skipp has joined #openstack-infra18:32
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: Improve support in fake builds
openstackgerritJames E. Blair proposed openstack-infra/zuul: Restore test_parallel_changes
openstackgerritJames E. Blair proposed openstack-infra/zuul: Remove unused Inheritable class
*** diltram has joined #openstack-infra18:36
diltramhey, is there any way to debug docs-gate?18:36
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: remove FakeWorker
diltramor where I can find info how to setup my own docs gate18:37
jeblairdiltram: do you have a link to a failed build?18:37
*** amotoki has quit IRC18:37
openstackgerritMerged openstack-infra/tripleo-ci: Add opstools candidate repo
diltramjeblair: exactly here18:38
*** ayoung has quit IRC18:38
diltramwe should have an image generated using pydotplus and there is nothing18:39
*** annegentle has joined #openstack-infra18:39
diltramand we don't know what is going one18:39
*** ccamacho|out is now known as ccamacho18:39
*** ddieterly is now known as ddieterly[away]18:40
prometheanfirebkero: no, that's just so everything is happy with puppet failing18:42
*** piet has quit IRC18:43
*** rbuzatu has joined #openstack-infra18:43
prometheanfirethat's git stash material, not commit :D18:43
bkeroI was wondering if it was necessary to make gentoo work with system-config, and if it was, if it was going to land18:45
prometheanfireit's not, I need to start work on the broken puppet peices is all18:45
bkeroI should be familiar with that if I can get my build to go that far :)18:46
prometheanfirebkero: what's odd is that you can see in the logs that puppet apply is failing, but DIB doesn't care18:46
*** julim has quit IRC18:46
bkeroprometheanfire: that is odd18:46
prometheanfirebut for me it works out, since that's not what I'm targeting for a fix yet18:48
*** rbuzatu has quit IRC18:48
fungidiltram: the associated job logs for that build can be found by replacing docs-draft with logs like so:
diltramfungi: great, thank you very much :)18:49
fungidiltram: this looks suspect...
*** piet has joined #openstack-infra18:50
fungi"dot exited with error: Warning: using box for unknown shape cylinder18:50
fungidiltram: ^18:50
diltramfungi: yep - Database [shape=cylinder];18:51
fungiand then "*** stack smashing detected ***: dot terminated" (not sure if the warning was related to the stack smashing)18:51
rcarrillocruzShrews , mordred : shade is back to passing func no tests, revert on designate code fixed it18:52
rcarrillocruzmugsie: ^ thx!18:52
mordredrcarrillocruz: woot!18:52
*** _sarob has joined #openstack-infra18:53
*** amitgandhinz has joined #openstack-infra18:54
openstackgerritMerged openstack-infra/zuul: Add docstrings to the model
*** jed56 has quit IRC18:55
*** adrian_otto has joined #openstack-infra18:56
openstackgerritMerged openstack-infra/zuul: Add an internals doc to the documentation
openstackgerritMerged openstack-infra/zuul: Rename BasePipelineManager to PipelineManager
*** sarob has quit IRC18:56
*** claudiub has joined #openstack-infra18:56
*** eharney has joined #openstack-infra18:58
*** aviau has quit IRC18:58
*** aviau has joined #openstack-infra18:58
*** amitgandhinz has quit IRC18:58
*** karthik__ has joined #openstack-infra19:01
*** claudiub has quit IRC19:02
prometheanfirebkero: let me know if anything else comes up :D19:02
*** jsavak has quit IRC19:03
bkeroprometheanfire: Sure thing. I'm still running into the perm denied because of /dev/shm19:03
prometheanfirebkero: your output from earlier didn't seem to include my debug code19:04
*** xarses_ is now known as xarses19:04
bkeroprometheanfire: Yeah, that's why I usually put some obvious non-whitespace characters in it, to emphasize it19:04
prometheanfirebkero: there :P
bkeroyeah, running now19:06
*** fifieldt has quit IRC19:07
prometheanfireagain, gate is working...19:07
bkeroThat's fantastic19:07
*** tonytan4ever has joined #openstack-infra19:08
*** EricGonczer_ has joined #openstack-infra19:09
*** rbrndt has joined #openstack-infra19:09
*** mctaylor has joined #openstack-infra19:10
*** EricGonc_ has quit IRC19:13
*** tonytan4ever has quit IRC19:13
prometheanfirebkero: ah I wonder if it's exiting because it returned 119:14
anteayafungi: was taking a personal day today then it occured to me that you or any other infra root could pull together the electroral roll for the requirements ptl election that is scheduled to begin on friday19:14
prometheanfireit didn't do the stat for instance19:14
anteayafungi: so i thought I would come online with a reminder and to express my gratitude for your help19:15
anteayaso thank you19:15
*** Sukhdev has joined #openstack-infra19:15
diltramfungi: ok, I know that this error doesn't matter, but I found out that this docs gate is not running tox -edocs from project but some "gate" scripts in venv only19:15
anteayaany time you have a chance to create that and email it to myself and dhellmann between now and friday morning is wonderful19:15
bkeroI commented out the grep19:16
prometheanfireso no stat output?19:16
prometheanfirethat seems odd19:17
prometheanfiremaybe ls -l /dev and see what it says shm is?19:17
prometheanfirein the debug19:17
*** ddieterly[away] is now known as ddieterly19:17
*** fifieldt has joined #openstack-infra19:19
prometheanfirebkero: looks like it's a normal directory19:19
prometheanfirethought you said it was mounted :|19:19
*** jsavak has joined #openstack-infra19:19
bkeroit is19:19
prometheanfirethen why doesn't /proc/mounts know about it19:20
bkerotmpfs           5.9G  1.3M  5.9G   1% /dev/shm19:20
bkeroit does19:20
bkero[bkero@Whitefall shm]$ grep /dev/shm /proc/mounts19:20
bkerotmpfs /dev/shm tmpfs rw,nosuid,nodev 0 019:20
prometheanfirebut in the script it returned 119:20
bkerolooks like it returned 0 to me19:20
prometheanfireline 448
bkerodib overall returned 1, yes19:21
prometheanfirewell, why did it exit before19:21
prometheanfireline 440 is the first19:21
prometheanfirethere we did the grep19:22
prometheanfiregrep returned nothing (caused it to exit 1), causing the script to exit and then dib to exit19:22
prometheanfireall fails19:22
*** _ari_ has quit IRC19:22
* bkero runs with -x19:22
*** mctaylor has quit IRC19:22
fungianteaya: gladly! i'll get that running now19:22
bkerogreghaynes: did you say you created a dib IRC channel?19:22
greghaynesbkero: we need to propose the channel creation stuffs19:23
greghayneswe decided were going to do it, no one has19:23
*** zul_ has joined #openstack-infra19:23
prometheanfiregreghaynes: I'd join :D19:23
* bkero sitting in #diskimage-builder19:23
anteayafungi: thank you19:23
fungidiltram: correct, the job basically runs `tox -e venv python build_pshinx` per the consistent testing interface documented on governance.openstack.org19:23
greghayneswe really should request access before folks join otherwise its a pain to get ops back19:24
fungidiltram: so you should in theory be able to reproduce the problem that way19:24
prometheanfiregreghaynes: if it's unregistered that helps19:24
anteayafungi: I think you already have all the information you need to run that, yes?19:24
diltramfungi: this is a problem :P it's not running our additionall script which is generating that images19:24
fungianteaya: correct, i'll simply rerun it the same way i did last time19:24
jeblairdiltram: sorry, i got pulled away from the keyboard! thanks fungi for jumping in :)19:24
anteayafungi: okay great, thank you19:24
anteayafungi: going offline again19:24
fungidiltram: right, you probably need to implement it as a sphinx extension19:25
fungidiltram: rather than additional commands in tox or a wrapper script19:25
diltramok, thx fungi, jeblair for your help, I'm gonna take a look into that19:25
bkerogreghaynes: what's the process for registering a new channel? just says ask in here or on ML19:25
greghaynesbkero: you now know more than I do about it :)19:26
bkerojeblair: thank you!19:26
jeblairbkero: all of the services we help run are documented here:
fungior should be, and if they're not, that's a bug ;)19:27
* bkero nods19:27
*** asettle has joined #openstack-infra19:27
bkerobah, meeting. I'll do the channel stuff when I get back19:28
jeblairfungi, mtreinish: is there an outstanding patch to add mqtt docs?19:28
*** mctaylor has joined #openstack-infra19:28
*** piet has quit IRC19:28
fungijeblair: i think the spec is incomplete until they're added, at a minimum19:28
jeblairfungi: yes, the spec calls for them:
*** piet has joined #openstack-infra19:29
funginow that it's actually running, i agree it's time (even overdue) for service documentation in system-config19:29
fungii'll whip something up and get mtreinish to correct it19:29
jeblairfungi: i just thought i saw something from mtreinish along the lines of "i think that's all the work items in the spec done"19:29
fungioh! i'll check outstanding changes first then19:30
*** jsavak has quit IRC19:30
jeblairwhich, i suppose may technically be true since there isn't a work item for "write docs" but i would expect that to be implied by "deploy server"19:31
*** jsavak has joined #openstack-infra19:31
fungiyeah, not finding any but starting to flesh it out now19:31
*** tesseract- has joined #openstack-infra19:31
*** rockyg has quit IRC19:33
*** HeOS has joined #openstack-infra19:34
*** clenimar has quit IRC19:34
*** jcoufal has quit IRC19:34
*** clenimar has joined #openstack-infra19:35
*** zul_ has quit IRC19:35
*** gouthamr has quit IRC19:35
*** zul_ has joined #openstack-infra19:35
*** tesseract- has quit IRC19:37
rhedlindzuul question: zuul was killed by the system because it ran out of memory. Now whenever I try to restart zuul it won't create the file. No hints in the log files as to why. Is there another location where zuul keeps a persistent state that might make a new zuul process think another copy is already running?19:38
jeblairrhedlind: you may need to delete a stale .pid file19:39
jeblairrhedlind: ours is in /var/run/zuul/zuul.pid19:39
rhedlindjeblair: checked there. nothing in that folder19:39
*** eeiden_ has quit IRC19:39
*** rwsu has joined #openstack-infra19:40
jeblairrhedlind: you might try running as the zuul user with '-d' to tell it not to daemonize and see what happens19:40
*** bhavik has quit IRC19:41
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic: Make ironic-multitenant-network a voting job
*** ihrachys has joined #openstack-infra19:41
mordredalso, are you accidentally trying to start it as the wrong user?19:42
*** asettle has quit IRC19:42
rhedlindI'm trying to start it as root19:43
*** gouthamr has joined #openstack-infra19:45
*** ayoung has joined #openstack-infra19:46
*** rockyg has joined #openstack-infra19:46
*** roxanaghe has quit IRC19:48
*** rbuzatu has joined #openstack-infra19:49
*** tonytan4ever has joined #openstack-infra19:49
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add undercloud job in cistatus
*** Goneri has joined #openstack-infra19:51
mtreinishfungi: I forgot to push the doc patch, give me one sec I need to double check it and then I'll push it up19:51
fungimtreinish: oh, you already have one? i'll pop it back off my queue and review yours in that case19:52
mtreinishfungi: yeah, it's been sitting in my local repo since yesterday, I just never pushed it19:53
mtreinishfungi: of course I doubt it's complete enough :) But it's a starting point if nothing else19:53
*** rbuzatu has quit IRC19:53
*** amitgandhinz has joined #openstack-infra19:54
*** jpr1 has quit IRC19:54
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add docs for firehose.o.o
mtreinishfungi: ^^^19:55
therveI just discovered openstack-health thanks to Armando's email19:56
mordredmtreinish: so, can I use that mqtt broker for my home automation hub?19:56
therveHow reliable is the data in there?19:56
therveIt seems to be incorrect for heat at least19:56
*** diltram has left #openstack-infra19:56
*** mhickey has joined #openstack-infra19:56
*** Ravikiran_K has quit IRC19:57
mtreinishmordred: well you can because you have heira access. But we locked down write access to a single user19:57
mordredmtreinish: ah - ok19:57
mordredmtreinish: I did not catch the user existence from the doc19:57
mtreinishmordred: feel free to -1 on that if you think I should mention it's read only for external users19:58
*** dprince has quit IRC19:58
*** amitgandhinz has quit IRC19:59
*** asettle has joined #openstack-infra19:59
*** asettle has quit IRC19:59
mtreinishtherve: the data is fine, there are a few limitations though. It's based on subunit2sql which means it only has data if the run emitted subunit. So if something failed before devstack (or the test runner in non-dsvm jobs) a db record may not have been created for it19:59
mordredmtreinish: nope! I +2'd it already19:59
mordredmtreinish: I think it's great, because I now know more than I did before I read it19:59
mordredall docs can always be improved :)19:59
*** rossella_s has quit IRC20:00
thervemtreinish, Hum, ok. I wonde if we emit subunit properly then20:00
therveBecause we didn't have a failure today, which sounds suspicious :)20:00
*** rossella_s has joined #openstack-infra20:00
mtreinishtherve: do you have a link to a run which failed?20:01
*** permalac has quit IRC20:02
pabelangerfungi: jeblair: if projects wanted to send data to graphite.o.o from their test jobs, is that something we'd entertain?20:02
therveThere is almost empty testrepository.subunit.gz file20:02
mtreinishtherve: yeah so the subunit output there (in logs/testrepository.subunit) only has devstack20:03
* therve nods20:03
fungipabelanger: what sort of data would they send?20:03
fungipabelanger: also, curious how we'd wrangle access controls around that20:03
mordredpabelanger: SpamapS was working on a thing to collect some metrics and then have a something read the results and inject them into graphite iirc20:03
mtreinishtherve: whatever you're using to run testr has to do something like:
mordredfungi: ^^20:03
*** permalac has joined #openstack-infra20:03
mordredSpamapS: did you ever get anywhere with that?20:03
thervemtreinish, Awesome, thank you20:04
mtreinishtherve: to put the subunit output in the right place20:04
pabelangerfungi: for example statsd gauges, or timers20:04
mtreinishmordred: SpamapS's qa bluebrint/spec is still open. He never finished it :(20:04
pabelangermordred: SpamapS: interesting20:04
* pabelanger googles20:04
mtreinishpabelanger: ^^^20:05
mordredyah. that20:05
mordredthat's basically looking at the same problem space20:05
fungipabelanger: statsd gauges and timers of what though?20:05
pabelangercool, okay so there has been some discussion are that20:05
mordredfungi: well, in the case that SpamapS was working on, it was db usage counters from running tests - so that things like total number of rows or total number of queries could be tracked20:05
pabelangerfungi: in the case of tripleo-ci, they are using it do track how long sub tasks run when a job is running20:06
mordredas a way to spot significant usage increases20:06
fungipabelanger: could they record timing data in a subunit file?20:06
*** matrohon has joined #openstack-infra20:07
fungithis sounds more like the stuff we're stashing in the subunit2sql db20:07
pabelangerfungi: for example:
pabelangerfungi: I think if we proposed a different solution, they would be open to it20:07
pabelangerfor now, they have some random graphite server on the public web collecting the data20:07
pabelangerwhich, I have no idea, what they use it for20:07
pabelangertrying to track that down now20:07
*** bhunter71 has quit IRC20:08
*** EricGonc_ has joined #openstack-infra20:08
mtreinishpabelanger: that does sound a lot like what subunit2sql is tracking for test runs20:08
fungiSpamapS' partial work notwistanding, we don't currently have jobs tracking granular timing data in graphite but we _do_ have them tracking it in subunit2sql (openstack-health, stackviz, et cetera)20:08
mordredmtreinish: ++20:08
pabelangerthen I can start with that20:08
pabelangerya, stackviz would work well I think20:09
fungithey they wrote their tests as a tempest plugin (no idea how feasible that is) they could get it almost automagically20:09
*** EricGonczer_ has quit IRC20:09
fungier, if they20:09
mtreinishfungi: if it uses python unittest there is no real barrier20:10
*** Goneri has quit IRC20:10
*** daemontool has joined #openstack-infra20:10
fungithis is one of those places where rewriting their ci implementation on their own from scratch has penalized them and cut them out of opportunities to take advantage of tooling we've built on top of our standardized solutions20:10
pabelangermtreinish: sadly this would be using bash20:11
*** coolsvap has quit IRC20:11
mtreinishpabelanger: heh, then yeah using tempest plugins probably won't work20:11
pabelangerhowever, I am trying to bring them more inline with openstack-infra tooling, so changing it could be something that needs to be done20:12
timothyb89pabelanger: you can also generate subunit from bash with subunit-output20:12
mtreinishtherve: that or generate-subunit (which is what devstack uses)20:12
timothyb89oh, cool, I hadn't seen that20:13
mtreinishit's basically a much simpler subunit-output, but it supports setting timestamps20:14
timothyb89ah, yeah, that is a pretty vital feature missing from subunit-output :)20:14
pabelangertimothyb89: mtreinish: okay great. Looks like I know who to ping with questions20:15
bnemecDoing what devstack does makes sense, since we're doing essentially the same thing as it is.20:15
bnemecTrying to wrap tripleo-ci in Tempest or Python unit tests would be a very weird fit.20:15
mtreinishtimothyb89: only open for 8 months :)20:16
phschwartzjeblair: Hey, you around? I have a nodepool issue that I want to talk to you about.20:17
phschwartzfungi: ^ I haven't found the cause, but found a nodepool issue that has lead to issues for us. You might want to look at that log because you never know if infra will hit the issue.20:18
jeblairphschwartz: howdy20:18
jeblairphschwartz: the issue is a node ran 2 jobs?20:19
phschwartzjeblair: that log I posted shows 2 unique jobs both being run by jenkins on the same node.20:19
phschwartzjeblair: we are seeing it happy in about 40-50% of our runs now20:19
jeblairphschwartz: i don't see OFFLINE_NODE_WHEN_COMPLETE in the parameters list there20:19
jeblairphschwartz: is that not being set by zuul for some jobs?  maybe a job regex in zuul's layout.yaml accidentally caused the function that adds that to stop being applied to some jobs?20:20
fungiwhich is the first thing i looked for in there as well20:20
*** ihrachys has quit IRC20:20
phschwartzjeblair: hmm, I think a change might have caused that to go away. I missed it completely. ty20:21
phschwartzfungi: you as well :)20:21
fungiphschwartz: missing that parameter is just about always the reason for jobs to get rerun on the same nodes. another common (but less frequent) cause is a jenkins master restart causing it to bring nodes back online that haven't been deleted yet20:23
*** adrian_otto has quit IRC20:24
phschwartzfungi: I didn't realize the second. Good to know. nibalizer and I were discussing this earlier and the thought then would be OFFLINE_NODE_WHEN_COMPLETE20:24
fungiyeah, jenkins is really diligent about assuming that nodes are only ever accidentally offline and will try to bring them online again if restarted20:25
fungiso before we ceased using jenkins entirely, as procedure any time we restarted a jenkins master we'd remove slave entries from its config on disk20:25
*** eeiden has joined #openstack-infra20:26
fungiannoying, but did the trick20:26
*** HeOS has quit IRC20:28
*** tchaypo has joined #openstack-infra20:30
*** ddieterly is now known as ddieterly[away]20:32
*** mhickey has quit IRC20:33
*** karthik__ has quit IRC20:33
*** Jeffrey4l_ has joined #openstack-infra20:35
*** Jeffrey4l has quit IRC20:35
fungimtreinish: 350764 looks good. would also be nice to have the quick one-liner examples of tools you've been using to test its straight mqtt and websockets interfaces20:36
fungimakes it a lot easier for people to check that it's working as intended20:36
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove test-environments from tripleo-ci
mtreinishfungi: like the mosquitto_sub command I was using?20:38
fungimtreinish: yep20:39
*** roxanaghe has joined #openstack-infra20:39
fungiand maybe a sentence mentioning the package names that provide those client tools20:39
*** baoli_ has quit IRC20:39
*** tchaypo_vacay has quit IRC20:39
*** ddieterly[away] is now known as ddieterly20:40
*** baoli has joined #openstack-infra20:40
mtreinishfungi: I can do that, want me to respin or followon patch it20:40
fungimtreinish: whichever is easier for you. i'm happy to write it too, but that would take research on my part to piece together which you likely already have handy20:42
*** csomerville has joined #openstack-infra20:42
*** mctaylor has quit IRC20:43
mtreinishfungi: well I was probably just gonna link to:
*** csomerville has quit IRC20:44
mtreinishand then just put "mosquitto_sub -h --topic '#'" as an example command20:45
*** csomerville has joined #openstack-infra20:45
fungiand you had another command for testing the websockets interface right?20:45
*** mctaylor has joined #openstack-infra20:46
*** jsavak has quit IRC20:46
*** cody-somerville has quit IRC20:46
mtreinishor a python script that uses paho-mqtt to establish a connection over websocket20:46
*** jsavak has joined #openstack-infra20:46
*** daemontool has quit IRC20:48
fungiindeed, the mosquitto-clients package on debian seems to get me a working mosquitto_sub for that20:50
mtreinishif you want to use websocks and not via a random js app hosted on github I found from a google search20:51
*** Sukhdev has quit IRC20:52
fungibut that sounds so fun20:52
*** Sukhdev has joined #openstack-infra20:53
fungifor the what-mqtt page, what does your ws:// url look like? i'm not having a lot of luck guessing20:53
*** amitgandhinz has joined #openstack-infra20:55
*** furlongm has joined #openstack-infra20:57
mtreinishfungi: ws:// is what I was using before20:57
mtreinishit still works20:57
*** furlongm_ has quit IRC20:57
mtreinishyou've got to hit connect first, then subscribe20:57
*** jsavak has quit IRC20:58
fungiyeah, tried that with # as the topic, similar to your mosquitto_sub example20:59
*** acoles_ is now known as acoles20:59
*** amitgandhinz has quit IRC21:00
fungimight just be my browser is terrible21:00
mtreinishfungi: meh, don't worry about it, the python script I pasted should work just as well21:01
mtreinishalthough I should try running it first :)21:01
fungiahh, it's never even giving me "connection open" when i try to connect21:01
fungiso could be some browser plugin interfering21:01
fungimaybe privacy badger or something21:01
*** matt-borland has quit IRC21:03
fungithere we go, that was it21:04
fungieff privacy badger didn't like it opening a websocket to another domain i think21:04
mtreinishfungi: that would make sense. Normally you probably wouldn't want that happening21:05
mtreinishfungi: hmm, so maybe that python script doesn't work. It doesn't seem to actually be establishing a connection for me21:05
fungirunning afoul of privacy python, no doubt ;)21:06
*** dkranz has quit IRC21:06
mtreinishfungi: hmm, the script works fine without setting it to be a websocket transport (on the regular mqtt port)21:08
*** yamamoto has joined #openstack-infra21:09
mtreinishfungi: oh, but if I set the transport to websocket and don't set it to port 80 it works fine but with regular mqtt traffic21:10
mtreinishI suspect something is quite right with the library21:10
fungiquite right indeed21:10
*** rbuzatu has joined #openstack-infra21:10
fungior the not kind of quite right maybe21:10
*** jkilpatr has quit IRC21:11
mtreinishheh, oops. Yeah missed a not there21:11
jeblairpabelanger: is the only use of ZUUL_COMMIT in project-config....21:11
mtreinishfungi: haha, it's 'websockets' not websocket'21:11
jeblairpabelanger: how do the files get there in the first place?21:11
jeblairpabelanger: (that usage is downloading from that URL -- where are they uploaded to that path?)21:11
mtreinishfungi: works21:12
*** ldnunes has quit IRC21:13
mtreinishI don't know why it doesn't raise an exception when you specify an invalid transport type though21:13
mtreinishthat's kinda annoying21:13
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove test-environments from tripleo-ci
fungimtreinish: cool, that's pretty trivial indeed21:13
*** rbuzatu has quit IRC21:15
*** fifieldt has quit IRC21:15
pabelangerjeblair: current is it using the deb-pkgs publisher to upload:
fungiahh, the package on pypi is paho-mqtt21:16
*** ddieterly is now known as ddieterly[away]21:16
pabelangerjeblair: and zigo as logic in his build script to setup the path for publisher21:16
jeblairpabelanger: where's that script?21:16
zigopabelanger: Hi there! How do I get deb-auto-backports to publish to the right repo?21:17
*** kgiusti has left #openstack-infra21:17
pabelangerjeblair: deb-openstack-pkg-tools currently21:17
*** yamamoto has quit IRC21:18
jeblairpabelanger, zigo: is that script copied to every branch in that repo?21:18
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
pabelangerjeblair: I think only newton today21:19
mtreinishfungi: ah, yeah it's a bit confusing21:19
*** EricGonc_ has quit IRC21:19
mtreinishto make it a bit more confusing it's part of the mosquitto project too :)21:19
pabelangerjeblair: I think we could move that logic into project-config, if you see a problem with it being in deb-openstack-pkg-tools21:19
mtreinisherr, or maybe not. It's definitely also part of eclipse21:20
mtreinishI can't keep track21:20
*** EricGonczer_ has joined #openstack-infra21:21
zigopabelanger: Can you help me to get to publish to the correct backport repo?21:22
zigopabelanger: That infra patch is the last one before we really cat get going.21:22
fungimtreinish: yeah, i couldn't find distro packages. closest i have is: golang-eclipse-paho-dev - Go package implementing MQTT client library by Eclipse Paho21:22
*** bhunter71 has joined #openstack-infra21:22
pabelangerzigo: yes, we need to update the reprepro-import job to accept a different dist21:23
zigopabelanger: Oh, so that's something for you then? Or should I try?21:23
fungimtreinish: confirmed websockets poc script works for me as well21:24
*** gouthamr has quit IRC21:24
pabelangerzigo: you should be able to use the reprepro-import-debian-openstack-jessie-backports job and it works21:24
mtreinishfungi: I think this is it:
*** seldenr has joined #openstack-infra21:25
*** fifieldt has joined #openstack-infra21:25
mtreinishor maybe not, looking at the file list I don't know what that is21:26
pabelangerjeblair: do you see an issue with the implementation?21:26
zigopabelanger: How to do this?21:27
*** baoli has quit IRC21:27
zigopabelanger: Currently, we only have one job for publishing, called deb-pkgs21:27
zigopabelanger: How is that linked to the post job that import what's pushed to tarballs/packaging-deb ?21:28
pabelangerzigo: I commented on the review, should be the only change you need21:29
zigopabelanger: Oh, thanks so much !21:30
openstackgerritMerged openstack/os-client-config: Add release notes for 1.19.0 release
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add automatic backports
*** matrohon has quit IRC21:32
*** rhallisey has quit IRC21:32
*** _sarob has quit IRC21:35
openstackgerritJay Faulkner proposed openstack-infra/project-config: Fix reversed logic in Ironic jobs
*** sarob has joined #openstack-infra21:35
*** ddieterly[away] is now known as ddieterly21:36
*** Qiming has quit IRC21:37
zzzeekhey, gerrit is marked as -1 when only a non-voting job fails, all others pass?
*** chem has quit IRC21:39
zzzeekoh.  POST_FAILURE, what is that.21:39
openstackgerritJay Faulkner proposed openstack-infra/devstack-gate: Explicitly set IRONIC_BUILD_DEPLOY_RAMDISK
*** Qiming has joined #openstack-infra21:39
jeblairpabelanger: i was considering dropping it in zuulv3... but i *think* i can keep it if it's useful -- is it desirable that the actual commit sha be used, or is it just an arbitrary identifier unique to the build (ie, we could use the zuul ref instead)?21:40
*** rwsu has quit IRC21:40
*** thorst has quit IRC21:40
jeblairzzzeek: something went wrong uploading the artifacts (eg build log, tarball, etc) after the main part of the job succeeded21:40
openstackgerritJay Faulkner proposed openstack-infra/devstack-gate: Explicitly set IRONIC_BUILD_DEPLOY_RAMDISK
zzzeekjeblair: K so doing a 'recheck'21:41
jeblairspeaking of which -- pabelanger should we start upgrading the launchers?21:41
jeblairzzzeek: yep.  there is a change in the newest release of ansible we hope will eliminate, or reduce, or at least illuminate the underlying problem21:41
zzzeek+1 for illumination21:42
jeblairpabelanger: do you want me to do a smoke test of the new ansible with my local zuul-launcher setup?21:43
*** sdake has quit IRC21:43
*** mctaylor has quit IRC21:44
JayFI'd apprecaite a review on as it fixes an Ironic gate issue where we build deploy ramdisks for every job, causing more unneccessary random failures + eating up more gate resources than needed21:45
pabelangerjeblair: I am happy to update our script to use something else, sha1 was the first suggestion we talked in Austin about it.  If you think zuul_ref is better, lets do that21:45
pabelangerjeblair: sure, if you'd like to do the test21:46
jeblairpabelanger: okay, let me think about the zuul_commit thing some more21:46
jeblairpabelanger: i'll run the test in a min.21:46
jeblairJayF: that change appears only to remove whitespace21:47
JayFjeblair: egad21:47
JayFjeblair: 1s21:48
openstackgerritJay Faulkner proposed openstack-infra/project-config: Fix reversed logic in Ironic jobs
JayFjeblair: ^ fixed the change, sorry about that21:48
jeblairJayF: you're not using that programming language where the tokens are just spaces and tabs, are you? :)21:48
JayFjeblair: no, I'm using a faulty brain which sometimes swaps values twice, making them the same, lol21:49
*** shamail has quit IRC21:49
JayFjeblair: I'm not convinced that's a good fix anymore, going to -W it and dig in again, make sure I fully understand21:52
JayFsomething is rotten in that project-config flow and I can't quite find it :(21:52
*** vvalderrv has left #openstack-infra21:52
*** burgerk_ has joined #openstack-infra21:53
*** sarob has quit IRC21:53
*** sarob has joined #openstack-infra21:54
*** rockyg has quit IRC21:55
*** amitgandhinz has joined #openstack-infra21:56
*** burgerk has quit IRC21:56
*** burgerk_ has quit IRC21:58
*** markvoelker has quit IRC21:58
*** markvoelker has joined #openstack-infra21:59
jeblairpabelanger: wfm21:59
*** rlandy has quit IRC21:59
*** esberglu has quit IRC22:00
*** amitgandhinz has quit IRC22:00
*** sarob has quit IRC22:01
*** mriedem has quit IRC22:01
*** eharney has quit IRC22:02
*** rwsu has joined #openstack-infra22:03
*** florianf has quit IRC22:03
*** eharney has joined #openstack-infra22:03
*** thorst has joined #openstack-infra22:03
*** markvoelker has quit IRC22:03
*** mctaylor has joined #openstack-infra22:04
*** signed8bit is now known as signed8bit_Zzz22:04
*** mctaylor has quit IRC22:04
*** mctaylor has joined #openstack-infra22:05
*** piet has quit IRC22:05
pabelangerjeblair: great, ready here. 349103 needs +A22:05
*** adrian_otto has joined #openstack-infra22:06
*** mdrabe_ has quit IRC22:07
jeblairpabelanger: done22:07
*** thorst has quit IRC22:07
*** mmandel has quit IRC22:08
*** acoles is now known as acoles_22:09
openstackgerritJay Faulkner proposed openstack-infra/project-config: Forbid ironic grenade job from building ramdisk
JayFjeblair: ^ that fixes the real issue, sorry for getting it wrong the first time22:10
JayFWhat's supposed to be the proper interaction between project-config and devstack-gate? Like I'm not sure why that intermediate DEVSTACK_GATE_IRONIC_BUILD_RAMDISK exists at all, vs just setting it in devstack configs through the gate22:10
*** rbuzatu has joined #openstack-infra22:12
openstackgerritK Jonathan Harker proposed openstack-infra/project-config: Add integration tests between system-config and logstash-filters
*** rwsu has quit IRC22:13
*** annegentle has quit IRC22:13
*** tphummel has quit IRC22:14
*** signed8bit_Zzz is now known as signed8bit22:16
*** adrian_otto has quit IRC22:16
*** rbuzatu has quit IRC22:17
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: re-register functions after disconnect
mtreinishfungi: were you working on the doc patch for the client side mqtt stuff? If not I can write something up in a bit22:18
jeblairpabelanger: we should probably merge that -- it's important, ^ but i'm not sure we need to hold up the current ansible-upgrade restart for it.22:18
*** signed8bit has quit IRC22:18
openstackgerritMerged openstack-infra/puppet-zuul: Upgrade to ansible
pabelangerjeblair: ack22:20
*** dimtruck is now known as zz_dimtruck22:20
jeblairJayF: i think there's an effort to batch up some of those pass-through variables, though i don't have further details at hand... sdague may know?22:20
jeblairJayF: (but yes, in principle, d-g really only needs to know the ones it needs to know or change via its branch-selection logic)22:21
mtreinishjeblair: well we want to move a lot of it to just write directly to the config files22:24
*** rwsu has joined #openstack-infra22:24
mtreinishjeblair: sdague has a series up for using a local.conf to enable that:
mtreinishwhich has been sitting for a while22:24
mtreinisha lot of those intermediate variables in d-g predate the existence of the devstack passthrough var being in devstack-gate22:25
openstackgerritJay Faulkner proposed openstack-infra/devstack-gate: Explicitly set IRONIC_BUILD_DEPLOY_RAMDISK
jeblairmtreinish: yeah, they originated when external configuration of devstack other than what d-g explicitly provided was not a thing (during the time of 'core projects' :)22:27
JayFmtreinish: will put this here as well; I'll file a bug in Ironic for that to get deprecated out of devstack in this case; I just lost an hour+ tracking down gate weirdness b/c I didn't know how all the misdirection worked :)22:27
JayF(although now I know, which is nice)22:28
*** yamamoto has joined #openstack-infra22:29
*** gordc has quit IRC22:29
*** rfolco has quit IRC22:30
*** ddieterly has quit IRC22:31
*** yamamoto_ has joined #openstack-infra22:31
*** yamamoto has quit IRC22:31
*** ddieterly has joined #openstack-infra22:32
fungimtreinish: happy to write it up, but probably won't be until tomorrow22:33
*** karthik__ has joined #openstack-infra22:34
mtreinishfungi: ok cool, thanks. I probably wouldn't get to it any sooner either22:34
*** ddieterly has quit IRC22:35
*** julim has joined #openstack-infra22:37
*** Teresita-Warrior has quit IRC22:38
*** rwsu has quit IRC22:40
pabelangerjeblair: looks like zlstatic01.o.o is upgrade. Did we want to start with that?22:40
jeblairpabelanger: ++22:41
jeblairpabelanger: i'm watching logs; you want to hit the button?22:41
*** kien-ha has joined #openstack-infra22:43
pabelangerjeblair: stopped now22:43
pabelangerand starting22:43
pabelangerjeblair: actually, do we need to restart zuul-launcher if ansible-playbook is launched as a process?22:44
*** cody-somerville has joined #openstack-infra22:44
jeblairpabelanger: erm, i think you're right, we probably don't22:44
pabelangerYa, I see errors in zlstatic01.o.o about ansible-playbook missing22:45
pabelangerwhich pip uninstalled22:45
pabelangerin that case, we just monitor now22:45
*** rwsu has joined #openstack-infra22:45
*** zul_ has quit IRC22:46
jeblairso zuul will probably relaunch a bunch of jobs because of that22:46
pabelangerYa, I can see that in grafana22:46
*** csomerville has quit IRC22:47
pabelangernot so rolling upgrades22:47
*** mctaylor has quit IRC22:47
*** zz_dimtruck is now known as dimtruck22:48
pabelangerjhesketh: we just upgrade zuul-launcher to ansible
jheskethpabelanger: oh neat :-)22:49
*** thiagop has quit IRC22:51
jeblairjhesketh: is this patch which i accidentally self-approved (sorry!) what you had in mind in your earlier review comment?
jheskethjeblair: without having read it in detail, that looks like a good improvement :-)22:54
*** EricGonczer_ has quit IRC22:54
jheskethjeblair: the less we fake out the better, and just tidying up runAnsible seems like a great point to poke at things22:55
*** xyang1 has quit IRC22:56
*** amitgandhinz has joined #openstack-infra22:56
*** asettle has joined #openstack-infra22:57
jeblairpabelanger: i'm guessing the nodepool spike is the ansible upgrade (zuul fast-aborting and re-launching jobs when ansible was broken)22:59
jeblairpabelanger: it's returned to normal now which is good22:59
pabelangerjeblair: Yes, I agree with that23:00
*** amitgandhinz has quit IRC23:01
*** sdake has joined #openstack-infra23:03
*** seldenr has quit IRC23:07
*** sarob has joined #openstack-infra23:07
*** tpsilva has quit IRC23:08
*** asettle has quit IRC23:09
*** PollyZ has quit IRC23:09
*** pt_15 has quit IRC23:10
*** sarob has quit IRC23:11
*** rbrndt has quit IRC23:13
*** pt_15 has joined #openstack-infra23:14
JayFWe just had an Ironic job fail, with seemingly nothing wrong in it:
JayFno tests failed, even seeing Task exit code: 0, but at the end is result: FAILURE23:17
pabelangerjeblair: possible ansible issue^23:19
pabelangerThe async task did not return valid JSON: expected string or buffer23:19
pabelangerJayF: we just upgraded ansible on zuul-launchers, so this could be a side affect23:19
JayF this change is the one, a passing example + 2 failing examples23:20
JayF(the pep8 failure is real23:20
openstackgerritCraige McWhirter proposed openstack-infra/infra-specs: Initial commit of the Pholio spec
pabelangerJayF: try a recheck23:21
pabelangerJayF: I am checking out logs to see when we installed ansible
pabelangerAug  3 22:40:40 zl01 puppet-user[11566]: (/Stage[main]/Zuul::Launcher/Package[ansible]/ensure) ensure changed '' to ''23:22
pabelangerI wonder if the running ansible-playbook process, which was, started using the bits from the install which caused the issue23:24
JayFI'm going to fix the pep8 issue on that patch and repush23:24
pabelanger2016-08-03 22:44:03,713 p=19990 u=zuul |  fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "The async task did not return valid JSON: expected string or buffer"}23:25
pabelangerthe failure is 4mins after ansible was upgraded23:25
pabelangerso, timeframe matches up23:25
openstackgerritJay Faulkner proposed openstack-infra/project-config: Forbid ironic grenade job from building ramdisk
*** sdague has quit IRC23:29
*** mrmartin has quit IRC23:33
*** tonytan4ever has quit IRC23:33
*** fernnest_ has quit IRC23:34
*** dingyichen has joined #openstack-infra23:34
*** fernnest has quit IRC23:35
osnexusIf i have a devstack environment, how would I go about trying to make sure I can ssh into compute nodes? It seems I can only see the network node neutron spins up, but not its compute nodes23:37
docaedoosnexus: devstack is an all-in-one openstack environment, so the VM you run it on has all the services there23:41
*** Swami has quit IRC23:41
osnexusdocaedo right, but I can't see what the services are creating. I'm trying to use it as part of a 3rd party CI system, but the CI node needs to get at the compute nodes created by devstack23:43
osnexusBut nobody can see them23:43
osnexusexcept for the one neutron network node23:43
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
docaedowhen you say "compute nodes created by devstack" do you mean VMs devstack launches?23:44
*** zhurong has joined #openstack-infra23:45
*** _ari_ has joined #openstack-infra23:45
osnexusdocaedo yes, nodepool and jenkins on the CI VM send images over to the devstack VM which then spins them up23:46
osnexusAs VMs inside itself I believe23:46
osnexusVM inside a VM23:46
jeblairpabelanger, JayF: well that's the error than we hope the ansible upgrade will fix, so it's also possible that was just one more incidence of the error and proximity to the upgrade event is a coincidence.23:47
docaedoosnexus: ok, I kind of follow - but I'll leave third party CI debugging questions to the experts :) I thought maybe it was just a simple devstack question but I think this is a little more complicated23:47
jeblairmordred: is in merge conflict23:47
*** roxanaghe has quit IRC23:47
*** jamielennox is now known as jamielennox|away23:48
*** _ari_ has quit IRC23:48
osnexusya.... the complication never ends on this....23:49
JayFjeblair: we had two tests fail on the same patch in that way, so it's either an impact or we're very, very unlucky. I buy either explanation, and it doesn't matter anyway. The best kinds of bugs to discover are those someone else already fixed for you \o/23:51
*** sdake has quit IRC23:51
*** zhurong has quit IRC23:54
docaedoosnexus: don't know if this will help, but if devstack neutron uses provider network, VMs can be on the same network as devstack itself (which would reduce routing complexity at least), then make sure security groups are configured to allow traffic in to the vms23:54
*** sdake has joined #openstack-infra23:55
*** amitgandhinz has joined #openstack-infra23:57

Generated by 2.14.0 by Marius Gedminas - find it at!