Friday, 2016-06-03

fungier, actually exactly what you said00:00
funginow that i reread you00:00
fungiif we modify the copied tree, that would actually be more work on the launcher than merely indexing it, i think?00:01
jeblairfungi: that's a good point, a bit more io00:02
*** sdake_ has joined #openstack-infra00:02
*** sdake has quit IRC00:02
fungijust modifications to the filesystem index and not actually moving blocks, but still it's more writes00:02
*** thorst has joined #openstack-infra00:03
*** Sukhdev has joined #openstack-infra00:03
*** Sukhdev has quit IRC00:03
jeblairso maybe my ordered favorites are: (a) rsync down with embedded find, rsync up; (b) remote find, synchronize down, rsync up; (c) synchronize down, local mutation, rsync up00:04
openstackgerritMerged openstack-infra/project-config: Remove FIPs graphing from OSIC
patrickeasthi, maybe folks here have some ideas, i'm trying to change my ssh key for gerrit through and its getting stuck on 'Working ..." and eventually giving a 50000:05
patrickeastany suggestions?00:05
patrickeastderp nvm00:06
jeblairpatrickeast: wait a few minutes until the database backups have completed :(00:06
patrickeastsigning out and back in fixed it00:06
jeblairpatrickeast: (everything breaks for a few minutes at 00:00 utc)00:07
patrickeastjeblair: ohh, that would do it00:07
pabelangerfungi: have a minute to +A Removes our skip test logic based on the conversation just now with clarkb00:07
patrickeastjeblair: thanks!00:07
clarkbpabelanger: its possible I am the only one that feels the hand curating of these rules over time isn't worth it. Not sure how muh feedback we want to get. I just know that neutron and nova and other projects that have used them regularly run into similar problems00:08
clarkbpabelanger: in fact, I bet that those projects will be similarly broken if they switch to using other requirements00:08
*** thorst has quit IRC00:09
*** annegentle has quit IRC00:10
*** thorst has joined #openstack-infra00:10
fungijeblair: i was actually pondering (d) synchronize down, rsync up with embedded find. the volume of data copied will still be basically the same (we're just not creating subdirectories below the target in the second phase) and it's less change to the existing code00:11
fungiclarkb: pabelanger: i've long argued that the job filters based on changed file patterns is a misfeature which creates confusion through added complexity and increases the risk of not thoroughly testing a change00:12
*** sflanigan has joined #openstack-infra00:15
*** thorst has quit IRC00:15
pabelangerclarkb: fungi: I do like the functionality, for the simple fact we can limit our resources depending on the patch.  However, I also agree it does open us to the exposure of failures. I'm happy to switch it based on your comments.00:16
pabelangernow that osic is back online, we have more resources too00:16
*** rlandy has quit IRC00:16
*** masayukig has quit IRC00:16
*** Sukhdev has joined #openstack-infra00:17
*** xarses has quit IRC00:17
clarkbpabelanger: ya the issue is that computers are cheap and humans aren't. What that feature really does is externalize the cost of testing to infra when things don't work as expected00:17
clarkbpabelanger: which is great when you are working on neutron I guess but not so great for us00:17
*** masayukig has joined #openstack-infra00:18
pabelangerclarkb: I can agree with that logic00:18
*** rlandy has joined #openstack-infra00:19
*** jerryz has quit IRC00:19
openstackgerritGoutham Pacha Ravi proposed openstack-infra/project-config: Add manila-specs to specs.yaml
*** ayoung has joined #openstack-infra00:20
*** sarob has quit IRC00:20
*** Sukhdev has quit IRC00:21
*** piet_ has quit IRC00:21
openstackgerritMerged openstack-infra/puppet-openstack_health: Use the distributed memcached locked dbm cache backend
jeblairfungi: ok that works00:23
*** Swami has quit IRC00:23
*** yamahata has joined #openstack-infra00:24
jeblairfungi: i mostly agree, though i think "files" has a place, and requirements is a pretty good one for it.  how we define requirements changes once every 3 years on average.00:25
*** david-lyle has quit IRC00:25
jeblairi will sign up for reviewing changes to that once every 3 years00:25
jeblairmonthly is probably too much00:25
*** rhallisey has joined #openstack-infra00:26
*** david-lyle has joined #openstack-infra00:26
jeblair(skip-if, on the other hand, i generally find far too complex to use and maintain and will not sign up for reviewing those patches)00:27
*** banix has quit IRC00:29
lifelessclarkb: fungi: re the pylint discussion earlier; pylint is one of the blacklisted things we don't constrain because linters00:30
*** david-lyle has quit IRC00:30
*** xarses has joined #openstack-infra00:30
*** xarses has quit IRC00:33
*** xarses_ has joined #openstack-infra00:33
*** mriedem has joined #openstack-infra00:34
*** baoli has joined #openstack-infra00:36
*** r-mibu has quit IRC00:36
fungijeblair: yeah, i shouldn't have said "misfeature" but it does serve as an attractive nuisance for people with a compulsion to optimize away every last corner case where some job might not be entirely necessary00:36
*** Sukhdev has joined #openstack-infra00:37
*** kushal has quit IRC00:39
*** amitgandhinz has joined #openstack-infra00:39
*** markvoelker has joined #openstack-infra00:40
*** amotoki has joined #openstack-infra00:40
*** banix has joined #openstack-infra00:41
*** r-mibu has joined #openstack-infra00:42
*** amitgandhinz has quit IRC00:44
*** andreww has joined #openstack-infra00:45
*** amotoki has quit IRC00:45
*** tqtran has quit IRC00:46
*** asselin_ has quit IRC00:46
*** ffrank has quit IRC00:47
*** pvaneck has quit IRC00:47
*** xarses_ has quit IRC00:48
cloudnullclarkb fungi -- Will work for beer :P00:49
*** akshai_ has quit IRC00:50
*** Julien-zte has joined #openstack-infra00:51
openstackgerritIan Wienand proposed openstack-infra/project-config: Add dib-utils func test
*** rhallisey has quit IRC00:56
cloudnulllet me know how things go with the OSIC and the provider net setup. If anything needs to be adjusted I should be available for the next few days before I go back on holiday .00:56
*** fedexo has joined #openstack-infra00:57
*** annegentle has joined #openstack-infra00:58
clarkbcloudnull: will do but initial results look great00:59
clarkbI have some extra deschutes pine drop in my refrigerator if you want some00:59
clarkbit may not last very long00:59
bkerommmm deschutes01:00
*** mtanino has quit IRC01:01
clarkbbkero: I am trying to drink something other than wanderlust01:01
bkeroclarkb: Get some DeGarde01:01
clarkbI am actually planning to try having a keg of wanderlust available at all times at new house01:01
bkerokegerator investment01:01
*** Qiming has joined #openstack-infra01:02
*** Goneri has joined #openstack-infra01:05
*** zhurong has joined #openstack-infra01:07
*** sputnik13 has joined #openstack-infra01:08
*** sputnik13 has quit IRC01:08
*** mixos has joined #openstack-infra01:09
*** Sukhdev has quit IRC01:09
*** mixos_ has joined #openstack-infra01:10
*** mixos has quit IRC01:13
*** Sukhdev has joined #openstack-infra01:13
*** Sukhdev has quit IRC01:15
cloudnullclarkb: I love pine drop01:18
* cloudnull just discovered it01:19
cloudnullHEB had a family pack on sale the other day and it was a tasty investment :)01:19
*** cody-somerville has quit IRC01:21
* bkero is busy burning through his stash of Sticky Hands.01:21
bkeroThe consistently best IPA I've ever tasted.01:21
clarkbdo they distribute up here?01:22
*** baoli has quit IRC01:23
bkerothey do, they just started canning pints01:24
bkeroThey don't produce in volume enough to be in grocery stores yet sadly01:24
jheskethjeblair: couple of comments on if you're still around01:27
*** baoli has joined #openstack-infra01:27
clarkbjhesketh: for your first comment execution would break the while, hit the end of with execute the exit and return from the function01:30
clarkbnot sure I grok the concern there01:30
*** baoli has quit IRC01:30
*** baoli has joined #openstack-infra01:31
*** tphummel has quit IRC01:32
jheskethclarkb: err yes, I've failed there... and that may make the rest of my comments wrong... gimme a sec01:32
*** Daisy has joined #openstack-infra01:32
*** Qiming__ has joined #openstack-infra01:33
*** Qiming__ has quit IRC01:33
jheskethclarkb: actually I think we may still want to rejoin the thread01:33
openstackgerritJeff Grafton proposed openstack-infra/jenkins-job-builder: Update Git SCM so that extensions are configured correctly.
clarkbjhesketh: ya especially since both threads write to the console01:34
*** jerryz has joined #openstack-infra01:34
*** tlian2 has joined #openstack-infra01:35
*** akshai has joined #openstack-infra01:35
*** baoli has quit IRC01:36
openstackgerritJamie Lennox proposed openstack-infra/puppet-jenkins: Use servername and apache_name in vhost
*** yanyanhu has joined #openstack-infra01:37
*** zz_dimtruck is now known as dimtruck01:38
*** tlian has quit IRC01:38
*** amitgandhinz has joined #openstack-infra01:40
*** akshai has quit IRC01:40
*** amitgandhinz has quit IRC01:44
*** Sukhdev has joined #openstack-infra01:46
openstackgerritMerged openstack-infra/zuul: When a playbook fails, don't succeed in the error_block
*** woodster_ has quit IRC01:48
*** Qiming_ has joined #openstack-infra01:49
*** Qiming has quit IRC01:49
*** roxanaghe has joined #openstack-infra01:54
*** mriedem has quit IRC01:56
*** roxanaghe has quit IRC01:58
*** PollyZ has joined #openstack-infra01:59
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: [WIP] Add mulitnode CI job support
*** mtanino has joined #openstack-infra02:01
*** sarob has joined #openstack-infra02:01
*** Apoorva has quit IRC02:03
*** sarob has quit IRC02:06
*** baoli has joined #openstack-infra02:06
*** gouthamr has quit IRC02:09
*** baoli has quit IRC02:09
*** Qiming_ is now known as Qiming02:10
*** shashank_hegde has quit IRC02:10
cloudnullclarkb: -- great beer :)02:14
openstackgerritJeremy Stanley proposed openstack-infra/zuul: Support keep-hierarchy in launcher rsync publisher
fungijhesketh: first stab ^02:14
* jhesketh looks02:15
*** baoli has joined #openstack-infra02:15
fungijhesketh: not tested, but relatively trivial adjustment and should work based on my reading of the ansible docs02:16
jheskethfungi: why shell not command?02:16
fungijhesketh: because needing ` support02:16
fungijhesketh: command does not run under a shell02:17
jheskethfungi: so by adding that trailing slash, I don't think rsync will copy the folder name (just its contents)02:19
jheskethalthough maybe I'm remembering wrong02:19
jheskeththat looks like what we want though (given scproot is a tmpdir), but it's different to what was happening02:19
jheskethwere we getting tmpdir names copied across before?02:19
fungijhesketh: the trailing / was already there02:19
jheskethoh, so it was!02:20
jheskethjust in the command02:20
jheskethfungi: thanks, lgtm02:20
funginow i have some release automation to hack on02:20
*** PollyZ has quit IRC02:21
*** dimtruck is now known as zz_dimtruck02:28
*** csomerville has joined #openstack-infra02:30
*** gildub has joined #openstack-infra02:30
*** gyee has quit IRC02:36
*** shashank_hegde has joined #openstack-infra02:39
*** amitgandhinz has joined #openstack-infra02:39
*** amitgandhinz has quit IRC02:40
*** mixos_ has quit IRC02:41
*** sflanigan is now known as sflanigan_afk02:41
*** amotoki has joined #openstack-infra02:41
*** amitgandhinz has joined #openstack-infra02:42
*** csomerville has quit IRC02:44
*** mtanino has quit IRC02:46
*** amotoki has quit IRC02:46
*** amitgandhinz has quit IRC02:46
*** amitgandhinz has joined #openstack-infra02:47
*** yamamoto_ has joined #openstack-infra02:48
*** yuanying has quit IRC02:50
*** r-daneel has quit IRC02:53
*** sflanigan_afk is now known as sflanigan02:55
*** fedexo has quit IRC02:55
*** piet has joined #openstack-infra02:57
*** sputnik13 has joined #openstack-infra03:04
*** anteaya has quit IRC03:05
*** yamahata has quit IRC03:05
*** liusheng has quit IRC03:07
*** fedexo has joined #openstack-infra03:08
*** Daisy has quit IRC03:09
*** Daisy has joined #openstack-infra03:09
*** piet has quit IRC03:11
*** Daisy has quit IRC03:14
*** banix has quit IRC03:21
*** jerryz has quit IRC03:24
*** Sukhdev has quit IRC03:28
*** Sukhdev has joined #openstack-infra03:29
*** amitgandhinz has quit IRC03:30
*** shashank_hegde has quit IRC03:33
*** Sukhdev has quit IRC03:33
*** piet has joined #openstack-infra03:35
*** jamielennox is now known as jamielennox|away03:39
*** amotoki has joined #openstack-infra03:42
*** markvoelker has quit IRC03:44
*** yamamoto_ has quit IRC03:45
*** tlian2 has quit IRC03:46
*** amotoki has quit IRC03:47
*** yuanying has joined #openstack-infra03:49
*** roxanaghe has joined #openstack-infra03:55
*** hparekh has quit IRC03:55
*** baoli has quit IRC03:59
*** roxanaghe has quit IRC04:00
*** javeriak has joined #openstack-infra04:01
*** javeriak_ has joined #openstack-infra04:04
*** Sukhdev has joined #openstack-infra04:04
*** javeriak has quit IRC04:05
*** piet has quit IRC04:06
*** hparekh has joined #openstack-infra04:07
*** harlowja_at_home has joined #openstack-infra04:08
openstackgerritMin Pae proposed openstack-infra/project-config: Add octavia periodic jobs with oslo master
*** sree has joined #openstack-infra04:15
*** ayoung has quit IRC04:20
*** shashank_hegde has joined #openstack-infra04:25
*** Daisy has joined #openstack-infra04:29
*** yamamoto has joined #openstack-infra04:30
*** amitgandhinz has joined #openstack-infra04:30
*** vaidy has quit IRC04:31
*** Daisy has quit IRC04:34
*** amitgandhinz has quit IRC04:35
*** psachin has joined #openstack-infra04:35
*** links has joined #openstack-infra04:38
*** hdaniel has joined #openstack-infra04:39
jeblairjhesketh, clarkb: responded to comment on 32488904:40
*** amotoki has joined #openstack-infra04:43
*** markvoelker has joined #openstack-infra04:44
*** hdaniel has quit IRC04:44
*** nwkarsten has joined #openstack-infra04:45
*** e0ne has quit IRC04:46
*** e0ne has joined #openstack-infra04:46
*** e0ne has quit IRC04:46
*** Sukhdev has quit IRC04:47
*** amotoki has quit IRC04:47
jeblairjhesketh, clarkb: and again04:49
*** markvoelker has quit IRC04:49
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Write console in thread
jeblairjhesketh, clarkb: ^04:54
*** flwang has quit IRC04:56
*** roxanaghe has joined #openstack-infra04:56
*** harlowja_at_home has quit IRC04:56
*** sputnik13 has quit IRC04:56
jheskethjeblair: cool, responded04:57
*** Daisy has joined #openstack-infra04:58
*** rcernin has joined #openstack-infra04:58
*** nwkarsten has quit IRC05:00
*** roxanaghe has quit IRC05:01
*** nwkarsten has joined #openstack-infra05:01
*** vaidy has joined #openstack-infra05:02
openstackgerritMerged openstack-infra/project-config: [translations] Fix path for firstapp
jeblairjhesketh: i would like that, but i know of no good way to break out of readline like that; the alternative would be a select loop in python where we read 1 character at a time.  readline is, i believe, implemented in C outside of the GIL and so is much more efficient.  i think it's worth sticking with readline -- there is ultimately no difference between the fd being closed by python before sys.exit or during.  :)05:03
*** vaidy has quit IRC05:04
*** rcernin has quit IRC05:04
*** nadya has joined #openstack-infra05:05
*** nwkarsten has quit IRC05:06
openstackgerritMerged openstack-infra/project-config: Fix translation proposals
*** maishsk_ has joined #openstack-infra05:10
jheskethjeblair: works for me, but won't we need to make the thread a daemon so that the program can exit?05:11
jeblairjhesketh: it is05:11
jeblairjhesketh: line 81 in patch05:11
jheskethheh, indeed it is... I'm very blind today sorry05:12
*** maishsk has quit IRC05:12
*** maishsk_ is now known as maishsk05:12
jheskethjeblair: I haven't heard from pabelanger about his progress on the timeout issue... do you know if he had started it05:13
* jhesketh doesn't want to duplicate effort05:13
*** Julien-zte has quit IRC05:16
*** yuanying has quit IRC05:17
jeblairjhesketh: my understanding is that he was focusing on trying to understand it more, particularly focusing on getting more real-time logs from ansible (rather than just logging at the end of the run)05:18
*** elynn__ has joined #openstack-infra05:18
jeblairjhesketh: you might consider working on a watchdog for ansible, where if the ansible playbook hasn't finished within the timeout we gave it (plus a bit of slop), we forcibly terminate ansible (using the mechanism we use for aborting a job perhaps)05:19
jeblairjhesketh: that might be a good complimentary thing to work on while pabelanger continues to try to uncover the underlying cause05:20
jheskethpossibly a worthwhile addition to what pabelanger is doing05:20
jheskethokay, I'll take a look at that05:20
jeblairand i will go to bed :)05:20
jeblairthx, night!05:20
jheskethheh, isn't it super late?05:20
jheskethno worries05:20
* jhesketh makes no promises though05:20
jheskethseeya :-)05:20
elynn__If I want to skip a gate job in stable/mitaka release, how to do that in openstack-infra/openstack-config? Any references?05:21
*** nwkarsten has joined #openstack-infra05:22
openstackgerritSteven Dake proposed openstack-infra/project-config: Add security-analysis docuemntation repository
openstackgerritSteven Dake proposed openstack-infra/project-config: Add security-analysis docuemntation repository
*** gongysh has joined #openstack-infra05:28
*** psachin_ has joined #openstack-infra05:29
*** psachin_ has quit IRC05:30
*** amitgandhinz has joined #openstack-infra05:31
*** pcrews has quit IRC05:33
*** yamahata has joined #openstack-infra05:34
*** pcrews has joined #openstack-infra05:36
openstackgerritSteven Dake proposed openstack-infra/project-config: Add security-analysis documentation repository
openstackgerrityolanda.robla proposed openstack-infra/system-config: Update documentation about infra cloud
*** amitgandhinz has quit IRC05:38
*** nwkarsten has quit IRC05:40
*** nwkarsten has joined #openstack-infra05:40
*** nadya has quit IRC05:42
*** amotoki has joined #openstack-infra05:44
*** nwkarsten has quit IRC05:45
*** rcernin has joined #openstack-infra05:46
sdake_elynn__ you can make it nonvoting05:46
sdake_elynn__ grep the repo for nonvoting05:47
sdake_I need to hit the sack or I'd help more - night :)05:47
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: rename temporary directories
elynn__sdake_, thanks, have a good night :)05:47
*** Daisy has quit IRC05:48
*** amotoki has quit IRC05:49
*** Daisy has joined #openstack-infra05:49
openstackgerritMerged openstack-infra/zuul: Support keep-hierarchy in launcher rsync publisher
openstackgerritMerged openstack-infra/project-config: Part two of kolla-mesos retirement
*** slaweq_ has joined #openstack-infra05:51
*** Daisy has quit IRC05:54
*** zhurong has quit IRC05:56
openstackgerritMerged openstack-infra/project-config: Remove skip test logic for project-config
*** zhurong has joined #openstack-infra05:57
*** nadya has joined #openstack-infra05:58
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup source-repositories output
*** liusheng has joined #openstack-infra06:00
*** amitgandhinz has joined #openstack-infra06:00
*** fedexo has quit IRC06:02
*** yuanying has joined #openstack-infra06:05
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** amitgandhinz has quit IRC06:05
*** aeng has quit IRC06:06
*** vaidy has joined #openstack-infra06:08
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Ansible launcher: add watchdog for ansible
jheskethjeblair, pabelanger: ^ first pass at a watchdog06:09
openstackgerritMerged openstack-infra/project-config: Run Ironic Inspector jobs with TinyIPA
*** vaidy has quit IRC06:10
*** slaweq_ has quit IRC06:15
*** slaweq_ has joined #openstack-infra06:15
*** Daisy has joined #openstack-infra06:15
*** openstackgerrit has quit IRC06:17
*** openstackgerrit has joined #openstack-infra06:18
*** elynn has joined #openstack-infra06:18
*** elynn__ has quit IRC06:19
openstackgerritMasahito Muroi proposed openstack-infra/irc-meetings: Add meeting chair to Congress meetings
openstackgerritMerged openstack-infra/puppet-zuul: Remove launcher socket option
*** bhavik has joined #openstack-infra06:27
*** yuanying has quit IRC06:28
*** cgross has quit IRC06:28
*** yuanying has joined #openstack-infra06:29
*** lmiccini has quit IRC06:29
*** mkoderer has quit IRC06:30
*** yuanying has quit IRC06:31
*** mkoderer has joined #openstack-infra06:31
*** yuanying has joined #openstack-infra06:31
*** cgross has joined #openstack-infra06:31
*** slaweq_ has quit IRC06:32
*** lmiccini has joined #openstack-infra06:32
openstackgerritMerged openstack-infra/project-config: Create separate ACL groups for murano-apps project
*** nadya has quit IRC06:32
*** yuanying has quit IRC06:33
*** yuanying has joined #openstack-infra06:34
*** andreas_s has joined #openstack-infra06:35
openstackgerritMerged openstack-infra/project-config: Add mitaka to gerritbot for OpenStack-Chef
openstackgerritMerged openstack-infra/project-config: Enable lbaasv2 jobs into check and gate queues
*** harlowja has quit IRC06:35
*** nadya has joined #openstack-infra06:36
openstackgerritMerged openstack-infra/project-config: Enable the Ubuntu binary gate on liberty branch
*** dingyichen has quit IRC06:38
openstackgerritMerged openstack-infra/project-config: Revert "Revert "Actually use the service domain in V3 jobs""
openstackgerritMerged openstack-infra/project-config: Run {name}-announce-release for Puppet OpenStack modules
openstackgerritMerged openstack-infra/project-config: Add openstack-server-release-jobs to Mistral in zuul
openstackgerritMerged openstack-infra/project-config: Add dib-utils func test
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup source-repositories output
AJaegerelynn: you can easily skip a job - just add a branch condition to it.06:41
AJaegerelynn: do you need a pointer?06:41
*** nadya has quit IRC06:42
*** asselin_ has joined #openstack-infra06:42
openstackgerritMerged openstack-infra/project-config: Allow core reviewers to push merge commits to fuel-plugin-cinder-netapp
elynnAJaeger, yes06:42
openstackgerritMerged openstack-infra/project-config: Update gate-ironic-staging-drivers-dsvm-nv job
elynnAJaeger, for example, I just want to run tempest job on master branch and mitaka branch won't support tempest tests since it's newly added.06:43
*** vaidy has joined #openstack-infra06:43
*** vaidy has quit IRC06:44
AJaegerelynn: that one blacklists kilo - but the principal is the same06:45
*** amotoki has joined #openstack-infra06:45
AJaegerJust have a regex that skips mitaka, liberty and kilo06:45
elynnAJaeger, Cool! thanks!06:45
AJaegerelynn: and make a proper comment on your commit so when we review it we check that it does the right thing ;)06:45
*** markvoelker has joined #openstack-infra06:45
elynnAJaeger, got it ;)06:46
bhavikHi, I need help to implement process with gerrit+zuul  ... we have two git repos .. let's say upstream and internal, right now we sync branches from upstream and publish to internal, as it is .. now got a requirement that get branch commits from upstream and pass them to gerrit and on approval submit to internal repo. any suggestion pointers appreciated.06:46
*** shashank_hegde has quit IRC06:49
*** amotoki has quit IRC06:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add support for bindep
*** rlandy has quit IRC06:53
bhavikI've wrote script to convert upstream commits to understand by gerrit at
*** roxanaghe has joined #openstack-infra06:57
*** gildub has quit IRC06:58
*** markvoelker has quit IRC06:58
*** yamada-h has joined #openstack-infra06:59
AJaegersdake_: is security-analysis a document repository or do you have code and will release it?07:00
*** nadya has joined #openstack-infra07:00
openstackgerritMerged openstack-infra/project-config: Make keystone functional tests job voting
sdake_AJaeger i just answered thatin the review07:00
sdake_with a "I am not certain"07:00
*** dizquierdo has joined #openstack-infra07:00
AJaegerbhavik: sorry, can't help with that. There's a 3rd party meeting, see - best to ask there07:00
*** asselin_ has quit IRC07:01
openstackgerritMerged openstack-infra/project-config: Add back to tripleo-centos-7
sdake_I think it will jut be RST files with SVG files07:01
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
sdake_but who knwos what will happen with this repository :)07:01
sdake_I'm not really on the security team so I don't have details07:01
bhavik@AJaeger , thanks for pointer07:01
*** roxanaghe has quit IRC07:01
*** harlowja_at_home has joined #openstack-infra07:02
AJaegersdake_: if its a doc repo, you won't tag and upload to pypi. Right now it has no tag permissions, thus it will never upload to pypi.07:02
*** amitgandhinz has joined #openstack-infra07:02
*** harlowja_at_home has quit IRC07:02
AJaegersdake_: I suggest for now to remove the pypi jobs - and keep the pypi entry...07:02
*** ggnel_t has joined #openstack-infra07:02
sdake_it is a doc repo07:02
*** openstackgerrit has quit IRC07:03
sdake_AJaeger would you mind throwing me a bone and leaving exact instructions on wht ot remove, yo uwould think I'd know this stuff by now but i struggle with every new project add :)07:03
*** openstackgerrit has joined #openstack-infra07:03
AJaegersdake_: Ok, let me give more details..07:03
sdake_I don't totaly understand how all the jobs work is most of the problem07:03
sdake_or part of it atleast07:04
sdake_AJaeger i've bee nrolling for 18 hours, need to hit the sack07:04
sdake_I'll submit a new version tomorrow morning07:04
sdake_thanks for taking a look :)07:04
*** flepied has quit IRC07:04
*** vincentll has joined #openstack-infra07:06
AJaegersdake_: have a good night!07:06
sdake_AJaeger one quick question, I have a upstream line in there in that review07:06
sdake_will that pull from my github repo that I have specified there?07:06
AJaegeryes, it will import *once* from your github repo.07:06
sdake_I see all the upstream lines have been removed, so I was concerned maybe that function was removed07:06
sdake_ya I want just one import07:06
sdake_its a cookiecutter built repo anyway07:06
*** amitgandhinz has quit IRC07:07
AJaegerso, that's fine07:07
AJaegerNo, we remove them after the import - to reduce confusion. Some people thought it would sync somehow...07:07
sdake_git-upstream ftw ;)07:08
*** arxcruz has joined #openstack-infra07:09
sdake_113degrees F tomorrow07:10
* sdake_ GROANS07:10
AJaegersdake_: sleep now, you can worry tomorrow ;)07:10
*** ociuhandu_ has quit IRC07:10
*** nadya has quit IRC07:15
*** sorantis has joined #openstack-infra07:15
*** nadya_ has joined #openstack-infra07:15
*** mikelk has joined #openstack-infra07:15
*** vaidy has joined #openstack-infra07:16
*** amoralej|off is now known as amoralej07:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix VENV temporary path
AJaegerinfra-root: I fear we have lots of unused tmp directories in /tmp on the proposal slave since the translation setup was broken, see above ^. An "rmdir /tmp/tmp.*" on the proposal slave might be a nice idea to get rid of those created but unused directories07:21
*** jistr is now known as jistr|mtg07:22
openstackgerritYanyan Hu proposed openstack-infra/project-config: Tune rally gate jobs for senlin
*** yamahata has quit IRC07:23
*** watanabe_isao has joined #openstack-infra07:25
*** nadya_ has quit IRC07:25
*** watanabe_isao has quit IRC07:26
*** watanabe_isao has joined #openstack-infra07:27
*** elynn has quit IRC07:27
AJaegerah, that was with my change that merged yesterday - so we won't have that many unused directories...07:28
*** shashank_hegde has joined #openstack-infra07:28
*** yaume has joined #openstack-infra07:28
*** watanabe_isao has quit IRC07:29
*** watanabe_isao has joined #openstack-infra07:29
pleia2AJaeger: yeah, just seeing a few from june 3 in tmp/tmp.Mt3pWOJVvb etc07:30
openstackgerritMartin Mágr proposed openstack-infra/tripleo-ci: Add opstools candidate repo
AJaegerpleia2: yeah, just from today - not yesterday.07:30
AJaegerpleia2: what time is it for you?07:30
*** matrohon has joined #openstack-infra07:30
*** esikachev has joined #openstack-infra07:31
AJaegerlet's first merge 325090 to not create more...07:31
*** javeriak_ has quit IRC07:31
*** elynn has joined #openstack-infra07:31
pleia2AJaeger: 3:30AM07:32
*** tmorin has joined #openstack-infra07:32
*** dizquierdo has quit IRC07:32
AJaegerpleia2: that's a crazy time to be online ;(07:32
AJaegerpleia2: nothing urgent here - just didn't want it forgotten...07:33
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Increase tempest baremetal deploy/unprovision timeouts
*** armax has joined #openstack-infra07:34
pleia2AJaeger: which change introduced this?07:35
*** yaume has quit IRC07:36
*** elynn_ has joined #openstack-infra07:36
AJaegerpleia2: mine ;(
AJaegerI should have removed the two extra lines when I changed the mktemp call ;(07:36
*** ramishra has quit IRC07:37
*** elynn has quit IRC07:37
pleia2aha, it happens :)07:38
*** jlanoux has joined #openstack-infra07:38
*** Daisy has quit IRC07:38
*** ramishra has joined #openstack-infra07:39
*** Daisy has joined #openstack-infra07:39
AJaegeryeah :(07:39
AJaegerthanks for review, pleia2 !07:39
*** jlanoux has quit IRC07:42
*** Daisy has quit IRC07:43
*** jlanoux has joined #openstack-infra07:44
*** flepied has joined #openstack-infra07:44
*** jlanoux has quit IRC07:45
*** amotoki has joined #openstack-infra07:45
openstackgerritJan Hruban proposed openstack-infra/zuul: Support for dependent pipelines with github
openstackgerritJan Hruban proposed openstack-infra/zuul: Merge pull requests from github reporter
openstackgerritJan Hruban proposed openstack-infra/zuul: GitHub file matching support
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow github trigger to match on branches/refs
openstackgerritJan Hruban proposed openstack-infra/zuul: Encapsulate determining the event purpose
openstackgerritJan Hruban proposed openstack-infra/zuul: Log GitHub API rate limit
openstackgerritJan Hruban proposed openstack-infra/zuul: support github pull reqeust labels
openstackgerritJan Hruban proposed openstack-infra/zuul: Better merge message for GitHub pull reqeusts
*** watersoul has quit IRC07:47
*** javeriak has joined #openstack-infra07:49
*** amotoki has quit IRC07:50
*** yuanying has quit IRC07:50
*** roxanaghe has joined #openstack-infra07:50
*** roxanaghe has quit IRC07:53
*** watersoul has joined #openstack-infra07:53
*** mhickey has joined #openstack-infra07:54
*** timello has quit IRC07:55
*** bhavik has quit IRC07:58
*** bhavik has joined #openstack-infra07:58
*** nikhil has quit IRC07:58
*** monester has quit IRC07:58
*** jed56 has quit IRC07:58
*** kgalanov has quit IRC07:58
*** teran has quit IRC07:58
*** jaranovich has quit IRC07:58
*** igorbelikov has quit IRC07:58
*** madhuvishy has quit IRC07:58
*** ogelbukh has quit IRC07:58
*** fmccrthy has quit IRC07:58
*** jpmaxman has quit IRC07:58
*** yonglihe has quit IRC07:58
*** dham1 has quit IRC07:58
*** evgenyl has quit IRC07:58
*** kozhukalov has quit IRC07:58
*** rajinir has quit IRC07:58
*** zxiiro has quit IRC07:58
*** tpeoples has quit IRC07:58
*** rguillebert has quit IRC07:58
*** auggy has quit IRC07:58
*** Daisy has joined #openstack-infra07:59
*** bexelbie has quit IRC07:59
*** bexelbie has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** yaume has joined #openstack-infra08:00
*** kzaitsev_mb has joined #openstack-infra08:00
*** zzzeek has joined #openstack-infra08:01
*** zeih has joined #openstack-infra08:02
*** amitgandhinz has joined #openstack-infra08:02
*** sdake_ has quit IRC08:03
openstackgerritEthan Lynn proposed openstack-infra/project-config: Don't run gate-senlin-dsvm-tempest-api on mitaka branch
*** daemontool has joined #openstack-infra08:06
*** ifarkas has joined #openstack-infra08:06
*** amitgandhinz has quit IRC08:07
*** flwang has joined #openstack-infra08:08
*** permalac has joined #openstack-infra08:08
*** fesp has joined #openstack-infra08:10
*** lucas-afk is now known as lucasagomes08:11
*** derekh has joined #openstack-infra08:13
*** ihrachys has joined #openstack-infra08:15
*** fesp has quit IRC08:16
*** dizquierdo has joined #openstack-infra08:16
*** yaume has quit IRC08:18
openstackgerritMartin Mágr proposed openstack-infra/project-config: Retiring monitoring-for-openstack step 2
*** e0ne has joined #openstack-infra08:24
*** javeriak has quit IRC08:25
openstackgerritEthan Lynn proposed openstack-infra/project-config: Make gate-senlin-dsvm-tempest-api job voting
*** esikachev has quit IRC08:28
*** dtantsur|afk is now known as dtantsur08:28
penguinologHello! Zuul feel bad? I see a lot of UNSTABLE statuses on checks08:29
penguinologExample: gate-fuel-web-docs: UNSTABLE (non-voting) -> no logs available08:31
*** sflanigan has quit IRC08:31
AJaegerIs any infra-root around to check our log servers? ^08:31
AJaegerthanks penguinolog for noticing us.08:32
*** esikachev has joined #openstack-infra08:32
Adri2000hi, is there a reason for not having https only? I'd "file a bug" for that but I'm not sure how to do it in storyboard :) (new story?)08:35
*** andreww has quit IRC08:35
*** andreww has joined #openstack-infra08:35
*** betherly_ has joined #openstack-infra08:36
*** phschwartz_ has joined #openstack-infra08:36
*** MarkAtwood_ has joined #openstack-infra08:36
*** nemish has quit IRC08:36
*** fkautz_ has joined #openstack-infra08:37
vsaienkoplease help to merge the patches are blockers for ironic grenade08:37
*** fkautz_ has quit IRC08:37
*** ggillies_ has joined #openstack-infra08:37
*** senk_ has joined #openstack-infra08:37
*** ildikov_ has joined #openstack-infra08:38
*** reed_ has joined #openstack-infra08:38
*** fkautz_ has joined #openstack-infra08:38
*** briancli1e has joined #openstack-infra08:38
*** matbu_ has joined #openstack-infra08:38
*** daemontool has quit IRC08:38
*** nemish has joined #openstack-infra08:38
*** e0ne has quit IRC08:38
*** jzerebec1i has joined #openstack-infra08:38
*** e0ne has joined #openstack-infra08:39
*** terryw has quit IRC08:39
*** fkautz has quit IRC08:39
*** ildikov has quit IRC08:39
*** sshnaidm|afk has quit IRC08:39
*** reed has quit IRC08:39
*** johnthetubaguy has quit IRC08:39
*** MarkAtwood has quit IRC08:39
*** senk has quit IRC08:39
*** bradjones has quit IRC08:39
*** AJaeger has quit IRC08:39
*** rackertom has quit IRC08:39
*** phschwartz has quit IRC08:39
*** morgabra has quit IRC08:39
*** briancline has quit IRC08:39
*** jzerebecki has quit IRC08:39
*** watanabe_isao has quit IRC08:39
*** sulo has quit IRC08:39
*** ggillies has quit IRC08:39
*** pblaho has quit IRC08:39
*** mescanef_ has quit IRC08:39
*** jamespage has quit IRC08:39
*** tonyb has quit IRC08:39
*** betherly has quit IRC08:39
*** matbu has quit IRC08:39
*** dasm has quit IRC08:39
*** verdurin has quit IRC08:39
*** jamespage has joined #openstack-infra08:39
*** sulo has joined #openstack-infra08:39
*** tonyb has joined #openstack-infra08:39
*** johnthetubaguy has joined #openstack-infra08:39
*** jamespage has quit IRC08:39
*** jamespage has joined #openstack-infra08:39
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add virtualbmc from source non-voting test
*** ildikov_ is now known as ildikov08:39
*** otherwiseguy has joined #openstack-infra08:39
*** yuanying has joined #openstack-infra08:39
*** reed_ is now known as reed08:39
*** bradjones has joined #openstack-infra08:39
*** bradjones has quit IRC08:39
*** bradjones has joined #openstack-infra08:39
*** morgabra_ has joined #openstack-infra08:40
*** AJaeger has joined #openstack-infra08:40
*** MarkAtwood_ is now known as MarkAtwood08:40
*** pblaho has joined #openstack-infra08:40
*** verdurin has joined #openstack-infra08:40
*** dasm has joined #openstack-infra08:40
*** rackertom has joined #openstack-infra08:41
*** betherly_ is now known as betherly08:41
*** mescanef has joined #openstack-infra08:41
*** penguinolog has quit IRC08:42
*** fkautz_ is now known as fkautz08:43
*** fkautz is now known as Guest9930108:43
*** jistr|mtg is now known as jistr08:43
*** _degorenko|afk is now known as degorenko08:45
*** armax has quit IRC08:45
*** chem has joined #openstack-infra08:45
*** amotoki has joined #openstack-infra08:46
*** kzaitsev_mb has quit IRC08:47
*** markusry has joined #openstack-infra08:48
*** esikachev has quit IRC08:49
*** penguinolog has joined #openstack-infra08:49
*** Daisy has quit IRC08:51
*** amotoki has quit IRC08:51
*** jaosorior has joined #openstack-infra08:51
*** roxanaghe has joined #openstack-infra08:51
*** sshnaidm|afk has joined #openstack-infra08:51
*** Daisy has joined #openstack-infra08:51
*** madhuvishy has joined #openstack-infra08:53
*** esikachev has joined #openstack-infra08:54
*** hdaniel has joined #openstack-infra08:54
*** rajinir has joined #openstack-infra08:54
*** roxanaghe has quit IRC08:56
*** AJaeger has quit IRC08:57
*** kozhukalov has joined #openstack-infra08:57
*** evgenyl has joined #openstack-infra08:57
*** AJaeger has joined #openstack-infra08:58
*** dizquierdo has quit IRC08:58
*** jaranovich has joined #openstack-infra08:58
*** zxiiro has joined #openstack-infra09:00
*** hdaniel has quit IRC09:01
*** esikachev has quit IRC09:01
*** armax has joined #openstack-infra09:01
*** teran has joined #openstack-infra09:01
*** rguillebert has joined #openstack-infra09:02
*** liusheng has quit IRC09:03
*** kgalanov has joined #openstack-infra09:03
*** liusheng has joined #openstack-infra09:03
*** amitgandhinz has joined #openstack-infra09:03
*** flwang has quit IRC09:03
*** Daisy_ has joined #openstack-infra09:04
*** chem` has joined #openstack-infra09:05
*** infra-red has joined #openstack-infra09:05
*** monester has joined #openstack-infra09:06
*** chem has quit IRC09:06
*** tpeoples has joined #openstack-infra09:08
*** amitgandhinz has quit IRC09:08
*** Daisy has quit IRC09:08
*** sambetts|afk is now known as sambetts09:08
*** ogelbukh has joined #openstack-infra09:08
*** esikachev has joined #openstack-infra09:09
*** yonglihe has joined #openstack-infra09:11
*** jed56 has joined #openstack-infra09:11
*** jpmaxman has joined #openstack-infra09:12
*** auggy has joined #openstack-infra09:12
*** skolekonov has left #openstack-infra09:13
robcresswellSomething up with the gate? Patches exploding with UNSTABLE. See :)09:13
*** dham1 has joined #openstack-infra09:14
*** fmccrthy has joined #openstack-infra09:14
*** igorbelikov has joined #openstack-infra09:14
*** nikhil_ has joined #openstack-infra09:17
*** nikhil_ is now known as Guest2723109:17
*** yaume has joined #openstack-infra09:18
*** vdrok has joined #openstack-infra09:19
AJaegerrobcresswell, yeah something looks broken - we ned an infra-root to check the log server09:20
jaosorioryep, same is going on for TripleO jobs now09:21
*** yamada-h has quit IRC09:21
*** jaosorior has quit IRC09:21
*** rpodolyaka is now known as roman09:23
*** shashank_hegde has quit IRC09:25
*** jaosorior has joined #openstack-infra09:26
*** Daisy_ has quit IRC09:27
*** hashar has joined #openstack-infra09:28
*** Daisy has joined #openstack-infra09:28
ttxAJaeger: maybe send alert ?09:30
ttxno point in rechecking09:30
*** rcernin is now known as rcernin|lunch09:31
AJaegerttx, go ahead - I don't have permissions and seems nobody else is around ;(09:32
AJaegerttx, indeed no point in rechecking09:32
robcresswellAJaeger: Ah, understood. Thanks!09:35
ttx#status notice CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.09:35
openstackstatusttx: sending notice09:35
* ttx looks up how to add AJaeger to the statusbot whitelist09:36
-openstackstatus- NOTICE: CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.09:37
openstackstatusttx: finished sending notice09:38
*** armax has quit IRC09:40
*** sdague has joined #openstack-infra09:41
*** maishsk has quit IRC09:41
*** kzaitsev_mb has joined #openstack-infra09:41
openstackgerritThierry Carrez proposed openstack-infra/system-config: Add AJaeger to statusbot auth_nicks
*** elynn_ has quit IRC09:42
ttxAJaeger: that may do it ^09:42
AJaegerah, ttx...09:42
*** sree has quit IRC09:42
openstackgerritamrith proposed openstack-infra/project-config: Add channel notification for trove-image-builder
AJaegerthanks, ttx09:43
*** pcaruana has joined #openstack-infra09:44
amrithAJaeger, my apologies re: I should've re-read the instructions when you reviewed and asked about channel notifications.09:45
AJaegeramrith: no worries, we can do this in two steps... - but see the notice that the bot just send and don't recheck when you get UNSTABLE jobs for now09:46
*** amotoki has joined #openstack-infra09:47
amrithyes, saw that09:48
amrithin #openstack-trove09:48
sdagueAJaeger: I have a docs job question when you get a chance09:50
*** Na3iL has joined #openstack-infra09:51
*** amotoki has quit IRC09:51
*** bhavik has quit IRC09:51
*** sorantis has quit IRC09:52
*** roxanaghe has joined #openstack-infra09:52
*** abregman|afk has quit IRC09:52
*** penguinolog has quit IRC09:54
*** penguinolog has joined #openstack-infra09:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Filter non-public fields when searching for users
*** roxanaghe has quit IRC09:56
AJaegersdague: I'm here now...09:57
sdagueAJaeger: so... with the os-api-ref code what I'd *really* love to be able to do is have a non-voting job that renders the api-ref of another project with master of it09:57
*** logan_ has joined #openstack-infra09:57
sdagueso when we propose a change, we can see how it looks with another project automatically09:58
sdagueI'm doing that entirely locally right now09:58
*** logan_ is now known as Guest6297109:58
AJaegersdague: yeah, some kind of integration testing.09:58
sdagueyeh, basically09:58
AJaegersome options:09:58
*** charz_ has joined #openstack-infra09:59
aspiersam I misunderstanding something, or is Jenkins broken? it's giving a -1 for reasons I can't understand since there's no logs either:
AJaeger1) Do it like bandit in gate-bandit-integration-{target-project}09:59
*** logan- has quit IRC09:59
AJaegeraspiers: see recent notice from the bots - please wait, our infrastructure is broken09:59
AJaegersdague: jenkins/jobs/bandit.yaml10:00
aspiersAJaeger: ok thanks!10:00
*** javeriak has joined #openstack-infra10:00
*** hferenc has quit IRC10:00
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Filter non-public fields in users when updating
aspiersAJaeger: BTW your reply could have been 3usec quicker you know ;-)10:00
*** frickler has quit IRC10:01
AJaegersdague: 2) do that with some scripts in your repository. You can easily checkout e.g. nova, cd in there and create the docs...10:01
*** hogepodge has quit IRC10:01
*** charz has quit IRC10:01
*** Guest62971 is now known as logan-10:01
AJaegeraspiers: you owe me some drinks next week now ;)10:02
*** jed56 has quit IRC10:02
*** kgalanov has quit IRC10:02
*** igorbelikov has quit IRC10:02
*** ogelbukh has quit IRC10:02
*** fmccrthy has quit IRC10:02
*** kozhukalov has quit IRC10:02
*** jpmaxman has quit IRC10:02
*** rguillebert has quit IRC10:02
*** evgenyl has quit IRC10:02
*** yonglihe has quit IRC10:02
*** jaranovich has quit IRC10:02
*** auggy has quit IRC10:02
*** rajinir has quit IRC10:02
*** tpeoples has quit IRC10:02
*** dham1 has quit IRC10:02
*** Guest27231 has quit IRC10:02
*** teran has quit IRC10:02
*** monester has quit IRC10:02
*** madhuvishy has quit IRC10:02
*** zxiiro has quit IRC10:02
aspiersAJaeger: I think I already did ;-)10:02
*** Mmike has quit IRC10:02
rakhmerovhi, some gates are in UNSTABLE status, do you know the reason?10:02
sdagueAJaeger: ok, I'll look. I was kind of wondering if there was a good example of checking out another repo in the system. I understand the full stack jobs a lot better than these :)10:02
AJaegersdague: periodic-python-jobs-with-oslo-master is another example10:02
*** Mmike has joined #openstack-infra10:03
*** frickler has joined #openstack-infra10:03
AJaegerrakhmerov: known issue, see previous notice by the bots10:03
aspiersAJaeger / ttx: maybe it's worth putting the Jenkins outage in $TOPIC?10:03
rakhmerovAJaeger: ok10:03
aspierswhat does the openstackstatus bot do, exactly?10:03
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Allow superusers to edit other users successfully
ttxI just issues a notice rather than an later10:03
ttxso it just spoke in channels10:03
ttxrather than an alert, I mean10:04
AJaegeraspiers: send notices to all subscribed channels, or update topic as well...10:04
aspiersit did not alert #openstack-ha so I guess I did not configure that channel ccorrectly?10:04
ttxAJaeger: should I reissue it as an alert ?10:04
*** amitgandhinz has joined #openstack-infra10:04
ttxaspiers: probably not :)10:04
AJaegerttx, it's so broken right now that an alert seems a good idea...10:05
*** tdasilva has quit IRC10:05
ttx#alert notice CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.10:05
AJaegerttx, than we can just write a bot that welcomes each newcomer with: Please read topic first ;)10:05
*** zeih_ has joined #openstack-infra10:05
*** zeih has quit IRC10:05
ttx#status alert CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.10:05
openstackstatusttx: sending alert10:05
*** bastelfreak has quit IRC10:06
*** tdasilva has joined #openstack-infra10:07
*** bhavik has joined #openstack-infra10:08
-openstackstatus- NOTICE: CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.10:08
*** ChanServ changes topic to "CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience."10:08
*** bastelfreak has joined #openstack-infra10:08
AJaegeraspiers: updated system-config/hiera/common.yaml and add openstack-ha there10:08
aspiersAJaeger: already done it10:08
AJaegergreat, aspiers10:08
*** ihrachys has quit IRC10:09
*** amitgandhinz has quit IRC10:09
*** zhurong has quit IRC10:10
openstackgerritAdam Spiers proposed openstack-infra/system-config: Add #openstack-ha to statusbot
openstackstatusttx: finished sending alert10:11
*** pkoniszewski has quit IRC10:12
*** isq has quit IRC10:12
*** spiffxp has quit IRC10:12
*** pkoniszewski has joined #openstack-infra10:12
*** isq has joined #openstack-infra10:12
*** anthonyper_ has joined #openstack-infra10:12
*** andymccr_ has joined #openstack-infra10:13
*** SlickN1k has joined #openstack-infra10:13
*** spiffxp has joined #openstack-infra10:14
aspiersAJaeger: should I add a link from to ? currently channel creation is undocumented? if so it's easy to forget things like setting up statusbot/meetbot10:14
*** madhuvishy has joined #openstack-infra10:14
*** rhallisey has joined #openstack-infra10:14
aspiersIMHO it would be better to include the setup process in the docs, rather than having it on the wiki10:14
AJaegeraspiers: better ask ttx for project-team-guide ^10:14
*** ihrachys has joined #openstack-infra10:14
*** rajinir has joined #openstack-infra10:15
AJaegeraspiers: see also which links to the system-config10:15
AJaegeraspiers: do you want your channel to be logged as well? Or only status messages?10:15
aspiersAJaeger: it's already logged10:15
*** javeriak has quit IRC10:16
*** Daisy has quit IRC10:16
*** jaosorior has quit IRC10:16
*** permalac has quit IRC10:16
*** mkoderer has quit IRC10:16
*** gothicmindfood has quit IRC10:16
*** slagle has quit IRC10:16
*** anthonyper has quit IRC10:16
*** SlickNik has quit IRC10:16
*** andymccr has quit IRC10:16
*** anthonyper_ is now known as anthonyper10:16
*** SlickN1k is now known as SlickNik10:16
AJaegeraspiers: indeed, found it now...10:16
aspiersbut I never knew about statusbot until now, hence the question10:16
*** andymccr_ is now known as andymccr10:16
*** zeih_ has quit IRC10:17
*** zeih has joined #openstack-infra10:18
*** kozhukalov has joined #openstack-infra10:19
*** evgenyl has joined #openstack-infra10:19
*** rcernin|lunch is now known as rcernin10:20
*** jaranovich has joined #openstack-infra10:21
aspiersAJaeger: ah, I was getting confused between the Project Team guide and the Project Creator's Guide10:22
aspiersAJaeger: perhaps partially because there no link from the former to the latter?!10:23
*** yujunz has joined #openstack-infra10:23
*** sshnaidm|afk has quit IRC10:23
*** tdasilva has quit IRC10:23
ttxaspiers: It's tricky because projects should be created on infra before they are openstack projects10:24
ttxso the egg comes before the chicken10:24
*** nadya has joined #openstack-infra10:24
aspiersttx: doesn't that mean people need the Creator's Guide first?10:24
*** zxiiro has joined #openstack-infra10:25
ttxyes, they do10:25
aspiersin which case, if they mistakenly arrive at the Project Team guide first, they need a way to find it10:25
AJaegeraspiers: has a link to Project Team Guide10:25
ttxyeah, some warning in project team guide that you don't have to be official to get a repo would prevent things like
aspiersAJaeger: I'm talking about a link in the opposite direction10:26
*** teran has joined #openstack-infra10:26
AJaegeraspiers: ah ;)10:26
*** tdasilva has joined #openstack-infra10:26
aspiersttx: right10:26
*** flepied has quit IRC10:26
* ttx lunches10:26
*** yujunz has quit IRC10:26
*** rguillebert has joined #openstack-infra10:27
*** kgalanov has joined #openstack-infra10:27
*** nadya has quit IRC10:27
*** gongysh has quit IRC10:28
*** hdaniel has joined #openstack-infra10:29
*** jlanoux has joined #openstack-infra10:29
*** andreykurilin has joined #openstack-infra10:29
aspiersAJaeger: any suggestions in which section I should put such a link?10:29
AJaegeraspiers: not really ;(10:30
aspiersAJaeger: a new extremely short "Project creation" section before the Intro section?10:30
*** monester has joined #openstack-infra10:30
*** dtantsur is now known as dtantsur|lunch10:30
*** jzerebec1i is now known as jzerebecki10:32
*** yanyanhu has quit IRC10:32
*** tpeoples has joined #openstack-infra10:32
*** ogelbukh has joined #openstack-infra10:33
AJaegeraspiers: I really don't know what's best, I don't have time to dig into that now. Propose something and see what others say ;)10:33
aspiersAJaeger: or a single sentence at the very top of the intro section?10:33
*** jaosorior has joined #openstack-infra10:34
*** yonglihe has joined #openstack-infra10:35
*** yamamoto has quit IRC10:35
*** jed56 has joined #openstack-infra10:35
*** jpmaxman has joined #openstack-infra10:36
*** auggy has joined #openstack-infra10:36
andreykurilinAJaeger: hi! I found that novaclient's pypy job started failed while installing requirements. Should pypy be updated on our gates?10:37
*** sshnaidm|afk has joined #openstack-infra10:38
*** dham1 has joined #openstack-infra10:38
*** fmccrthy has joined #openstack-infra10:38
*** igorbelikov has joined #openstack-infra10:39
*** banix has joined #openstack-infra10:41
*** flepied has joined #openstack-infra10:41
*** Guest27231 has joined #openstack-infra10:41
* dims reads the topic first :)10:43
*** permalac has joined #openstack-infra10:45
*** schipiga has joined #openstack-infra10:45
*** sorantis has joined #openstack-infra10:46
schipigaHello, all! Some problems with CI ? Almost all builds have status "unstable" without any reports
*** amotoki has joined #openstack-infra10:48
BobBallAny ideas why10:48
*** lucasagomes is now known as lucas-lunch10:49
*** hogepodge has joined #openstack-infra10:49
*** Daisy has joined #openstack-infra10:49
*** mkoderer has joined #openstack-infra10:49
*** gothicmindfood has joined #openstack-infra10:49
*** slagle has joined #openstack-infra10:49
*** changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"10:49
*** jamespage has quit IRC10:49
sdaguehmmm... right the Unstable thing?10:50
*** gnuoy has quit IRC10:50
*** gnuoy has joined #openstack-infra10:51
robcresswellschipiga: Its in all the room topics.10:51
*** jamespage has joined #openstack-infra10:51
robcresswellschipiga: Or was, until 30 secs ago.10:51
*** jamespage has quit IRC10:51
*** jamespage has joined #openstack-infra10:51
sdagueright, something set it back on a rejoin10:52
AJaegerargh ;(10:52
*** amotoki has quit IRC10:52
sdagueirc topic is not really persistent status :)10:52
*** hashar is now known as hasharAway10:52
*** roxanaghe has joined #openstack-infra10:53
AJaegerschipiga: I can't help with pypy at all...10:53
robcresswellNo, but I can imagine its a little frustrating for infra to have to repeat it to everyone who joins the channel for 2 minns :)10:53
*** banix has quit IRC10:53
aspiershrm, gerritbot broken too, or just slower than normal in here?10:54
aspiersI just submitted two reviews to project-team-guide10:54
aspiersmaybe just need to be more patient10:54
AJaegeraspiers: project-team-guide is not announced here.10:55
aspiersah ok10:55
AJaegerrobcresswell: I'll just say now: Read ;)10:55
*** markvoelker has joined #openstack-infra10:55
robcresswellAJaeger: Oh I wasn't aware of this. Saves me bothering you every few days. Thanks!10:56
*** armax has joined #openstack-infra10:57
*** roxanaghe has quit IRC10:57
*** markvoelker has quit IRC11:00
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job
*** nadya has joined #openstack-infra11:01
*** raildo-afk is now known as raildo11:02
*** nadya has quit IRC11:02
*** nadya_ has joined #openstack-infra11:02
*** nadya_ has quit IRC11:03
*** nadya has joined #openstack-infra11:04
*** javeriak has joined #openstack-infra11:04
*** amitgandhinz has joined #openstack-infra11:05
sdagueAJaeger: are any infra-root looking into it, or do we need to wait for folks to wake up?11:07
AJaegersdague: we have to wait for one to wake up11:07
*** Daisy_ has joined #openstack-infra11:09
*** sree has joined #openstack-infra11:09
*** amitgandhinz has quit IRC11:09
*** flepied has quit IRC11:10
*** nadya has quit IRC11:10
*** vdrok has quit IRC11:10
*** Daisy has quit IRC11:12
*** vdrok has joined #openstack-infra11:12
*** sree has quit IRC11:13
*** Daisy_ has quit IRC11:16
*** sshnaidm|afk has quit IRC11:16
*** Daisy has joined #openstack-infra11:16
*** penguinolog has quit IRC11:17
*** penguinolog has joined #openstack-infra11:18
*** yamamoto has joined #openstack-infra11:18
*** rhallisey has quit IRC11:19
*** sshnaidm|afk has joined #openstack-infra11:20
*** mrunge has quit IRC11:20
*** Daisy has quit IRC11:21
*** xarses_ has joined #openstack-infra11:23
*** yamamoto has quit IRC11:24
*** flepied has joined #openstack-infra11:25
*** mrunge has joined #openstack-infra11:26
*** sorantis has quit IRC11:26
*** andreww has quit IRC11:26
*** gordc has joined #openstack-infra11:29
*** erlon has joined #openstack-infra11:32
*** sorantis has joined #openstack-infra11:32
*** ociuhandu has joined #openstack-infra11:33
*** tesseract has joined #openstack-infra11:33
*** d0ugal has quit IRC11:34
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: WIP: Add a OVB te-broker server
ttx#status alert CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.11:39
openstackstatusttx: sending alert11:39
*** gordc has quit IRC11:39
* ttx resends alert11:39
*** hdaniel has quit IRC11:39
AJaegerthanks, ttx11:40
*** gordc has joined #openstack-infra11:40
-openstackstatus- NOTICE: CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience.11:41
*** ChanServ changes topic to "CI is experiencing issues with test logs, all jobs are currently UNSTABLE as a result. No need to recheck until this is fixed! Thanks for your patience."11:41
*** bhavik has quit IRC11:41
*** gordc has quit IRC11:43
openstackstatusttx: finished sending alert11:44
*** pcaruana has quit IRC11:45
*** amotoki has joined #openstack-infra11:48
*** thorst has joined #openstack-infra11:50
*** thorst has quit IRC11:50
*** thorst has joined #openstack-infra11:50
*** ldnunes has joined #openstack-infra11:50
*** baoli has joined #openstack-infra11:51
*** yamamoto has joined #openstack-infra11:52
*** lucas-lunch is now known as lucasagomes11:52
*** yamamoto_ has joined #openstack-infra11:52
*** amotoki has quit IRC11:53
*** baoli_ has joined #openstack-infra11:53
*** roxanaghe has joined #openstack-infra11:54
*** rhallisey has joined #openstack-infra11:54
*** yamamoto_ has quit IRC11:55
*** gordc has joined #openstack-infra11:55
*** baoli has quit IRC11:56
*** yamamoto has quit IRC11:56
*** roxanaghe has quit IRC11:59
*** andymaier has joined #openstack-infra12:01
*** sorantis has quit IRC12:02
*** vincentfrancoise has joined #openstack-infra12:02
*** psilvad has joined #openstack-infra12:03
*** aysyd has joined #openstack-infra12:05
*** claudiub has joined #openstack-infra12:05
*** amitgandhinz has joined #openstack-infra12:06
*** banix has joined #openstack-infra12:09
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Only update changed fields when editing tasks
*** yaume_ has joined #openstack-infra12:09
SotKjeblair, fungi: that patch fixes the disappearing task notes bug you encountered yesterday12:10
krotschecktimothyb89: Ohai, you ready for some subunit?12:11
*** amitgandhinz has quit IRC12:11
*** markvoelker has joined #openstack-infra12:11
*** dtantsur|lunch is now known as dtantsur12:11
*** trown|outtypewww is now known as trown12:11
*** aarefiev has quit IRC12:12
*** yaume has quit IRC12:12
*** aarefiev has joined #openstack-infra12:13
*** dprince has joined #openstack-infra12:14
*** markvoelker has quit IRC12:15
*** markvoelker has joined #openstack-infra12:16
*** daemontool has joined #openstack-infra12:16
*** nadya has joined #openstack-infra12:16
*** tmorin has quit IRC12:16
*** zhurong has joined #openstack-infra12:17
*** yaume_ has quit IRC12:20
*** yaume has joined #openstack-infra12:21
*** nadya has quit IRC12:21
*** sree has joined #openstack-infra12:22
*** tmorin has joined #openstack-infra12:22
*** asselin_ has joined #openstack-infra12:23
*** banix has quit IRC12:24
BobBallAny clues why might not be merging?  It's got +2 and +A; and I've attempted to reapply +A and reverify (since it says "Needs Verified") but still no joy.12:24
*** amoralej is now known as amoralej|lunch12:25
*** yamamoto has joined #openstack-infra12:25
*** asselin_ has quit IRC12:28
*** mdrabe has joined #openstack-infra12:29
mugsieBobBall: CI is having issues - that could be causing the merge to fail12:30
BobBallI suppose it could.  Will wait until they are fixed before asking again :P12:30
persiaBobBall: I think it may also need to merge first, if I'm reading gerrit correctly12:32
persia(or be rebased, etc.)12:33
AJaegerBobBall: see Topic12:33
BobBallOk.  Yes.  It might.12:33
BobBallAJaeger: I saw that; but my only jenkins job is a noop which is reporting Success12:33
BobBallBut I suspect persia may be right - I hadn't spotted that.12:33
AJaegerBobBall: Ah;) Yes, persia's observation is on the spot12:34
*** deadnull_ has joined #openstack-infra12:34
* BobBall hides his head in embarassment12:34
persiaNote that /topic may mean fixing that doesn't let it merge, of course :)12:34
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement propose_rdo_promote CI periodic job
*** yamamoto has quit IRC12:34
jrollis it known that is down?12:36
*** julim has joined #openstack-infra12:36
*** rlandy has joined #openstack-infra12:37
sdaguejroll: it's not down here, also that's not infra maintained12:39
jrollsdague: oh, just came back up12:40
jrollbut, noted. thanks12:40
*** alaski is now known as lascii12:40
*** iurygregory has quit IRC12:42
*** markusry has quit IRC12:42
*** nadya has joined #openstack-infra12:44
*** piet has joined #openstack-infra12:48
*** amotoki has joined #openstack-infra12:49
*** sdake has joined #openstack-infra12:49
*** daemontool has quit IRC12:50
*** matt-borland has joined #openstack-infra12:51
*** infra-red has quit IRC12:53
*** amotoki has quit IRC12:54
*** katomo has joined #openstack-infra12:54
*** markusry has joined #openstack-infra12:54
*** roxanaghe has joined #openstack-infra12:54
*** andreas_s has quit IRC12:54
*** d0ugal has joined #openstack-infra12:55
*** infra-red has joined #openstack-infra12:55
*** Julien-zte has joined #openstack-infra12:55
*** jcoufal has joined #openstack-infra12:56
*** tonytan4ever has quit IRC12:56
Adri2000is there a reason for not having https only? I'd "file a bug" for that but I'm not sure how to do it in storyboard :) (new story?)12:57
*** infra-red has quit IRC12:57
*** psachin has quit IRC12:59
*** roxanaghe has quit IRC12:59
*** rodrigods has quit IRC12:59
SotKAdri2000: creating a new story is indeed the correct way to file a bug12:59
*** rodrigods has joined #openstack-infra12:59
SotKI'm not sure which project you should pick to file it against in this situation12:59
*** andymaier has quit IRC13:00
Adri2000SotK: puppet-cgit ?13:00
SotKthat would be my best guess13:00
*** psilvad has quit IRC13:00
*** infra-red has joined #openstack-infra13:01
*** xyang1 has joined #openstack-infra13:03
Adri2000done at!/story/200062413:04
*** amitgandhinz has joined #openstack-infra13:06
fungiinfra-root: what's the situation with log uploads? do we have a problem with static.o.o?13:07
*** amitgandhinz has quit IRC13:07
*** amitgandhinz has joined #openstack-infra13:07
fungioh yeah, tons of i/o errors from at least one cinder volume13:07
*** binbincong has quit IRC13:08
*** _ari_ has joined #openstack-infra13:08
*** amitgandhinz has quit IRC13:08
fungii'll see if i can get the filesystem brought back into service13:08
fungiis anybody else working on it?13:08
fungii'm guessing not, as i'm the only user ssh'd into the server13:08
*** phschwartz_ is now known as pschwartz13:08
AJaegerfungi, you're the first infra-root to show up to action.13:08
*** amitgandhinz has joined #openstack-infra13:09
AJaegerfungi, thanks for looking into it!13:09
*** pschwartz is now known as phschwartz13:09
*** amotoki has joined #openstack-infra13:09
openstackgerritAndreas Jaeger proposed openstack-infra/zuul: Fix turbohipster git repo
*** infra-red has quit IRC13:12
*** zqfan has quit IRC13:13
pleia2good morning13:15
AJaegermorning, pleia2 !13:15
*** nadya has quit IRC13:16
*** anteaya has joined #openstack-infra13:16
*** daemontool has joined #openstack-infra13:16
*** yamamoto has joined #openstack-infra13:17
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes button clearer
fungidocs-draft and tarballs need a quick fsck unfortunately13:19
pleia2fungi: need some help?13:19
*** amoralej|lunch is now known as amoralej13:20
fungipleia2: nah, it's under control at this point. everything is almost back online again13:20
fungithanks though!13:20
fungitarballs is already cleaned up, docs-draft is taking a few minutes to fsck because it's larger13:20
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make vertical alignment of task table cells consistent
fungithe other volumes are back in service again, jenkins ssh access reenabled, puppet reenabled and apache's started back up13:21
pleia2thanks fungi!13:21
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes read-only by default
* pleia2 gets to translations checksite debugging13:22
*** nadya has joined #openstack-infra13:24
*** mriedem has joined #openstack-infra13:24
*** akostrikov has joined #openstack-infra13:24
*** annegentle has quit IRC13:24
*** sorantis has joined #openstack-infra13:25
*** tonytan4ever has joined #openstack-infra13:26
*** rbrndt has joined #openstack-infra13:29
fungiugh, docs-draft fsck is _still_ going13:30
AJaegersdague: could you review , please? That was something you run into in the past AFAIR in openstackdocstheme13:30
*** tonytan4ever has quit IRC13:31
mugsieAJaeger: yeah, that was for the custom CSS files13:32
*** PollyZ has joined #openstack-infra13:32
*** tlian has joined #openstack-infra13:33
mugsieAJaeger: was the patch that fixed that.13:33
*** zeih has quit IRC13:33
*** nadya has quit IRC13:34
*** Goneri has quit IRC13:34
openstackgerritYury Tsarev proposed openstack-infra/zuul: Set filter according to PR/Change in URL
AJaegermugsie: ah, thanks!13:35
*** zeih has joined #openstack-infra13:35
*** katomo has quit IRC13:36
*** PollyZ has left #openstack-infra13:36
*** tlian has quit IRC13:38
mnaserfungi: if you're having issues with block storage, we can start providing a few nodes for static infra as we spoke a while ago ^13:39
fungimnaser: well, statistically we're skirting danger by having a logical volume residing on 12 cinder volumes of 1tb each13:40
*** tempo has joined #openstack-infra13:40
*** edmondsw has joined #openstack-infra13:41
mnaserfungi: forgive me if this is a silly question but .. why not a single 12tb volume..?13:41
fungiso if a network blip interrupts connectivity between the hypervisor and any of the backends serving those, we end up with an outage13:41
fungimnaser: rackspace sets a 1tb maximum limit on cinder volumes13:41
mnaseroh i see13:41
fungimax 1tb size per cinder volume that is13:41
*** gouthamr has joined #openstack-infra13:41
*** zeih has quit IRC13:42
mnaseryeah given we're using ceph we don't really have problems with however big volumes you want/need13:42
fungiand actually it's 14 cinder volumes, i misremembered13:42
*** zeih has joined #openstack-infra13:42
*** trown is now known as trown|brb13:42
fungibasically the maximum number of additional block devices their xen deployment will support attaching to a single domu guest13:42
anteayajroll: here is the place to file bugs with the website: we don't maintain the servers or have any say on the content13:42
sdagueAJaeger: yeh, I proposed that before for the same reasons that mugsie is13:43
fungimnaser: but yeah, that's tempting. we do end up with a lot of disruption from this particular configuration13:43
sdaguemugsie is working on trying to get os-api-ref working with docs theme13:43
fungioh finally docs-draft has gotten past fsck stage 2 and is repairing links13:43
sdaguemugsie: how did you deal with conflicting jquery?13:43
*** vincentfrancoise has left #openstack-infra13:44
mnaserfungi: let me know if -infra decides to move things over so i can make sure the appropriate resources are allocated13:44
fungimnaser: thanks, i'll bring it up and see if there's ideas around how we might sync that data over gracefully13:45
*** thiagop has joined #openstack-infra13:45
*** Guest27231 has quit IRC13:45
mugsiesdague: eh, I am pretty sure I just dropped one of them13:45
mnaserfungi: sounds good, if there's anything we can do to help out on a provider level id be more than happy to13:46
*** Guest27231 has joined #openstack-infra13:46
*** vdrok has quit IRC13:46
*** tlian has joined #openstack-infra13:47
*** Guest27231 is now known as nikhil13:48
*** nwkarsten has joined #openstack-infra13:48
*** nwkarsten has quit IRC13:48
*** nwkarsten has joined #openstack-infra13:49
*** bauzas is now known as bauwser13:49
*** piet has quit IRC13:50
bswartzInfra team: can we start to use xenial for our jenkins jobs yet, or are those images not yet ready for use?13:50
*** dansmith is now known as superdan13:51
fungibswartz: they are ready for use as far as i know, though we haven't moved all jobs for master branches over to them yet (i think there may still be some devstack/tempest job issues getting worked out? i don't know the exact state there)13:52
pabelangeragreed. Other projects are using them, we still need to schedule python-jobs update13:52
bswartzfungi: we may test it out with one of our jobs -- thanks13:52
fungipabelanger: you did the eavesdrop server upgrade, right? i've got a snapshot generating for the old offline server and will delete the server once the image is complete13:53
*** mriedem is now known as mriedem_meetings13:53
*** dprince has quit IRC13:53
*** dprince has joined #openstack-infra13:54
pabelangerfungi: yes, I did the new server. Thanks, I didn't generate a snapshot13:54
*** piet has joined #openstack-infra13:54
funginp, just happened to be in there looking for tickets about block storage disruption overnight and spotted that straggler13:55
*** tempo has quit IRC13:55
fungiokay, docs-draft is finally repaired13:55
*** roxanaghe has joined #openstack-infra13:55
*** mriedem has joined #openstack-infra13:56
*** trown|brb is now known as trown13:57
fungi#status ok Cleanup from earlier block storage disruption on has been repaired, and any jobs which reported an "UNSTABLE" result or linked to missing logs between 08:00-14:00 UTC can be retriggered by leaving a "recheck" comment.13:57
openstackstatusfungi: sending ok13:57
*** mriedem_meetings has quit IRC13:58
anteayafungi: thank you13:58
*** amitgandhinz has quit IRC13:58
*** amitgandhinz has joined #openstack-infra13:58
fungiyou're welcome!13:59
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"13:59
-openstackstatus- NOTICE: Cleanup from earlier block storage disruption on has been repaired, and any jobs which reported an "UNSTABLE" result or linked to missing logs between 08:00-14:00 UTC can be retriggered by leaving a "recheck" comment.13:59
*** roxanaghe has quit IRC13:59
*** e0ne has quit IRC13:59
*** e0ne has joined #openstack-infra14:00
*** eharney has joined #openstack-infra14:00
*** ayoung has joined #openstack-infra14:00
openstackstatusfungi: finished sending ok14:02
pabelangerfungi: was there a specific volume that had the problems?14:03
*** sree has quit IRC14:03
tdasilvagood morning, has anybody run into this error with bindep?
*** salv-orlando has joined #openstack-infra14:05
fungipabelanger: yes, according to dmesg, at least several of them14:05
pabelangertdasilva: does other-requirements.txt exist?14:06
tdasilvapabelanger: yes14:06
pabelangerlooks like an issue with lsb_release if I am reading this right14:06
*** sorantis has quit IRC14:06
pabelangerfungi: thanks14:07
fungipabelanger: i count 7 of the 14 attached volumes reporting i/o errors at 08:34 utc today14:07
tdasilvapabelanger: i'm trying to test this patch
*** sorantis has joined #openstack-infra14:08
fungipabelanger: xvdd, xvdf, xvdg, xvdh, xvdi, xvdk, xvdl14:08
*** Na3iL has quit IRC14:08
pabelangertdasilva: let me try the patch14:10
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add support for bindep
tdasilvapabelanger: this fixed it:
tdasilvapabelanger: thanks for the pointer :-)14:12
pabelangertdasilva: Great, tested it locally and it works too14:12
tdasilvapabelanger: great! thanks14:13
*** abregman has joined #openstack-infra14:13
*** xarses_ has quit IRC14:13
*** hongbin has joined #openstack-infra14:14
*** nadya has joined #openstack-infra14:14
*** e0ne has quit IRC14:14
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes read-only by default
*** e0ne has joined #openstack-infra14:14
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix spelling mistake on profile page
*** nadya has quit IRC14:16
*** weshay has joined #openstack-infra14:17
fungiSotK: here's an interesting one... i'm guessing it's a rendering issue, but the content in the comment i added a few minutes ago isn't displayed though you can see it quoted if you hit the reply button!/story/200033614:21
*** denisra has quit IRC14:21
tdasilvapabelanger: also, just a heads up, the launchpad link referenced here: doesn't exist14:22
* SotK takes a look14:22
SotKvery weird14:22
*** sigmavirus24_awa is now known as sigmavirus2414:22
* SotK investigates further14:22
*** timello has joined #openstack-infra14:22
openstackgerritMerged openstack-infra/project-config: Fix VENV temporary path
pabelangertdasilva: Yes, we should add something there too14:23
*** Goneri has joined #openstack-infra14:23
SotKfungi: argh, the html sanitiser is trying to parse the email addresses in the comment as html elements14:24
fungitdasilva: pabelanger: oops, that's supposed to be a link to!/project/81114:24
tdasilvapabelanger: just to clarify, when I add other-requirements.txt and define a jenkins job, will those dependencies be automatically installed before tests are run?14:25
*** r-daneel has joined #openstack-infra14:25
fungiSotK: i wondered if it might be something like that. the preview was displaying the comment but eating the < and > symbols (even if i replaced them with &lt; and &gt;)14:25
tdasilvafungi: I can update that link, I was in the middle of fixing a little typo I found14:26
fungitdasilva: only if the job includes the install-distro-packages builder14:26
pabelangertdasilva: yes, you need to use the install-distro-packages macro to use other-requirements.txt14:26
anteayafungi: SotK I replied to the not visible comment to see what would happen, now I can see both fungi's comment and my own14:26
pabelangerthat fungi said14:26
tdasilvafungi, pabelanger: awesome, thanks, will give it a try...14:26
jeblairugh not again14:26
jeblairfungi: anything need doing?14:26
anteayafungi: SotK yes I had noticed that the comment parser consumes any angle brackets14:27
fungijeblair: i think we're all patched up now on the static.o.o front14:27
*** mdrabe has quit IRC14:27
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: A newer version of virtualenv is already installed
SotKanteaya: yeah, it seems to have turned the email addresses into mailto: links in your comment14:28
fungijeblair: i pushed up a proposed launcher fix for keep-hierarchy feature parity last night 324996 (looks like yolanda approved it already?)14:28
openstackgerritThiago da Silva proposed openstack-infra/bindep: fix small typo and link
anteayaSotK: fungi interesting, do we want email addresses in comments turned into mailto: links?14:29
*** ri0 has joined #openstack-infra14:31
*** Na3iL has joined #openstack-infra14:32
*** e0ne has quit IRC14:33
*** zz_dimtruck is now known as dimtruck14:33
*** xarses_ has joined #openstack-infra14:34
*** matrohon has quit IRC14:35
anteayaI think that comments shouldn't try to be too smart, they should just render the characters as they are input14:35
*** vdrok has joined #openstack-infra14:35
*** links has quit IRC14:36
anteayaelse we risk losing being able to accurately communicate what we are trying to say14:36
SotKthey currently render as markdown, I don't want to lose the rich-text functionality14:36
fungianteaya: well, it's tricky with web interfaces. you have to translate/sanitize certain things to keep browsers from reinterpreting things like <script>14:36
anteayafair enough14:36
anteayaSotK: okay14:37
fungiSotK: i likely just didn't realize that was trying to interpret markdown. i'll read up on the syntax for it (i'm only used to restructuredtext and raw html)14:37
*** e0ne has joined #openstack-infra14:37
*** Daisy has joined #openstack-infra14:38
*** kzaitsev_mb has quit IRC14:38
*** tonytan4ever has joined #openstack-infra14:38
*** PollyZ has joined #openstack-infra14:38
jeblairfungi: i will post-hoc review it, thanks :)14:38
fungijeblair: thanks, it seemed trivial, but i did not test it yet14:39
*** Mmike has quit IRC14:39
*** Mmike has joined #openstack-infra14:39
SotKfungi: yeah, I keep meaning to add a note mentioning it and a link to some syntax guide but haven't got round to it14:40
*** bryan_att has joined #openstack-infra14:40
*** mdrabe has joined #openstack-infra14:41
*** jcoufal has quit IRC14:42
jeblairfungi: neat.  it even works with more than one matching file14:42
*** deadnull_ has quit IRC14:42
fungithat was the goal14:42
fungibecause some of our jobs may depend on that, not sure14:43
*** Daisy has quit IRC14:43
jeblairfungi: that's my job definition, and then what showed up14:43
*** jcoufal has joined #openstack-infra14:43
*** csomerville has joined #openstack-infra14:43
jeblairfungi: anything else you think i should test while i have this set up?14:44
fungimy only concern is that _if_ you need the hierarchy flattened _and_ there are uploaded files with spaces in their names, it may (probably will) break. if we null-separate the find output and tell rsync the file list is null-separated that mighth be an improvement, but would need some further fiddling14:44
jrollSotK: I like the way github does it, a little note under the comment box with a link to a guide14:45
jeblairfungi: do you think that might provide a shell escape vulnerability?  we are running this on the launcher...14:45
*** hasharAway is now known as hashar14:45
clarkbjeblair I would test **/whatever with multiple dir levels14:46
*** zeih has quit IRC14:46
fungijeblair: oh, good point... i bet we could tease find -print into formatting the paths with " wrapping?14:46
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Deprecated snaphot image building
jeblairclarkb: ya -- i think there's a tox macro that will do that -- i'll base a test on that.14:46
*** rcernin has quit IRC14:46
jeblairi'm trying the space thing now14:46
*** kzaitsev_mb has joined #openstack-infra14:47
fungii expect it to assume the spaces are delimiting different paths14:47
fungiand fail miserably14:47
*** EricGonczer_ has joined #openstack-infra14:47
*** Guest86121 has quit IRC14:48
timothyb89krotscheck: hi, sorry I'm a bit late, let me know if you still wanted to discuss subunit things14:48
krotschecktimothyb89: No worries, gimme 30 minutes, the munchkin is up and needs to go to daycare.14:48
timothyb89krotscheck: sure, no problem14:49
anteayajroll: I'm going to start by creating a gui guide, currently there isn't one and they putting it in there:!/story/200062714:49
jrollanteaya: ++14:49
jrollanteaya: I like the little link because I often don't realize I need the markdown guide until I'm writing something14:50
anteayajroll: then we can work on links in the gui, actually can you add your comment to my story?14:50
*** Hal has joined #openstack-infra14:50
anteayajroll: fair enough14:50
sdaguealso, d-g core folks - - it's blocking things that would make liberty actually pass14:50
anteayajroll: thank you14:50
*** amitgandhinz has quit IRC14:50
jeblairfungi, clarkb: yep, on spaces:
*** Hal is now known as Guest7782014:50
mriedemcan we get another infra core to +W this d-g change from sdague so we can get live migration testing working in master?
*** amitgandhinz has joined #openstack-infra14:51
jeblairfungi, clarkb: that was "a file.tar.gz" and "b file.tar.gz"14:51
acabothi, I just submitted a new release for Watcher project (freshly accepted in the big tent) #link but tox cannot validate it because of this msg "deliverables/newton/watcher.yaml release 0.3.0 is missing openstack/watcher from the governance list"14:51
acabotdo I need to update a specific file on openstack-infra ?14:52
*** jaosorior has quit IRC14:52
*** Goneri has quit IRC14:52
AJaegeracabot: for release, best ask on #openstack-release14:53
*** raildo is now known as raildo-afk14:53
fungijeblair: after the release meeting, i'll see if i can come up with a good quoting solution for the find output (unless you're already hacking one out)14:53
acabotAJaeger : sure thx14:53
jeblairfungi: i am not14:53
*** xarses_ is now known as xarses14:54
fungicool, should have something in the next hour then14:54
*** dizquierdo has joined #openstack-infra14:54
*** Kaiyan has joined #openstack-infra14:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix the profile submenu
*** chem` has quit IRC14:55
*** roxanaghe has joined #openstack-infra14:56
*** sorantis has quit IRC14:57
*** javeriak has quit IRC14:58
*** arxcruz has quit IRC14:59
fungiunfortunate that find -printf doesn't seem to support %q14:59
fungithat would have been a trivial solution15:00
*** javeriak has joined #openstack-infra15:00
*** roxanaghe has quit IRC15:00
fungimay need to use find -print0 | rsync -0 ...15:01
*** ifarkas has quit IRC15:01
fungithat should get us around any shell injection risk, without opening up format string vulnerability risks in the process15:01
*** fedexo has joined #openstack-infra15:03
openstackgerritMerged openstack-infra/system-config: Remove double slash for individual links
*** mtanino has joined #openstack-infra15:04
openstackgerritPaul Belanger proposed openstack-infra/zuul: Stream subprocess output for ansible-playbook
pabelangerjeblair: ^ Tests the best I could locally, but think that gives realtime output from ansible15:04
AJaegerjeblair, project-config cores: There's a change by dhellmann to introduce a new queue into Zuul - a release-check one. jeblai, I'd like your review on that one, please15:04
AJaegerjeblair: change is
*** zul has quit IRC15:05
*** zul has joined #openstack-infra15:05
*** Goneri has joined #openstack-infra15:05
*** e0ne has quit IRC15:05
*** lezbar has quit IRC15:05
*** csomerville has quit IRC15:06
openstackgerritMerged openstack-infra/project-config-example: Email notifications should only be for silent checks.
*** fedexo has quit IRC15:06
*** tphummel has joined #openstack-infra15:06
krotschecktimothyb89 ALrightey, let's take a look. You'd like to land this in openstack-health, yes? DId you want the karma-subunit-reporter to be managed by infra as well?15:07
*** mriedem1 has joined #openstack-infra15:07
*** e0ne has joined #openstack-infra15:07
*** e0ne has quit IRC15:08
*** tmorin has left #openstack-infra15:08
*** mriedem has quit IRC15:08
timothyb89krotscheck: right, looking to add to openstack-health/stackviz15:08
timothyb89krotscheck: as for management, wherever the best home will be :)15:08
openstackgerritMerged openstack-infra/ansible-puppet: Remove dead code
openstackgerritThiago da Silva proposed openstack-infra/project-config: add liberasurecode gate jobs
*** mriedem1 is now known as mriedem15:09
timothyb89krotscheck: there's also to worry about as well15:10
pabelangerjeblair: we may need to pass PYTHONUNBUFFERED = 1 to env, for ansible too.  We currently do that when running ansible from jenkins to enable live streaming in ansible15:11
krotschecktimothyb89: Well, governance is orthogonal to getting it into the project.15:13
*** javeriak has quit IRC15:13
krotscheckSo let's start with making it work, and then bikeshed in the QA channel as to whether we want those two projects in Gerrit :D15:13
timothyb89krotscheck: okay, sounds like a plan!15:14
timothyb89krotscheck: I've already got a preliminary patch up for openstack-health:
krotschecktimothyb89: That patch works, eh?15:16
timothyb89krotscheck: it should, though we'll need patches to project-config and the log client to actually collect results15:16
*** dprince has quit IRC15:17
*** jcoufal has quit IRC15:17
krotschecktimothyb89: Makes sense, and should be easy enough. When you say "collect" though, you mean having it stream back to "<insert something here>", yes?15:18
*** r-daneel has quit IRC15:18
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a note mentioning markdown is supported under the comment box
timothyb89krotscheck: publishing to the log server during npm-run jobs for one, but also collecting for subunit2sql15:19
krotscheckI'm digging through project-config right now to figure out where that's done in other places.15:21
*** piet has quit IRC15:22
jeblairAJaeger: that change looks techincally correct, so i left a comment to that effect.  i would like to abstain from weighing in on whether it's something we want to do.15:22
timothyb89krotscheck: I want to say subunit collection is configured here:
*** vincentll has quit IRC15:23
*** thiagop is now known as thiagop-afk15:23
krotscheckAh, that's system-config15:23
* krotscheck was looking the wrong place15:23
*** pradk has quit IRC15:24
AJaegerjeblair: thanks for reviewing. Let's see what others think15:24
*** pradk has joined #openstack-infra15:24
*** pradk has quit IRC15:24
pabelangercloudnull: morning15:24
eharneywhat should i do if i'm getting "Code Review - Error" "file not found in revision 317643,4" when trying to post comments on a review?   i've somehow written comments that can't be saved?15:24
pabelangerI noticed a few SSH timeouts this morning on osic nodes, for example15:24
pabelanger2016-06-03 15:20:56,577 ERROR nodepool.NodeLauncher: Timeout launching node id: 1405352 in provider: osic-cloud1 error: Timeout waiting for ssh access15:24
pabelangercloudnull: we usually see that when we cannot connect into the public IP of a server for whatever reason15:25
*** pradk has joined #openstack-infra15:25
jeblairfungi: i've been attempting to make a shell injection vulnerability, but so far i can't seem to create a file with backticks in its name.  i may not be very good at haxoring.15:25
krotscheckAnyone familiar with how subunit/elasticsearch is set up in infra that can give me a quick overview of how our builds send streams?15:26
jeblairfungi: er, s/vulnerability/exploit/15:26
sdagueclarkb: ug, after all that the d-g change fails on rax networking at the ansible phase -
fungijeblair: how about files with > or | or ; in their names?15:26
kashyapHeya folks, I'm sure I've asked this before here, but I forget, in a series of changes, when Jenkins gives throws this: "This change depends on a change that failed to merge."  What could be the root cause?
clarkbkrotscheck: the builds dont send anything but a completion event, another daemon picks that up and pulls logs off the log server and processes them15:27
jeblairfungi: okay, i'll try that15:27
pabelangercloudnull: and example of how often it happens:
fungijeblair: or &15:27
*** migi is now known as migi_afk15:27
*** migi_afk is now known as migi15:27
*** nwkarsten has quit IRC15:27
*** woodster_ has joined #openstack-infra15:27
krotscheckclarkb: Gotcha, so copying the subunit to /logs/testrepository.subunit and ensuring that they're uploaded would be sufficient.15:27
krotscheckclarkb: Everything else is automagic?15:27
*** zhurong has quit IRC15:27
*** nwkarsten has joined #openstack-infra15:28
fungijeblair: really, any character which could trigger overwriting arbitrary files or terminate the current shell command and initiate a new one15:28
clarkbkrotscheck: anno you still have to co figure the backend to do things with it, I lack context for wjat you are trying to do15:28
krotscheckclarkb: We have a subunit reporter for javascript builds now.15:28
AJaegerkashyap: rebase your change15:30
kashyapAJaeger: So that Zuul re-queus it?15:31
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Upload karma.subunit to log server
krotscheckclarkb: Here's the start ^^15:31
AJaegerkashyap: I'm currently double checking your change...15:31
*** rcernin has joined #openstack-infra15:31
kashyapAJaeger: Not mine, asking for another contributor on -nova15:31
AJaegerkashyap: those message come from 3rd party CIs - nothing you can do besides informing the 3rd party CIs.15:32
*** vhosakot has joined #openstack-infra15:32
*** nwkarsten has quit IRC15:32
AJaegerWait, Jenkins also reports it. Let me dig deeper...15:32
*** sree has joined #openstack-infra15:33
*** mdbooth has joined #openstack-infra15:34
AJaegerkashyap: I'm confused now - hope somebody else can help...15:34
kashyapAJaeger: mdbooth is the author here15:35
mdboothAJaeger: Hey, kashyap is abandoning his attempts to repurpose himself as a message broker15:35
kashyapmdbooth: Messages you might've missed above: AJaeger says what you observed: Jenkins reports too, and that he's not sure of it...15:35
kashyapOkay, out for real :-) Gotta clean the dishes and run to the tram station15:36
mdboothSome of the patches which were kicked out of zuul are now back in15:36
mdboothSo it might just be resolving itself15:36
openstackgerritTim Buckley proposed openstack-infra/system-config: Collect 'karma.subunit' files
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Get translation jobs into healthcheck
timothyb89krotscheck, clarkb: ^^ I think this should be the changes for subunit2sql collection as well15:37
clarkbtimothyb89: krotscheck right so that will get your subunit log files on the log server. It will not do anything to elasticsearch15:38
clarkbwhich is fine but I have a feeling you are looking for something more?15:38
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Upload karma.subunit to log server
mdboothAJaeger: I suspect a race in whatever queues the jobs meant that they were initially queued in the wrong order.15:39
*** d0ugal has quit IRC15:39
timothyb89clarkb: are there any other places subunit results are used?15:39
*** rcernin has quit IRC15:39
clarkbtimothyb89: subunit2sql slurps them up but only for tempest I think15:40
clarkbmdbooth: no I don't think that is it15:40
EmilienMfungi: I'll let you take decision on this new periodic job:
timothyb89clarkb: I think this patch should configure collection for subunit2sql?
clarkbmdbooth: I think what happened was you pushed a new patchset to a parent change which kicked the child out of the gate because its parent updated15:40
mdboothclarkb: I pushed the whole series at once.15:41
openstackgerritJeremy Stanley proposed openstack-infra/zuul: Make ansible launcher rsync flattening more secure
clarkboh hrm ya they all went up about an hour ago15:41
fungijeblair: so there's an option ^15:41
*** chem` has joined #openstack-infra15:41
openstackgerritamrith proposed openstack-infra/project-config: Add channel notification for trove-image-builder
*** asettle has quit IRC15:42
*** Na3iL has quit IRC15:43
*** ri0 has quit IRC15:43
*** csomerville has joined #openstack-infra15:44
notmorganfungi: always print filenames exactly as is? isn;t that the opposite of what you want?15:44
funginotmorgan: is it?15:45
notmorganfungi: wouldn't -ls/fls be more correct?15:45
*** ihrachys has quit IRC15:45
notmorganfungi: -print0: Always print the exact filename, unchanged, even if the output is going to a terminal.15:45
funginotmorgan: it's not going to a terminal though, it's going to a pipe15:45
fungiand rsync is instructed to read the list from a pipe15:45
fungiso no shell to interpret any of that15:46
clarkbthe trivial explanation would be that nova master has moved ahead around when you did your rebase and introduced a new conflict15:46
clarkbbut I haven't tested that15:46
notmorganfungi: hmmmm.15:46
notmorganfungi: oh i missed the '|'15:46
*** infra-red has joined #openstack-infra15:47
notmorganfungi: yeah thats fine then :)15:47
funginotmorgan: it's the classic solution for avoiding shell injection (and misinterpretation of filenames in general)... get rid of the shell when passing data15:47
notmorganfungi: i thought it was just -print015:47
*** sree has quit IRC15:47
* fungi was hoping the code comment there was explicit about it15:47
notmorganfungi: pre-coffee15:47
notmorganfungi: comments look like line noise ;)15:48
fungii'm about to disappear for lunchies anyway15:48
notmorganfungi: ooh yum. enjoy15:48
anteayafungi: enjoy lunchies15:48
fungiafter i have a look at EmilienM's change15:48
EmilienMfungi: this is not urgent, lunch is > than my patch ;-)15:49
notmorganEmilienM knows whats up. Lunch > lots of things.15:49
openstackgerritTim Buckley proposed openstack-infra/system-config: Collect 'karma.subunit' files
*** csomerville has quit IRC15:49
EmilienMnotmorgan: I'll let you imagine "French lunch > ?" ;-)15:50
fungitime is an illusion, lunchtime doubly so15:50
notmorganEmilienM: i don't need to imagine.15:50
notmorganEmilienM: French Lunch > all15:50
pleia2chocolate bread with cheese and wine15:50
*** Na3iL has joined #openstack-infra15:51
jeblairfungi, notmorgan: hrm, that's not working -- rsync is outputing the usage message15:51
notmorganpleia2: oooooooooh15:51
krotscheckclarkb, timothyb89: So, plus plus equals subunit-in-elasticsearch, yes?15:51
fungijeblair: oh, i wonder if the '{source}' replacement being two parameters is causing it to misinterpret15:51
clarkbkrotscheck: there is no subunit in elasticsearch anywhere15:52
notmorganfungi: very well might be.15:52
clarkbkrotscheck: which is sort of why I am asking about what you want to achieve15:52
jeblairfungi: i think you still need SRC with files-from15:52
*** EricGonczer_ has quit IRC15:52
fungijeblair: aha, i bet that's it15:53
krotscheckclarkb: That's why I asked for explanation as to what the pieces in infra were earlier wrt subunit, because I don't understand the system well enough to know what's possible, and what's currently being done.15:53
jeblair"The filenames that are read from the FILE are  all  relative  to the  source  dir"15:53
*** EricGonczer_ has joined #openstack-infra15:53
*** bhavik has joined #openstack-infra15:53
pleia2while we have some folks around, etherpad for rename in ~4 hours could use some eyes (anteaya looked+added comments yesterday), the changes are prepped and ready15:53
fungijeblair: yeah, then i'll likely need to rework that also so that find prunes the parent dirs15:53
fungii can take a stab at fixing after lunch15:53
krotscheckclarkb: In short: We have jobs that generate subunit. What do those jobs actually do right now? Do they send a stream of results to a collector? How is that collector determined? Do they just upload it to the log server? ¯\_(ツ)_/¯15:54
anteayafungi: are we altering the timing of the rename at all for release folks? pleia215:54
fungianteaya: only if they're not done with release activities by the start of the window15:55
*** markmcd has quit IRC15:55
*** pblaho has quit IRC15:55
anteayafungi: pleia2 grand thank you15:55
*** pblaho has joined #openstack-infra15:56
*** schipiga has quit IRC15:56
*** roxanaghe has joined #openstack-infra15:56
EmilienMok I have a dumb question, is there a mechanism in place in our infra that +2 +A a patch that is passing CI on a patch proposed by our Proposal Bot?15:56
*** markmcd has joined #openstack-infra15:57
fungiEmilienM: nope, by design we expect actual humans to review proposed changes and be responsible for interpreting them and their job results before approving them to merge15:58
EmilienMok, makes sense thx15:58
*** mikelk has quit IRC15:58
*** infra-red has quit IRC16:00
anteayaEmilienM: we had a long chat about this in Austin, what you propose would be a vector for exploitation16:01
EmilienMok cool no prob, just asking16:01
jeblairfungi: fwiw, with the current state (not your proposed pipe patch), when i create "a&foo.tar.gz" i end up with "a&foo.tar.gz" at the destination.16:02
anteayaEmilienM: sure16:02
notmorganjeblair: sounds like it isn't breaking out of the shell in the find.16:03
notmorganjeblair: which is good news (tm)16:03
fungijeblair: that's a good sign. the additional pipe might just be overkill16:03
fungiokay, heading out. bbiab16:03
*** Swami has joined #openstack-infra16:03
notmorganfungi: i would worry about >, |, and `` still but if & isn't causing issues i'm not sure the others would.16:04
notmorganfungi: and/or $()16:04
*** bdemers has quit IRC16:05
*** jlanoux has quit IRC16:05
cloudnullpabelanger: ping. sorry was AFK. Do you happen to have the ip addresses for a few of the nodes that we/are failing to get SSH access?16:07
krotschecktimothyb89: seems like clarkb's busy, so let's see what we can do about getting those tests into openstack-health. How does openstack-health accept subunit data?16:08
timothyb89krotscheck: it queries it directly through subunit2sql16:08
timothyb89krotscheck: mtreinish would know for sure, but I think should be enough for that16:09
*** javeriak has joined #openstack-infra16:09
*** sree has joined #openstack-infra16:09
krotschecktimothyb89: Oh good!16:09
jeblairclarkb, fungi: fungi's initial solution also works for the  '.tox/*/log/*' and '**/*testrepository.subunit.gz' cases16:10
krotschecktimothyb89: Alright, let's go ahead and see what the gate thinks, and if we're lucky it will all work like magic :D16:11
timothyb89krotscheck: cool!16:11
openstackgerritThiago da Silva proposed openstack-infra/project-config: add liberasurecode gate jobs
* krotscheck pushes the "That was easy" button16:11
*** piet has joined #openstack-infra16:12
pabelangercloudnull: Let me look16:12
*** PollyZ has quit IRC16:12
*** bdemers has joined #openstack-infra16:12
pabelangercloudnull: 2016-06-03 15:23:34,005 DEBUG nodepool.NodeLauncher: Node id: 1405539 is running, ipv4:, ipv6:16:13
timothyb89krotscheck: while we're on the topic, do you have any thoughts about proper translation of karma spec names to something more subunit-y?16:13
*** javeriak_ has joined #openstack-infra16:13
jeblairnotmorgan, fungi:
timothyb89krotscheck: right now it's dot-separating and converting16:13
*** trown is now known as trown|lunch16:13
jeblairnotmorgan, fungi: those all seem to show up in the destination as they are in the source16:13
timothyb89krotscheck: ... and converting ' ' to '_', resulting in names like this: GroupedRunsController.should_process_chart_data_correctly16:13
krotschecktimothyb89: What would be a better subunit-y format?16:14
notmorganjeblair: ok cool16:14
*** moravec has quit IRC16:14
timothyb89krotscheck: well, for openstack-health it works alright, but for e.g. ironic-webclient the results are less good16:14
notmorganjeblair: i'm happy with that (I can take a stab at other awful shell things as well after i get checked out of my hotel if you want - but it seems pretty sane)16:14
timothyb89krotscheck: that translation doesn't work so well for 'Unit: OpenStack Ironic API Transformation Methods'16:15
*** degorenko is now known as _degorenko|afk16:15
*** reed has quit IRC16:15
*** reed has joined #openstack-infra16:15
jeblairnotmorgan: yeah, the more the merrier on something like this :)16:15
krotschecktimothyb89: Ah, yes. That was during my verbose phase16:15
*** sambetts is now known as sambetts|afk16:15
*** javeriak has quit IRC16:16
openstackgerritMerged openstack-infra/project-config: Add manila-specs to specs.yaml
*** mriedem is now known as mriedem_away16:16
krotschecktimothyb89: Also, we may have other projects that use things like protractor, which'll have an entirely different output.16:16
krotscheck(protractor-subunit anyone?)16:16
openstackgerritMerged openstack-infra/project-config: Add missing release jobs for instack & instack-undercloud
timothyb89krotscheck: heh, I think that's less an issue with your code, and more a problem with my quick-and-dirty normalizing16:16
timothyb89krotscheck: that should be doable as well :)16:17
*** aarefiev has quit IRC16:17
*** bnemec has quit IRC16:17
Shrewsjeblair: remind me again why we call rsync directly16:18
*** bhavik1 has joined #openstack-infra16:18
*** nwkarsten has joined #openstack-infra16:18
*** bdemers_ has joined #openstack-infra16:18
*** tesseract has quit IRC16:19
openstackgerritMerged openstack-infra/project-config: Implement propose_rdo_promote CI periodic job
jeblairShrews: because we're copying from controller to third-party site.  synchronize module has no option to copy stuff that does not include the target host as either the source or destination.16:19
*** PollyZ has joined #openstack-infra16:19
krotschecktimothyb89: The jasmine naming pattern is (thing)+ (should behave like X). The naming I use in ironic-webclient asumed that the test categorization needed to happen in the test description, which I've since learned isn't really necessary.16:19
krotschecktimothyb89: Ultimately, quick-and-dirty normalization becomes a usability issue, so it's probably easier to address issues as they come up.16:20
*** xarses has quit IRC16:20
*** aarefiev has joined #openstack-infra16:20
cloudnullpabelanger: would it be possible to keep one of the bad actor nodes online for a while so I can poke at things ?16:20
anteayamorning zaro16:20
timothyb89krotscheck: fair enough, right now the plugin just leaves those as off-by-default options, probably best to keep it that way16:21
cloudnulllooks like the VM using was cleaned up16:21
*** nadya has joined #openstack-infra16:21
clarkbkrotscheck: aiui its tempest only aubunit in health16:21
anteayazaro: would you have any time in perhaps getting the storyboard gerrit plugin working on review-dev?16:21
Shrewsjeblair: ah right16:21
*** bhavik has quit IRC16:21
anteayazaro: if we can get it working on review-dev that means that Zara doesn't have to take her time figuring out how to stand up a gerrit16:21
*** david-lyle has joined #openstack-infra16:22
pabelangercloudnull: I'm not sure it is possible.  Basically, if we cannot ssh into the VM, we delete it right away16:22
pabelangerlet me look at the code quickly16:22
*** bdemers has quit IRC16:22
*** bhavik1 has quit IRC16:23
*** bnemec has joined #openstack-infra16:23
pabelangercloudnull: Ya, we delete it on exception. So I'd need to randomly hold nodes currently in the build state in an attempt to catch one16:24
*** PollyZ has quit IRC16:24
pabelangercloudnull: let me try that16:24
cloudnullif its too much ill try and attack it another way16:25
*** mhickey has quit IRC16:25
jeblairpabelanger: it looks like jhesketh didn't get around to doing a watchdog for the ansible timeout.  have you started working on that, or should i?16:25
jeblairpabelanger: (also see comment on 325356)16:25
pabelangercloudnull: Ya, I have no way to know which will hold, until after it fails16:26
pabelangerjeblair: Oh, I thought I see some watchdog code16:26
jheskethjeblair: actually I did16:26
jeblairjhesketh: oh neat!16:26
jheskethShould be tagged with v2516:26
zaroanteaya: just got back from pycon.  this is my priority: 1 gerrit git gc research, 2 find solutions for gerrit memory issues, 3 figure out how we can force online reindex for project renames.  i might be able to do storyboard in parallel but would be lower priority16:26
jheskethNot sure if it's what you had in mind16:26
jeblairgah it's right there16:27
jeblairjhesketh: sorry i missed it16:27
jheskethIt's quite crude16:27
jheskethAll good :-)16:27
anteayazaro: fair enough, welcome back from pycon, I hope you had a good time, thank you for sharing your priority list with me, I appreciate it16:27
pabelangerjeblair: thanks for the comment on 325356, I didn't even think of doing that.16:27
anteayajhesketh: what are you doing up?16:27
*** cody-somerville has joined #openstack-infra16:27
jheskethThat is a good question16:28
cloudnullpabelanger:  no worries. I'll figure something else out16:28
anteayajhesketh: glad I asked16:29
krotscheckclarkb: Gotcha. I think OpenStack Health wants to report on itself :)16:29
jeblairjhesketh: since we're using threads now, i don't think we can use sigalarm (we'll have a bunch of those going at once)16:30
*** nwkarsten has quit IRC16:31
*** shashank_hegde has joined #openstack-infra16:31
jeblairjhesketh: how about i rework that patch to spawn a timeout thread while you sleep?16:31
pabelangercloudnull: would a list of IPs help?16:31
zaroanteaya: np, sorry i think i missed some discussion around gerrit on storyboard channel, lost my backscroll because i switched my irc server.16:31
*** nwkarsten has joined #openstack-infra16:31
*** rcernin has joined #openstack-infra16:31
cloudnullpabelanger: that would16:31
cloudnullIDK if the issue is IP specific but i'll try to rule that out16:32
*** Apoorva has joined #openstack-infra16:32
*** dprince has joined #openstack-infra16:32
anteayazaro: I understand, fortunately meetbot was mostly working
*** kushal has joined #openstack-infra16:33
pabelangercloudnull: here is the most recently failures:
pabelangercloudnull: I can write a quick script to generate a list of all failures too if needed16:33
*** PollyZ has joined #openstack-infra16:33
pabelangercloudnull: there does seem to be a pattern16:34
jheskethjeblair: hmm okay. Yeah I was about to say having the watcher launch another thread would be easy enough. Im happy to do it but won't be able to until tomorrow  (or possibly my Sunday). You're welcome to though if you'd like16:34
*** nwkarsten has quit IRC16:36
openstackgerritMerged openstack-dev/hacking: Updated from global requirements
jeblairjhesketh: yeah, i'll give it a shot16:37
*** deadnull_ has joined #openstack-infra16:37
*** nadya has quit IRC16:37
*** xarses has joined #openstack-infra16:37
*** nadya has joined #openstack-infra16:37
*** PollyZ has quit IRC16:38
*** nadya_ has joined #openstack-infra16:38
*** piet has quit IRC16:39
*** piet has joined #openstack-infra16:39
*** r-daneel has joined #openstack-infra16:42
*** nadya has quit IRC16:42
*** dtantsur is now known as dtantsur|afk16:42
*** lucasagomes is now known as lucas-afk16:42
*** nadya_ has quit IRC16:42
anteayajeblair: I've commented on the comments should not paginate story:!/story/200062016:43
*** derekh has quit IRC16:44
anteayaI just learned about the red gear myself allowing a user to set event timeline preferences16:44
jeblairanteaya: yes, i set those settings a while ago, but i do not think it is reasonable to expect every user to do that.16:45
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Add python logger configuration
*** shashank_hegde has quit IRC16:46
*** javeriak has joined #openstack-infra16:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Simplify translation sync
*** johnny has joined #openstack-infra16:49
*** johnny is now known as Guest1118216:49
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Remove redundant job suffix for rally jobs
*** javeriak_ has quit IRC16:50
*** nwkarsten has joined #openstack-infra16:50
anteayajeblair: understood, have you time to say so in the story or shall I copy paste?16:51
anteayaah you commented, thank yoi16:52
*** yamahata has joined #openstack-infra16:53
*** tonytan4ever has quit IRC16:53
*** bnemec is now known as beekneemech16:54
*** harlowja has joined #openstack-infra16:55
openstackgerritTim Hinrichs proposed openstack-infra/irc-meetings: Add meeting chair to Congress meetings
*** ociuhandu has quit IRC16:58
*** Guest77820 has quit IRC16:59
*** yaume has quit IRC17:00
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Deprecated snaphot image building
jeblairpabelanger: i think i need your zuul patch in final form to do the watchdog thing; do you have time to update it real quick?17:01
*** PollyZ has joined #openstack-infra17:03
*** kencjohnston has joined #openstack-infra17:05
*** deadnull_ has quit IRC17:06
*** amitgandhinz has quit IRC17:06
*** amitgandhinz has joined #openstack-infra17:07
*** _vsaienko has joined #openstack-infra17:09
openstackgerritPaul Belanger proposed openstack-infra/zuul: Stream subprocess output for ansible-playbook
pabelangerjeblair: sorry, did the work. Forgot to git-review17:09
jeblairpabelanger: woot, thx!17:09
pabelangercloudnull: that is the list of IPs for all our failure since switching to public IPs17:09
*** sarob has joined #openstack-infra17:10
*** amitgandhinz has quit IRC17:11
*** csomerville has joined #openstack-infra17:11
*** sree has quit IRC17:11
*** cody-somerville has quit IRC17:11
*** salv-orlando has quit IRC17:11
*** amitgandhinz has joined #openstack-infra17:11
*** sree has joined #openstack-infra17:11
*** salv-orlando has joined #openstack-infra17:12
*** shashank_hegde has joined #openstack-infra17:12
pabelangerjeblair: Feel free to ping me when the next round of timeout failures once we are running watchdog and streaming ansible17:12
pabelangerjeblair: I'm going to search more in ansible about timeout not being respected17:13
*** timello has quit IRC17:13
*** sree has quit IRC17:16
*** flepied has quit IRC17:18
KaiyanHello infra team, is ready for review again (adding tempest tests on each monasca-repo as non-voting check jobs). It would be great if i can get some reviews on that :-D Thanks in advance!17:18
*** timello has joined #openstack-infra17:18
cloudnullpabelanger:  thanks !17:19
*** ccamacho has quit IRC17:21
*** _vsaienko has quit IRC17:21
*** rvasilets__ has joined #openstack-infra17:22
*** rbrndt has quit IRC17:23
*** kzaitsev_mb has quit IRC17:25
notmorgandevananda: storyville is good coffee, you are right.17:29
*** shashank_hegde has quit IRC17:29
devanandanotmorgan: isn't that the best?!17:29
devanandanotmorgan: also - good morning17:29
notmorgandevananda: yah, and happy friday!17:29
devanandanotmorgan: mind hopping into -ironic for a minute? we were just talking about keystone stuff17:30
notmorganoh sure sec.17:30
*** Na3iL has quit IRC17:30
*** jerryz has joined #openstack-infra17:30
*** deadnull_ has joined #openstack-infra17:31
*** notmorgan is now known as morgan17:32
* morgan switches to "casual nick friday" nick.17:32
*** JayF is now known as notJayF17:32
*** devananda is now known as deva17:32
*** notJayF is now known as JasonF17:32
*** deva is now known as devananda17:33
*** devananda is now known as deva17:34
dmsimardIs there a spec for other-requirements.txt format ? I'd like to perhaps put comments in there but afraid it'd break something.17:34
*** nwkarsten has quit IRC17:34
*** deva is now known as devananda17:34
*** mriedem_away has quit IRC17:36
dmsimardpabelanger: cool, ty17:37
timothyb89pabelanger: by any chance do you know the status of stackviz building in nodepool?17:39
*** mriedem has joined #openstack-infra17:40
pabelangertimothyb89: I thought we merge it17:41
*** esikachev has quit IRC17:42
timothyb89pabelanger: it seems to be, but I'm not seeing the dib element in the nodepool logs?
*** csomerville has quit IRC17:42
*** tonytan4ever has joined #openstack-infra17:42
*** piet has quit IRC17:43
pabelangertimothyb89: I think we need to restart nodepool-builder17:44
pabelangertimothyb89: I can do that today17:44
*** daemontool has quit IRC17:44
fungijeblair: excellent news on the original rsync patch then. i'm going to consider my follow-on change as mostly paranoid overengineering and put it on the back burner for now in case we discover a need for it later17:44
timothyb89pabelanger: oh okay, cool, thank you!17:44
pabelangertimothyb89: Once our images finish uploading, I can restart nodepool-builder and fire off a build, confirming it works17:45
pabelangertimothyb89: should be in about 4 hours17:45
anteayapersia: ah we did capture the task id in gerrit commit message story after all:!/story/200058417:45
*** cody-somerville has joined #openstack-infra17:46
timothyb89pabelanger: sounds good, I'll keep an eye out for it!17:47
*** sputnik13_ has joined #openstack-infra17:48
fungimorgan: for me, every day is casual nick friday (and hawaiian shirt day i guess)17:48
anteayamorgan: oh you are you today17:48
anteayamorgan: I wanted to include you in the story I highlighted persia on, but I couldn't find you17:49
morgananteaya: hehe17:49
anteayafungi: are you wearing a hawaiian shirt?17:49
morgananteaya: i'm hiding17:49
fungimorgan is sneakily hiding in plain sight17:49
*** shashank_hegde has joined #openstack-infra17:49
anteayamorgan: well it makes it harder to include on things I thought you were interested in17:49
fungianteaya: i was when i went to lunch, but i've changed back into my work uniform now (pajamas)17:49
AJaegeranteaya: could you abandon , please? It's not needed anymore17:49
anteayafungi: that's my work uniform!17:50
anteayafungi: either that or bathrobe17:50
morgananteaya: hehe i was just a joke for the -ironic folks. i'll switch back :(17:50
*** morgan is now known as notmorgan17:50
fungiit's trendy, for sure17:50
*** flepied has joined #openstack-infra17:50
anteayaAJaeger: done17:50
*** EricGonczer_ has quit IRC17:50
pleia2my nephew is pretty jealous of my star wars angry bird pants17:50
anteayanotmorgan: I don't care what you pick as long as it is somewhat consistent so I can find you17:51
fungipleia2: he's forgiven you for using his band-aids then?17:51
anteayapleia2: photo!17:51
pleia2fungi: yeah, short memory :)17:51
anteayaI would like star trek pajamas17:51
anteayathen I really could be wearing a uniform17:51
anteayaplaid doesn't make for a convincing uniform17:52
*** EricGonczer_ has joined #openstack-infra17:52
AJaegerthanks, anteaya17:52
fungii bet someone did market pajamas looking like star trek uniforms in the 60s/70s17:52
anteayaAJaeger: welcome, thanks for cleaning up'17:52
beisnerhi all.  we init'd a new project and seem to have forgotten the upstream: bit in the project definition.  we have a nice, shiny empty repo as one would expect.   how much of a pita is it for us to re-init that and pull in its code at this point?17:52
anteayawhere are they now?17:52
*** vdrok has quit IRC17:53
beisnerie. should we revert the infra change and start a-new, or can we have someone manually poke something for us?17:53
AJaegerbeisner: which change was that?17:53
beisnerAJaeger ^17:53
notmorgananteaya: i try not to change the nick, but it's a 50/50 chance which one i land on after netsplits.17:54
beisnerAJaeger, the missing piece was:     upstream:
notmorgananteaya: it will always be either morgan or notmorgan17:54
*** piet has joined #openstack-infra17:54
anteayanotmorgan: that is fair17:54
anteayanotmorgan: okay I will try to tab complete both in future before I give up17:54
anteayafigured you were around17:54
*** pblaho has quit IRC17:55
AJaegerbeisner: Ah - didn't review that one. I normally double check that an empty repo is fine (which is fine often).17:55
AJaegerbeisner: you need to persuade an infra-root to do it manually for you17:55
beisnerAJaeger, and I even cross reviewed this /o\  -   yah, i suspect empty repos are not entirely uncommon for folks to start with.17:56
notmorgananteaya: i try and keep it notmorgan because i overlap with notmyname in fewer channels than mordred17:58
*** nwkarsten has joined #openstack-infra17:58
anteayapleia2: nice17:58
*** sputnik13_ has quit IRC17:58
*** permalac has quit IRC17:58
anteayapleia2: exactly17:58
anteayabeisner: an empty repo is a choice some folks choose to make17:59
*** yolanda has quit IRC17:59
anteayabeisner: the commit message should inform if the choice is a conscious choice17:59
openstackgerritMerged openstack-infra/tripleo-ci: Stop rebuilding locales on centos-7
beisneranteaya, yep i imagine so.  ack re: indicating that in the commit msg.18:00
beisneranteaya, we'd prefer not to dump a huge commit in the repo to preserve commit history.   can you help us manually>18:01
anteayabeisner: so it isn't an option to revert creating a repo18:01
anteayapersonally no I cannot18:01
beisneranteaya, ps. apologies for the extra work on this one.  we just missed it.   any idea who i would need to ask?18:01
anteayaanyone who can is listening and if they want to will speak up18:01
anteayaI'm not speaking for anyone else18:01
anteayathe option at present is to create an initial commit to the repo to import the contents18:02
tdasilvaAJaeger: re your comment here: how can I make sure that tarball will be uploaded to
anteayathe import of the current repo is a one time step that is available during repo creation, it can't be repeated after the repo is created18:02
beisnerright, we'd prefer to preserve the commit history naturally18:02
*** PollyZ has quit IRC18:03
openstackgerritMerged openstack-infra/system-config: Have zuul use the copied jenkins jobs directory
anteayahow many commits do you have to the project?18:03
anteayawe had someone do this with a 177 chained commits that took out our testing system18:03
beisneranteaya, 22718:03
anteayaokay so our system is pretty good with about 20 commit at a time18:04
anteayaif you can offer them as patches in gerrit in finite packages of 20 or less chained commits that preserves your history and will get your code into the repo18:04
*** andymaier has joined #openstack-infra18:05
mtreinishtimothyb89, krotscheck: that patch will work assuming you store the subunit file in the root of the log dir18:05
*** sarob has quit IRC18:05
anteayapabelanger yolanda ^^ in future could you make a point of ensuring starting wtih an empty repo is the conscious choice of the new project patch owner?18:05
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add watchdog for ansible
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: A newer version of virtualenv is already installed
*** sarob has joined #openstack-infra18:05
*** sarob has quit IRC18:05
*** sarob has joined #openstack-infra18:06
*** psilvad has joined #openstack-infra18:06
AJaegertdasilva: tarballs are only uploaded if you have a job for that - and you did not specify one...18:06
AJaegertdasilva: since you have a C project, our usual tarball jobs will not work.18:06
*** tqtran has joined #openstack-infra18:06
AJaegertdasilva: so, you need to either create a custom job or update jenkins/scripts/
jeblairpabelanger, jhesketh: 325061 builds on pabelanger's change and does timeout using a watchdog thread18:07
AJaegertdasilva: Your change does not add a tarball job at all. Let's do this in two steps ;)  Get yours in with my suggestions and then work on a tarball job...18:08
tdasilvaAJaeger: sounds good! thank you18:08
* fungi assumes a tarball job for a c/autotools based project would call `make dist`18:08
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Ironic: always install tempest plugin from master
openstackgerritThiago da Silva proposed openstack-infra/project-config: add liberasurecode gate jobs
pabelangeranteaya: beisner: Sure, but I think we do a good job documenting it already in:, which we do note if you don't configure upstream source it will create an empty repo18:10
*** sarob has quit IRC18:10
pabelangerjeblair: jhesketh: great18:10
AJaegertdasilva: Looks fine now.18:10
anteayapabelanger: I do think it is worth asking the question during the review to ensure intent if it is not explicitly stated in the commit message18:11
tdasilvaAJaeger: thx18:11
krotscheckmtreinish: Awesome. Like this?
* AJaeger agrees with anteaya18:11
*** trown|lunch is now known as trown18:12
* AJaeger sometimes forgets asking as well...18:12
*** nwkarsten has quit IRC18:12
pabelangerAJaeger: anteaya: ack18:12
beisnerpabelanger, indeed you have doc'd it quite well.  we got ~27 of them right :)18:12
anteayapabelanger: great, thank you18:12
*** PollyZ has joined #openstack-infra18:12
beisnerpabelanger, sorry for the extra work.  we'll definitely double/triple check that on any future projects.18:12
anteayawell I am a big fan of very clear commit messages, in my opinion they prevent a host of fix up work after the fact18:13
anteayabeisner: at this point the extra work is all yours18:14
*** mixos has joined #openstack-infra18:14
*** andymaier has quit IRC18:14
*** sdague has quit IRC18:15
pabelangerI think there is a way for an infra-root to force push up the contents of the repo, but I've not done it18:15
pabelangermaybe not18:15
*** nwkarsten has joined #openstack-infra18:16
fungipabelanger: yes, clone the intended original repo state, add a gerrit remote, put your account in the "Project Bootstrappers" group in gerrit, and then git push --force gerrit18:17
*** yamada-h has joined #openstack-infra18:18
pabelangerbeisner: where is the upstream repo?18:18
fungithat will replace the boilerplate initial repo with the other desired commits18:18
pabelangerfungi: sure, let me try that18:18
jeblairbeisner, pabelanger: just don't tell anyone you're doing it -- we have a reputation to maintain!  ;)18:20
beisnerjeblair, pabelanger  /o\ so sorry.  you won't see us do that again. ;)18:21
*** kzaitsev_mb has joined #openstack-infra18:21
*** rbrndt has joined #openstack-infra18:22
pabelangerbeisner: pushed18:22
pabelangerfungi: great, thanks18:22
*** cody-somerville has quit IRC18:23
*** sarob has joined #openstack-infra18:23
beisnerpabelanger, i owe ya a beer, cookie, or whatever, whenever some time ;-)18:23
*** akostrikov has quit IRC18:23
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Create a StoryBoard gui manual
*** thiagop-afk has quit IRC18:24
anteayafunny openstackgerrit missed one of my patches, in #storyboard it reported both, the original and the commit message fix18:24
beisnertyvm, all18:25
anteayajroll: my first patch on the storyboard gui guide:
AJaegerbeisner: now add a proper .gitreview to your repo!18:26
beisnerAJaeger, on that now.18:26
AJaegerbeisner: if it's not there yet...18:26
jrollanteaya: thanks, I'll add myself as a reviewer but can't promise I can take the time to make sure it's right (totally unfamiliar with storyboard right now)18:26
*** deadnull_ has quit IRC18:26
anteayajroll: yup, I get it18:26
anteayajroll: actually you are the audience for the guide18:26
anteayatotally unfamiliar is what I am looking for here18:27
anteayajroll: and thank you18:27
jrollanteaya: indeed, just unsure when I'll have the time to go through it :)18:27
jrollI will try!18:27
*** cody-somerville has joined #openstack-infra18:27
anteayaoh I understand, prioritize accordly18:27
*** yamada-h has quit IRC18:28
*** javeriak has quit IRC18:30
*** javeriak has joined #openstack-infra18:31
jeblairclarkb, fungi, jhesketh, pabelanger, mordred, notmorgan, Shrews: i think all of the blockers we identified are either merged or in review; can you take a look at 324889 325356 325061?18:31
jeblairclarkb, fungi, jhesketh, pabelanger, mordred, notmorgan, Shrews: the ansible async timeout failing to function is still a mystery -- 325356 is intended to get us a bit more information next time it happens and 325061 is designed to avoid disruption when it fails18:32
* mordred waves at the channel of lovely people18:32
*** Na3iL has joined #openstack-infra18:33
pabelangerjeblair: looking18:33
notmorganjeblair: i shall look.18:33
notmorganmordred: oh look it's a mordred!18:33
Shrewsjeblair: ack18:33
* mordred hands notmorgan some mud from the dead sea18:33
*** kzaitsev_mb has quit IRC18:34
jeblairthat mud is dead.18:34
* bkero waves back, happily mud-free.18:34
*** jed56 has quit IRC18:35
mordredpabelanger: I see something in scrollback about switching to public ips with cloudnull ... what did I miss?18:35
fungimordred: osic has levelled up18:35
*** Swami has quit IRC18:36
*** nwkarsten has quit IRC18:36
mordredfungi: zomg. we have non-floaters there?18:36
pabelangerWe've reached the final boss on OSIC18:37
*** nwkarsten has joined #openstack-infra18:37
* mordred hands cloudnull a bag of gourmet salt from the dead sea18:37
pabelangermordred: No FIPs, but we are timing out SSHing to a few public IPs.18:37
sridhar_raminfra-team - i'm looking for the procedure / process to add "openstackstatus" to the #tacker channel. Just so we don't miss any important annoucements - like the one this morning on CI stability.18:38
mordredpabelanger: cool. that would still be a thing we're handling right, right?18:38
mordredpabelanger: like, we time out and don't leak resources18:38
pabelangermordred: Right, we look good on that front18:38
* mordred does a dance18:38
notmorganjeblair: the first of those were pretty easy to review, looking at the watchdog one now.18:41
openstackgerritMerged openstack-infra/project-config: Add DefCore guideline gate checks for json lint and schema
anteayamordred: I'll take salt from the dead sea18:41
anteayaI have a salt collection18:41
anteayaso far none from the dead sea18:42
*** sjmc7 has joined #openstack-infra18:43
*** Swami has joined #openstack-infra18:43
krotscheckThere's some intermittent gate failures on NPM-based things due to a low file-descriptor limit in xenial. Any cores around to tell me this isnt' crazy?
*** dimtruck is now known as zz_dimtruck18:44
openstackgerritMerged openstack-infra/project-config: Add pushMerge for oslo.messaging core for feature branches
sridhar_ramAJaeger: thanks ,will look it up!18:44
anteayaAJaeger: if you feel so inclined I would be grateful if you would be willing to review this storyboard documentation patch of mine:
anteayaAJaeger: things like when to use full stops, when to use hyphens things of that nature for consistentcy18:45
*** dprince has quit IRC18:45
*** zul has quit IRC18:45
anteayaAJaeger: as well as content of course18:45
*** chem`` has joined #openstack-infra18:46
*** dprince has joined #openstack-infra18:46
*** asettle has joined #openstack-infra18:47
*** chem` has quit IRC18:47
*** PollyZ_ has joined #openstack-infra18:48
*** sdague has joined #openstack-infra18:49
mgagneis there a way to tell nodepool to not spawn more of an image? I mean max-servers is shared across all images. Can it be per image?18:50
*** PollyZ has quit IRC18:50
*** zul has joined #openstack-infra18:51
Shrewsjeblair: the big pink warning above this ( concerns me a bit with what you are doing in 32535618:52
*** dizquierdo has quit IRC18:52
*** asettle has quit IRC18:52
pabelangerkrotscheck: left comment18:52
rvasilets__AJaeger: Hi, could you reply Thx18:53
bkeromordred: are you doing DevOpsDays Israel?18:53
jeblairShrews: i believe that's saying "don't call when you also may have data waiting on stderr".  in our case, we only have one outbound pipe, and we're reading from it.  we have no inbound pipe.  so there should be nothing to deadlock.18:54
jeblairpabelanger: ^ (your change :)18:54
mordredbkero: OpenStack Israel18:55
bkeroah right on, didn't realize that was coinciding18:55
openstackgerritSridhar Ramaswamy proposed openstack-infra/system-config: Add #tacker to statusbot
*** Na3iL has quit IRC18:56
*** zul has quit IRC18:56
*** zul has joined #openstack-infra18:56
AJaegeranteaya: thansk for the writeup, that's an important document! I left some comments, some apply for the whole text18:57
AJaegerrvasilets__: this is best done here ;)18:57
anteayaAJaeger: wonderful thank you18:57
pabelangerjeblair: sorry, not following18:58
Shrewsjeblair: hopefully, but, yeah, i think your thinking may be valid there18:58
AJaegerrvasilets__: If you update zuul/layout.yaml in project-config, it's very convenient to know whether a job is voting or not. That's why we use the -nv, it's a handy notation for us as reviewers as well...18:58
jeblairpabelanger: Shrews was asking about your change; wanted to make sure you saw it :)18:59
sridhar_raminfra-team, AJaeger - I pushed a patchset to enable #tacker for statusbot. BTW, a quick follow on IRC question - how can I change the title of the channel, similar to the one infra (& other projects has)  - "Discussion of ... " | docs .. | logged ..18:59
pabelangerjeblair: reading now18:59
notmorganShrews: i *think* jeblair is correct on that18:59
notmorganit sounds right?18:59
AJaegersridhar_ram: #tacker should have an operator that has those permissions...18:59
jeblairnotmorgan, Shrews, pabelanger: amusingly, communicate has a complimentary warning that's basically "don't use communicate if there's lots of data".  it doesn't suggest what to do instead.  but i'd probably do this.  :)19:00
sridhar_ramAJaeger: hmm.. i'm the PTL but i don't remember getting it from the last person who ran this19:00
notmorganjeblair: lol oi19:00
jeblair(or if you need to have separate stdout/stderr streams, select i guess)19:00
pabelangerwarnings each way you go :)19:00
notmorganjeblair: that is probably the right answer19:01
notmorganjeblair: but.. yeesh19:01
sridhar_ramAJaeger: is there way to check who holds the operator permission for #tacker channel ?19:01
pabelangerinfra-root: I have to step away for 90mins to run an errand. Feel free to take over any of my zuulv25 patches in flight19:01
AJaegersridhar_ram:  run "/msg chanserv access #tacker list" and ask one of those folks for help...19:01
pabelangerwill be back online after19:01
jeblairyeah.  i think we're good here.  at least, i'm willing to take it for a spin :)19:01
sridhar_ramAJaeger: will do.. thanks19:01
notmorganjeblair: it wont be worse than what we have. and we can pull it together into something else if we need [select loops]19:02
notmorganjeblair: or whatever.19:02
fungipleia2: i'm going to go ahead and stop puppet on the puppetmaster in preparation for our maintenance19:03
fungialso checking in with the release team on their situation19:03
pleia2fungi: thanks19:03
*** markusry has quit IRC19:03
pleia2fungi: I figure in 30 minutes I'll send out the statusbot notice19:04
pleia2(once we confirm with release folks)19:04
fungisounds good19:04
*** EricGonc_ has joined #openstack-infra19:04
rvasilets__AJaeger: Ou. Now I get. Its always -nv...19:05
rvasilets__AJaeger: If its help to infra revivers then I will update our job names19:06
rvasilets__AJaeger: Is it needed?19:06
*** amoralej is now known as amoralej|off19:06
*** nwkarsten has quit IRC19:06
*** nwkarsten has joined #openstack-infra19:07
*** EricGonczer_ has quit IRC19:08
rvasilets__AJaeger: I just not understand why we can't write it explisitly)19:08
rvasilets__by the link19:08
AJaegerrvasilets__: it's a nice convention. It's not required but appreciated19:09
mgagneMy understanding is that there is a maintenance in progress with Gerrit? Is it normal I get an error with the contact store at this time?19:09
fungimgagne: maintenance isn't starting for another ~hour19:10
anteayamgagne: what error?19:10
mgagnethe usual "Error contacting contact store"19:10
jrollooo gerrit maintenance in an hour? yay early friday19:11
notmorganfungi: was lunchies good?19:11
*** javeriak has quit IRC19:11
funginotmorgan: fish burrito and some local(ish) cerveza, so yes19:11
notmorganperfect fiday lunch imnsho.19:12
fungisridhar_ram: AJaeger: it looks like permissions on #tacker are already set correctly to me19:12
* AJaeger signs off for the day and hopes for a smoothless gerrit maintenance...19:12
pleia2have a nice weekend, AJaeger19:12
AJaegerfungi: what needs to be done to change the topic there?19:12
fungithanks AJaeger!19:12
AJaegerthanks, pleia2 and fungi!19:13
pleia2AJaeger: as you said, they just need to ask an existing operator to do it19:13
sridhar_ramfungi: i was poking around few chanserv cmds as well to understand this...19:13
pleia2oh, looks like we're the only ones with access?19:13
mtreinishkrotscheck: yep, that should do it19:14
pleia2we can add one of the tacker folks to the list19:14
pleia2sridhar_ram: I'll see about adding you, as the ptl and all19:14
fungiAJaeger: sridhar_ram: accessbot is set correctly at but statusbot configuration still needs adding
sridhar_rampleia2: thanks!19:15
pleia2sridhar_ram: you can now /msg chanserv topic #tacker your topic here19:15
fungiif this was about the original question of how to get statusbot updating topics in #tacker, anyway19:15
pleia2fungi: they want to update their channel topic19:15
sridhar_ramfungi: i just pushed a patchset to add tacker to statusbot ..19:15
*** salv-orl_ has joined #openstack-infra19:15
*** penguinolog has quit IRC19:16
fungiah! i misread the original question. reread it just now19:16
fungisorry for the confusion19:16
*** piet has quit IRC19:16
*** kzaitsev_ws has quit IRC19:16
sridhar_rampleia2: trying the chanserv cmd now...19:17
mgagnefound the problem, coworker didn't register the right email at the foundation =)19:17
pleia2mgagne: story of our life :)19:17
*** pramchan has joined #openstack-infra19:17
pramchanhow do I create a an openstack repo?19:18
*** salv-orlando has quit IRC19:18
pramchanI would like to create the openstack/meghdwar repo for a project19:18
anteayamgagne: yeah, I wondered if you were getting that error, that wasn't making sense to me19:18
sridhar_rampleia2: woot, it worked ! thanks everyone..19:18
pleia2pramchan: we have instructions here:
pleia2sridhar_ram: great!19:18
openstackgerritBoris Deschenes proposed openstack-infra/glean: changed the name of the interface on which to add the default gw in gentoo
sridhar_rampleia2: so, just so i understand.. you've added my nick to #tacker channel operators ?19:18
pleia2sridhar_ram: yes19:19
pramchanI attempted to do it through the official openstack process, but was told by the governance code review that I should create the repo first and have some development on it first before making it an official openstack project19:19
sridhar_rampleia2: cool, thx!19:19
mgagnesmallest change ever, 1 character :D19:20
pleia2pramchan: link to the governance change? some projects do start with no repository to import19:21
anteayamgagne: wooo19:21
pleia2but I'd need to see context, some projects may have different rules19:21
pramchanhere's the link to the comment19:22
sridhar_rampleia2: infra-team: along the same lines, our github repo message is awfully wrong at   How can  I have that changed ?19:22
pleia2pramchan: he's not saying you can't create an empty repo, he's saying you can't submit it to governance until it's an active project19:23
*** PollyZ_ has quit IRC19:23
pramchanand there's the governance repo gerrit code review19:23
*** linuxmaniac has joined #openstack-infra19:23
*** linuxmaniac_ has quit IRC19:23
pramchanah, so how do I create an empty repo? I couldn't find that piece of information in the project creation guide19:23
pramchanI would like to create the openstack/meghdwar repo19:23
anteayapramchan: I'll suggest as I did last night, perhaps contributing to some current projects and understanding our workflow might be of use to you19:23
anteayapramchan: would save everyone conserned a lot of time if you learned our workflow first19:24
pleia2pramchan: your change 319466 adds the empty repo, you need to abandon the governance change and remove it from Depends-on: in the commit message19:24
pleia2pramchan: but anteaya makes a good point, it's very unusual for a first commit to be a project creation, being familiar with the tools in the OpenStack ecosystem by contributing to other projects is important19:26
pabelangerand back online, errand cut short19:27
*** yolanda has joined #openstack-infra19:27
*** pramchan has quit IRC19:28
*** csomerville has joined #openstack-infra19:28
*** piet has joined #openstack-infra19:30
Shrewsjeblair: qq... where do you actually set the 'timeout' module variable used by 'async'?19:30
Shrewsjeblair: b/c i'm not seeing that  :(19:30
*** Jeffrey4l_ has quit IRC19:30
*** pramchan has joined #openstack-infra19:31
pramchananteaya: pleaia2: point taken, I will find a bug in Neutron or Nova to fix to get familiar with the current workflow. If you have a specific bug you need help with, please let me know as well19:31
*** cody-somerville has quit IRC19:32
Shrewsjeblair: oh, command line level. nm19:32
anteayapramchan: well we aren't in those projects and we don't know what you know, but yes that is the best direction for learning our workflow19:32
anteayapramchan: thank you for listening19:32
pleia2fungi: any word from release folks?19:33
fungipleia2: dhellmann is still working on flushing some final releases through. we may need to push it out a bit depending on if he thinks they'19:33
fungill be wrapped up in the next hour19:33
pleia2ok, I'll hold off on the notice then19:33
pleia2I can't stay too late tonight (having dinner here, but would like to do the family thing a bit before I fly home tomorrow :))19:34
dhellmannI'm waiting for zuul to merge the patch I approved 15 minutes ago -- and it just did :-)19:34
*** Jeffrey4l_ has joined #openstack-infra19:35
fungipleia2: no worries, i can babysit the offline reindex and bring everything back online once finished, so duck out whenever you need to19:35
pleia2also thinking I should run the reindex as root under screen, so anyone else can attach to it19:35
*** slaweq_ has joined #openstack-infra19:35
pleia2(I'm on a Verizon hotspot out here, it's been good, but who knows!)19:35
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Create a StoryBoard gui manual
fungipleia2: well, not as root (as gerrit2) but otherwise yes19:35
anteayadhellmann: yay!19:36
pleia2ah, we do run it as gerrit2 already19:36
openstackgerritPrakash Ramchandran proposed openstack-infra/project-config: add new project meghdwar
pleia2so, I do this hacky thing of opening up a second screen to prevent the first screen from closing when the job is done, how am I supposed to do this? :)19:37
pleia2must be some "don't close" screen option...19:37
* pleia2 man page19:37
*** pramchan has quit IRC19:37
anteayathere is a man page for pleia2?19:38
anteayaI wonder if it says anything about star wars uniforms/pajamas19:38
pleia2haha, if only19:38
bkeroI made one of those for myself once. The man syntax is archaic19:39
*** sarob has quit IRC19:39
*** sarob has joined #openstack-infra19:39
*** roxanaghe has quit IRC19:39
pleia2oh, I should just screen, then run the command19:40
anteayawould be interesting practice for writing man pages19:40
pleia2etherpad updated19:40
pleia2bkero: you never fail to make me feel old :)19:40
*** nwkarste_ has joined #openstack-infra19:41
*** nwkarsten has quit IRC19:41
anteayathey exist!
anteayared gold or blue?19:41
*** banix has joined #openstack-infra19:41
anteayaso many decisions19:41
pleia2definitely not red19:41
bkeropleia2: my bad ._.19:42
anteayaI could give red out though19:42
*** Kaiyan has quit IRC19:43
*** slaweq_ has quit IRC19:43
* bkero checks to see that it is reasonably worksafe:
pleia2haha, cute19:44
pleia2good at parties: -d19:45
fungipleia2: yeah, just start screen by itself and you'll have a shell in it. then whatever command you run in that shell will return you to the shell in screen rather than causing screen to terminate19:45
pleia2fungi: nods19:45
*** Apoorva has quit IRC19:45
*** sarob has quit IRC19:45
*** timello has quit IRC19:46
*** sputnik13_ has joined #openstack-infra19:47
*** zz_dimtruck is now known as dimtruck19:49
*** amrith is now known as _amrith_19:49
fungianteaya: blue of course! science is always a good career choice19:50
anteayablue it is19:50
bkeroanteaya: redshirts are a whole thing19:51
fungiyeah, that too. you want to make sure you come back from your away missions19:51
fungired shirts were like phaser magnets19:52
anteayathe black market yellow dye19:52
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Create a StoryBoard gui manual
*** sputnik13_ has quit IRC19:56
*** aimeeu has quit IRC19:57
pleia2sounds like we're on track to being "soon" so I'll send out the notice, we'll do the alert when we actually begin19:57
anteayapleia2: sounds good to me19:58
pleia2#status notice The infrastructure team is taking Gerrit offline for maintenance this afternoon, beginning shortly after 20:00 UTC. We aim to have it back online around 00:00 UTC.19:58
openstackstatuspleia2: sending notice19:58
*** asettle has joined #openstack-infra19:58
*** asettle has quit IRC19:58
*** pvaneck has joined #openstack-infra19:58
-openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance this afternoon, beginning shortly after 20:00 UTC. We aim to have it back online around 00:00 UTC.19:59
*** aimeeu has joined #openstack-infra19:59
*** e0ne has joined #openstack-infra20:00
openstackstatuspleia2: finished sending notice20:01
pleia2"Also add changes to update projects.txt in all branches of the requirements repo" <-- I don't remember what this means20:01
*** hashar has quit IRC20:01
anteayain the requirements repo if any of the projects are listed in projects.txt we have to offer patches to the requirements repo20:01
anteayaalso we have to check all branches in the requirements repo20:02
* timrc does his routine Friday check-in and see Star Trek references and notes all is good.20:02
anteayaI did not do that step20:02
*** kushal has quit IRC20:02
* pleia2 does not see either in the projects.txt20:02
anteayapleia2: yay, thank you for completing that step20:02
anteayapleia2: did you want to use strikethrough on that step?20:03
pleia2the first bit of the step did need to be done, and it is20:03
*** cbader has quit IRC20:03
*** banix has quit IRC20:04
*** _amrith_ is now known as amrith20:04
*** timello has joined #openstack-infra20:04
anteayapleia2: announce it with statusbot is our version of 'say it with flowers'20:05
pleia2"since none of you read pleia2's emails"20:06
*** baoli_ has quit IRC20:06
anteayaI do20:06
anteayaI read some20:06
pleia2thank you :)20:06
fungiwhat was the etherpad url again?20:06
pleia2fungi: want to do zuul and github?20:06
pleia2I can do gerrit and gitXX20:07
fungican do20:07
pleia2#status alert Gerrit is offline for maintenance until 00:00 UTC20:07
openstackstatuspleia2: sending alert20:07
fungii guess as soon as the alert is done updating topics everywhere i'll dump zuul check/gate and stop it20:08
*** nwkarste_ has quit IRC20:09
fungijeblair: any special handling needed on zuul.o.o with your launcher testing?20:09
-openstackstatus- NOTICE: Gerrit is offline for maintenance until 00:00 UTC20:09
*** ChanServ changes topic to "Gerrit is offline for maintenance until 00:00 UTC"20:09
*** nwkarsten has joined #openstack-infra20:09
*** sputnik13_ has joined #openstack-infra20:10
openstackstatuspleia2: finished sending alert20:12
fungitook it a while longer than i'm used to expecting20:12
fungiokay, dumping zuul check/gate and stopping now20:13
* pleia2 pats openstackstatus 20:13
anteayamaybe it has more channels to alert these days20:13
fungiconfirmed zuul daemon is no longer running20:13
pleia2ok, gerrit time20:14
fungianteaya: that's my assumption too20:14
fungianteaya: and possibly also chanserv being slow to act20:14
anteayaI don't have any sense of chanserv's pace20:14
*** Swami has quit IRC20:15
fungii interact with it only infrequently as well20:15
pleia2ok, gerrit is off20:15
*** rcernin has quit IRC20:15
pleia2database now20:15
*** Apoorva has joined #openstack-infra20:15
*** gyee has joined #openstack-infra20:16
pleia2hmm, not matching anything20:17
fungithat's not unusual (at least the account project watches)20:17
fungiit's changes which hopefulyl matches something20:18
fungibut even that _might_ not if there were never any changes for a project20:18
pleia2Query OK, 0 rows affected (0.36 sec)20:18
pleia2Rows matched: 0  Changed: 0  Warnings: 020:18
fungiif neither table is matching for one of those projects, might be a good idea to check on the filesystem and make sure that's _really_ it20:18
anteayais account project watches the people watching the project?20:18
fungianteaya: yes20:18
fungiit's a table of which accounts have set to watch that project in their preferences20:19
pleia2fungi: mind jumping on to see if I'm missing something in this database?20:20
pleia2mysql> select * from changes where dest_project_name="ansible-puppet";20:20
pleia2Empty set (0.38 sec)20:20
fungipleia2: missing namespaces on those queries20:21
fungipleia2: and on your filesystem commands too20:21
fungiet cetera20:22
mordredpleia2, fungi: btw - I am here. although it's 23:22 and I have spotty internets20:22
*** sjmc7 has left #openstack-infra20:22
*** Swami has joined #openstack-infra20:22
anteayamordred: glad you are here20:22
*** bauwser is now known as bauzas20:22
anteayamordred: you can hand out a round of cats if you would like20:22
pleia2Rows matched: 3  Changed: 3  Warnings: 020:22
anteayawe haven't had a round lately20:22
pleia2much better :)20:22
*** kgiusti has quit IRC20:22
anteayapleia2: yay!20:23
pleia2ok, database is done20:23
* mordred hands pleia2 a few of the cats he found on the street in Tel Aviv20:23
pleia2moving git repositories20:23
pleia2mordred: \o/20:23
anteayamy cat is standing guard to my left20:24
anteayashe just came in to stand beside me for a bit20:24
pleia2ok, #8 is done, time to reindex20:24
fungipleia2: don't forget to use a screen session for the reindex20:24
anteayawooo, the fun bit20:24
* pleia2 nods20:24
*** eumel8 has joined #openstack-infra20:25
pleia2Cannot open your terminal '/dev/pts/1' - please check.20:25
pleia2yay sudo20:25
fungiscript /dev/null20:25
fungiit's a hacky workaround to clear the parent pty20:26
fungilooks like you're in. i'm attached to it as well now20:26
* dasm is reading it as a good thriller book20:26
pleia2ok, it's running20:27
*** PollyZ has joined #openstack-infra20:27
fungiyep, i see it too. good deal20:27
anteayapleia2: yay20:27
* pleia2 does favor and reduces terminal size20:27
fungii'll get started on the github bits now while you work on the git farm i guess20:27
pleia2sounds good20:27
anteayanow we watch grass grow and paint dry for four hours or so, yeah20:27
*** _ari_ has quit IRC20:28
fungianteaya: well, it's _possible_ that our upgrade to a faster server will also make the reindex go more quickly20:28
*** banix has joined #openstack-infra20:28
anteayafungi: ohhhh, wouldn't _that_ be lovely20:28
fungii'm not banking on it, but here's hoping20:29
anteayaalways room for hope20:29
*** abregman has quit IRC20:29
*** jerryz has quit IRC20:29
jeblairi guess we're not doing the groovy online reindex thing?20:29
pleia2git farm is done20:30
*** amotoki has quit IRC20:30
*** nadya has joined #openstack-infra20:30
pleia2jeblair: I guess not :\ I didn't know we were planning to20:30
anteayaI thought we had concluded during the Feb rename that the online reindex wouldn't work20:30
jeblairi don't think we necessarily were planning to20:30
jeblairzaro said he had a solution that involved using groovy to force it to do a full online reindex20:31
fungigithub renames are done now20:31
jeblairmy question would have been perhaps better phrased as "i guess zaro's work on groovy-based online reindex has not progressed to the point where we are ready to try it for this rename?"20:31
fungijeblair: yeah, but we'd need to add the groovy script plugin to be able to do that20:31
anteayaetherpad from Feb:
fungijeblair: after some back and forth, zaro concluded that getting it added as an option in the admin plugin might make more sense and he proposed (or was going to propose, can't remember if he did yet) a patch upstream for that20:32
anteayamy read of zaro's update of his priorities he gave me today in channel was that is on his list of things to address20:32
*** roxanaghe has joined #openstack-infra20:33
fungiapparently online reindexing is completely implemented, but it only triggers on a gerrit upgrade and so an additional little bit was required to be able to get it to trigger arbitrarily even when there were no changes to the gerrit version/db schema20:33
*** persia has quit IRC20:33
*** nwkarsten has quit IRC20:34
*** dimtruck is now known as zz_dimtruck20:34
pleia2my favorite part of renames is when I decide to go back to reviewing to pass the ...oh20:34
mordredpleia2: ++20:35
fungithe missing blobs are, i guess, still not completely cleaned up20:35
jeblairpleia2: yeah, it's nice to be able to use gertty to review when gerrit's offline, right?20:35
*** nwkarsten has joined #openstack-infra20:35
*** persia has joined #openstack-infra20:35
fungigertty is working great for me, other than indicating it's got a sync error20:36
anteayafungi: the missing blobs that were supposed to be cleaned up in Feb?
*** baoli has joined #openstack-infra20:37
jeblairfungi: oh, huh, i think there's a bug...20:37
jeblairwell, sort of20:37
jeblairi don't think we handle 503's as indicating it's offline20:37
fungianteaya: we cleaned up some ond half-deleted/half-renamed projects from ancient times, if that's what you're recalling20:38
pleia2hah, yeah, haven't kept my gertty synced this trip20:38
*** rhallisey has quit IRC20:38
*** flwang1 has quit IRC20:38
pleia2(on pay-by-the-gig internet, doing everything I can to control what I use)20:38
fungithe missing blobs are actual git objects gerrit can't find which it thinks it has references to20:38
fungijeblair: that 'splains why it's not transitioned to offline mode for me20:38
anteayafungi: yes I think that is what I am recalling20:38
*** amotoki has joined #openstack-infra20:39
jeblairfungi: yeah, i'll whip up a patch.  i've got 4 hours to test it.20:39
anteayafungi: ah sorry20:39
*** eharney has quit IRC20:39
fungijeblair: if we're lucky, maybe only three!20:39
fungiReindexing changes: projects: 45% (474/1044), 33% (106889/322853)20:39
*** amotoki has quit IRC20:39
fungiit's making okay time so far, but as we know this is nonlinear20:40
*** roxanaghe has quit IRC20:40
*** roxanaghe has joined #openstack-infra20:40
*** kushal has joined #openstack-infra20:41
*** PollyZ has quit IRC20:41
*** tonytan4ever has quit IRC20:41
pleia2in the proud tradition of status bars20:42
fungimicrosoft would be proud20:42
*** dprince has quit IRC20:43
*** julim has quit IRC20:44
*** gouthamr has quit IRC20:44
*** ayoung has quit IRC20:44
jeblairfungi:  if you want to manually apply20:45
jeblairfungi: or, better yet, i'll mail that in to dr. dobbs and you can type it in from the next issue.20:45
*** flwang has joined #openstack-infra20:46
*** e0ne has quit IRC20:46
*** flwang1 has joined #openstack-infra20:47
fungijeblair: worked!20:48
fungiOffline Sync: 920:48
jeblairnow i just need gertty to remember to push that up for me in 4 (3?) hours20:49
fungiin gerrit watched pot news... Reindexing changes: projects: 54% (566/1044), 47% (151776/322853)20:49
jeblairi approved all the zuul ansible things20:51
jeblairhopefully we can switch that back on later today or over weekend or something20:51
fungisounds good20:51
*** nwkarsten has quit IRC20:53
notmorganjeblair: at <4hrs from now> <run gertty again> ? ;)20:53
*** nwkarsten has joined #openstack-infra20:53
notmorganjeblair: aslo next week can i bug you for a gear release?20:53
notmorgani'll remind you next week (of course)20:54
*** eumel8 has quit IRC20:54
jeblairnotmorgan: oh, gertty will push up reviews automatically, i was joking i wanted to extend it to push up changes too20:54
notmorganjeblair: ahhhh20:54
jeblairnotmorgan: absolutely -- i looked yesterday, and there was a change i wanted to go ahead and get in; i approved it, and then the zuul launcher stuff...20:54
jeblairlemme see if it merged20:54
notmorganjeblair: its so i can close out all but the socket issues with getting nodepool to py3 and voting (i'll play rebase chasing after that)20:55
jeblairnotmorgan: what are the socket issues?20:55
notmorganjeblair: gear is not reliably starting/stopping in the fixture under py320:55
jeblairoh :(20:56
jeblairnotmorgan: i want to land
notmorganjeblair: the server. its... a bit baffling, and i'm about to need to open python in gdb20:56
jeblairnotmorgan: i just approved it20:56
*** rhochmuth has joined #openstack-infra20:56
notmorganbecause pdb can't break on socket things very well.20:56
notmorganalso threading.20:56
jeblairnotmorgan: does the server reliably stop/start on its own in py3?20:57
notmorganjeblair: ah cool will look when gerrit is back up.20:57
notmorganjeblair: yep. as far as i can tell it works fine under py2720:57
notmorganand i can't replicate the hang outside of the fixture and test cases20:57
notmorganso, something weird is happening with threading, fixture, gear, test case.20:57
notmorganbut it worries be because it's consistently unreliable enough i dont want to vote on py3 things before it can be resolved.20:58
*** nwkarsten has quit IRC20:58
*** nadya has quit IRC20:58
notmorganjeblair: or at least this happens way less under py27, it might aslo account for some spurrious errors i've seen in nodepool on py27.20:58
*** piet has quit IRC20:59
* notmorgan is trying to avoid a fixture that spins up a full separate process gearman and then kills it -- it shou;dn't be an issue.20:59
*** gouthamr has joined #openstack-infra20:59
jeblairthat definitely should not be necessary21:02
*** Goneri has quit IRC21:03
jeblairnotmorgan: this works for me and exits cleanly with only the main thread running at the end:
anteayawhat fiction is our progress bar reporting?21:03
notmorganjeblair: exactly21:04
*** Swanson has joined #openstack-infra21:04
notmorganjeblair: works fine with something like that. but consistently fails in the nodepool tests (the worst offender is the disconnect handling test)21:04
notmorganjeblair: and the other tests will occasionally just not start with the fixture.21:04
notmorganjeblair: the disconnect test never closes the socket/listener on linux (using netstat to track it, and with artificial sleeps after the shutdown/join is calleD)21:05
jeblairnotmorgan: and gear has py3 tests which have a server fixture too21:05
notmorganjeblair: yup. and it's always worked outside of the nodepool tests. so something magic extra is happening in nodepool that is hanging up the socket.21:06
*** flwang1 has quit IRC21:06
*** xyang1 has quit IRC21:06
*** salv-orl_ has quit IRC21:06
notmorganjeblair: i'll chase it down next week for sure - i'm sure it's going to be something silly.21:06
notmorganand really trivial once the fix is "known"21:06
* notmorgan is looking at the final failing py27 test for zuul getting a bunch of py3 fixes in too21:07
*** salv-orlando has joined #openstack-infra21:07
notmorganneed to see what is happening with some urllib.urlopen oddity21:07
anteayafungi: pleia2 any indexing update?21:08
fungiReindexing changes: projects: 73% (772/1044), 65% (212392/322853)21:10
*** csomerville has quit IRC21:10
*** jianghuaw has quit IRC21:10
*** psilvad has quit IRC21:10
anteayafungi: thank you21:11
*** flwang1 has joined #openstack-infra21:13
*** pradk has quit IRC21:14
*** pt_15 has joined #openstack-infra21:14
*** amrith is now known as _amrith_21:15
*** flwang1 has quit IRC21:17
*** flwang1 has joined #openstack-infra21:17
*** harlowja has quit IRC21:20
*** ldnunes has quit IRC21:21
*** baoli has quit IRC21:22
jeblairand i wrote a change to gracefully stop the zuul launcher21:23
*** baoli has joined #openstack-infra21:24
*** thorst has quit IRC21:24
jeblairfungi, clarkb: you'll like this -- the graceful stop includes a step where we release all idle nodes back to nodepool.  that's also something we can run on its own.21:24
*** thorst has joined #openstack-infra21:24
*** matt-borland has quit IRC21:24
jeblair(graceful is basically: pause + release + wait + stop)21:25
fungioh, speaking of, i guess i should stop nodepool and start it again once zuul21:26
fungi's back online?21:26
fungii don't think our rename process has been updated to account for nodepool's use of gearman21:26
jeblairfungi: well, you shouldn't need to stop it21:27
*** thorst_ has joined #openstack-infra21:27
jeblairfungi: but if you do, i think it has a bug that will mean you have to wait until it's back online21:27
fungijust restart i guess21:27
jeblairfungi: shouldn't even need to do that21:27
*** lascii is now known as alaski21:27
*** _amrith_ is now known as amrith21:27
jeblairfungi: yeah, it's happily idle.21:28
jeblairfungi: what we *should* do however is set all nodes to delete state.  i can do that if you want21:28
fungioh, i thought we'd had issues with nodepool not reestablishing its connection to gearman or something21:28
jeblairfungi: yes, that's only on startup21:28
*** thorst has quit IRC21:29
jeblairfungi: if it also has problems when running, that's a new bug21:29
fungiaha, got it21:29
fungithen yeah, i can set all nodes to delete state. which integer value is that?21:29
jeblairfungi: i just did it.  4.21:29
notmorganmordred, jeblair: woo found my urllib issue, whee will have that posted once gerrit comes back :)21:30
fungiupdate node set state=4;21:30
fungigot it21:30
jeblairnotmorgan: neat!21:30
notmorganjeblair: turns out it was in the FakeURLOpener and i missed a req=urllib.request.Request conversion21:30
*** trown is now known as trown|outtypewww21:30
*** daemontool has joined #openstack-infra21:31
*** thorst_ has quit IRC21:31
fungiReindexing changes: projects: 99% (1036/1044), 88% (285553/322853)21:35
fungigetting close (maybe?)21:35
pleia2or 2 hours at 88% :)21:35
* pleia2 fingers crossed21:35
*** baoli has quit IRC21:37
*** banix has quit IRC21:38
*** PollyZ has joined #openstack-infra21:39
*** bryan_att has quit IRC21:43
*** thorst has joined #openstack-infra21:46
*** erlon has quit IRC21:46
*** rvasilets__ has quit IRC21:46
*** ffrank has joined #openstack-infra21:48
*** esikachev has joined #openstack-infra21:50
*** thorst has quit IRC21:50
pleia2Reindexing changes: projects: 99% (1040/1044), 92% (300021/322853) (\)21:55
pleia2dinner is ready, so I'm going to attend to that21:55
pleia2I'll check in after21:56
anteayathanks pleia221:56
anteayaenjoy dinner and family21:56
*** _vsaienko has joined #openstack-infra21:57
*** sbelous has left #openstack-infra21:57
*** yolanda has quit IRC21:57
*** shashank_hegde has quit IRC21:58
fungii'm still around and keeping tabs on it anyway21:59
fungiso we're covered21:59
fungitake your time21:59
*** ffrank has quit IRC22:00
*** timello has quit IRC22:04
*** mriedem has quit IRC22:04
*** JasonF is now known as JayF22:05
*** timello has joined #openstack-infra22:09
*** EricGonc_ has quit IRC22:10
fungiReindexing changes: projects: 99% (1042/1044), 95% (306856/322853)22:10
*** shashank_hegde has joined #openstack-infra22:11
*** gordc has quit IRC22:15
*** gouthamr has quit IRC22:16
notmorganso close!22:16
anteayathe last 1% takes the longest22:17
timrcY'all know how to do Friday, 'round here.22:17
anteayaHappy Friday!22:18
timrcWhen Gerrit says Service Unavailable what it's really trying to tell me is, "Go get a beer Tim.  You deserve it!"22:18
anteayanow you speak gerrit22:19
notmorgananteaya: so.. nova?22:19
anteayanotmorgan: yeah22:19
notmorganthat makes me a little sad22:19
anteayanotmorgan: it has made us sad for some time22:19
anteayait is a known issue in upstream22:19
notmorganalmost time to call it a day here and head up to visit w/ friends :)22:19
notmorgani was hoping to rebase a change but... alas22:20
notmorgani think i wont get to22:20
anteayanotmorgan: sounds reasonable for me22:20
anteayayeah have some Friday22:20
anteayaI don't expect reindexing to finish in the next hour22:20
notmorganeah and i am leaving in ~10min or so22:21
notmorganoh blah, my weather widget stopped working :(22:23
*** mdrabe has quit IRC22:23
*** earlephilhower has quit IRC22:23
anteayawould you consider looking out the window?22:23
anteayaradical idea, I know22:24
notmorgananteaya: i use it for things not-where i am22:24
notmorganaka what is the weather at home, where i am headed to (travel wise) etc.22:24
notmorganso i know what to pack for.22:24
* dtroyer checks in to the weather bureau: 'sun shining here'22:24
*** _vsaienko has quit IRC22:24
notmorgandtroyer: lol22:24
anteayadtroyer: ha ha ha22:24
anteayanotmorgan: where are you going? maybe someone there can look out the window?22:25
*** yamahata has quit IRC22:25
anteayasun shining here too, btw22:25
dtroyerand thanks to Gerrit's indexing, I get to enjoy it sooner than planned!22:25
notmorgananteaya: i am in seattle now was looking for home weather22:25
notmorgananteaya: Portland, OR 24 hr temps: 82.3°F |▇████▇▇▅▅▃▃▃▁▁_▁▃▅▅▇▇██| 90.9°F  Range: 64.1°F - 90.9°F22:25
anteayadtroyer: wooo, spread the happiness22:25
anteayanotmorgan: nice22:25
anteayayes warm it is22:25
anteayalow 20's here22:26
*** daemontool has quit IRC22:26
sputnik13_what's the dealio with gerrit?22:26
sputnik13_oh, should look at the topic maybe22:26
anteayasputnik13_: it is down22:27
timrcnotmorgan: Sounds like you're having a hotter day than we are in Austin... which is sort of crazy to consider.22:27
fungiReindexing changes: projects: 99% (1042/1044), 96% (312543/322853)22:27
*** xarses has quit IRC22:27
* SpamapS has visions of Office Space22:28
anteayafungi: yay another 1%22:28
notmorgantimrc: record temp in PDX for today is 91, 198722:28
*** xarses has joined #openstack-infra22:28
SpamapSLumberg is coming22:28
notmorganSpamapS: ... PC Load Letter22:28
*** markvoelker has quit IRC22:31
*** PollyZ has quit IRC22:34
*** kdas__ has joined #openstack-infra22:35
*** esikachev has quit IRC22:36
*** kushal has quit IRC22:37
*** esikachev has joined #openstack-infra22:39
*** EricGonczer_ has joined #openstack-infra22:39
*** edmondsw has quit IRC22:41
timrcnotmorgan: Feel marginally bad for anyone that doesn't have A/C there!22:42
timrcI have to keep my house at a mild 72-ish at all times, otherwise I get sad.22:43
*** nwkarsten has joined #openstack-infra22:44
*** rlandy has quit IRC22:44
*** rbrndt has quit IRC22:44
*** xarses has quit IRC22:47
*** xarses has joined #openstack-infra22:47
*** nwkarsten has quit IRC22:49
*** ffrank has joined #openstack-infra22:51
*** harlowja has joined #openstack-infra22:51
*** mriedem has joined #openstack-infra22:52
*** aysyd has quit IRC22:52
*** nwkarsten has joined #openstack-infra22:56
*** sdague has quit IRC22:57
fungiReindexing changes: projects: 99% (1043/1044), 97% (314387/322853)22:57
anteayaburning through that last few percent23:00
*** nwkarsten has quit IRC23:01
*** hongbin has quit IRC23:03
*** r-daneel has quit IRC23:04
*** kdas__ has quit IRC23:05
*** harlowja has quit IRC23:10
*** harlowja has joined #openstack-infra23:10
*** shashank_hegde has quit IRC23:11
jlvillalIs there a known issue with Zuul status?
anteayajlvillal: well zuul is stopped23:12
jlvillalI get a "Loading..." message and it doesn't bring up the running jobs23:12
jlvillalAh, that would be the issue then :)23:13
* jlvillal would vote for the Zuul status page to say so23:13
jlvillalanteaya, Thanks!23:13
anteayalikely yes, watch the channel topic change and if the issue persists once the maintenance is over, do let us know23:13
*** esikachev has quit IRC23:14
fungiugh, at the current pace of advancement indexing changes (~1/sec), we're looking at almost 2 hours to go23:17
fungimaybe it will speed back up again23:19
anteayathat would be over 5 hours this time23:19
anteayafungi: anything besides you optimisim in support of that possibility?23:19
SpamapSfungi: is it CPU locked?23:20
SpamapSMight be good to do these on a baremetal box if so23:20
*** vhosakot has quit IRC23:21
*** tphummel has quit IRC23:24
*** kdas__ has joined #openstack-infra23:25
pleia2anteaya: for the storyboard one #15 we say update the db, but it seems I can rename it through the UI23:27
*** kdas__ has quit IRC23:27
anteayapleia2: oh can you?23:27
pleia2not sure if we couldn't edit it that way before and noted the db, or if there's something else we need to do23:27
fungianteaya: pure optimism i'm afraid23:28
anteayapleia2: perhaps use the gui first then check the db and see what it says?23:28
anteayafungi: go go pure optimism23:28
anteayafungi: I love your optimism23:28
pleia2I wouldn't even know where to begin with the db :)23:28
pleia2could poke around, but if it's something specific I'd likely miss it23:28
anteayawhen I need some I picture myself standing beside you and then I feel better23:28
anteayapleia2: hmmm, I don't know then23:29
anteayapleia2: I wonder if fungi has any thoughts on the storyboard rename bit23:29
fungiSpamapS: it seems to be mostly single-cpu bound (whatever its parallelism scheme is, it's not great... maybe split by project?)23:29
pleia2anteaya: ok, I'll rename through the UI first then, we'll figure it out if something else needs to be edited23:29
anteayapleia2: sound good, thank you23:30
fungiSpamapS: zaro is working on a solution to allow us to use an online background reindexing feature in gerrit for this in the future, at least23:30
fungipleia2: go for the gui--i think that feature didn't exist when we added that item in the instructions23:31
fungipleia2: though the schema is simple--there's just one table with a name to integer index mapping for projects anyway23:32
pleia2fungi: ah, that's good23:32
*** tphummel has joined #openstack-infra23:33
anteayaI think I'm going to grab some dinner since we are here for another few hours anyway23:33
fungiSpamapS: yeah, it's sad... top shows 16 cpus on this server, 15 of which are sitting idle and 1 which is slammed23:33
zaroSpamapS: yeah, wasn't able to get that feature onto our gerrit this time around but probably next time23:34
zarothis is why reindex is slow
*** EricGonczer_ has quit IRC23:34
fungiahh, yep. that was my guess23:35
*** EricGonczer_ has joined #openstack-infra23:35
*** tphummel has quit IRC23:35
*** EricGonczer_ has quit IRC23:35
fungiexplains why it goes more quickly at first but then when we get down to the last dozen or so projects it starts to slow considerably23:35
fungiif it prioritized the project order by decreasing number of changes then it would do a much better job of maximizing the scheduling23:36
*** mtanino has quit IRC23:37
*** EricGonczer_ has joined #openstack-infra23:37
*** EricGonczer_ has quit IRC23:38
*** xarses has quit IRC23:39
*** EricGonczer_ has joined #openstack-infra23:41
zarofungi: i don't see how that would make it any faster?23:41
*** weshay has quit IRC23:41
*** Swami has quit IRC23:41
fungizaro: at least with a large number of projects of varying change count like we have now, it would be utilizing all available processors all the way up to the end23:43
*** gyee has quit IRC23:43
*** flaviof has joined #openstack-infra23:43
*** rhochmuth has quit IRC23:44
funginova gets scheduled to cpu 0, neutron on cpu 1, cinder on cpu 2, glance on cpu 3... glance will finish before the others and cpu 3 will move on to horizon, then cinder will finish and cpu 2 will move on to ironic...23:44
*** EricGonczer_ has quit IRC23:45
funginova's still churning on 0 and neutron on 1 possibly for most of the duration, but the other cpus are getting scheduled faster and faster projects to complete as the situation progresses23:45
fungiso if you still end up with only cpu 0 in use at the end it's because you eventually ran out of other projects to index, and the max time is however long it takes to process nova23:47
fungibut if instead nova gets saved until near the end of the project list... then it's however long it took to index the other projects _plus_ how long it took to index nova23:48
*** ffrank has quit IRC23:48
*** roxanaghe has quit IRC23:49
fungiReindexing changes: projects: 99% (1043/1044), 98% (317401/322853)23:50
zaroi see what mean, that may help.  i think it's alpha order now.23:50
zarohere's the disussion on possible ways to improve, maybe post your idea?
zarofungi: while we are on the topic.23:52
zarofungi: i'm wondering how we should test the forcing of online reindex?23:52
*** rguillebert has quit IRC23:53
*** johnthetubaguy has quit IRC23:53
*** johnthetubaguy has joined #openstack-infra23:53
*** evgenyl has quit IRC23:53
zaroi don't think anybody else has done it maybe beside the guys at Gerritforge23:53
*** rguillebert has joined #openstack-infra23:54
zaroi'm slightly concerned that it might do something bad.23:54
*** banix has joined #openstack-infra23:55
*** evgenyl has joined #openstack-infra23:55
fungizaro: just replied to that thread (it's awaiting moderation now). thanks for the suggestion23:59

Generated by 2.14.0 by Marius Gedminas - find it at!