Thursday, 2016-02-11

openstackgerritTony Breeds proposed openstack-infra/project-config: Force the gate-irc-meetings-tox-ical postbuildscript to always run
anteayamnaser: enjoy00:00
mnaserthanks for the help mordred clarkb jeblair anteaya ! :D00:00
mordredmnaser: any time!00:00
mgagneclarkb: you win all the internets sir thanks!00:00
zaroclarkb: what is xvdc and xvde in these graphs?
tonybanteaya: commit message updated, let me knwo what you think about the other comment.00:00
zaroclarkb: also xvda and xvdb?00:01
clarkbzaro: they are block devices, xvde is the ephemeral device and xvdc is the ssd volume we put our repos on00:01
clarkbzaro: xvda is / and I don't know what b is maybe b is ssd and c is something else00:01
clarkbmordred: their ssl cert is not yet valid, is there a way with oscc/shade to specify untrusted ssl is ok?00:01
anteayatonyb: will do00:02
clarkbmordred: not sure we have much alternative until they fix that (I guess we could hold off merging the osic change until they have proper cert and domain)00:02
anteayafirefox 44.0 is just not gerrit compatible00:02
Shrewsclarkb: verify=False in clouds.yaml, iirc00:02
zaroclarkb: if we are using those for disks then what's sda1 and sda2?00:02
clarkbShrews: danke00:02
clarkbI grepped for ssl and insecure but not verify00:03
*** nwkarsten has joined #openstack-infra00:03
mordredclarkb: yes00:03
Shrewsclarkb: irc grep usually works :)00:03
*** crinkle has quit IRC00:03
mordredoh - Shrews got you00:03
*** crinkle_ has joined #openstack-infra00:03
clarkbzaro: the sda things may be historical? from when gerrit was on the legacy cloud00:04
clarkbzaro: not sure how cacti cleans that up00:04
*** FallenPegasus has quit IRC00:04
*** crinkle_ is now known as crinkle00:04
clarkbmordred: Shrews that would not go in the profile right? thats a per clouds.yaml setting? makes sense considering individual users will wnat to set what they are comfortable with for security00:04
zaroclarkb: and which one is the review_site on?  it looks like it might be either xvdb or xvde but can't tell from the graph.00:04
*** hdd has joined #openstack-infra00:05
JayFI ^c'd git review in the middle (it had been hanging for a long time), and now my patch appears to be in a weird limbo. doesn't have my change, but git-review fails now with ! [remote rejected] HEAD -> refs/publish/master/bug/1526918 (no new changes) (full logs:
mordredclarkb: no- it should be a per-cloud setting00:05
mordredclarkb: in fact, it certainly is, because I know unitedstack had broken certs for a while00:06
jeblairJayF: yep, you'll want to amend it and push a new patchset00:06
mordredclarkb:   ustack:00:06
mordred      profile: unitedstack00:06
mordred      #verify: False00:06
clarkbzaro: looks like xvda is /, xvde is /opt, and /home/gerrit2 is on xvdb and c via lvm00:06
mordredclarkb: that's from my clouds.yaml00:06
JayFjeblair: so just amend my change with something enough to change the sha1 and repush? WFM00:06
jeblairJayF: yep.  the issue was gerrit database backup at 0000utc00:06
mordredclarkb: I think we should wait a bit before merging a verify: false in oscc itself00:06
JayFjeblair: aha, thanks :)00:07
mordredclarkb: so for now maybe yo just put it in your clouds.yaml00:07
clarkbmordred: agreed or never at all and just do it in clouds.yaml00:07
mordredI'm not sure why they don't have real certs00:07
clarkbmordred: ok I think I have what I need, time to instll my change and openstack client and see if it works00:07
mordredbut I won't ask that questoin just right now00:07
clarkbmordred: because they don't have a domain00:07
clarkbI am told it is in progress00:07
jeblairmordred: i think we may have outgrown a simple mysqldump and maybe it's time for replication or some other backup strategy.  :)00:07
*** nelsnels_ has joined #openstack-infra00:07
mordredjeblair: agree00:07
*** nelsnelson has quit IRC00:08
mordredjeblair: maybe it's time to make a plan for that00:08
jeblairmordred: well, at this time of day, i can only make a plan to make a plan for that.00:09
anteayatonyb: good enough, thank you00:09
mordredjeblair: can we make a plan for the plan making plan?00:10
tonybanteaya: Thanks.00:10
*** armax has joined #openstack-infra00:10
anteayamordred: you'll be on a plane00:10
*** denisra has joined #openstack-infra00:12
tristanCclarkb: what if I need multiple jobs to complete before doing a publish job for example, is this possible ?00:13
*** Daisy has joined #openstack-infra00:14
*** sabeen3 has quit IRC00:14
tristanCthat is, in the same pipeline00:14
clarkbtristanC: it is not00:15
clarkbI believe that is one of the features that zuulv3 will add00:15
*** rocky_g has quit IRC00:15
tristanCso it's only one to many with zuulv200:15
*** annegentle_ has joined #openstack-infra00:15
clarkbmordred: woot I have openstackclient working with a clouds.yaml and my vendor profile so will push this up momentarily00:15
clarkbtristanC: yes00:15
tristanCclarkb: thank!00:15
*** nwkarsten has quit IRC00:15
clarkbtristanC: you can serialize them all if it is really important00:15
tristanCclarkb: well the use-case would be to do publish job in the gate, so that if the publish failed, the change isn't merged00:16
*** Daisy has quit IRC00:18
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services
*** annegentle_ has quit IRC00:19
tristanC(though this also mean the publish would have to wait to be on top of the gate to actually happen)00:20
*** annegentle_ has joined #openstack-infra00:20
clarkbok have my first devstack tempest run going on osic on a base ubuntu image00:23
clarkbwill sort out getting one of our qcow2's up there next00:23
*** dchen has joined #openstack-infra00:25
*** hashar_ has quit IRC00:25
*** rbrndt_ has quit IRC00:26
*** rkukura has quit IRC00:27
clarkbjeblair: this script is like the best thing ever00:28
*** jsavak has quit IRC00:28
*** xyang1_ has joined #openstack-infra00:28
*** xyang1 has quit IRC00:28
*** xyang1_ is now known as xyang100:28
clarkbE: Unable to locate package liberasurecode-dev thats interesting00:29
*** ganso has quit IRC00:29
armaxhi folks, we have a couple of graphite links bookmarked here: but they look a bit broken right now..doesn’t anyone have pointers?00:29
*** ybathia has quit IRC00:30
clarkbis that in backports?00:30
*** claudiub has joined #openstack-infra00:31
armaxum, now I get a notfound right of the bat00:31
*** vmbrasseur is now known as vmb_off_record00:32
*** vmb_off_record is now known as vmbrasseur00:32
clarkbarmax: first thing I would check is if a local build of the docs produce those two files00:32
armaxclarkb: ok, it did in the past…that was working a minute ago, but the graphite graphs looked wrong00:33
clarkbarmax: but I am guessing that the docs site has some htpassword magic preventing those two file "extensions" ?00:33
armaxclarkb: let me double check00:34
armaxclarkb: I guess this has worked by accident so far?00:34
clarkbarmax: it would not surprise me if the docs team updated their htpassword rules00:34
jeblairclarkb: yay!00:35
clarkbbut first thing is to confirm the builds actually make those files00:35
armaxclarkb: checking00:35
armaxI get a build error locally now00:35
clarkbI uncommented all the repos in sources.list fwiw and liberasurecode-dev showed up00:35
armaxsomething merged recently and broke the docs job00:35
clarkbaptitude says it is in universe which I thought was enabled beofre so maybe universe backports?00:36
jeblairclarkb: yeah, trusty-backports universe00:36
armaxclarkb: I am still waiting on
clarkb"Running bdist_wheel for numpy: still running..." how nice of it to tell me it is still spinning the cpu :)00:39
clarkbjeblair: but this should give us an upper bound on how bad a run is without wheels and cached info and so on00:39
clarkbI guess I could've added the wheel mirror first, oh well00:40
mordredclarkb: woot!00:40
jeblairclarkb: i think comparing the tempest runtimes as output by devstack is the best way to compare00:40
jeblairclarkb: eliminates all of that setup concern00:41
clarkbjeblair: good point, the tempest runtime should be ya that00:41
jeblairor rather, the tempest runtime as output by tempest00:41
jeblair(ran x tests in ...s)00:41
openstackgerritEric Brown proposed openstack/requirements: Revert "Only install mock on Python < 3.3"
armaxclarkb: the page came back now:
armaxbut the graph looks funny00:41
armaxI am puzzled00:42
clarkbarmax: that has the .html, the old links didnt00:42
*** rkukura has joined #openstack-infra00:42
clarkbwhich lends some weight to the htaccess theory00:42
armaxclarkb: that’s bizzarre indeed00:42
clarkbfwiw, you might find using our grafana instance is a better place for that stuff?00:42
armaxbecause I am simply browsing from my browser00:42
jeblairarmax: you can add dashboards to grafana if you want00:42
jeblairclarkb: jinx00:43
*** arthus has quit IRC00:43
armaxjeblair: tell me more :)00:43
jeblairarmax: http://grafana.openstack.org00:43
jeblairarmax: i made a graph similar to what's on that page here:
jeblairarmax: it's mostly a stub -- but it's an example00:44
armaxjeblair: thanks for the pointers00:44
jeblairarmax: look at some of the other dashboards there for other examples00:44
* armax looks00:44
jeblairarmax: and each of those has a link to the file in project-config that configured the dashboard (all yaml)00:44
jeblairarmax: eg:
*** Swami has quit IRC00:44
armaxjeblair: that’s excellent00:45
jeblairarmax: main thing it gets you over raw graphite, aside from pretty, is zooming and correlation00:45
armaxjeblair: plus you get the nice URLs00:46
jeblairoh look, if you zoom out far enough on the nodepool graphs, you can actually see the devstack runtime drop from the wheel mirrors :)00:46
jeblair(it takes a lot of data to get that to average out into something smooth)00:47
armaxjeblair: so I can just post a patch adding a neutron.yaml file?00:47
jeblairarmax: yep!00:47
armaxright on00:48
*** annegentle_ has quit IRC00:48
armaxthe CI jobs will also show how it looks? or does it need to be deployed to become live?00:48
*** thorst_ has joined #openstack-infra00:49
jeblairarmax: we haven't quite worked that out yet; though i think pabelanger had some ideas.  it will validate syntax.00:49
armaxjeblair: ok00:49
armaxjeblair: mine was just idle curiousity00:49
jeblairarmax: also, you can use grafyaml to make the json and actually paste it into grafana (or edit json that's already there)00:49
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance
armaxjeblair: noted00:50
*** sabari has joined #openstack-infra00:50
*** browne has joined #openstack-infra00:50
armaxdougwig: ^^ this is useful info00:50
mtreinishmordred: ^^^00:50
mtreinishit's not really ready yet though, it needs tests (and I need to test it)00:51
brownehi all.  i did a revert of the g-r mock change.  looks like its breaking py34 jobs for some projects00:51
*** Sukhdev has quit IRC00:51
mordredmtreinish: neat!00:51
mordredmtreinish: I hear that's a thing people do00:51
*** jlk has quit IRC00:54
mtreinishI know that I've done it before :)00:54
*** jlk has joined #openstack-infra00:54
*** jlk has quit IRC00:54
*** jlk has joined #openstack-infra00:54
*** [1]Thelo has joined #openstack-infra00:55
*** Jeffrey4l has joined #openstack-infra00:56
*** rpioso has quit IRC00:56
*** Thelo has quit IRC00:57
*** [1]Thelo is now known as Thelo00:57
mtreinishmordred: also: I'm guessing the wheel mirror went live on the 5th00:58
mordredmtreinish: nice!00:58
mordredjeblair: ^^00:58
jeblairmtreinish: yep00:59
*** ajmiller has quit IRC01:00
slogan621anteaya: is there anything I need to do to improve so it can get approval by you? I see you are real busy, and I'm trying to not bother you but feeling a little unsure if you are waiting for me on something01:02
tonyblifeless: again not trying to be annoying but can you look at:  I think that releaseing fixtures 1.2.1 is the right compromise position01:02
openstackLaunchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New]01:02
tonyblifeless: FWIW I'm drafting an email re install -U but that's a tangent to the bug above01:03
*** eil397 has quit IRC01:03
*** thorst_ has quit IRC01:05
*** thorst_ has joined #openstack-infra01:06
tonybbrowne: Can you include a bug ot logs for the py34 jobs affected by the mock change?01:07
*** Jeffrey4l has quit IRC01:07
*** esker has joined #openstack-infra01:08
*** annegentle_ has joined #openstack-infra01:08
*** Sukhdev has joined #openstack-infra01:08
brownetonyb: sure01:09
clarkbjeblair: is osic with 4 cpu01:11
anteayaslogan621: I'm not waiting for you on anything01:12
clarkbjeblair: is the change I got the script form01:12
clarkbjeblair: it seems to hold up extremely well01:12
anteayato be honest I'm not the best person to review new project patches01:12
anteayaas I feel new folks make far better use of their time if they spend time on current projects01:12
anteayahelping with bugs and testing and documentation01:12
anteayato learn our workflow01:13
anteayabut the fashion seems to just launch your own project01:13
anteayawhich I can't stop01:13
anteayabut I'm not a supporter01:13
anteayaand thanks for fixing the commit message01:13
anteayaanyway I'm at the end of my day01:13
anteayagood night01:14
*** thorst_ has quit IRC01:14
sabaribrowne: other projects like glance didn't break because they depend on oslotest that still depends on mock.01:14
sabaribrowne: hasn't merged btw01:14
brownesabari: ah!01:14
jeblairclarkb: not shabby01:15
jeblairclarkb: is that 8cpu or 4?01:15
clarkbjeblair: 4vpcu 8GB ram and 96GB /01:15
jeblairthen we'll just be ip limited :)01:16
clarkbjeblair: well they gave us the IPs so ya01:16
brownetonyb: where are bugs opened on g-r?01:16
clarkbneed that oscc change in then I can propose a chnage to our clouds.yaml01:16
clarkband I need to make a mirror01:16
jeblairclarkb: link to change?01:16
tonybbrowne: normally in the reporting project01:17
*** sridhar_ram has quit IRC01:17
tonybbrowne: it's just helpful for others to follow the bug -> resolution path as we tent not to document that in the commit message very well.01:17
jeblairclarkb: are non public (or semi public) clouds going into oscc?01:17
clarkbjeblair: mordred said I should push it, so I guess so?01:18
clarkbjeblair: there is a bluebox.json too which is only private clouds01:18
*** annegentle_ has quit IRC01:18
*** harlowja has quit IRC01:18
mordredwe have one semi-public and one vendor of private clouds01:18
brownetonyb: understood. normally i'd go to launchpad, but doesn't appear to be anything for openstack/requirements projects01:18
mordredcatalyst is semi-public01:18
*** chlong has joined #openstack-infra01:18
mordredthey are - but also they focus on government, so there's weird ip access restrictions01:19
jeblairmordred: who focuses on government?01:19
tonybbrowne: I'd use:
*** Jeffrey4l has joined #openstack-infra01:20
brownetonyb: ok01:20
*** angdraug has quit IRC01:20
mordredjeblair: catalyst01:20
*** achanda has quit IRC01:21
tonybbrowne: I know I butted in, and I'm not being super helpful :/01:21
jeblairanyway, i'm not sure it's the call i would have made (seems like a lot of churn for not much benefit), but i don't think there's much point arguing against it.  :)01:21
mordredclarkb: comment on the patch - it's missing docs entries01:21
*** annegentle_ has joined #openstack-infra01:22
clarkboh thats not all generated from the json?01:22
clarkbI can add01:22
jeblairclarkb: you can, of course, proceed without the osic oscc change and switch to it later.01:22
clarkbjeblair: that is true01:23
clarkband its not too bad to do that since it is very defaulty01:23
jeblairclarkb: when you spin up the mirror -- be aware that if their trusty image is built from the server image rather than the cloud image, it may have the lts enablement packages installed; our puppet will remove them but there may be some post-trusty kernel packages installed that need to be removed for openafs to compile01:24
clarkbjeblair: ok01:24
*** sflanigan has joined #openstack-infra01:24
*** sflanigan has joined #openstack-infra01:24
jeblair(this has only happened on internap so far)01:24
clarkbgoing off of what my test box is the kernel looks older so probably not a problem01:24
clarkbLinux clarkb-test 3.13.0-66-generic #108-Ubuntu01:25
jeblairoh you used one of their images rather than upload.  easy to check then :)01:25
jeblairversion looks right for trusty (though is 10 builds behind?)01:26
openstackgerritEric Brown proposed openstack/requirements: Revert "Only install mock on Python < 3.3"
tonybbrowne: thanks01:26
jeblairclarkb: this will be the first cloud where using it will not be a multi-day process...01:27
*** esker has quit IRC01:27
jeblairclarkb: all we need to do is upload existing images and create the mirror server -- we don't need to *update* the images (assuming they use one of the existing formats), and the mirror doesn't need to build anything01:28
jeblairyou could have it all done tomorrow :)01:28
*** annegentle_ has quit IRC01:28
clarkbmordred: change updated01:29
clarkbjeblair: yup uses qcow2 so images should be easy01:29
clarkbits just a matter of lining everything up so that nodepool can talk to it and start booting things01:30
clarkbI will get stuff in hiera tomorrow, propose clouds.yaml and nodepool.yaml changes01:30
clarkbstart with 1 VM and go from there01:30
nibalizerclarkb: you can either parent on the infracloud stuff or not01:31
nibalizerit hasn't landed yet but I think it can, its got positive votes at least01:32
clarkbin that cas emaybe I should start by reviewing infracloud stuff01:32
clarkbso that it can go in then do osic changes01:32
*** lane_kong has joined #openstack-infra01:32
*** achanda has joined #openstack-infra01:33
nibalizerI think is the important one, since once that is in I/we can build a mirror in that region01:33
nibalizerit might be cool to try to build mirrors together01:33
*** jasondotstar has quit IRC01:35
*** raginbajin has quit IRC01:35
*** eliqiao_ has quit IRC01:36
clarkbjeblair: re comment on it has been the case that we haven't been able to use the same name01:36
clarkbthat stopped being the case with hpcloud01:36
clarkbso we could do that for now01:36
*** dstufft has quit IRC01:36
clarkbuntil some random cloud has a conflict again01:36
clarkboh and for password I am not sure01:36
jeblairclarkb: i specifically meant for both omfra east and west01:36
*** eliqiao_ has joined #openstack-infra01:36
*** dims_ has joined #openstack-infra01:37
clarkboh gotcha01:37
*** jasondotstar has joined #openstack-infra01:37
jeblairmy argument is that we use the same creds for rax iad/dfw/ord01:37
clarkbooh looks like horizon has built in password changing now01:37
jeblairso we should use the same creds for omfra east/west01:37
clarkbjeblair: ++01:37
*** raginbajin has joined #openstack-infra01:37
clarkbI should maybe change the passwds before we properly use this thing01:37
jeblairi chose not to push the argument though and left a +201:37
nibalizerI'm open to flattening it, really01:37
*** annegentle_ has joined #openstack-infra01:38
*** dstufft has joined #openstack-infra01:38
nibalizerif I get time before Im done working today I can flatten it01:38
jeblairnibalizer and i are both being equal parts stubborn and gracious ;)01:38
clarkbbut now I need to go wrangle children01:38
*** dims has quit IRC01:38
*** dims_ has quit IRC01:41
*** gildub has quit IRC01:41
*** apoorvad has quit IRC01:41
*** zz_dimtruck is now known as dimtruck01:42
*** alex_xu has quit IRC01:43
*** alex_xu has joined #openstack-infra01:45
nibalizersome of the IC nodes do not apply puppet01:47
nibalizerthey do not apply puppet because they don'thave the secrets01:47
nibalizerbecause they don't have infracloud.yaml copied over01:47
nibalizerbecause puppet_get_hiera_file_list is not returning 'group/infracloud.yaml' in its array01:48
nibalizerwhich makes very little sense because some hosts are working just fine01:48
*** gyee has quit IRC01:48
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Add neutron tempest run using neutron-lib master to neutron exp
*** dims has joined #openstack-infra01:49
*** hdd has quit IRC01:49
*** pvaneck has quit IRC01:50
*** rlandy has quit IRC01:51
*** annegentle_ has quit IRC01:51
openstackgerritLingxian Kong proposed openstack-infra/project-config: Add distil jobs back
*** baoli has joined #openstack-infra01:53
lane_kongping AJaeger01:53
*** tiswanso has joined #openstack-infra01:53
slogan621anteaya: ok, that clears things up, thanks.01:55
slogan621can anyone suggest a core reviewer who can tackle the simple process of approving a new project commit? I've got a few thousand lines of code to commit to that project, once it exists, so I would appreciate some help. Not trying to be pushy, but I'm blocked until that project gets created, so appreciate some help.01:58
slogan621 is the project01:59
*** akshai has quit IRC01:59
*** tphummel has quit IRC01:59
*** eliqiao_ has quit IRC02:00
*** eliqiao_ has joined #openstack-infra02:00
*** baoli has quit IRC02:00
*** Daisy has joined #openstack-infra02:00
*** rbrndt has joined #openstack-infra02:01
*** rbrndt has quit IRC02:02
*** rossella_s has quit IRC02:03
*** Daisy has quit IRC02:03
*** edmondsw has quit IRC02:03
*** rossella_s has joined #openstack-infra02:03
*** asahlin has joined #openstack-infra02:04
*** mtanino has quit IRC02:04
*** Sukhdev_ has joined #openstack-infra02:06
*** browne has quit IRC02:07
*** dchen has quit IRC02:07
*** slogan621 has quit IRC02:07
*** alex_xu has quit IRC02:08
*** Jeffrey4l has quit IRC02:08
*** shashank_hegde has quit IRC02:08
*** ggillies has joined #openstack-infra02:09
*** megm_ has joined #openstack-infra02:10
*** alex_xu has joined #openstack-infra02:10
*** HeOS__ has joined #openstack-infra02:11
*** daemontool has quit IRC02:11
*** Jeffrey4l has joined #openstack-infra02:11
mesterymordred: Any idea why would be stuck and not goign into the merge queue? The dependent patch in neutron has merged.02:12
*** annegentle_ has joined #openstack-infra02:12
*** thorst_ has joined #openstack-infra02:12
*** dpaterson has quit IRC02:14
*** fbo_ has joined #openstack-infra02:14
*** terryw has joined #openstack-infra02:14
*** Sukhdev has quit IRC02:15
*** bpokorny has quit IRC02:15
*** ctlaugh has quit IRC02:15
*** fbo has quit IRC02:15
*** otherwiseguy has quit IRC02:15
*** hamzy has quit IRC02:15
*** amitgandhinz has quit IRC02:15
*** openstackgerrit has quit IRC02:15
*** HeOS_ has quit IRC02:15
*** vsaienko has quit IRC02:15
*** megm has quit IRC02:15
*** amitgandhinz has joined #openstack-infra02:15
*** chlong has quit IRC02:15
*** annegentle_ has quit IRC02:17
*** baoli has joined #openstack-infra02:17
*** sridhar_ram has joined #openstack-infra02:19
*** thorst_ has quit IRC02:19
jeblairmestery: it is likely because neutron and neutronclient do not share a change queue02:20
jeblairmestery: that means that there is no job that the two projects have in common02:20
jeblairmestery: that is... surprising to me.02:20
tonybmestery: It's in both check and gate now.02:21
jeblairmestery: (in other words, there is no job that tests both things together).02:21
*** hamzy has joined #openstack-infra02:21
jeblairmestery, tonyb: adding (or toggling) workflow causes it to be enqueued02:21
*** ctlaugh has joined #openstack-infra02:23
*** baoli has quit IRC02:23
*** vsaienko has joined #openstack-infra02:23
*** daemontool has joined #openstack-infra02:23
*** sergmelikyan has joined #openstack-infra02:23
*** openstackgerrit has joined #openstack-infra02:24
sergmelikyanfolks, any ideas why change-request may be stuck in zuul POST pipeline for 13 hours?02:24
*** baoli has joined #openstack-infra02:24
sergmelikyanI have several change-requests stuck for openstack/yaql there :(02:24
*** slogan621 has joined #openstack-infra02:25
sergmelikyanon build-tarboll job02:25
*** Jeffrey4l has quit IRC02:25
clarkbsergmelikyan: the post queue is a lower priority than check and gate02:25
*** tphummel has joined #openstack-infra02:25
clarkbas long as there is check and gate demand post is deferred02:25
sergmelikyanyeah, but 13 hours... it's kinda a lot02:25
clarkbyes because there has been 13 hours of check and gate demand02:26
sergmelikyangot it, so we never start processing Post unless we have no more staff in queue for check and gate02:26
mesterythanks for the explanation jeblair02:26
sergmelikyanclarkb: thank you02:26
* sergmelikyan is not really worried about building tarball, just was worried about the fact that jobs seemed to be stuck for 13 hours02:28
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add Neutron Failure Rate dashboard to Grafana
clarkbsergmelikyan: yes the way the scheduker works is gatr and check must have all queued jobs running before post will get nodes02:29
*** chlong has joined #openstack-infra02:29
*** xyang1 has quit IRC02:29
*** baoli has quit IRC02:29
*** tpsilva has quit IRC02:36
*** roaet- has quit IRC02:41
*** Sukhdev_ has quit IRC02:41
*** roaet_ has joined #openstack-infra02:42
*** armax has quit IRC02:43
*** woodster_ has quit IRC02:46
*** Jeffrey4l has joined #openstack-infra02:49
*** slogan621 has quit IRC02:56
openstackgerritSpencer Krum proposed openstack-infra/system-config: Delete json reports when we delete yaml reports
*** tphummel has quit IRC03:03
*** browne has joined #openstack-infra03:06
*** rguillebert has quit IRC03:07
*** annegentle_ has joined #openstack-infra03:08
*** daemontool has quit IRC03:08
*** links has joined #openstack-infra03:10
*** fawadkhaliq has joined #openstack-infra03:16
*** yamahata has quit IRC03:17
*** thorst_ has joined #openstack-infra03:18
*** lucas-afk has quit IRC03:19
*** lucasagomes has joined #openstack-infra03:20
*** chaitu has joined #openstack-infra03:21
*** annegentle_ has quit IRC03:21
*** sc68cal has quit IRC03:23
nibalizerinfra-root I am takign the hiera lock03:24
*** thorst_ has quit IRC03:24
nibalizerinfra-root unlocked03:25
*** annegentle_ has joined #openstack-infra03:25
openstackgerritSpencer Krum proposed openstack-infra/ansible-puppet: Remove dead code
clarkbspeaking of dead code I saw hpcloud things in oscc03:27
prometheanfireianw: the diff/patch does not need the dep, but overall, the element does03:27
openstackgerritSpencer Krum proposed openstack-infra/ansible-puppet: Remove dead code
clarkbmordred: ^ any idea if anyone could possibly be using that somehow or should we push a cleanup?03:27
*** baoli has joined #openstack-infra03:27
*** esker has joined #openstack-infra03:32
lifelesstonyb: I'll look again first thing tomorrow03:34
*** shashank_hegde has joined #openstack-infra03:34
mordredclarkb: aroo?03:36
mordredclarkb: oh - yeah - I have already pushed an hp cleanup to oscc03:37
*** annegentle_ has quit IRC03:40
tonyblifeless: Thanks.  There is also a thread on os-dev re pip install -U ... You shoudl chime in03:41
tonyblifeless: Thanks for looking at the mock stuff.03:41
*** asahlin has quit IRC03:43
*** asahlin has joined #openstack-infra03:44
*** annegentle_ has joined #openstack-infra03:44
*** harlowja_at_home has joined #openstack-infra03:47
openstackgerritMatthew Kassawara proposed openstack-infra/project-config: networking-ovn: Add install job for kuryr
lifelesstonyb: np03:49
*** amotoki has joined #openstack-infra03:50
*** genediazjr has joined #openstack-infra03:50
*** piet has joined #openstack-infra03:51
*** esker has quit IRC03:53
openstackgerritSpencer Krum proposed openstack-infra/system-config: Don't run infracloud puppet twice
*** slogan621 has joined #openstack-infra03:56
*** amotoki has quit IRC03:57
*** esker has joined #openstack-infra03:58
*** hdd has joined #openstack-infra04:01
*** esker has quit IRC04:03
*** Sukhdev has joined #openstack-infra04:06
*** arthus has joined #openstack-infra04:08
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Fixed #! linein javascript jobs
*** tphummel has joined #openstack-infra04:12
*** boris-42 has quit IRC04:14
*** dahoo has quit IRC04:14
*** dimtruck is now known as zz_dimtruck04:17
*** tiswanso has quit IRC04:18
*** genediazjr has quit IRC04:18
*** Daisy has joined #openstack-infra04:18
*** sc68cal has joined #openstack-infra04:20
*** thorst_ has joined #openstack-infra04:22
*** woodster_ has joined #openstack-infra04:23
*** Daisy has quit IRC04:23
*** annegentle_ has quit IRC04:25
dougwighi infra, i've noticed jobs taking a lot longer than normal.  on 268232 e.g., rally looks to be restarting multiple times. is that a known issue?04:27
*** browne has left #openstack-infra04:28
clarkbusually that means the job is crashing in such a way tat the slave is no longer useable, zuul will restart assuming neworking went away04:28
*** claudiub has quit IRC04:29
*** amotoki has joined #openstack-infra04:29
*** thorst_ has quit IRC04:30
dougwigi've been watching this one all day (not all the same jenkins run), and the last job always seems to be the neutron rally job, with those resets. i wonder if there's a problem affecting all neutron runs around that job.04:31
prometheanfirey/win 104:31
*** jgriffith is now known as jgriffith_away04:31
clarkbdougwig: have an example console log?04:32
dougwigclarkb: no, because i figured i was nuts thinking the progress bar was jumping backwards all day, until i just now saw it go from halfway to queued to starting.  now i've got this one open to see if i can catch it:
*** armax has joined #openstack-infra04:34
dougwigclarkb: or i am just losing my mind. that's always highly possible.  :)04:34
clarkbno its a thing, it may be related to using a sparse swapfile on clouds iwthout a swap device04:35
clarkb will in theory make those fail hard when they run out of fsik04:35
clarkb*out of disk04:35
clarkbrather than get into an undebuggable restart loop04:36
dougwigclarkb: is that why the check queue has been so backed up lately?04:36
clarkbI think queues are mostly backed up due to all of the recent breaking regressions04:37
clarkbnova + eventlet, oslo config? + tempest, the mock thing etc04:37
clarkbwhen that happens are we reset the gate frequently we take resources away from the other queues04:37
dimsclarkb : oslo.config => cliff04:38
clarkbdims: thanks, I wasn't sure which thing did it and thought someone said oslo config04:38
dimsclarkb : yep, we had a work around which went in last night, i just tracked down the cliff commit and dean has a revert up too04:39
*** baoli has quit IRC04:39
clarkbbut some of it certainly is cloud growing pains04:39
dougwigclarkb: ahh, got it.  thanks for the info.04:40
*** salv-orl_ has joined #openstack-infra04:41
*** amotoki has quit IRC04:42
*** piet has quit IRC04:42
*** dims has quit IRC04:43
*** arthus has quit IRC04:43
*** salv-orlando has quit IRC04:44
clarkbdougwig: is another I think04:45
*** annegentle_ has joined #openstack-infra04:46
dougwigclarkb: that's a non-voting job. would it help sort things out if we disable it until the issue is sorted?04:46
*** fawadkhaliq has quit IRC04:46
*** sergmelikyan has quit IRC04:48
clarkbdougwig: if the fix is my devstack-gate change then not really, that will get us rolling again04:49
clarkband it is almost merged04:49
dougwigclarkb: ok04:49
openstackgerritMerged openstack/diskimage-builder: Correct rhel-common for rhel6
*** ajmiller has joined #openstack-infra04:49
openstackgerritIan Wienand proposed openstack/diskimage-builder: yum-minimal: pre-install base packages
*** shashank_hegde has quit IRC04:50
*** ajmiller has quit IRC04:50
*** isaacb has quit IRC04:50
*** annegentle_ has quit IRC04:51
*** armax has quit IRC04:51
dougwigclarkb: yep, my change just hit that reset again as well. i'll be patient and check after your fix merges. thanks.04:51
clarkbif you look at the log jenkins is losing the connection to the slave04:51
clarkbit is possible that that is an independnet issue to what my d-g change addresses since tended to hve the slaves completely lock up04:52
clarkbbut could be different symptoms for the same problem04:52
*** arthus has joined #openstack-infra04:54
fungione thing we can do, if it persists, is hold a worker (in rackspace if possible so we also have easyish access to an out of band console) and then see if we can spot why the worker is getting completely trashed04:56
slogan621quick questions: where would I find the list of those who can give +2? Is there an alias or similar that addresses all of them on IRC? I seem to be suffering from having not done any reviews or work for infra according to earlier comments here, and so I'm not getting much support it seems. but I have answered questions in in the past and have done code reviews, just not for infra. Does that count for anyt04:59
slogan621Finally, if I must do work for infra, what kind of work is someone outside of infra capable of doing?04:59
slogan621My experience is with neutron04:59
dougwigslogan621: in gerrit, top menu 'people' -> groups -> then type in the name of the group you're looking for. but unless something is broken, most of those folks are quite busy. what is it you need reviewed?05:00
*** julim has joined #openstack-infra05:00
dougwigslogan621: and contacting cores directly, if you don't already know them, is a pretty good way to NOT get a review done.  :)05:00
clarkbright one of the big issues right now is we are all swamped keeping several boats afloat05:00
slogan621clarkb: got it :-)05:01
fungisome of us are bailing water out the back while the others are trying to keep rowing ;)05:01
*** tzn has joined #openstack-infra05:01
fungiand some of us are falling asleep. midnight here05:01
*** rbrndt has joined #openstack-infra05:01
clarkbbetween trying to spin up new clouds to backfill hpcloud going away, nodepol builders that allow us to push images to those clouds, and feeding babies I have no time for my "pet" projects like upgrading the ELK stack05:01
dougwigindeed, infra cores are among the best at getting to reviews unsolicited. if they don't, you can almost bank on their being a fire going.05:01
clarkband that doesn't even account for the fires :)05:02
clarkbslogan621: but that is why the suggestion is to help out, if you can review a few changes and either ACK or NACK them that can help alleviate the pressure on everyone else05:02
clarkbslogan621: and in return they will ahve mor etime to look at your change(s)05:02
slogan621dougwig: It's a project create that initially submitted maybe 6 days back. I think we have it, and we have 1 +2 already.
slogan621so, what can I review, then. Point me at something and glad to look.05:03
*** markvoelker has quit IRC05:03
clarkbslogan621: that project-config repo typically has 10s of changes open for review. If you are familira with how to add projects or jenkins jobs you can review those05:04
dougwigslogan621: i'm not a core, but i'll take a peek.05:04
clarkbdougwig: more info, the two tests we had links for ran in different ovh regions. OVH is one of hte places where we use a sparse swap file so it could still be that issue05:04
slogan621Like I said, new to neutron but did learn a small amount with this tiny change we made05:04
slogan621er, new to infra05:04
*** rbrndt has quit IRC05:04
clarkbdougwig: it will be telling if we find a test fial in that way on rax because rax is using a proper swap device not a fiel05:04
*** armax has joined #openstack-infra05:05
*** piet has joined #openstack-infra05:05
clarkbdougwig: and ya when it isn't a few minutes before bed time I/we can work on holding a test node and poke at it more directly05:05
slogan621let me see if I can handle any of it, but bad reviews are worse than no reviews, so I won't take on anything other than looking for the obvious things like alphabetization issues and so on05:05
*** jamielennox is now known as jamielennox|away05:06
dougwigclarkb: ok, sleep well, and let me know anything i can do to help.05:06
*** tzn has quit IRC05:06
clarkbthe other theory I had was that the recent ubuntu trusty kernel may be causing us grief but that was backed by almost no data05:07
*** sridhar_ram has quit IRC05:08
*** fawadkhaliq has joined #openstack-infra05:11
*** piet has quit IRC05:18
slogan621wow, there is a lot of reviews going on05:20
*** jgriffith_away is now known as jgriffith05:21
*** bpokorny has joined #openstack-infra05:22
*** thorst_ has joined #openstack-infra05:27
*** hdd has quit IRC05:28
*** hdd has joined #openstack-infra05:31
*** jamielennox|away is now known as jamielennox05:31
*** hdd has quit IRC05:32
dougwigclarkb: three fails in a row on ovh nodes. this time it got a rax node, so it'll be interesting to see what happens.05:34
*** thorst_ has quit IRC05:34
*** infra-red has joined #openstack-infra05:35
*** yamahata has joined #openstack-infra05:36
*** infra-re_ has joined #openstack-infra05:36
*** piet has joined #openstack-infra05:36
*** armax has quit IRC05:37
*** infra-red has quit IRC05:39
*** hdd has joined #openstack-infra05:44
*** hdd has quit IRC05:44
chaituHi, While trying to add slave agent in the jenkins node i am facing following error can someone please help me out with it.05:46
*** fedexo has joined #openstack-infra05:52
*** fedexo_ has joined #openstack-infra05:53
*** piet has quit IRC05:54
*** vgridnev has joined #openstack-infra05:58
openstackgerritJoseph Robinson proposed openstack-infra/irc-meetings: Add the new US meeting time to the User Guide Team yaml
*** infra-re_ has quit IRC06:02
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance
*** rossella_s has quit IRC06:03
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance
*** rossella_s has joined #openstack-infra06:03
*** markvoelker has joined #openstack-infra06:04
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance
*** nadya has joined #openstack-infra06:05
*** infra-red has joined #openstack-infra06:05
*** nadya is now known as Guest2572706:05
*** jaosorior has joined #openstack-infra06:10
dougwigclarkb: success on the rax node. hopefully that means swap is the root issue there as well.06:11
*** shashank_hegde has joined #openstack-infra06:13
*** Guest25727 has quit IRC06:16
*** infra-red has quit IRC06:17
jklarehey, is something wrong with the gerrit-zuul communication? there were no gate jobs started for this (and some other) patches after the workflow +106:18
*** infra-red has joined #openstack-infra06:20
*** bpokorny has quit IRC06:22
*** cp16net has joined #openstack-infra06:23
cp16netsounds like the apt mirrors went live and about that time causes some trouble for our trove gate jobs earlier in the day.06:24
cp16netlooks like at least one of the mirrors i looked at does not have the ubuntu package python-software-properties06:24
cp16netlooked into this mirror and didnt see it.
*** infra-red has quit IRC06:29
openstackgerritTony Breeds proposed openstack-infra/elastic-recheck: Add query for missing mock in py34 tests bug 1544377
openstackbug 1544377 in python-glanceclient "proposed change to test-requirements breaks py34" [Undecided,New]
*** thorst_ has joined #openstack-infra06:32
*** markvoelker has quit IRC06:34
*** kdas_ has quit IRC06:36
*** woodster_ has quit IRC06:36
*** tphummel has quit IRC06:37
openstackgerritMerged openstack/requirements: Revert "Only install mock on Python < 3.3"
*** thorst_ has quit IRC06:39
*** zeih_ has joined #openstack-infra06:41
*** harlowja_at_home has quit IRC06:43
*** vgridnev has quit IRC06:43
*** maishsk has quit IRC06:46
*** maishsk has joined #openstack-infra06:47
*** flepied has joined #openstack-infra06:55
*** vgridnev has joined #openstack-infra06:55
*** sc` has joined #openstack-infra06:56
*** jaosorior has quit IRC06:56
*** rvasilets_ has joined #openstack-infra06:58
*** zeih_ has quit IRC06:59
*** sc68cal has quit IRC07:03
*** flepied1 has joined #openstack-infra07:05
*** infra-red has joined #openstack-infra07:06
*** esikachev has joined #openstack-infra07:07
*** infra-re_ has joined #openstack-infra07:08
slogan621question: is the jobs: section in jenkins/jobs/projects.yaml for a project a requirement?07:08
*** flepied has quit IRC07:09
slogan621if I don't want to run any voting jobs, is there some minimum entry or can it be completely removed?07:09
*** vgridnev has quit IRC07:10
slogan621the docs suggest it "should" be there but not "must" be there07:10
*** nadya has joined #openstack-infra07:13
*** infra-red has quit IRC07:13
*** nadya is now known as Guest1354407:13
*** fedexo has quit IRC07:14
*** Sukhdev has quit IRC07:15
*** maishsk has quit IRC07:15
*** maishsk has joined #openstack-infra07:16
*** lmiccini|gone is now known as lmiccini07:16
*** vgridnev has joined #openstack-infra07:18
*** jaosorior has joined #openstack-infra07:20
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** kozhukalov has joined #openstack-infra07:23
*** abregman has joined #openstack-infra07:24
*** abregman|afk has quit IRC07:25
*** ianw has quit IRC07:30
*** ianw has joined #openstack-infra07:30
*** jaosorior has quit IRC07:32
*** jtomasek has joined #openstack-infra07:36
*** mrmartin has joined #openstack-infra07:36
*** scheuran has joined #openstack-infra07:37
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
*** thorst_ has joined #openstack-infra07:37
*** amotoki has joined #openstack-infra07:38
*** kozhukalov has quit IRC07:38
*** kozhukalov has joined #openstack-infra07:38
*** kozhukalov has quit IRC07:40
*** kozhukalov has joined #openstack-infra07:41
*** kozhukalov has quit IRC07:42
*** kozhukalov_ has joined #openstack-infra07:42
*** kozhukalov_ has quit IRC07:42
*** kozhukalov_ has joined #openstack-infra07:42
*** kozhukalov has joined #openstack-infra07:43
*** kozhukalov has quit IRC07:43
*** flepied1 has quit IRC07:44
*** kozhukalov has joined #openstack-infra07:44
*** thorst_ has quit IRC07:45
*** zeih_ has joined #openstack-infra07:51
*** electrofelix has joined #openstack-infra07:51
*** max_lobur has quit IRC07:54
yolandagood morning08:00
*** infra-red has joined #openstack-infra08:02
*** abregman_ has joined #openstack-infra08:04
jklarecan somebody help me understand why this patch is not merging ? something wrong with the gates?08:04
jklareother patches from other projects seem to merge fine08:04
*** infra-re_ has quit IRC08:05
*** arthus has quit IRC08:05
*** jaosorior has joined #openstack-infra08:05
jklarebut there are multiple cookbook patches (different projects) that should run gate jobs, but nothing happens08:06
*** vgridnev has quit IRC08:06
*** achanda has quit IRC08:11
*** arthus has joined #openstack-infra08:12
*** hashar has joined #openstack-infra08:13
*** achanda has joined #openstack-infra08:14
*** flepied1 has joined #openstack-infra08:14
*** kozhukalov has quit IRC08:14
*** kozhukalov has joined #openstack-infra08:15
*** kozhukal` has joined #openstack-infra08:15
*** kozhukal` has quit IRC08:16
*** kozhukalov has quit IRC08:16
*** kozhukalov has joined #openstack-infra08:17
yolandajklare, can you try giving another +workflow? someone with +w powers has to send another one08:17
jklaredone and seems to work08:19
jklareok, so i will do this for all patches that failed to trigger the gate jobs?08:19
jklareits no big deal, just asking if that is the right approach to the issue08:20
yolandajklare, i don't know if there were some problems with zuul yesterday. But some times, seems that the event is not picked properly08:20
*** kozhukalov_ has quit IRC08:20
yolandaand the way to retrigger is to send another +w08:20
jklareyolanda ok, thanks08:20
*** esikachev has quit IRC08:23
*** mikelk has joined #openstack-infra08:24
*** jlanoux has joined #openstack-infra08:25
AJaegerjklare: your job has a depends-on on another project08:25
jklareAJaeger sure, but why is that an issue?08:26
AJaegerIf your change gets +W first and these two changes do not share a testing queue, there's no notification that the first one merged and the second one can go in08:26
jklareAJaeger ahhhh, i see08:26
*** shashank_hegde has quit IRC08:26
*** jed56 has joined #openstack-infra08:26
AJaegerjklare: So, what I'm doing: Whenever I see a depends-on: Wait until the dependent job is in and only then approve.08:27
jklareAJaeger yeah, i think that would have been easier08:27
AJaegerOr if there's an approval, just trigger it again08:27
jklareAJaeger next time :D08:27
*** hashar has quit IRC08:27
yolandaoh , that's the other reason.. sorry, i missed the depends-on08:27
*** kozhukalov has quit IRC08:28
AJaegeryolanda: can I push these three on your review queue for later today, please? and Thanks!08:29
yolandasure, i will look in a while08:29
*** nmagnezi has joined #openstack-infra08:32
*** esikachev has joined #openstack-infra08:33
*** kozhukalov has joined #openstack-infra08:33
*** abregman_ has quit IRC08:33
*** abregman has quit IRC08:33
*** rcernin has joined #openstack-infra08:35
*** tzn has joined #openstack-infra08:35
*** kozhukalov has quit IRC08:36
*** isaacb has joined #openstack-infra08:36
*** kozhukalov has joined #openstack-infra08:36
*** maishsk has quit IRC08:37
*** abregman has joined #openstack-infra08:38
*** kozhukalov has quit IRC08:39
*** kozhukalov has joined #openstack-infra08:39
*** maishsk has joined #openstack-infra08:39
*** shardy has joined #openstack-infra08:40
*** kozhukalov has quit IRC08:40
*** kozhukalov has joined #openstack-infra08:40
*** kozhukalov_ has joined #openstack-infra08:41
*** shardy has quit IRC08:41
*** yaume has joined #openstack-infra08:41
*** shardy has joined #openstack-infra08:42
*** tzn has quit IRC08:42
*** e0ne has joined #openstack-infra08:42
*** thorst_ has joined #openstack-infra08:42
*** mrmartin has quit IRC08:43
*** achanda has quit IRC08:44
*** claudiub has joined #openstack-infra08:44
*** amotoki has quit IRC08:45
*** matrohon has joined #openstack-infra08:46
*** openstackgerrit has quit IRC08:47
*** openstackgerrit_ has joined #openstack-infra08:47
*** openstackgerrit_ is now known as openstackgerrit08:48
*** alex_xu_ has joined #openstack-infra08:49
*** abregman has quit IRC08:49
*** thorst_ has quit IRC08:50
*** abregman has joined #openstack-infra08:50
openstackgerritMerged openstack-infra/project-config: Force the gate-irc-meetings-tox-ical postbuildscript to always run
yolandaAJaeger, would you say that fits under networking scope, and needs +1 from neutron liaisons?08:50
*** dingyichen has quit IRC08:51
*** ildikov has quit IRC08:51
*** arthus has quit IRC08:51
openstackgerritMerged openstack-infra/project-config: Templatize experimental bindep jobs
*** maishsk has quit IRC08:52
*** jtomasek has quit IRC08:52
*** alex_xu has quit IRC08:52
AJaegeryolanda: no, it's independent. But looking at comments: Wait with your +A another day, there's still some discussion going on whether this is the right approach08:56
yolandaok we'll wait for that08:57
*** jtomasek has joined #openstack-infra08:57
*** fawadkhaliq has quit IRC08:59
*** jordanP has joined #openstack-infra09:00
*** derekh has joined #openstack-infra09:00
*** fhubik has joined #openstack-infra09:00
*** fhubik is now known as fhubik_brb09:00
*** sshnaidm has quit IRC09:00
*** BobBall_AWOL is now known as Bobball09:01
*** Bobball is now known as BobBall09:01
*** zeih_ has quit IRC09:01
*** fhubik_brb is now known as fhubik09:05
*** arthus has joined #openstack-infra09:05
*** mrmartin has joined #openstack-infra09:06
*** placko_ has joined #openstack-infra09:06
*** hashar has joined #openstack-infra09:06
*** ihrachys has joined #openstack-infra09:07
*** jordanP has quit IRC09:12
jheskethAJaeger: so I'm confused about the linters stuff sorry... I thought the issue was the pep8 target was poorly named since it might actually be running bashate (for example). So I thought the idea was to rename pep8 to linters effectively?09:12
*** fhubik is now known as fhubik_brb09:13
*** ifarkas has joined #openstack-infra09:14
AJaegerjhesketh: feel free to read backscroll from 14 hours ago where I discussed with jeblair and fungi09:16
AJaegerbasically: Yes, that was the idea moving forward.09:16
AJaegerBut I forgot about the PTI - and that we have run-pep8 which does some extra tests09:16
hashariris they started with 'pep8'  then switched to use flake8  but to ease transition kept 'pep8' as an entry point09:17
hasharwhich is really an alias for 'lint stuff'09:17
AJaegerCiting from last night:09:17
AJaeger1) Updating PTI, 2) fixing linters job to do the same as pep8, 3) adding linters to python-jobs with lots of skips/non-voting, 4) updating official repos to have linters target that does pep8 job on master (not on stable), 5) removing pep8 and skips/non-voting09:17
AJaegerPlan B: 1) Revert changes for bandit, keystone 2) Remove linters job from python-jobs in jenkins . Leave ansible, system-config, project-config as is (using linters)09:17
AJaegerAnd for plan A, I don't have the time to commit driving this through.09:18
* AJaeger would be willing to review but getting this is in properly was more than I can do09:18
*** mrmartin has quit IRC09:18
*** fhubik_brb is now known as fhubik09:18
jheskethAJaeger: right, okay, thanks09:19
jheskethAJaeger: so from plan A.2), what do linters do differently as pep8 presently?09:19
jhesketh(I think this is where my confusion is)09:19
AJaegerjhesketh: So, yes, pep8 is a bad name.09:19
AJaegerjhesketh: Check run-pep8 ;) It adds sdist check as well, to check that tarballs are useable09:19
openstackgerritMerged openstack-infra/project-config: Fixed #! linein javascript jobs
AJaegerjhesketh: that'S the easiest part to change ;)09:20
AJaegerjhesketh: If you want to help cleanup, feel free to review: and Thanks!09:21
AJaegerjhesketh: step 4 is a lot of mechanical work- and step 3 and step 5 might be iterative..09:22
jheskethAJaeger: right, I agree it's a bit of work to get it entirely there09:23
AJaegerjhesketh: right now we have 380+ repos using pep8 - but not all of them are official projects, so we would only need to update the official ones since those are under PTI09:23
jheskethwe could try tackling it in a backwards compatible way... for example by having the CTI allow use of either linters or pep8 for a while09:23
AJaegerjhesketh: and then you need to figure out what to do with stable branches.09:24
AJaegerDo we want - like keystone had: pep8 in stable but linters in master?09:24
AJaegerjhesketh: yes, transition time would be needed - but if it's too long, we have a mess in zuul/layout.yaml09:24
AJaegerbest would be to do it atomic: Prepare changes, mass merge ;)09:25
*** vgridnev has joined #openstack-infra09:25
jheskethindeed and getting buy in would take time09:25
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone
*** ildikov has joined #openstack-infra09:26
AJaegerfrom the feedback so far, buy in will take time but it resonated well to rename pep8. So, that's the simplest part09:26
openstackgerritOndřej Nový proposed openstack/requirements: Bump PyECLib global-requirement to >= 1.2.0
*** eliqiao_ has quit IRC09:29
openstackgerritMerged openstack-infra/project-config: Add periodic job to autogenerate devstack plugins list
openstackgerritAlistair Coles proposed openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post
*** markvoelker has joined #openstack-infra09:31
*** ildikov has quit IRC09:33
*** sheeprine has quit IRC09:33
*** sheeprine has joined #openstack-infra09:35
*** eliqiao_ has joined #openstack-infra09:35
*** markvoelker has quit IRC09:36
*** scheuran has quit IRC09:37
*** gildub has joined #openstack-infra09:38
*** mrmartin has joined #openstack-infra09:39
openstackgerritMerged openstack-infra/project-config: Add requirements check for watcher
*** sfinucan has joined #openstack-infra09:41
*** sshnaidm has joined #openstack-infra09:41
*** sshnaidm has quit IRC09:42
openstackgerritMerged openstack-infra/project-config: Remove gate-nova-pylint
*** HeOS has quit IRC09:44
*** ociuhandu has quit IRC09:44
*** HeOS has joined #openstack-infra09:44
*** claudiub has quit IRC09:45
*** gildub has quit IRC09:45
derekhhey, has something changed in the behaviour of ZUUL  and how it sets ZUUL_CHANGES09:45
*** ildikov has joined #openstack-infra09:45
derekhe.g. on this patch
*** sshnaidm has joined #openstack-infra09:46
derekhthere is a depends on for a patch in openstack/heat09:46
derekhbut CI has ZUUL_CHANGES=openstack/tripleo-heat-templates:master:refs/changes/56/278556/109:46
derekhno mention of openstack/heat ?09:46
derekhthis didn't used to be the case09:47
shardySame problem with which has a heat Depends-On09:47
*** thorst_ has joined #openstack-infra09:48
*** max_lobur has joined #openstack-infra09:49
*** andymaier has joined #openstack-infra09:51
*** ganso has joined #openstack-infra09:51
*** scheuran has joined #openstack-infra09:51
*** kaisers has joined #openstack-infra09:54
*** thorst_ has quit IRC09:55
*** andymaier has quit IRC09:56
derekhshardy: I've rechecked this patch, it work as expected yesterday (as in ZUUL_CHANGES was set) , lets see what happens
*** tzn has joined #openstack-infra09:57
*** scheuran has quit IRC10:00
*** cgalan has joined #openstack-infra10:01
*** gildub has joined #openstack-infra10:02
*** scheuran has joined #openstack-infra10:02
*** rossella_s has quit IRC10:03
*** reed_ has joined #openstack-infra10:03
*** vgridnev has quit IRC10:03
*** rossella_s has joined #openstack-infra10:03
*** vgridnev has joined #openstack-infra10:08
*** vgridnev has quit IRC10:09
*** yamamoto has quit IRC10:10
*** yamamoto has joined #openstack-infra10:10
*** vgridnev has joined #openstack-infra10:11
*** vgridnev has quit IRC10:12
*** vgridnev has joined #openstack-infra10:12
*** vgridnev has quit IRC10:13
*** kzaitsev_mb has joined #openstack-infra10:14
*** vgridnev has joined #openstack-infra10:15
*** dtantsur|afk is now known as dtantsur10:20
*** mase_x200 has joined #openstack-infra10:20
*** gildub has quit IRC10:21
*** gildub has joined #openstack-infra10:22
*** esikachev has quit IRC10:23
*** fedexo_ has quit IRC10:24
*** matrohon has quit IRC10:26
*** Daisy has joined #openstack-infra10:26
*** esikachev has joined #openstack-infra10:26
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Allow zuul-release to push merge commits
*** Daisy has quit IRC10:31
*** esikachev has quit IRC10:31
*** vgridnev has quit IRC10:34
*** esikachev has joined #openstack-infra10:36
*** tpsilva has joined #openstack-infra10:36
*** yaume has quit IRC10:38
*** yamamoto has quit IRC10:38
*** salv-orlando has joined #openstack-infra10:42
*** vgridnev has joined #openstack-infra10:42
*** dizquierdo has joined #openstack-infra10:44
*** achanda has joined #openstack-infra10:44
*** salv-orl_ has quit IRC10:45
*** vgridnev has quit IRC10:45
*** vgridnev has joined #openstack-infra10:46
*** chaitu has quit IRC10:47
*** dims has joined #openstack-infra10:48
*** yamamoto has joined #openstack-infra10:49
*** yamamoto has quit IRC10:49
*** ldnunes has joined #openstack-infra10:49
*** scheuran has quit IRC10:50
*** rfolco has joined #openstack-infra10:51
*** achanda has quit IRC10:51
*** thorst_ has joined #openstack-infra10:52
kzaitsev_mbIs there a place, where we can find info about 'meta' tags in git tags?10:53
openstackgerritSean Dague proposed openstack-infra/project-config: create openstack/service-registry
*** [1]Thelo has joined #openstack-infra10:56
*** cgalan has quit IRC10:57
*** cgalan has joined #openstack-infra10:57
*** arthus has quit IRC10:58
*** Thelo has quit IRC10:58
*** [1]Thelo is now known as Thelo10:58
*** thorst_ has quit IRC11:00
openstackgerritSean Dague proposed openstack-infra/project-config: remove large-ops testing
therveHum I have an issue with osprofiler in stable/kilo for Heat11:01
therveIt seems to depend on a recent oslo.utils version11:01
therveShould I cap osprofiler in kilo?11:02
*** arthus has joined #openstack-infra11:02
*** ildikov has quit IRC11:03
sdagueshardy / derekh is HEAT in the project list for those jobs? I know the PROJECTS list was being pruned for the default case to speed up build times11:03
*** aysyd has joined #openstack-infra11:03
*** scheuran has joined #openstack-infra11:05
derekhsdague: not sure, I can check, but would it matter ? heat is getting mentioned nowhere in the jenkins job
shardywe don't test much in tempest anymore since we moved everything in-tree for our functional tests11:06
sdaguederekh: which job is the issue?11:06
derekhsdague: I see the problem (I think), shardy thee patch has merged already11:06
*** Guest13544 has quit IRC11:07
shardyderekh: aha - I was trying to work around the old current-tripleo pin11:07
*** kushal has joined #openstack-infra11:07
shardyI assumed we could override it on a per-package basis via Depends-On11:07
derekhshardy: looks like zuul is smart enough to no let you do that11:08
derekhsdague: thanks for looking11:08
*** esikachev has quit IRC11:08
sdaguesure, np11:08
derekhshardy: maybe a patch to add heat to the package list, or add it to zuul changes11:08
sdagueyou guys sorted now?11:08
shardysdague: thanks, and sorry I think I misinterpreted your question ;)11:08
sdagueshardy: ok, no prob11:09
derekhsdague: yup, we're sorted11:09
sdaguecool :)11:09
*** claudiub has joined #openstack-infra11:12
openstackgerritAlexander Tivelkov proposed openstack/requirements: Bump yaql version to 1.1.0
*** smanjara has joined #openstack-infra11:13
openstackgerritThomas Herve proposed openstack/requirements: Cap osprofiler
*** reed_ has quit IRC11:15
*** fawadkhaliq has joined #openstack-infra11:16
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling
*** dizquierdo has quit IRC11:17
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer
*** beisner has quit IRC11:19
openstackgerrityolanda.robla proposed openstack-infra/infra-specs: Add spec to provide an Infra Root CA
*** beisner has joined #openstack-infra11:20
*** flepied1 has quit IRC11:20
*** nadya has joined #openstack-infra11:21
*** daemontool has joined #openstack-infra11:21
*** nadya is now known as Guest7479611:22
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin
*** cgalan has quit IRC11:24
*** cgalan has joined #openstack-infra11:25
*** shardy has quit IRC11:26
*** shardy has joined #openstack-infra11:28
*** mase_x200 has quit IRC11:29
*** dizquierdo has joined #openstack-infra11:32
*** markvoelker has joined #openstack-infra11:32
*** fhubik is now known as fhubik_brb11:33
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling
openstackgerritMerged openstack/os-testr: Updated from global requirements
*** fhubik_brb is now known as fhubik11:34
*** markvoelker has quit IRC11:37
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling
*** esikachev has joined #openstack-infra11:37
openstackgerritAlexander Charykov proposed openstack-infra/gerritlib: Add limit for bulk_query request
*** amotoki has joined #openstack-infra11:41
*** fhubik is now known as fhubik_brb11:41
*** shardy has quit IRC11:46
*** achanda has joined #openstack-infra11:48
*** yamamoto has joined #openstack-infra11:50
*** Nikolay_St has joined #openstack-infra11:50
Nikolay_Sthi all11:50
Nikolay_Stwe have a strange update from OpenstackBot in python-muranoclient:
Nikolay_Stis it expected that we decrese python version to <3.3?11:51
*** achanda has quit IRC11:52
sdagueNikolay_St: see mailiing list, that was a mistake11:52
*** dprince has joined #openstack-infra11:55
openstackgerritSean Dague proposed openstack-infra/project-config: remove large-ops testing
*** vgridnev has quit IRC11:56
*** vgridnev has joined #openstack-infra11:56
*** thorst_ has joined #openstack-infra11:58
*** yamamoto has quit IRC11:58
*** Guest74796 has quit IRC11:59
*** amotoki has quit IRC12:00
Nikolay_Stsdague: ok, thx12:02
*** thorst_ has quit IRC12:02
*** amotoki has joined #openstack-infra12:03
*** pblaho has quit IRC12:06
*** kzaitsev_mb has quit IRC12:06
Nikolay_Stsdague: what is the title of the letter in ML?12:08
*** kushal has quit IRC12:10
*** rguillebert has joined #openstack-infra12:11
openstackgerritMartin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0
*** ig0r_ has joined #openstack-infra12:12
*** gildub has quit IRC12:14
*** pradk_ has quit IRC12:24
*** |-paul-| has joined #openstack-infra12:25
*** pc_m has quit IRC12:26
*** cgalan has quit IRC12:26
*** nadya has joined #openstack-infra12:27
*** nadya is now known as Guest2552612:27
*** Guest25526 has quit IRC12:27
*** nmagnezi has quit IRC12:28
openstackgerritMartin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0
*** weshay has joined #openstack-infra12:28
*** _amrith_ is now known as amrith12:28
*** doug-fish has joined #openstack-infra12:29
*** pc_m has joined #openstack-infra12:30
*** rvasilets_ has left #openstack-infra12:32
therveMy requirement patch fails with "You must put some 'source' URIs in your sources.list"12:32
therveIs that something intermittent?12:32
*** smanjara has quit IRC12:33
*** markvoelker has joined #openstack-infra12:33
*** aspiers_ is now known as aspiers12:35
*** mase_x200 has joined #openstack-infra12:35
*** flepied1 has joined #openstack-infra12:36
*** markvoelker has quit IRC12:37
*** pradk_ has joined #openstack-infra12:38
*** thorst_ has joined #openstack-infra12:38
*** daemontool has quit IRC12:39
*** fhubik_brb is now known as fhubik12:39
*** thorst_ has quit IRC12:39
*** daemontool has joined #openstack-infra12:39
*** dims has quit IRC12:40
*** nmagnezi has joined #openstack-infra12:40
*** thorst_ has joined #openstack-infra12:41
*** amotoki has quit IRC12:41
*** arthus has quit IRC12:42
*** Nikolay_St has quit IRC12:42
*** flepied1 is now known as flepied12:44
*** baoli has joined #openstack-infra12:47
*** arthus has joined #openstack-infra12:47
*** placko_ has quit IRC12:47
*** baoli_ has joined #openstack-infra12:48
*** pblaho has joined #openstack-infra12:48
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder: Do not remove python-dev from ironic-agent image
*** daemontool_ has joined #openstack-infra12:49
*** daemontool__ has joined #openstack-infra12:49
*** kzaitsev_mb has joined #openstack-infra12:49
*** daemontool has quit IRC12:51
*** baoli has quit IRC12:51
*** |-paul-| has quit IRC12:51
*** daemontool_ has quit IRC12:54
*** zeih_ has joined #openstack-infra12:55
*** jaypipes has joined #openstack-infra12:56
*** nadya has joined #openstack-infra12:56
*** nadya is now known as Guest6533312:57
*** rlandy has joined #openstack-infra13:01
*** annegentle_ has joined #openstack-infra13:01
*** mase_x200 has quit IRC13:02
*** jcoufal has joined #openstack-infra13:06
*** annegentle_ has quit IRC13:08
*** amotoki has joined #openstack-infra13:09
*** zeih_ has quit IRC13:09
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: try hugely reduced MTU on br-ex
*** andreykurilin__ has joined #openstack-infra13:10
*** arthus has quit IRC13:10
*** jsavak has joined #openstack-infra13:12
*** shardy has joined #openstack-infra13:14
*** esp has joined #openstack-infra13:17
*** eharney has joined #openstack-infra13:18
*** tcammann has joined #openstack-infra13:18
*** NobodyCa1 has joined #openstack-infra13:18
*** esker has joined #openstack-infra13:20
*** dims has joined #openstack-infra13:20
*** jtomasek_ has joined #openstack-infra13:20
*** Guest57497 is now known as zeus13:22
*** zeus has joined #openstack-infra13:22
*** arthus has joined #openstack-infra13:22
*** daemontool__ has quit IRC13:23
*** markvoelker has joined #openstack-infra13:23
*** esp has quit IRC13:23
*** kushal has joined #openstack-infra13:23
*** tcammann has quit IRC13:23
*** kzaitsev_mb has quit IRC13:24
*** Daisy has joined #openstack-infra13:24
*** xyang1 has joined #openstack-infra13:24
*** NobodyCa1 has quit IRC13:24
*** moravec has quit IRC13:28
*** asahlin has quit IRC13:29
*** asahlin has joined #openstack-infra13:30
*** dpaterson has joined #openstack-infra13:30
*** woodster_ has joined #openstack-infra13:33
*** annegentle_ has joined #openstack-infra13:34
*** daemontool__ has joined #openstack-infra13:35
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Use release version of Ansible for testing
*** jtomasek_ has quit IRC13:37
*** kgiusti has joined #openstack-infra13:37
*** burgerk has joined #openstack-infra13:37
*** edmondsw has joined #openstack-infra13:38
*** rubasov has quit IRC13:42
*** daemontool_ has joined #openstack-infra13:42
*** amitgandhinz has quit IRC13:45
*** jpeeler has joined #openstack-infra13:46
*** daemontool__ has quit IRC13:46
*** pradk_ is now known as pradk13:46
*** links has quit IRC13:48
*** amrith is now known as _amrith_13:49
Shrewsmordred: hopefully that ^^^ unbreaks us, yet again13:50
*** cgalan has joined #openstack-infra13:50
mordredtherve: link?13:50
*** achanda has joined #openstack-infra13:50
*** rhallisey has joined #openstack-infra13:51
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Add fuel-virtualbox project
*** cbader has quit IRC13:52
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone
*** cgalan has quit IRC13:53
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Add fuel-virtualbox project
*** dkranz has joined #openstack-infra13:55
*** achanda has quit IRC13:55
*** daemontool_ is now known as daemontool13:56
*** rhallisey has quit IRC13:56
*** kzaitsev_mb has joined #openstack-infra13:57
*** arthus has quit IRC13:57
*** fhubik is now known as fhubik_brb13:57
*** ddieterly has joined #openstack-infra13:59
*** edmondsw has quit IRC13:59
*** ildikov has joined #openstack-infra14:02
*** rossella_s has quit IRC14:03
*** annegentle_ has quit IRC14:03
*** rossella_s has joined #openstack-infra14:03
*** jaosorior has quit IRC14:05
*** julim has quit IRC14:05
*** notnownikki has joined #openstack-infra14:05
*** jaosorior has joined #openstack-infra14:05
*** sheeprine_ has joined #openstack-infra14:07
*** esker has quit IRC14:08
*** sheeprine has quit IRC14:08
therve more specifically14:09
*** sheeprine_ is now known as sheeprine14:09
*** sheeprine has joined #openstack-infra14:09
*** arthus has joined #openstack-infra14:10
*** Daisy has quit IRC14:11
*** zz_dimtruck is now known as dimtruck14:11
*** smanjara has joined #openstack-infra14:12
*** amitgandhinz has joined #openstack-infra14:15
*** krotscheck_dcm is now known as krotscheck14:15
*** mrmartin has quit IRC14:16
*** mriedem has joined #openstack-infra14:17
sdaguedstufft: random pypi wishlist14:17
sdaguehave a way to let CI systems acknowledge themselves to it14:17
*** rhallisey has joined #openstack-infra14:18
sdagueto be able to understand downloads by people and by ci systems14:18
*** _amrith_ is now known as amrith14:18
*** esker has joined #openstack-infra14:18
*** rhallisey has quit IRC14:22
*** Daisy has joined #openstack-infra14:23
*** esker has quit IRC14:23
*** ajmiller has joined #openstack-infra14:23
*** tiswanso has joined #openstack-infra14:24
mordredtherve: thanks - looking14:24
mordredclarkb: updated - have tested locally that it works with rackspace14:25
mordredclarkb: at least, I have tested that it constructs a swift client that can connect to rackspace with the new parameter choices14:25
mordredclarkb: I have not tested if it helps the token re-get problem14:25
*** abregman is now known as abregman|mtg14:25
*** Daisy has quit IRC14:26
*** rhallisey has joined #openstack-infra14:26
*** thiagop has joined #openstack-infra14:28
*** links has joined #openstack-infra14:28
*** ajmiller has quit IRC14:30
*** rhallisey has quit IRC14:30
*** rhallisey has joined #openstack-infra14:30
openstackgerritMonty Taylor proposed openstack/requirements: Stop using build-dep for numpy/pyyaml depends
mordredtherve: that shold fix it ^^14:33
*** annegentle_ has joined #openstack-infra14:33
thervemordred, Awesome, thanks14:34
*** Nikolay_St has joined #openstack-infra14:34
*** annegentle_ has quit IRC14:34
openstackgerritPaul Belanger proposed openstack-infra/project-config: Import ansible-role-nginx
*** rhallisey has quit IRC14:35
*** scheuran has quit IRC14:35
yolandahi fungi, about the spec.. what we've done now is to create the self signed cert manually. We ended with a CA in west and another in east, because i didn't have the details, config files, etc... to recreate certs on east. My initial idea is to generate a OpenStack Infra CA, self-signed, in my computer. Then create a cert with alt names for east and west. But that's manual, config details are missing, and cannot be reproduced, as they are on my l14:35
*** rhallisey has joined #openstack-infra14:36
yolandaso when we need to do it again, how will we reproduce? create a ca again on laptop, config file, generate more certs?14:36
asselin__why do we install pip packages in puppet instead of using a project's requirements.txt?14:36
openstackgerritBob Ball proposed openstack-infra/puppet-openstackci: Don't require a simple log_server
*** mrmartin has joined #openstack-infra14:36
yolandafungi, why don't we have these config files, ca, etc... on a shared host like puppetmaster, so we can reuse when we need? not generally for infra, but we may need for infra cloud now, or for other future uses14:37
*** sabeen has joined #openstack-infra14:37
*** moravec has joined #openstack-infra14:38
pabelangerasselin__: install a new version not hosted on pypi?14:38
*** akshai has joined #openstack-infra14:38
*** mrmartin has quit IRC14:38
asselin__pabelanger, you cannot specify a repo in requirements.txt?14:39
*** nmagnezi has quit IRC14:39
*** ociuhandu has joined #openstack-infra14:39
*** annegentle_ has joined #openstack-infra14:39
rcarrillocruzasselin__: you mean by repo a git repo in requirements?14:39
pabelangerasselin__: right, maybe we 'infra' just want to dogfood from master over the release on pypi14:40
asselin__rcarrillocruz, yes14:40
rcarrillocruzyou can, it's called 'editable packages'14:40
pabelangerasselin__: I do that with diskimage-builder.  I install it from git, even though nodepool has it in requirements14:40
rcarrillocruzyou can for example install from let's say github/lolrepo , commit foo14:40
rcarrillocruzpabelanger: i'm intrigued by letsencrypt, looks pretty straightforward and easy to automate14:41
asselin__pabelanger, so how does puppet manage that? You have to put all of nodepools requirmeents in puppet?14:41
*** sabeen3 has joined #openstack-infra14:41
rcarrillocruznot sure what infra folks think about relying on that service tho14:41
*** rhallisey has quit IRC14:41
pabelangerrcarrillocruz: Ya, we've talked about it before. I think somebody just needs to do the work to try it out14:41
*** sabeen has quit IRC14:41
pabelangerasselin__: No, I only care about diskimage-builder and shade. I'm mostly dogfooding from master on them, while using nodepool from pip.14:42
openstackgerritMerged openstack/requirements: Updated from generate-constraints
yolandaletsencrypt was raised on some meeting, i think cody-somerville proposed that, but this was discarded14:43
*** alivigni has joined #openstack-infra14:44
pabelangeryolanda: happen to have an link do that discussion?14:44
yolandai can check14:45
asselin__pabelanger, is raising the issue. How do we manage what to get from pip and what to get from repos?14:45
fungiyolanda: we can't just use self-signed certs and put those certs/keys in hiera?14:45
yolandai think it was on prev 2 meetings14:45
*** nmagnezi has joined #openstack-infra14:45
yolandafungi yes, but the thing is, what's the way to generate them? manually from our laptops?14:45
yolandai think it will be better to have something in infra on place14:45
yolandaso we can share same CA, same key, config14:46
fungithe "let's use self-signed certs" decision was really meant to imply "let's not rely on a certificate authority for this" and not "let's run our own certificate authority for this!"14:46
yolandaok if you want to change the intention of the spec14:46
fungiyolanda: usually i just generate them on the puppetmaster14:46
fungiand then copy the results into hiera14:46
yolandaso yes, generate that on pupetmaster14:46
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check
fungiand let puppet put them onto the machines that need them14:46
yolandabut have the config on place, automated14:46
*** annegentle_ has quit IRC14:46
yolandato generate a CA you need to have openssl.cnf, same for intermediate14:47
*** cbader has joined #openstack-infra14:47
yolandaa CA key, etc.14:47
*** fhubik_brb is now known as fhubik14:47
*** nwkarsten has joined #openstack-infra14:47
BobBallasselin__: Is there a reason why we have a project_config_repo but no project_config_revision at the top level?  I see that project_config_revision is defaulted to 'master' in zuul.pp but the other classes that use project_config_repo don't seem to have a revision?14:47
yolandaso i propose at least have this setup on puppetmaster14:47
fungiyolanda: why do you need an intermediate/chain cert in this case? there's nothing to chain14:47
yolandathe same thing we do unofficially, have it on our repos, officially seutp14:47
fungia self-signed cert is exactly that. the private key is used to sign its own public component14:48
asselin__BobBall, I think that's something that started being added later. there's also the 'base' parameter14:48
fungithere's nothing to trust beyond the key associated with the cert14:48
yolandait was setup like that on prev versions with crinkle/nibalizer so i generated and intermediate as well on east. And our infra cloud uses chain files14:48
pabelangerasselin__: For me, it is at the discretion of the user using puppet IMO. If they want to rev up dependencies at their own pace, why stop them.  There is nothing stopping a user from doing pip install diskimage-builder even before running puppet. By the time puppet installs nodepool, it won't need to worry about diskimage-builder download, because it already exists on the system.  Assuming both upper14:48
pabelangerand lower requirements are met14:48
fungiyolanda: that's pure cargo-culting14:49
fungion everyone's part apparently14:49
rcarrillocruzhmm, so from now on, we shouldn't really put intermediate chain files14:49
pabelangerasselin__: especially because we are dogfooding a lot of projects from master, I really think both shade and diskimage-builder should be moved out of the nodepool class into their own.  Then allow the user to choose how to install them14:49
fungircarrillocruz: you should use them when necessary14:49
yolandafungi, that's how puppet is setup on infra cloud right now14:50
rcarrillocruzcos spinning up openstackci in test environments i found them just a pita14:50
fungircarrillocruz: not all certificate authorities need a chain cert, and self-signed certs _never_ need a chain cert because there's nothing to chain back to14:50
yolandaanyway, even if we just use our own CA, it should be nice to have a shared place for it14:50
yolandaso every self-signed cert shares the same14:50
fungircarrillocruz: some certificate authorities need chain certs because their trust is delegated from some other certificate authority14:50
openstackgerritMerged openstack/requirements: bump openstackclient in u-c to 2.1.0
openstackgerritMerged openstack/requirements: Bump futurist to 0.11.0
asselin__pabelanger, sound like you agree with my suggestion in that patch then right?14:50
*** zeih_ has joined #openstack-infra14:50
fungiyolanda: self-signed certs don't share an anything14:51
fungithey're self-signed14:51
*** abregman|mtg is now known as abregman14:51
funginot signed by a ca14:51
yolandafungi ok, then we need to change several things on our infra cloud setup. Can we make sure we are all on the same page?14:52
pabelangerasselin__: Right, I think moving them out is a good idea. I already do that with ansible (ansible-role-diskimage-builder). Then simply have nodepool include that role.  The same concept could apply to our puppet modules14:52
pabelangereg: puppet-diskimage-builder14:52
yolandafungi, what do you think about a self-signed cert with alt names? for * and *.hpuswest.ic.openstack.org14:52
fungiyolanda: what else needs to change beyond what was decided in the meeting when we said we were going with self-signed certificates?14:52
fungiyolanda: subject-altnames is fine, sure14:52
*** julim has joined #openstack-infra14:52
yolandafungi, well , puppet code needs to change, hiera values need to change14:53
fungior just *.ic.openstack.org14:53
yolandafungi, tested that one and was buggy, because we are on two levels14:53
yolanda* doesn't match controller.hpuseast.ic.openstack.org14:53
yolandaalt names worked fine14:53
fungiahh, then sure, subject-altnames is the way to go14:53
*** dizquierdo has quit IRC14:53
yolandaso this needs agreement with crinkle and nibalizer, because it needs changes14:54
fungii forgot some libraries don't consider wildcards to validate below one subdomain level14:54
yolandapart of the spec comes from the conversations we've had these days about how to handle certs14:54
fungii thought they agreed in the infra meeting?14:54
pabelangeryolanda: if you are using ubuntu, can you use the ssl-cert package and just setup snakeoil certs?14:54
pabelangereg: make-ssl-cert generate-default-snakeoil --force-overwrite14:54
fungior did they not understand what self-signed certs actually are?14:54
*** erlon has joined #openstack-infra14:55
yolandafungi, looks as we missed some bits there14:55
fungii wonder if everyone fixated on the not-buying-certs part of that discussion rather than the actual proposal which was to explicitly use self-signed certs (i don't mind buying certs when they're needed, but this use case didn't seem to need them)14:56
*** ccrouch has joined #openstack-infra14:56
*** rbrndt has joined #openstack-infra14:56
*** fhubik is now known as fhubik_brb14:57
yolandafungi the idea i ended having, from the conversation + the current puppet code, was have a CA for infra, and i think the other people working in infra cloud had the same idea14:57
fungiregardless, i'm probably missing something that makes the infra-cloud case not a matter of just having puppet apply blobs from hiera to create the usual certificate, key (and optionally chain cert when needed) files on the systems which need to serve them14:57
fungioh, and the tricky bit, which is extending the trust to include the self-signed certs for things it needs to connect to14:58
fungibut that same need exists even if you set up your own ca14:58
yolandafor extending the trust i created a patch
openstackgerritClaudiu Belu proposed openstack/requirements: Removes posix-ipc from global-requirements
yolandaand we've found today a bug in neutron with self signed certs, people in puppet-openstack are fixing now14:58
fungiawesome. i love it when the things we build are able to surface bugs in other projects they didn't know they had14:59
yolandawe are amazing beta-testers15:00
rcarrillocruzi think this is the great thing about the infracloud15:01
rcarrillocruzis that we dogfood a lot of stuff15:01
AJaegerfungi, sdague : Could you review a pep8/linters change for me, please?
rcarrillocruzso it's a win-win for the whole project15:01
*** burgerk has quit IRC15:01
*** zeih_ has quit IRC15:02
fungieveryone working on infra-cloud is doing an amazing job of finding rough edges in openstack. it's one of the most exciting things about the opportunity to build this15:02
*** tiswanso has quit IRC15:02
*** eharney has quit IRC15:02
*** Guest65333 is now known as nadya15:02
*** tiswanso has joined #openstack-infra15:03
pabelangerrcarrillocruz: hopefully we'll be able to dogfood other OSes at some point in the future :)15:03
rcarrillocruzi heard you on the meeting, yup15:03
openstackgerritMax Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository
*** nadya is now known as Guest3450715:03
*** Guest34507 is now known as _nadya_15:03
pabelangerrcarrillocruz: I actually download puppet-infracloud the other day and started adding rhel support.  But haven't pushed up any patches yet. Holding off until sprint to see if there is interest15:04
rcarrillocruzi believe we should probably focus on making it work and really automate the whole thing. We have way too many pieces and cherry picked changes wandering around on the jumphosts15:06
rcarrillocruzthen discuss HA15:06
rcarrillocruzthen discuss dogfooding other stuff15:06
rcarrillocruzthat's just me15:06
openstackgerritClaudiu Belu proposed openstack/requirements: Adds PyMI as a global-requirement
*** jaosorior has quit IRC15:06
*** piet has joined #openstack-infra15:06
yolandaso yes, right now manifest cannot work without patches. Several fixes going on15:06
*** sc68cal has joined #openstack-infra15:07
mriedemis the elastic-recheck bot dead?15:09
mriedemi'm not seeing it commenting on known tracked failures15:09
mriedemclarkb: ^?15:09
*** sshnaidm has quit IRC15:10
fungimriedem: i can look in a sec and see if it's throwing errors or something15:10
fungi2016-02-11 14:50:30,168 INFO recheckwatchbot: Looking for failures in 244328,20 on gate-tempest-dsvm-neutron-full15:11
fungiseems to be active at least15:11
*** annegentle_ has joined #openstack-infra15:11
*** sdake_ has joined #openstack-infra15:11
fungithe bot seems to be running since yesterday. looks like it got restarted, maybe by a new patch?15:12
*** piet has quit IRC15:12
*** fhubik_brb is now known as fhubik15:12
mriedemi guess it did show up this morning on something
*** piet has joined #openstack-infra15:12
mriedemElastic Recheck6:49 AM↩Patch Set 25:I noticed jenkins failed, I think you hit bug(s):15:12
fungiyeah, there was a pattern removal change which landed yesterday, so puppet restarted it for that i guess15:13
anteayaanyone know why patchbot it in -meeting?15:15
*** Daisy has joined #openstack-infra15:16
*** sdake_ is now known as sdake15:16
jeblairif someone wants to run a bot in meeting, they can add it to puppet15:17
*** sigmavirus24_awa is now known as sigmavirus2415:17
openstackgerritMerged openstack-infra/nodepool: Document clouds.yaml as the preferred configuration
asselin__BobBall, does this work?
*** Daisy has quit IRC15:17
jeblairnotmyname: is patchbot you?15:17
*** rhallisey has joined #openstack-infra15:18
*** arthus has quit IRC15:19
*** ig0r_ has quit IRC15:21
openstackgerritMax Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository
*** eharney has joined #openstack-infra15:21
jeblairnotmyname: if you want to run bots in common openstack channels, would you mind please adding them to the openstack-infra configuration?  you can find information on the relevant files here:
*** rhallisey has quit IRC15:23
jeblairnotmyname: in the mean time, i have removed patchbot from the meeting channel15:23
anteayajeblair: thank you15:23
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Rename fuel-plugin-astra to fuel-plugin-astara
*** ddieterly has quit IRC15:24
*** sshnaidm has joined #openstack-infra15:25
*** burgerk has joined #openstack-infra15:26
anteayayour kidding15:27
anteayaanother rename patch15:27
BobBallasselin__: I assume so - but TBH I uploaded too early; currently having difficulties with the deploy elsewhere.15:27
reedmore spam on the wiki15:27
jeblairsomeone should add a spellchecker lint that detects astara15:27
asselin__BobBall, my intuition says it won't work as-is.15:28
anteayareed: can you get rid of it? I recall last time you did15:28
anteayajeblair: agreed15:28
reedanteaya, I sure can :)15:28
anteayamarkmcclain: folks are having difficulty with the proper spelling of your repo name15:28
anteayamarkmcclain: just as a heads up15:29
mordredanteaya, markmcclain: although to be fair it's still a better name than jeepyb15:29
*** angdraug has joined #openstack-infra15:29
anteayait is, jeepyb is a terrible name, but it doesn't get mis-spelled15:30
anteayareed: thank you15:30
clarkbor renamed every other week15:30
anteayathat too15:30
*** daemontool_ has joined #openstack-infra15:32
*** jsavak has quit IRC15:32
*** dpaterson has quit IRC15:32
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Check for astra misspelling
AJaegeranteaya, jeblair: Not sure whether the above works ^15:33
*** lezbar has quit IRC15:33
anteayaha ha ha15:33
*** ajmiller has joined #openstack-infra15:33
anteayaalways at the ready with a linter change15:34
AJaegeranteaya, we have simple rules for Openstack etc ;)15:34
anteayawish I knew what they were15:34
AJaegeranteaya: check the change, the two lines above mine15:34
*** daemontool has quit IRC15:34
rbradforhi infra, I just wanted to know following on from the infra-cloud spec, puppet-infracloud cloud and infra-cloud topic details I've been reading what is a good next point to become more aware of work progress.  Is reviewing the -infra channel history the best place to look next?15:34
*** achanda has joined #openstack-infra15:34
*** kozhukal` has joined #openstack-infra15:35
anteayaAJaeger: nice15:35
*** rhallisey has joined #openstack-infra15:35
*** sshnaidm has quit IRC15:35
clarkbrbradfor: our meeting logs typically have updates (and for infracloud links to various changes and docs too)15:35
AJaegerI fear the check is only for the description...15:35
*** zul_ has joined #openstack-infra15:36
AJaegerindeed ;(15:36
*** kozhukalov has quit IRC15:36
*** zul has quit IRC15:36
anteayaAJaeger: :(15:36
rbradforclarkb, great thanks, meeting logs would be a much smaller volume place to start.15:36
AJaegerdescription is misspelled as well, so might have helped...15:36
*** woodster_ has quit IRC15:36
*** kozhukalov_ has quit IRC15:36
anteayaAJaeger: might have15:37
AJaegerexactly: *might* ;)15:37
anteayarbradfor: have you spent much time talking with crinkle?15:37
*** Daisy has joined #openstack-infra15:37
anteayaor nibalizer?15:37
*** piet has quit IRC15:37
anteayahere is the current etherpad for the infra cloud sprint:
rbradforanteaya, no, I've met crinkle and nibalizer  in Vancouver and I've seen the puppet work via repos only.15:38
*** FallenPegasus has joined #openstack-infra15:38
*** placko has joined #openstack-infra15:39
rbradforanteaya, thanks, another resource I'd not looked at to review, Tokyo or midcycle etherpads.15:39
* rbradfor adds these to my reading list15:39
*** rhallisey has quit IRC15:39
anteayaand here is the storyboard story for infra-cloud:!/story/200017515:39
*** mtanino has joined #openstack-infra15:40
*** jsavak has joined #openstack-infra15:40
rbradforanteaya, thanks had read through that15:40
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check
fungireed: i think i reverted the last spam on that page15:41
fungiit seems to be popular15:41
*** Daisy has quit IRC15:41
anteayarbradfor: wonderful15:41
rbradforanteaya, put perhaps I cannot drive storyboard that well, other then story tasks, I didn't find any details?15:41
fungiwas just last week if i'm remembering correctly15:41
anteayarbradfor: well storyboard is also under active development15:42
*** achanda has quit IRC15:42
*** fawadkhaliq has quit IRC15:42
anteayarbradfor: so there are some gaps in its functionality, one of them being the ablity to split out tasks within a story and add details to a specific task rather than the story as a whole15:42
anteayaZara: ^^15:43
*** fawadkhaliq has joined #openstack-infra15:43
*** edmondsw has joined #openstack-infra15:43
fungiworth noting, launchpad doesn't let you add task-specific details either15:43
anteayarbradfor: so I think you have done a lot of reading and you might be at the point of participating in discussion15:43
fungicomments in lp are on the entire bug, regardless of how many bugtasks it has15:44
* anteaya notes fungi's point about launchpad functionality15:44
anteayaso parity15:44
rbradforanteaya, I'll read the meeting notes from clarkb recommendation and I'll join the weekly -infra meeting to keep abreast more15:44
markmcclainanteaya: haha yeah I think lots of autocorrect change it astor15:44
rbradforanteaya, thank you for point me in the right direction15:44
anteayamarkmcclain: good to know15:44
funginame idea for my next project15:45
anteayamarkmcclain: mind adding that to
anteayaI like asteroids15:45
*** tiswanso has quit IRC15:45
yolandarbradfor, so one part of the work of infra cloud is work with bifrost, ironic, glean...for the deploy of servers. Then another work on top of that is ansible + puppet to install manifests for controller and compute, reusing puppet-openstack15:45
AJaegeranteaya: let me update my change, just tell me what to add.15:45
AJaegeranteaya: I'm sure it won't pass the testsuite as is...15:45
anteayarbradfor: wonderful, thanks for wanting to get involved, we need more folks like you helping out15:45
anteayarbradfor: thank you15:46
*** tiswanso has joined #openstack-infra15:46
rbradforyolanda, thanks. other than knowing the names of bifrost, ironic and glean I've never read in detail or tried to use. Seems a next starting point after I learn more about them.15:46
anteayaAJaeger: well markmcclain pointed out astor is a popular choice of autocorrect15:46
*** rhallisey has joined #openstack-infra15:46
AJaegeranteaya: yes - just added to my patch. And now I test locally ;)15:47
Zararbradfor: yolanda: there are also storyboard patches in review for linking resources to tasks:
anteayaAJaeger: awesome15:47
anteayaZara: knew you were working on something, yay!15:47
anteayaZara: you are awesome \o/15:47
* anteaya is looking forward to cake next week15:47
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance
*** placko has quit IRC15:48
Zararbradfor: yolanda: so those don't allow task descriptions within storyboard itself, but where there are more details than would fit sensibly in the story description, a user can link to those.15:48
*** ddieterly has joined #openstack-infra15:48
Zaraanteaya: aw, thank you! :)15:48
Zaraand me too! :)15:48
yolandaZara that's great, it's something i miss sometimes, the ability to explain the tasks deeper15:49
* fungi is missing out on teacakes15:49
*** jlanoux has quit IRC15:49
mordredfungi: NEVER skip teacakes15:49
yolandabtw, fungi, testing a deploy of a controller with self-signed certs, going to see what breaks15:49
fungibut best wishes for the storyboard midcycle15:49
* anteaya reviews Zara's posted patches15:49
clarkbpabelanger: +1 to snakeoio if you can get away with them15:50
clarkbso simple15:50
fungiyolanda: no doubt there's likely something. it sounds like there are multiple places the cert needs to be added to the trust set15:50
*** dangers_away is now known as dangers15:51
*** rhallisey has quit IRC15:51
AJaegeranteaya: added and fixed15:51
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Check for astra misspelling
*** kozhukalov_ has joined #openstack-infra15:52
*** amotoki has quit IRC15:52
fungibut if this gets worked out, it's a big win for anyone who wants to puppet a cloud test/poc deployment without bothering to get certs from an outside ca and without having to set up a ca of their own15:52
*** rubasov has joined #openstack-infra15:53
Zarayolanda: yes, it was really annoying me when I was updating stories! I mainly planned it as a way for people to link to things in gerrit and say 'patch here' (I was sick of putting that in descriptions/comments), but it should also work for docs.15:53
*** GonZo2000 has quit IRC15:53
anteayaAJaeger: wonderful thank you15:53
*** EmilienM has quit IRC15:53
*** dpaterson has joined #openstack-infra15:54
anteayaAJaeger: I think astara lowercase is acceptable, would you agree markmcclain?15:55
*** HenryG has quit IRC15:55
*** niska has quit IRC15:55
*** sshnaidm has joined #openstack-infra15:55
*** rhallisey has joined #openstack-infra15:55
AJaegeranteaya: it's just for that single file gerrit/projects.yaml...15:56
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone
*** jcoufal has quit IRC15:57
anteayaAJaeger: oh and just for the description15:57
fungii'm convinced we should be maintaining a global project name spell-checker, but i'll let someone else fight that battle15:57
anteayaAJaeger: okay15:57
fungier, not convinced15:57
*** infra-red has quit IRC15:58
clarkbTIL keystone uses rabbit15:58
anteayafungi: well this is the third time we are renaming astara things15:58
anteayaI'm open to options15:58
anteayafolks can't seem to spell it correctly, including me15:58
fungiat least we already had plans to rename some stuff tomorrow15:58
fungiwhich reminds me, i forgot i have an appointment with my accountant tomorrow afternoon, but i should still be back in time for the maintenance. i just won't be around for a couple hours prior15:59
rcarrillocruzthat's probably because of opel astra :D15:59
rcarrillocruzjust guessing15:59
*** rhallisey has quit IRC15:59
*** achanda has joined #openstack-infra15:59
anteayaand glad I saw the fuel rename patch as the person proposing it didn't add it to the wikipage:
anteayafungi: happy taxes16:00
*** mriedem is now known as mriedem_meeting16:00
*** rcarrillocruz has left #openstack-infra16:00
*** jordanP has joined #openstack-infra16:00
anteayazaro: did you get anywhere testing renaming on new gerrit?16:00
*** tiswanso has quit IRC16:00
*** GonZo2000 has joined #openstack-infra16:00
*** GonZo2000 has quit IRC16:00
*** GonZo2000 has joined #openstack-infra16:00
anteayazaro: any new steps or changed steps we should be aware of?16:01
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: rename ceilometer-specs to telemetry-specs
*** moravec has quit IRC16:01
* AJaeger just rebased the change above ^16:01
*** tiswanso has joined #openstack-infra16:01
*** HenryG has joined #openstack-infra16:02
*** achanda has quit IRC16:02
anteayaI see that16:02
anteayathank you16:02
*** kozhukal` has quit IRC16:04
*** links has quit IRC16:04
AJaegeranteaya: do you have time to review 278633, 278641, and 278627, please?16:04
*** kozhukalov has joined #openstack-infra16:05
*** kzaitsev_mb has quit IRC16:05
sdagueAJaeger: before I +A - whats the jjb job count on that?16:06
sdagueoh, I guess it's only a couple of instances16:06
*** EmilienM has joined #openstack-infra16:06
anteayaAJaeger: only if they are rename patches16:06
anteayaI want to get the rename things done without interruption if i can, so I can actually finish it16:07
*** moravec has joined #openstack-infra16:07
*** piet has joined #openstack-infra16:07
*** niska has joined #openstack-infra16:07
*** kzaitsev_mb has joined #openstack-infra16:07
*** jpr has joined #openstack-infra16:09
*** mattt has left #openstack-infra16:09
*** shardy has quit IRC16:10
*** kushal has quit IRC16:10
AJaegersdague: only touches zuul/layout16:10
AJaegerthanks, sdague.16:10
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Extend experimental-bindep-jobs to 14 projects
AJaegeranteaya: they're not.16:11
*** kushal has joined #openstack-infra16:11
AJaegerthen let me ask sdague for 278641, and 278627 as well, please ;)16:11
*** woodster_ has joined #openstack-infra16:11
AJaegerfungi: do you have a spec or larger explanation of what you do with the bindep changes?16:12
*** smanjara has quit IRC16:12
fungianteaya: AJaeger: looks like the author of that new rename patch replied to the maintenance announcement letting us know at least16:12
fungiAJaeger: this is part of the diskimage-builder rollout for nodepool images16:12
fungiAJaeger: need to make sure we can run jobs on dib-built ubuntu-trusty so we can get rid of bare-trusty16:12
anteayaAJaeger: okay thanks16:13
anteayafungi: well that is something, thanks16:13
clarkbfungi: sdague appears to be what caused the failure of the preallocated swapfile gate run last night16:13
fungiAJaeger: but confirming it with a bunch of experimental equivalents of jobs on major projects before we start switching over the main job templates to use these builder macros16:13
crinklefungi: yolanda the reason we've been using a CA and not just self-signed certs is that we were adding the CA to the system-trusted certificates, which was necessary because the puppet-openstack modules to not just roll with insecure certs16:14
clarkbfungi: sdague does that look like lack of disk space? the worlddump at the end of the run shows plenty of disk but that may be because tempest has stopped running16:14
yolandamordred, greghaynes, SpamapS, can you review ? it is the element for infra-cloud bridge16:14
fungicrinkle: you can't put the self-signed certs in as a "ca"?16:14
*** cbader has quit IRC16:14
*** rhallisey has joined #openstack-infra16:14
yolandacrinkle, i'm trying adding the cert on the ca.crt16:14
yolandalet's see if that works fine16:14
fungibecause they are, they're just a ca that only ever signed one cert (which happens to be themselves)16:14
*** vgridnev has quit IRC16:15
*** cbader has joined #openstack-infra16:15
AJaegerfungi, so part of - a spec that has far too many holes ;)16:15
crinklefungi: no I hadn't tried, didn't know that was a possibility16:15
fungicrinkle: there's really nothing special about a self-signed cert16:15
yolandafungi crinkle, so deploying with cert in chain file field now, so it adds it to the global trsut16:16
fungiit's still a "certificate authority" in the strictest sense16:16
*** dizquierdo has joined #openstack-infra16:16
fungijust happens to serve as its own certificate authority16:16
yolandabtw crinkle, i needed to add this to manifest to work
openstackgerritMerged openstack-infra/project-config: Use pep8 for doc projects
mordredfungi: ooh! I didn't realize we were that close to being able to test dib-ubuntu-trusty for everyting16:16
mordredfungi: that's exciting16:17
fungithe only place i would expect that to not work as intended is if something is explicitly disallowing self-signed certs (a certificate which is signed by a certificate authority that happens to be its own key)16:17
mordredthese certs are for?16:17
crinklefungi: yolanda in any case we've found that on liberty self-signed or signed-by-self-signed-ca certs don't work anyways16:17
clarkbmordred: the openstack services in infra cloud16:17
yolandacrinkle, i found a bug in neutron today, they are fixing it now16:17
mordredright - for the API endpoints?16:17
yolandadegorenko and EmilienM16:17
fungimordred: we've been that close for a while, i just hadn't gotten around to broadening the validation to more jobs, and nobody else has had time to pick it up and run with it either, so i finally found a few minutes16:18
clarkbmordred: yes16:18
mordredfungi: neat!16:18
mordredclarkb: nod16:18
crinkleyolanda: weren't you having issues with openstackclient too?16:18
EmilienMyolanda: nice spot!16:18
yolandacrinkle, i fixed those. I didn't notice that my key had a pass16:18
mordreddare I ask why we don't just get real certs? I'm guessing it's in the scrollback16:18
yolandaso openstackclient was failing for that16:18
mordredI mean, we're talking like 5 certs per cloud at most16:18
crinkleyolanda: ah ok16:18
yolandawhen i created that passwordless, opensatck client started working, the failure is on neutron16:18
fungicrinkle: can you elaborate? i believe that's just a matter of dealing with the fact that requests provides its own trust set rather than looking at the system trusts by default16:19
yolandafungi, crinkle, i found that exporting REQUESTS_CA_BUNDLE can help16:19
mordredfungi: for intra-cloud interactions, there is no way to configure things to use specific certs16:19
mordredfungi: this is a known problem with openstack services talking to each other16:19
crinklefungi: it sounds like yolanda figured it out16:19
mordredand with the way requests has decided to behave16:19
*** bdemers has joined #openstack-infra16:19
clarkbmordred: you can do the requests override16:20
fungimordred: right, you'd have to add them to the trusted certificate authorities16:20
clarkband do ^16:20
mordredyah - but without exporting the env var wrapped around the API service start16:20
fungibut my point was that's going to be the same problem whether you use self-signed certs or decide to run your own certificate authority16:20
mordredthere is no config file value to pass in the requests ca bundle args16:20
yolandaexport REQUESTS_CA_BUNDLE=/etc/ssl/ca-certificates.crt helped with requests16:20
mordredfungi: yes16:20
mordredfungi: which is why I was wondering why we don't just buy cheap certs16:21
bdemersCan I get a new channel created? "#openstack-networking-cisco" Is there anything I should do, other then just request it here ?16:21
*** bnemec has quit IRC16:21
jeblairmordred: well, honestly, this ought to work16:21
crinkleyolanda: in the puppet run?16:21
jeblairmordred: i think it's worth exploring to see if we can do it or if not, what's so hard about it16:21
mordredjeblair: it does not- the support for this in openstack is broadly broken16:21
fungiit seems like punting on something we could make openstack do better16:21
mordredbecause of reuqests16:21
yolandacrinkle or global var, just having it on the env16:21
mordredthis is a known problem16:21
jeblairmordred: yeah, but there's a workaround in requests that maybe could be applied here16:21
crinkleyolanda: okay awesome16:22
*** vgridnev has joined #openstack-infra16:22
mordredso - honestly fixing requests is the right thing -because requests' behavior is broken16:22
yolandamordred, i could make it work properly unti i hit the neutron bug16:22
mordredand the attempts to fix it openstack side involved heaping more workaround on top of more workarounds16:22
jeblairmordred: i understand that it's not a lot of money, but it also doesn't get us any benefit, so that's weird.16:22
clarkbmordred: but requests has explicitly said they will not accept those changes...16:22
mordredfor the fact that requests is categorically broken16:22
*** vgridnev has quit IRC16:22
mordredclarkb: I know16:22
clarkbbdemers: we have it documented somewhere, let me find that link16:22
yolandait was a funny bug, creating a network called InsecureRequestWarning :)16:22
*** [HeOS] has joined #openstack-infra16:23
mordredthe 'right' way to do it in creating the requests Session objects involves passing cacert options to the constructor16:23
anteayaclarkb: ^^16:23
clarkbanteaya: thanks16:23
*** HeOS has quit IRC16:23
mordredbut the various openstack services do not all provide config values in their files for specifying those in the areas where one configures intra-service communication16:24
anteayaclarkb: welcome16:24
anteayaclarkb: everybody is in the cert dustup16:24
mordredin any case - I actually do not think it is valuable to fix this in openstack16:24
jeblairi think it would be useful if they did, and i think it would be a good thing for someone to work on fixing that, even if it takes a little while.16:24
mordredsince it further validates requests ridiculous position of breaking SSL everywhere16:24
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move bindep-fallback to non-voting for trusty
pabelangerfungi: mordred: I thought I had pushed that up already ^16:25
mordredthere is a system on linux for dealing with this16:25
Clintis it called letsencrypt16:25
mordredand it works well16:25
mordredrequests has chosen to break that for everyone16:25
jeblairmordred: i understand the stand on principal, but i don't think it's going to make requests change their mind, so in the mean time, we hurt folks who can and want to operate their own ca16:25
*** jlanoux has joined #openstack-infra16:25
mordredand people doing work to support requsts position is damaging to the entire ecosystem16:25
jeblairmordred: which is potentially a lot of private cloud operators16:26
jeblair(lots of big shops operate their own cas)16:26
jeblair(actually, like *every* big shop operates their own ca)16:26
mordredjeblair: indeed. however- fixing that is a non-small task and one I don't understand why we think we should take on as infra16:26
mordredit involves adding a lot of config options to a lot of openstack services16:26
mordredand piping them through to  a lot of constructor calls16:26
jeblairmordred: i'm glad no one gave you that argument for any of the many quixotic tasks you have started ;)16:27
mordredjeblair: fair enough16:27
*** jordanP has quit IRC16:27
clarkbClint: only if letsencrypt is trusted by requests :/16:27
mordredI just don't understand why of all of the tasks we might want to undertake, we'd choose "fix the openstack workarounds to requests breakage" over "just get 5 real certs"16:27
clarkbClint: which it probably is, but you ahve to explicitly check because of how they manage their cert lists16:28
mordred_especially_ considering that we're running kilo16:28
bdemersanteaya, thanks!16:28
jeblairmordred: i don't fully understand the problem yet16:28
jeblairmordred: this is the first time i've heard about intra-cloud tasks being a problem -- how many points of interaction are there that we would run into?16:28
mgagnemordred: was about to say: so it can benefit to other people but then you said: "but we run kilo" :-/ =)16:28
*** nwkarsten has quit IRC16:29
*** harlowja_at_home has joined #openstack-infra16:29
clarkbjeblair: the big one are goint to be nova + glance, nova + neutron, and nova + cinder16:29
clarkband everyone talking to keystone16:29
*** nwkarsten has joined #openstack-infra16:29
fungimgagne: well, we have liberty in one region and kilo in the other i think16:29
yolandafungi, mgagne, i'm testing on liberty16:30
yolandaas it's not yet on production, i prefered to test the upgrade16:30
pabelangerclarkb: do you have a link to the requests issue about rejecting the changes? Curious to see that16:30
pabelangerclarkb: thanks16:31
*** bpokorny has joined #openstack-infra16:31
fungianyway, i'm still not ruling out the possibility that we may just end up obtaining (and continually keeping on top of renewing) certs for infra-cloud ssl/tls interactions, but at the very least we should be able to pin down where using your own ca (which is what self-signed certs ultimately are, the trust challenges will be the same) breaks for an openstack deployment so that someone else has more16:31
fungiinformation on what needs fixing even if we don't have the bandwidth to solve all the issues ourselves16:31
*** bpokorny has quit IRC16:31
anteayabdemers: welcome16:32
*** bpokorny has joined #openstack-infra16:32
jeblairfungi: ++ this has all been a surprise to me -- it certainly breaks my expectations of how the software should operate.  getting it documented and raising awareness of the problem is very useful.16:32
*** Swami has joined #openstack-infra16:32
fungialso, i suspect this is actually _not_ a problem for people deploying distro packages of python-requests, as i believe the distros rip out the bundled trust set and point requests at the system-provided one16:32
mgagnefungi: could it be a cross-project initiative? I mean, I don't believe infra should be the only people working on it.16:33
yolandaso i'm testing with self signed now (not with infra ca), and now i'm getting problems16:33
jeblairfungi: oh!16:33
mordredfungi: I recommend reading the openstack-dev thread "Use of self signed certs in endpoints"16:33
yolandafungi, with infra ca i managed to get it work, but with self-signed only, it complains about CA, even if the cert is added to trust16:33
mordredlemme find web link16:33
pabelangerfungi: Ya, believe that is the case. Going to check however16:33
crinkle( and are patches to upgrade btw)16:33
openstackgerritRussell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl.
crinkleyolanda: does that have to do with the config files pointing at a path for a ca?16:34
openstackgerritGábor Antal proposed openstack-infra/gerritbot: Reduced code duplication
pabelangerfungi: Yup: simply patch:
mordredfungi: ^^ that16:34
fungimgagne: infra _is_ a cross-project initiative16:34
*** bpokorny_ has joined #openstack-infra16:34
yolandacrinkle i removed the chainfile of apache16:34
yolandaand testing keystone now, that fails16:34
yolandai will check keystone conf as well16:34
mordredusing distro requests appropriate patches requests to behave like people expect it to behave on linux16:35
*** nwkarsten has quit IRC16:35
*** lmiccini is now known as lmiccini|away16:35
*** sshnaidm has quit IRC16:35
mgagnefungi: didn't meant that you weren't but that people outside the infra team can also contribute so you don't have to do all by yourself16:35
mordredbut pip installed requests does not honor system ssl bundles or additions made to them by admins16:35
clarkbone terrible option would be a pip package that monkey patches requests16:35
jeblairor a simple wrapper around it16:36
mordredclarkb: notmorgan and I actually poked around at that a little while ago16:36
crinkleyolanda: can you paste the errors?16:36
pabelangerwhat is the issue what using python-requests from the OS? Just the version that ships right?16:36
yolandajust with curl curl: (60) SSL certificate problem: unable to get local issuer certificate16:36
yolanda curl --cacert /etc/ssl/certs/ca-certificates.crt
yolandacurl: (60) SSL certificate problem: unable to get local issuer certificate16:36
jeblairmordred: i'm reading your thread now -- as you point out, we already "wrap" requests :)16:36
mordredpabelanger: mixing distro installed python packages and pip installed python packages leads to pain16:37
*** skath_ has quit IRC16:37
*** mkwiek has quit IRC16:37
mordredjeblair: we do - except we don't use that wrapper everywhere yet16:37
*** alkhodos has joined #openstack-infra16:37
yolandacrinkle, i pasted the full crt as trusted cert16:37
*** rguillebert has quit IRC16:37
*** nmagnezi has quit IRC16:37
mordredthis gets easier once we're done getting all of the projects to use keystoneauth116:37
mordredbut they don't yet (and certainly don't in kilo)16:37
*** samyewell has quit IRC16:37
*** tphummel has joined #openstack-infra16:37
pabelangermordred: true. But I thought we were using packages for infracloud? Or are we pip installing deps?16:37
*** ams_ has quit IRC16:37
*** bpokorny has quit IRC16:37
yolandapabelanger packages16:37
*** nikhil_k has joined #openstack-infra16:38
jeblairmordred: would we still need them each to pass in parameters?16:38
*** jbryce has quit IRC16:38
*** ganso has quit IRC16:38
*** ashtokolov has quit IRC16:38
*** tpsilva has quit IRC16:38
mordredjeblair: once we get keystoneauth to be the way services talk to each other, we could add a monkeypatch patch to ksa16:38
*** igorbelikov has quit IRC16:38
mordredjeblair: and then the need for parameters goes away16:38
*** wendar_ has joined #openstack-infra16:38
clarkbto clarify infra cloud should usedistro requests16:38
*** zhenguo_ has quit IRC16:38
mordredbecause as an admin you can just add ca bundles to the system16:38
*** Ng has quit IRC16:38
*** tpeoples has quit IRC16:38
*** comstud has quit IRC16:38
mordredif we're using distro requests16:38
jeblairyolanda: so that means the intra-cloud stuff can be taken care of by system ca-certs, and it's just when we use something external to that (eg nodepool) where we would have a problem?16:38
pabelangerclarkb: right, that's what I would expect16:39
clarkbbecause that is how the puppet stuff is installed16:39
*** mwhahaha has quit IRC16:39
mordredthen we should be able to add ca bundles to the system CA16:39
jeblair(intra-infra-cloud :)16:39
*** betherly has quit IRC16:39
*** jmccrory has quit IRC16:39
*** wendar has quit IRC16:39
*** piet has quit IRC16:39
*** mrmartin has joined #openstack-infra16:39
yolandajeblair i could make it work with and Infra CA16:39
*** asahlin has quit IRC16:39
*** ayoung has quit IRC16:39
*** nikhil has quit IRC16:39
mordredjeblair: yes. and telling nodepool to deal with the self-signed cert is easy16:39
anteayano spelling mistake possibilies there16:39
yolandabut fails for me with a simple self signed16:39
jeblairmordred: yep16:39
mordredyolanda: yah - because you need the CA to be able to add it ot the system CA bundle16:39
pabelangerYa, I think update-ca-certificates should work once the cert is in the proper location16:40
mordredpabelanger: ++16:40
jeblairyolanda: if you add the self-signed cert to the ca-bundle it doesn't work?16:40
*** rguillebert has joined #openstack-infra16:40
yolandamordred, but if i go with a simple self signed, i don't have the CA16:40
*** ganso has joined #openstack-infra16:40
*** skath_ has joined #openstack-infra16:40
yolandajeblair, no, i added16:40
*** krtaylor has quit IRC16:40
*** mkwiek has joined #openstack-infra16:40
yolandabut fails16:40
jeblairfungi: have i just moved my brain to where you were like 45 minutes ago?16:40
*** ashtokolov has joined #openstack-infra16:40
Clintclarkb: it does as of v2.8.116:40
yolandain nodepool we don't have the problem, because we can pass the cert, or the insecure flag16:41
*** armax has joined #openstack-infra16:41
mordredyolanda: ++16:41
*** ashleighfarnham has joined #openstack-infra16:41
yolandapain is to make it work with self signed, i thoguht that adding that to CA will do the trick16:41
yolandaat least is documented like that16:41
*** salv-orl_ has joined #openstack-infra16:41
*** asahlin has joined #openstack-infra16:41
*** betherly has joined #openstack-infra16:41
*** samyewell has joined #openstack-infra16:41
*** igorbelikov has joined #openstack-infra16:42
jeblairyeah, this seems like a fairly constrained problem -- we know adding *something* to the bundle should work, and it's a matter of working out if there's something unusual about adding a self-signed cert vs a ca-cert16:42
*** esker has joined #openstack-infra16:42
fungiyolanda: does /etc/ssl/certs/ca-certificates.crt include the self-signed cert?16:42
*** sridhar_ram has joined #openstack-infra16:42
yolandafungi yes16:42
*** scheuran has joined #openstack-infra16:42
*** mwhahaha has joined #openstack-infra16:43
*** tpeoples has joined #openstack-infra16:43
*** Ng has joined #openstack-infra16:43
*** piet has joined #openstack-infra16:43
fungisorry, too many conversations in different places, catching up again16:43
*** isaacb has quit IRC16:43
*** jmccrory has joined #openstack-infra16:44
*** jbryce has joined #openstack-infra16:44
*** comstud has joined #openstack-infra16:44
*** zhenguo_ has joined #openstack-infra16:44
*** salv-orlando has quit IRC16:44
yolandafungi, even if i instruct curl to use it directly i get the problem curl --cacert /etc/ssl/certs/
yolandacurl: (60) SSL certificate problem: unable to get local issuer certificate16:45
*** ams_ has joined #openstack-infra16:45
*** mikelk has quit IRC16:45
openstackgerritMax Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository
fungiyolanda: implies what you're doing should work16:46
*** dizquierdo has quit IRC16:46
*** scheuran has quit IRC16:47
*** mrmartin has quit IRC16:47
yolandamm, it says to add the name to /etc/ca-certificates.conf16:47
*** dtantsur is now known as dtantsur|afk16:47
openstackgerritBrian Demers proposed openstack-infra/project-config: Adding new channel #openstack-networking-cisco to bot lists
*** hdd has joined #openstack-infra16:48
madornfungi: do you by any chance know the status of getting the openstack-dev/sandbox +2 issue was this gerrit upstream bug btw:
yolandait that's on the bundle it shouldn't be like that16:48
Clintif you put it into /usr/local/share/ca-certificates you don't have to do that16:48
Clintbut in that case you need to make the extension .crt16:48
fungiyolanda: well, that's if you're not explicitly pointing to the bundle with --cacert16:49
*** smanjara has joined #openstack-infra16:49
yolandai tried several options an no one worked16:49
*** piet has quit IRC16:49
pabelangerClint: then update-ca-certificates16:49
fungimadorn: i believe zaro indicated that the cleanup will require a gerrit outage. we're planning on for friday afternoon, so perhaps we should try to squeeze that into our maintenance plan16:49
pabelangerthat's what I used to do16:49
yolandaopenssl_client also shows Verify return code: 21 (unable to verify the first certificate)16:49
Clintpabelanger: right16:49
fungier, planning one16:50
madornfungi: great thank you!16:50
*** nikhil_k is now known as nikhil16:50
pabelangeryolanda: are we able to download the cacert to import locally?16:50
yolandapabelanger yes, i can provide the details16:50
Clintmaybe it's checking the CA flag in the cert16:51
yolandagoing to paste key and cert16:51
fungimadorn: that is to say, it's my understanding the problem won't recur (not because of the upstream gerrit fix but because of the workaround we added to jeepyb to avoid triggering the bug), however it still left a mess in some dozens (hundreds?) of repositories which need to be cleaned up16:51
fungione of which is the openstack-dev/sandbox repo16:51
fungionce we complete that offline cleanup, voting in those repos should work properly again16:51
yolandapabelanger, or any others
*** r-daneel has joined #openstack-infra16:52
*** ayoung has joined #openstack-infra16:52
*** amotoki has joined #openstack-infra16:52
*** mriedem_meeting is now known as mriedem16:52
*** tpsilva has joined #openstack-infra16:53
*** kzaitsev_mb has quit IRC16:53
*** kzaitsev_mb has joined #openstack-infra16:53
prometheanfireanyone able to look at (dib)16:55
*** amrith is now known as _amrith_16:55
*** moravec has quit IRC16:55
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for horizon connection refused test bug 1544619
openstackbug 1544619 in OpenStack Dashboard (Horizon) "test_create_delete_instance race fails in gate-horizon-dsvm-integration with "error: [Errno 111] Connection refused"" [Undecided,Confirmed]
*** mrmartin has joined #openstack-infra16:56
*** rcernin has quit IRC16:56
*** bnemec has joined #openstack-infra16:56
yolandamm, i'm reading something about copying hash of cert16:56
yolandagoing to try16:56
openstackgerritMartin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0
*** krtaylor has joined #openstack-infra16:57
*** daemontool_ has quit IRC16:57
*** slogan621 has quit IRC16:58
madornfungi: thanks16:58
*** amotoki has quit IRC16:58
*** esker has quit IRC16:58
*** moravec has joined #openstack-infra16:59
*** nelsnels_ has quit IRC16:59
fungizaro: when you're around, we should see if we can squeeze cleanup of all those rogue "registered users" groups into the maintenance plan for tomorrow16:59
*** nelsnelson has joined #openstack-infra17:00
yolandano luck17:00
yolandainfra-root , if you want to play with that, controller is on
yolandayou have keys there17:01
*** samuelBartel has joined #openstack-infra17:01
*** tiswanso has quit IRC17:01
*** bpokorny_ has quit IRC17:01
anteayazaro: also I have an etherpad prepared with the current steps of our rename process before adding content, I want to confirm with you these steps are still accurate for 2.11:
*** esker has joined #openstack-infra17:02
*** bpokorny has joined #openstack-infra17:02
greghaynesprometheanfire: Do you have a use case in mind for that patch?17:02
greghaynesprometheanfire: I think its fine, just wondering17:02
*** tiswanso has joined #openstack-infra17:02
clarkbyolanda: how are you testing?17:03
prometheanfiregreghaynes: ya, atm the gentoo cleanup script is at 99-cleanup, which runs before 99-package-uninstalls17:03
greghaynesprometheanfire: ah, ok17:03
yolandaclarkb, tried wget or curl, and openssl s_client to verify17:03
clarkbyolanda: might help if we have a write up of the crrent steps that have been taken so that we know what was done17:03
clarkbyolanda: and how was the ca cert installed?17:04
yolandatesting the endpoint17:04
greghaynesprometheanfire: We have had an idea for a while we haven't followed through on that all our scripts should be between 10- and 90- so that users can always add things before and after17:04
greghaynesprometheanfire: not really important here, just remembered it though17:04
prometheanfiregreghaynes: that would be nice, would move it to 85 then17:04
greghaynesprometheanfire: I wouldn't do that yet, I think we should do the whole world at one time17:04
prometheanfirecleanup would be 8917:04
prometheanfireya, not yet17:04
yolandaclarkb so cert and key are configured in apache, for 35357 endpoint. the self signed cert has been copied to /etc/ssl/certs, and i ran update-ca-certificates17:04
prometheanfirejust saying how it would be if so17:05
greghaynesprometheanfire: yep17:05
yolandai am testing even passing the path, wget --ca-certificate=/etc/ssl/certs/
yolandaeven with that fails complaining with Unable to locally verify the issuer's authority.17:05
*** gyee has joined #openstack-infra17:05
*** sabeen3 has quit IRC17:06
*** esker has quit IRC17:07
BobBallasselin__: Any way to get the nodepool yaml file read the provider_username/password from heira?17:07
*** mrmartin has quit IRC17:08
asselin_BobBall, absolutelyyes17:08
asselin_in fact we need to make that the only way17:08
crinkleyolanda: can you put `openssl s_client -connect -CAfile /path/to/cert` in a paste?17:08
clarkbyolanda: you need to copy it to /usr/local/share/ca-certificates then run update-ca-certificates according to the manpage17:08
yolandaclarkb, i did that, i have a patch in puppet that does the same, and i even did manually to double check17:09
clarkbok that is not what you said you did above17:10
yolandayep, missed that bits17:10
clarkb(which is why I think it is important we clarify exactly what w ehave done to test)17:10
*** lucasagomes is now known as lucas-dinner17:10
asselin_BobBall, a bit of that here, but we should make that the default
*** sabeen has joined #openstack-infra17:11
*** mrmartin has joined #openstack-infra17:12
*** derekh has quit IRC17:12
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default
*** sdake has quit IRC17:12
*** ociuhandu has quit IRC17:12
*** sfinucan has quit IRC17:13
clarkbcrinkle: I think that paste was meant for you17:13
*** e0ne has quit IRC17:14
*** esker has joined #openstack-infra17:14
*** esker has quit IRC17:14
*** esker has joined #openstack-infra17:14
yolandaclarkb, the paste was for you, the output you requested17:14
clarkbthat was crinkle17:14
*** fhubik has quit IRC17:15
clarkbyolanda: the .pem that is in /etc/ssl/certs, did you copy that or was that the result of update-ca-certificates operating on /usr/local/share/ca-certificates?17:15
yolandapuppet on infra cloud, copies the certs direclty on /etc/ssl/certs17:16
clarkbI wonder if it would work if we put the crt in /usr/local/share/ca-certificates then run the update17:17
yolandalet me try17:17
*** apoorvad has joined #openstack-infra17:17
clarkb(basically I want to do what the manpage says you shoudl do, not depend on puppet)17:17
BobBallasselin_: Ah ok. Will look into that.  I saw oscc_file_contents but the template said it would be generated by puppet so I ignored it :)17:17
asselin_BobBall,'re the 2nd person to express confusion with that17:18
yolandaclarkb, same result17:18
asselin_BobBall, perhaps we should change it. That's to notify the reader of the deployed version not to update it. But it needs to be populated by a human17:18
clarkbyolanda: did you remove the -CAfile so that it would use the system default?17:19
clarkb(that should avoid permissions issues or file name mismatches)17:19
yolandaclarkb yes17:20
*** _nadya_ has quit IRC17:20
BobBallasselin_: FWIW defaulting log_server to undef did give a full deployment, but I'm currently trying to figure out why my jobs.yaml wasn't read by jjb :)17:21
BobBallasselin_: I'm going to get this CI fully set up before I remove the workflow -1 in case there are some error messages lurking somewhere...17:21
asselin_BobBall, I'm expecting issues with the zuul comment with the  log file links17:22
pabelangeryolanda: back now, going to give it a try17:22
yolandapabelanger thanks17:22
BobBallasselin_: Well I have to change that zuul comment in my setup17:22
yolandai need to go now , but i'll connect later at night17:22
asselin_BobBall, ok...perhaps you can use your patch to fully add swift support?17:23
asselin_I never tested that17:23
*** regXboi has joined #openstack-infra17:24
BobBallasselin_: my swift support is custom hackery because swift middleware doesn't (didn't?) let you set container types, so if you use the regular swift upload support then the log files look ugly and n-cpu.log.gz gets downloaded to your system rather than opening in a browser.17:25
pabelangeryolanda: works here17:25
pabelanger$ curl --cacert infra.crt
pabelanger{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}17:25
pabelangerthat is centos717:25
yolandagoing to try pasting that on my local box17:25
pabelangeryou testing from trusty?17:25
yolandapabelanger, can you add the cert to global trust, in centos and check without the cacert ?17:26
yolandayes, trusty17:26
pabelangeryolanda: sure, one sec17:26
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Periodic Devstack job against oslo.* from master
*** annegentle_ has quit IRC17:26
anteayawho is the un-named maroon person on
anteayaI'm guessing its fungi17:28
pabelangeryolanda: $ curl{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}17:29
pabelangerimport worked17:29
crinkleyolanda: this seems to be a problem in west as well now (it wasn't a problem two days ago)17:29
yolandathat's fine17:29
pabelangeryolanda: also, you'll need to recreate your cert, since you posted a private key :)17:29
fungianteaya: not me17:29
anteayafungi: okay thanks17:29
pabelangeryolanda: let me spin up a trusty box17:29
yolandapabelanger yes, no problems17:29
yolandait's just a dummy cert not used anywhere17:29
anteayamust be either clarkb or jeblair then17:29
fungianteaya: possibly zaro17:30
AJaegerteam, I saw yesterday - a liberty page on, today it's master again. The page had the 7.0.3 tag from yesterday's neutron release. Where can I find the log files from neutron from the release queue?17:30
pabelangeryolanda: fedora 23 works fine too, just tested17:30
anteayazaro: are you the un-named maroon person?17:30
*** smanjara has quit IRC17:30
*** esker has quit IRC17:31
clarkbnot me17:31
anteayafungi: ah thank you, yes you usually make a point of signing in, and are currently fancying black as your colour17:31
mtreinishanteaya, AJaeger: any ideas why the unit test jobs show UNSTABLE:
anteayaclarkb: okay thanks17:31
fungiAJaeger: i'm going to bet their docs job is using zuul-cloner and running on the tag, which z-c can't find so falls back to checking out master17:32
AJaegerfungi: that's what I want to figure out...17:32
clarkbmtreinish: possibly something funny on the logs server17:32
openstackgerritZara proposed openstack-infra/storyboard-webclient: WIP Search Changes
mtreinishclarkb: hmm, that's weird. I did 2 or 3 rechecks and they all were like that17:32
fungiAJaeger: do `git show-ref 7.0.3` to get the tag's sha, and then use that to formulate the url17:32
AJaegermtreinish: "ERROR: Failed to upload files: 2 No such file or directory" - no die17:33
*** samuelBartel has quit IRC17:33
zaroanteaya: i'm looking at the etherpad now17:33
*** ociuhandu has joined #openstack-infra17:33
AJaegerfungi: thanks, will do17:33
anteayazaro: thanks are you maroon?17:33
*** esker has joined #openstack-infra17:33
*** fawadkhaliq has quit IRC17:33
clarkbmtreinish: ps2's dir is owned by root and 75517:33
anteayaokay I'll buy fushia17:33
clarkbmtreinish: so the jenkins user cannot create subdirs17:33
anteayazaro: thanks, so the additions you made to the etherpad, are they improvements to our current workflow or changes/additions for 2.11?17:34
clarkbfungi: any idea why that may happen?17:34
mtreinishclarkb: well that would do it17:34
zarofungi: i think we may not need to do offline indexing this time.17:34
*** jsavak has quit IRC17:34
zaroanteaya: improvements.17:35
fungianteaya: i believe those additions are the cleanup for verify voting on openstack-dev/sandbox (and cleaning up all the various other repos that mentioned registered users in their acls too)17:35
anteayazaro: okay so not 2.11 related, great17:35
anteayafungi: ah thank you17:35
clarkbif I were going to guess we have a rogue find updating ownership/perms?17:35
*** arthus has joined #openstack-infra17:36
anteayazaro: did you come across any changes we should make now we are on 2.11?17:36
openstackgerritZara proposed openstack-infra/storyboard-webclient: WIP Task Links
fungiclarkb: that's the best idea i can come up with17:36
zarofungi: no, i haven't gotten to that yet.  that's cleanup for projects that are kept lingering from renames17:36
zaroanteaya: i don't see anything that should be different17:36
fungizaro: oh... now i'm looking at the script, yeah17:36
clarkbdrwxr-xr-x 3 root    root    4096 Jan 27 08:45 217:37
anteayazaro: so you did a rename workflow on review-dev and our current process (for the gerrit steps) work as is?17:37
fungizaro: so we still had some lingering cleanup beyond the deletions we performed during the upgrade?17:37
*** sshnaidm has joined #openstack-infra17:37
*** abregman has quit IRC17:37
zarofungi: i don't think so, but if you change foo to bar.  then foo is left in the db correct?  don't we want to clean that out?17:38
clarkbcrinkle: does that mean west has regressed and isn't functioning or just that some behavior has changed but it is clouding ok?17:38
*** jlanoux has quit IRC17:38
zarofungi: isn't that why we had to run that cleanup script on the last gerrit upgrade?17:38
*** esker has quit IRC17:39
fungizaro: i'm not following17:39
*** esker has joined #openstack-infra17:39
anteayafungi zaro here is the etherpad from the last rename for reference:
fungizaro: during the upgrade maintenance we ran that script you linked to clean up a mess left behind during some very old renames which had been done incorrectly17:40
zarofungi: so if we change asta to astana.  isn't asta still in the DB?17:40
clarkbmtreinish: fungi that is where the dir got created17:40
clarkbmtreinish: fungi but that runs as the jenkins user so would've been jenkins:jenkins at that point17:40
fungizaro: our rename process normally takes care of that17:40
*** mrmartin has quit IRC17:40
fungizaro: it happens during step #517:40
* anteaya notes another mis-spelling of astara, markmcclain17:40
zarofungi: ahh, ok.  i didn't know that included cleanup.  then no need for 5.517:41
mtreinishclarkb: hmm, so something changed the ownership between then and the next time the tests ran17:41
crinkleclarkb: yolanda it seemed like puppet was misbehaving due to ssl but I think I had environment variables set that were messing with it17:41
clarkbya, looking at our file maintenance script it only rm's files and dirs and gzips files17:42
clarkbnone of those operations should change ownership17:42
fungithough the documentation at is still some hybrid involving ansible playbooks which we've never actually tried17:42
zaroanteaya: i haven't tested a rename on review-dev but not sure that's necessary.17:42
crinkleclarkb: yolanda so puppet is functioning normally, the clients are functioning normally, but openssl does report return code 21 for neutron and nova endpoints17:42
fungii thought we'd reverted that17:42
*** jsavak has joined #openstack-infra17:42
Sam-I-AmAJaeger: hi17:42
*** kozhukal` has joined #openstack-infra17:42
anteayazaro: do you have time for a test today? would make me feel better if you did17:42
*** esikachev has quit IRC17:42
clarkbwell gzip may change ownership of the file but not that higher level dir17:42
crinkleyolanda: does puppet work for you?17:43
zaroanteaya: i think if that stuff in etherpad worked previoiusly then should continue to work.17:43
anteayazaro: would save us time if we hit something different tomorrow17:43
fungizaro: so the database schema between 2.8 and 2.11 hasn't really changed in ways which would impact the database queries we used to perform during renames?17:43
yolandacrinkle sorry, i was just stepping out. Have you tried puppet from scratch? when it needs to create all services in keystone? a fresh deploy?17:43
clarkbcrinkle: maybe openssl is acting stricter?17:43
yolandait worked for me with the Infra CA. Not working with self signed, complains in keystone17:44
*** kozhukalov_ has quit IRC17:44
zaroanteaya: that would be a big test.17:44
anteayazaro: what makes it big?17:44
*** kozhukalov has quit IRC17:44
crinkleyolanda: no I was verifying against the production one17:44
anteayaa usual rename takes 30 minutes or did on 2.817:44
AJaegerfungi, thanks, that helped. We indeed published to the wrong location17:44
zaroanteaya: all those steps.17:44
*** esker has quit IRC17:45
yolandacrinkle it will work, as is not contacting keystone to create tenants , endpoints, etc17:45
clarkbthe last change to maintenance script merged last september17:45
yolandato verify it needs to be tested on a fresh deploy17:45
zarofungi: where are the scripts/queries?  i can check those.17:45
yolandaat your point everything should be created so it will skip most of the steps17:45
AJaegerteam, I have a problem with The neutron 7.0.3 (liberty) release yesterday pushed content to both docs.o.o./developer/neutron/7.0.3 *and* docs.o.o/developer/neutron/ - thus overriding the master documents there.17:45
AJaegerProblem is
AJaegerWe check for the latest tag and then publish.17:45
crinkleyolanda: it contacts keystone to check for existing resources17:45
fungizaro: unless someone gets time to revert the documentation "update" that switched it to ansible we haven't actually used yet, we end up copying from previous maintenances like
yolandacrinkle, with self signed cert or infra ca as we have on hiera?17:46
clarkbmtreinish: I am stumped, short of someone modifying those file perms I don't know what could be doing that17:46
AJaegerWe shouldn't do that for neutron that publishes after each merge, only for clients that publish only from release job. Correct?17:46
fungizaro: that example is the stackforge rename maintenance, so the largest example we have17:46
crinkleyolanda: with nothing changed, will try without ca in a dev env17:46
yolandacrinkle so with ca and chain works here, with self signed fails17:47
yolandacan you test the same?17:47
clarkbthere was a volume error when I was out traveling17:47
clarkbyolanda: when was that? could something done then have changed file perms on the logs fs?17:47
anteayafungi zaro here is a smaller rename:
*** dtardivel has joined #openstack-infra17:47
fungizaro: mostly want to make sure steps 5-7 are still valid17:47
yolandaclarkb, so i tested this morning with infra ca. That works fine. Then, per conversation with fungi on the channel, i changed to simply self signed certs. That fail17:48
fungizaro: it sounds like you're saying step 7 is no longer needed?17:48
yolandai can reproduced and make it work again with infra ca + chain, system is ok17:48
clarkbyolanda: sorry I am crossing the streams. Last month the logs server lost a cinder volume17:48
fungiyolanda: what chain?17:48
yolandaclarkb, not talking about that17:49
clarkbyolanda: I know, I am17:49
fungiyolanda: how did the "infra ca" have a chain/intermediate certificate and what would even have been in it?17:49
*** mrmartin has joined #openstack-infra17:49
*** ifarkas has quit IRC17:49
clarkbyolanda: trying to debug mtreinish's log problem. There was a directory on the logs fs created on january 27t17:49
clarkbyolanda: it is owned by root and causing future uploads to that dir to fail bceause uploads happen as the jenkins user17:49
clarkbanyways we can come back to that to avoid too many conversations at once17:50
SpamapSfungi: I was just reading your response to the apt mirror questions on the ML. I am a bit confused. Are we running an actual mirror of Ubuntu, or a partial one with different lists files (otherwise, why wouldn't we have signed package lists?)17:50
fungiahh, right, the logs.o.o filesystem failure from a couple weeks ago17:50
*** annegentle_ has joined #openstack-infra17:50
zaroanteaya, fungi : thanks for narrowing it down.  i can verify those steps.  i'm pretty sure it's still ok on 2.11 but will verify anyway.17:50
yolandaclarkb, it can match the dates. However what i did is unmount, fsck + mount17:51
yolandanot updating anything else17:51
*** annegentle_ has quit IRC17:51
anteayazaro: thanks so much17:51
fungiSpamapS: it's possible reprepro simply doesn't copy the signatures. anyway it's not a _full_ mirror since we omit i386 and restricted/multiverse17:51
SpamapSwhy would we use reprepro?17:51
fungiSpamapS: and reprepro regenerates the indices17:51
fungiso that they're consistent17:51
clarkbyolanda: thanks17:51
zarofungi: yeah, step 7 might not be needed. we should be able to start gerrit then initiate the online indexing and let it index while gerrit is running.17:51
clarkbI wonder if it is possible for fsck to recover things that way17:51
yolandafungi, an infra ca i generated manually. I created a CA , and intermediate, for openstack-infra ca, just a dummy one. Then i created a cert in top of that. using that setup, it works17:52
fungiSpamapS: our biggest issue to date with "other people's mirrors" has been inconsitent indices17:52
zarofungi: initiate the index with ssh command.17:52
fungiyolanda: i'm still not understanding why you created (or even needed) an intermediate certificate17:52
SpamapSfungi: until we have the new apt from 16.04/unstable, there is no way to make apt consistent. :-/17:52
dimsAJaeger : one last job and i'll be done for now :) this time it's a devstack+tempest+oslo.* from master -
clarkbSpamapS: the system we have in place should be pretty darn ood17:53
fungiSpamapS: whatcha mean by that? an index generated of the packages you have should be consistent17:53
yolandafungi, following that
openstackgerritDmitriy Ukhlov proposed openstack-infra/project-config: Adds neutron tempest tests for oslo.messaging drivers
*** annegentle_ has joined #openstack-infra17:53
fungiSpamapS: rather than copying indices from somewhere else when they might be in flux17:53
yolandain my server, then pasting the generated contents on hiera17:53
*** esker has joined #openstack-infra17:53
mtreinishclarkb: I've never heard of fsck changing ownership, but I can't really rule it out17:53
AJaegerdims: famous last words ;)17:53
SpamapSfungi: caching fail. But, in our context, that is N/A, so I see where reprepro + having to reconfigure apt to respect infra's key might be better than dealing with apt-mirror fail.17:53
dimsAJaeger : i did say "for now" :)17:54
* AJaeger ignored that17:54
clarkbmtreinish: ya I hvaen't either17:54
*** mrmartin has quit IRC17:54
SpamapSfungi: note that Ubuntu's mirrors actually set Expires headers that avoid the problem 99.9% of the time.17:55
fungiyolanda: i see. yes this is a guide for people who are trying to maintain a secured certificate authority. adding a "signing key" below your primary ca key is certainly excessive for a low/no-security ca17:55
SpamapSfungi: it's just the other mirrors, that aren't, that don't do that right :-P17:55
clarkbI wonder how long a chown -R jenkins:jenkins /srv/static/logs would take17:55
SpamapSclarkb: ACK, I agree, it's actually quite close to perfect, and if you can guarantee no caching ever, then it should be 100%.17:56
fungiSpamapS: three nines of uptime for a package mirror is pretty terrible in our situation17:56
SpamapSfungi: has been that bad?17:56
zaroanteaya, fungi : improvement suggestion in etherpad.17:56
yolandafungi yes, but it worked without much problems with openstack, apart from neutron. If we can solve the problem with self-signed it will be better, but we can have this as an alternative if we hit a blocker17:56
yolandaanyway, just leaving home now,17:56
*** fawadkhaliq has joined #openstack-infra17:56
fungiSpamapS: bad enough that we have separate elastic-recheck patterns tracking failures in vs or whatever17:56
AJaegerdims: one question inline on your change...17:57
SpamapSfungi: trrrble. Anyway, that explains it, I didn't know we were reprepro'ing, thanks.17:57
dimsAJaeger : ah, good point, will do17:58
fungiSpamapS: also, we're currently _not_ signing the indices at all. it's something of a feature, as it discourages anyone using it outside our ci17:58
SpamapSfungi: yeah, +1 for not making our own ubuntu fork. :)17:59
fungiSpamapS: yesterday it turned up an ansible role testing bug, where the role was set to check whether you've disabled signature verification on your package lists. which means that the test was actually validating the apt configuration of the test host! crazy18:00
*** SumitNaiksatam has joined #openstack-infra18:00
*** sabeen has quit IRC18:01
SpamapSfungi: I replied to the ML for the diskimage-builder based things. apt-conf element, DIB_APT_CONF=/etc/apt/apt/conf .. done18:01
BobBallasselin_: I'm confused by using clouds.yaml with nodepool... What's the link from nodepool's yaml file into clouds.yaml to get the credentials?18:01
SpamapSthere are too many slashes there18:01
*** kozhukalov_ has joined #openstack-infra18:01
clarkbBobBall: the cloud: key under the provider. It names a cloud inside of your clouds.yaml18:01
fungiSpamapS: so many slashes18:02
asselin_BobBall, nodepool osclient usage looks at clouds.yaml file for credentials18:02
fungiSpamapS: but i read your reply, so good enough18:02
*** shardy has joined #openstack-infra18:02
*** Nikolay_St has quit IRC18:02
*** rossella_s has quit IRC18:03
BobBallclarkb / asselin_: Yes but, for example, suggests we should have a "cloud: rax" but there are 3 regions defined in the rax cloud.  Will nodepool just use all three regions? (whereas before I had to specify a region to use)18:03
*** rossella_s has joined #openstack-infra18:04
AJaegerdougwig: could you review , please?18:04
dougwigAJaeger: yes18:04
clarkbBobBall: no, you specify both18:04
mtreinishmordred, Shrews: I was looking at the results from the shade functional jobs and I noticed it's not copying the subunit stream from the test run18:04
mtreinishso the testr_results.html isn't really useful:
BobBallohhhhhhhhh - sorry.  Makes sense.  Thanks.18:05
*** kzaitsev_mb has quit IRC18:05
BobBallclarkb: I'd assumed that cloud: rax was enough to define the region+credentials etc.  Awesome, thanks.18:05
AJaegerdougwig: and also , please18:05
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job
clarkbBobBall: you can problaby make a cloud called rax-iad and another called rax-ord then drop the regions from nodepool config18:06
pabelangeryolanda: something is different with trusty, not working as I would expect18:06
BobBallok, thanks.  I'll test the better way first then fallback to that if I can't get it working.18:06
*** jsavak has quit IRC18:07
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Periodic Devstack job against oslo.* from master
dougwigAJaeger: done18:07
*** jsavak has joined #openstack-infra18:08
AJaegerthanks, dougwig !18:08
pabelangeryolanda: have to step out. But certs are good18:09
*** esker has quit IRC18:10
clarkbnibalizer: ping when you want to work on getting mirrors going (though I recognize this cert stuff may have to be sorted first)18:11
*** _nadya_ has joined #openstack-infra18:11
fungizaro: any thoughts on doing the "registered users" group cleanup in the database and in the groups files of all the affected repos?18:11
fungiyou had mentioned that due to the database change it needs to happen offline, so tomorrow might be a good time to get it out of the way while we're already going to be down to do project renames18:12
*** hashar has quit IRC18:13
*** infra-red has joined #openstack-infra18:13
AJaegerI have four patches for review: Frist two small devstack-gate cleanups, then two linters-pep renames one. Could any project-config core  review them, please? 278051, 278061, 278641, 278627 yolanda gave her +2s already18:13
*** esikachev has joined #openstack-infra18:13
fungiAJaeger: did you find the doc publishing log you were looking for? any insights? i probably missed it in scrollback18:14
*** mrmartin has joined #openstack-infra18:14
*** e0ne has joined #openstack-infra18:15
*** mrmartin has quit IRC18:15
clarkbmtreinish: I went ahead and chowned that particular dir so you should be able to recheck and get uploads to work18:15
mtreinishclarkb: ok, I'll do that now18:16
AJaegerfungi, yes, I mentioned it at the wrong time. Full story at
clarkbmtreinish: I basically wans't getting anywhere with further debug so preserving the old state wasn't helpful18:16
AJaegerfungi: So, found the problem thanks to the ref but I'm lost on how to fix it18:16
AJaegerany help is welcome!18:16
clarkbI couldn't find anything that indicated fsck may cause ownership to change18:16
*** rcernin has joined #openstack-infra18:17
*** infra-red has quit IRC18:18
*** esikachev has quit IRC18:18
*** infra-red has joined #openstack-infra18:19
*** dolphm has joined #openstack-infra18:19
*** baoli_ has quit IRC18:20
*** flepied has quit IRC18:20
fungiAJaeger: so it's a logic issue in
AJaegerfungi: that's my assessment18:21
AJaegerfungi: but I'd like confirmation for that - and an idea how to fix it (or a patch ;)18:21
*** esker has joined #openstack-infra18:21
NobodyCammorning infra folk18:22
fungiAJaeger: briefly skimming the conditional at it seems like it should do what you're wanting18:22
mtreinishclarkb: I'm fine with just chalking it up to cosmic rays18:22
fungiAJaeger: make a copy from branch to tag if the tag is latest, otherwise move content from branch to tag18:22
*** mriedem has quit IRC18:22
fungiclarkb: did you start a find to look for other incorrect ownership in that tree?18:23
clarkbfungi: I have not18:23
*** sabeen1 has joined #openstack-infra18:23
AJaegerfungi: tag is on a stable branch and neutron publishes from  master as well.18:23
Sam-I-AmAJaeger: hi18:23
AJaegerfungi: And thus the content gets published on the master location, overriding it18:23
AJaegerfungi: until the next master change merges and overrides again...18:24
*** piet has joined #openstack-infra18:24
fungiAJaeger: right, which suggests we're behaving like the "$TAG" = "$LATEST" part of teh conditional18:24
AJaegerfungi: yes, correct. We're in that if18:24
*** mriedem has joined #openstack-infra18:24
fungi"2016-02-10 17:09:55.917 | + LATEST=7.0.3"18:25
NobodyCamI have a question: I need to adjust the build_timeout for gate-tempest-dsvm-large-ops test but am unable to locate the correct place to do this. can someone point /me inthe right direction (this is for stable liberty btw)18:25
anteayaNobodyCam: hey there18:25
NobodyCamhey anteaya18:25
fungiAJaeger: so something is wrong with the definition of LATEST in line 43 of run-docs.sh18:25
AJaegerfungi, That if sounds correct to me for repos that use openstack-client-publish-jobs and do not publish from post. But here we use openstack-server-publish-jobs and do publish from post as well18:25
anteayaNobodyCam: why do you want to adjust the build_timeout?18:26
clarkbfungi: something like find /srv/static/logs -uid 0 -gid 0 ? should we exec a chown or just collect a list?18:26
AJaegerfungi: I don't think so. 7.0.3 is the lastest tag18:26
clarkbhrm may want to or those two18:26
AJaegerfungi: but you're right is a newer tag...18:27
NobodyCama backport is timing out at 196 seconds ... we need to bump it up18:27
fungiAJaeger: oh... yes this is a logic error, you're right. we should only update the branch if we're not normally updating it in post18:27
AJaegerfungi: but we would be in this situation even if there wouldn't be a newer tag18:27
NobodyCamso I'm looking for the correct place to set BUILD_TIMEOUT for the ironic test there18:27
*** eil397 has joined #openstack-infra18:28
*** zeih_ has joined #openstack-infra18:28
Sam-I-AmAJaeger: why isnt devref published off master like we do for things like the admin guide?18:28
AJaegerSam-I-Am: it is. We're just overriding it.18:28
fungiAJaeger: the regex on line 43 omits prerelease tag patterns (by design) which is why 7.0.3 is considered LATEST18:28
anteayaNobodyCam: okay I think this is a change you need to make within the test18:28
*** ihrachys has quit IRC18:28
anteayaNobodyCam: I wouldn't classify this as a build_timeout error from an infra perspective18:29
AJaegerfungi: but even if we just branched off 7.0 and have opened master with no tags on it, it would be wrong.18:29
anteayasince the build completed and tests were executed18:29
fungiSam-I-Am: the only good reason i can think of is that the commit for 7.0.3 would have been published in post as 7.0.2+ because the 7.0.3 tag wasn't pushed at the time18:29
fungibut if a newer commit than the tagged one has also merged we really shouldn't touch the master docs18:29
AJaegerfungi, do we have to create a new {name}-server-docs job?18:30
AJaegerfungi: and then change behaviour based on job name?18:30
*** Jeffrey4l has quit IRC18:30
fungiAJaeger: i think we should just be able to check whether the master branch tip is what the tag is pointing at. if so, publish master and tag docs, if not, only publish tag docs18:31
Sam-I-Amhere's a q - if someone does a devref update in liberty, does the job publish the liberty branch instead of master?18:31
NobodyCamanteaya: we do this for master branch tests:
*** sdake has joined #openstack-infra18:31
*** kozhukal` has quit IRC18:31
clarkbfungi: I am running that find with an -or on a subset of the logs dir18:32
AJaegerfungi: but tagging is not necessarily the tip - with releases repo, the team can tag an older change18:32
clarkbfind /srv/static/logs/24 \( -uid 0 -or -gid 0 \)18:32
NobodyCambut I not sure where to set it for the large opts test that is failing on the backport I put up18:32
clarkbnothing yet18:32
Sam-I-AmAJaeger: i think thats what i'm saying here18:32
anteayaNobodyCam: oh okay well you know more than I do then18:32
Sam-I-Amtags are arbitrary values18:32
*** gordc has joined #openstack-infra18:32
*** kozhukalov_ has quit IRC18:33
Sam-I-Amso latest may be 7.0.3 instead of 8.0.0b3 or whatever18:33
fungiclarkb: optionally you could have done -not \( -user jenkins -group jenkins \) i think18:33
clarkbfungi: good point18:33
fungiclarkb: but really, it's likely either correct or 018:33
fungiso what you have should be fine18:33
*** zhenguo_ has quit IRC18:33
AJaegerfungi: waht about: check if the tag is pointing at any change in master?18:33
sdaguedstufft: I'm finding really weird behavior with virtualenv 14.0.618:34
dstufftsdague: yea?18:34
sdagueit's scrubing out /usr/local... so it won't work on my linux box18:34
sdaguewith tox18:34
fungiAJaeger: well, it's possible to push a tag for something which isn't the branch tip, so that would cause it to roll back the master documentation to an earlier commit (though it would update the version number). i guess that's a trade-off, i don't really know which is better18:34
anteayaNobodyCam: sorry I'm not more help18:34
*** pvaneck has joined #openstack-infra18:34
*** ashtokolov has quit IRC18:34
Sam-I-Amfungi: devref should always reference master, imo18:35
*** kushal has quit IRC18:35
*** edwarnicke has quit IRC18:35
*** mtanino__ has quit IRC18:35
*** teran has quit IRC18:35
NobodyCamanteaya: don't be at all18:35
Sam-I-Ami dont think we publish stable branches of it18:35
Sam-I-Amor if there's even a reason18:35
*** esikachev has joined #openstack-infra18:35
anteayaNobodyCam: I hope you find what you need, though, if that is helpful18:35
*** rhallisey has quit IRC18:35
AJaegerSam-I-Am: it should - and that's what we discuss here. The problem is that a liberty change overwrote it18:35
*** mtanino__ has joined #openstack-infra18:35
*** teran has joined #openstack-infra18:35
AJaegerSam-I-Am: We publish the complete docs tree...18:35
*** zhenguo_ has joined #openstack-infra18:35
dstufftsdague: what do you mean it's "scrubing out /usr/local"18:36
fungiAJaeger: i think what Sam-I-Am is saying is we shouldn't run this job in the release pipeline, only post18:36
Sam-I-Amseems like a devref change for a prior release is pointless18:36
Sam-I-Amunless we're going to start publishing releases of it18:36
Sam-I-Ami'm curious which patch did that... could have been a backport?18:36
AJaegerSam-I-Am: we publish
AJaegerSam-I-Am: not only devref18:36
*** baoli has joined #openstack-infra18:37
*** ccrouch has quit IRC18:37
*** ashtokolov has joined #openstack-infra18:37
*** rhallisey has joined #openstack-infra18:37
*** shashank_hegde has joined #openstack-infra18:37
AJaegerfungi: but indeed, I wonder why we need these...18:37
Sam-I-AmAJaeger: how does one get to those other versions?18:38
Sam-I-Amthey dont use the conventional release names that docs.o.o does18:38
AJaegerSam-I-Am, fungi : That's why I wonder whether we need them - nobody knows about these...18:38
*** jlvillal has quit IRC18:38
Sam-I-Amto me, it seems developers would be updating docs for master because thats where the active development occurs18:38
AJaegerThere's also
Sam-I-AmAJaeger: which is even another tag heh18:38
*** angdraug has quit IRC18:39
AJaegerand we have and  -but no liberty18:39
Sam-I-AmAJaeger: i dont think we have ways to get to any of those "old" versions18:39
*** jlvillal has joined #openstack-infra18:39
fungiAJaeger: yeah, the logic in that job seems to assume all tags are related to the master branch, so stable branch docs are only updated for stable commits in post18:39
Sam-I-Amunless you happen to know the link exists18:39
*** mrmartin has joined #openstack-infra18:40
fungithe script relies on `echo $ZUUL_REFNAME | grep stable/` which will never match on a tag18:40
dstufftsdague: I bet this is something to do with the sys.path munging that virtualenv does :/18:40
fungisince $ZUUL_REFNAME in this case is refs/tags/7.0.3 not refs/heads/stable/liberty18:41
*** gordc has quit IRC18:41
clarkbSam-I-Am: AJaeger projects could publish links as part of their index on the master side that grows as new tags and releases are made18:41
*** rhallisey has quit IRC18:41
clarkbnot sure that that is a problem with our publishing of them18:42
sdaguedstufft: yeh, I don't know, I followed the homepage link for virtualenv and got - which is a 40418:42
AJaegerclarkb: I doubt any of them does...18:42
sdagueso I defaulted to asking you18:42
clarkbAJaeger: they don't, my point is we do our best to make sure the things are published18:42
clarkbAJaeger: indexing those things is up to the projects themselves18:42
fungiclarkb: the problem in this case is that the logic around tags is naive18:43
sdague14.0.5 seems to work18:43
dstufftsdague: open an issue on please, I have to run to the store but I'll take a look when I get back18:43
sdaguedstufft: ok, cool. It would be nice if that was linked in the pypi record somewhere18:43
AJaegerclarkb: yes, we give them the option - and I expect nobody knows about it18:43
*** moravec has quit IRC18:43
*** piet has quit IRC18:43
fungiclarkb: i _think_ someone tried to "fix" the script to work around the "bug" of docs for master not mentioning the latest release once a tag is pushed, but didn't think through the implications18:44
crinkleyolanda: I was able to reproduce your issue and solve it, i have a number of possible troubleshooting steps18:45
Sam-I-Amwe only need to publish devref changes to master18:45
fungiclarkb: AJaeger: Sam-I-Am: it may be easiest to just accept that cosmetic nuisance and make the script either publish tag documentation or branch documentation but not try to suss out when it should do both at the same time18:45
Sam-I-Amif someone backports a patch that includes a devref change... it should really do nothing?18:45
AJaegerfungi: but then python-XXXclient would never publish to developer/python-XXXclient ;(18:46
AJaegerSince the clients only publish from release queue18:46
Sam-I-Amsounds like we have 2 cases here. things that use a release queue, and things that should always be master?18:47
AJaegerSam-I-Am: for me devref is neutron/doc/source/devref - but we talk about all of neutron/doc/source.18:47
anteayafungi zaro I'm going to start populating that etherpad18:47
fungianteaya: thanks!18:47
Sam-I-AmAJaeger: thats what i mean doc/source18:47
anteayathank you18:47
fungiAJaeger: it may make more sense to not publish two copies of client docs, and instead publish a redirect to the tag (which gets updated when the tag is updated)18:48
*** kushal has joined #openstack-infra18:48
AJaegerfungi: you've lost me, I'm not following18:49
clarkbfungi: a good chunk of /srv/static/logs/24/271124/1 is also root owned18:49
fungiAJaeger: let me dig up a specific example, hold on18:49
AJaegerthanks, fungi18:49
*** edwarnicke has joined #openstack-infra18:50
*** eliqiao_ has quit IRC18:50
clarkbfour of the five gate jobs for that change have root owned dirs and files18:50
clarkbbut not all of them making it even weirder18:50
*** esker has quit IRC18:51
AJaegerclarkb: permissions? Non-writable directory?18:51
*** infra-re_ has joined #openstack-infra18:51
*** zeih_ has quit IRC18:51
fungiAJaeger: so could be a temporary (not permanent) http redirect to and when a tag for python-novaclient 3.2.1 is pushed, publish only and update the redirect to point at that version18:51
clarkbAJaeger: it is 755 and 644 so the jenkins user can't write to it18:52
*** eliqiao_ has joined #openstack-infra18:52
clarkbAJaeger: this is unrelated to the publishing of docs conversation18:52
clarkbfungi: dougwig I have held the slave running and ssh'd into it18:52
clarkbhopefiully we see what is going on18:52
AJaegerclarkb: yes, I know that you're fixing the logs issues18:52
fungiAJaeger: that way the docs are only updated when a new release happens, and "main" copy of the documentation (as determined by the http redirect) always mentions the right version number18:52
*** mrmartin has quit IRC18:53
*** gyee has quit IRC18:53
clarkbthey were also updated last at january 27 084518:53
zarofungi: will add steps to etherpad on the registered users cleanup.  i think it's pretty isolated and safe to do.  I will not be around tomorrow so will provide info so you can decide whether you'de like to do it.18:53
jeblairclarkb, fungi, AJaeger:
AJaegerfungi: yes, that would work for the clients - but not for the server18:53
clarkbsomething happened at 0845 to do this18:53
fungizaro: that's awesome, thanks for the help!18:53
AJaegerjeblair: tahnks18:53
clarkbjeblair: yup iirc annegentle_ didn't want to confuse people consuming the client iwth unreleased feature documentation18:53
*** mrmartin has joined #openstack-infra18:54
*** infra-red has quit IRC18:54
jeblairviewing that change required that i sign in -- is that true for other people too?18:54
*** daemontool has joined #openstack-infra18:54
jeblairclarkb: i don't see any discussion about why that wouldn't apply to server docs though...18:54
clarkbjeblair: you can "go anonymous"18:54
fungiAJaeger: right, i'm agreeing that we probably need _that_ behavior for client publication (things which only publish on release) and a branch-or-tag-but-not-both publication mechanism for "servers" (things that have branch versions of documentation as well as tag versions of documentation)18:54
AJaegerthe nice thing with our discussion is that we have a self-healing problem. The next merge for master will fix it ;)18:55
anteayajeblair: I can see 137725 in chrome without signing in18:55
pabelangerjeblair: I can view it and not logged in18:55
jeblairclarkb: oh ok, so not related18:55
*** amotoki has joined #openstack-infra18:55
fungiAJaeger: if we want to not duplicate too much of the logic, we just have two jobs which call the script with different parameters18:55
clarkbthe thought with the server dev docs was people consuming them are hacking on the tip of $brnach18:55
clarkbwhich is slightly different than someone wanting to consume the nova api with novaclient and installing the latest version on pypi18:56
AJaegerfungi, so a new {name}-server-docs job18:56
*** tzn has quit IRC18:56
fungijeblair: thanks, yep clarkb's change is what implemented the current behavior we see18:57
jeblairclarkb: no i understand that -- what i'm saying is that it looks like we had no discussion..a AHA!18:57
jeblairi said "(If we start doing something sensible with tags on the main projects, this will need updating since we probably don't want a tag overwriting trunk docs, but that's not a concern we have to solve now.)"18:57
clarkbfungi: dougwig and I have a tail -f on syslog as I figure if anything happens that is catchable that will catch it18:57
fungiand wow, 3.5 years ago18:57
jeblairapparently, at the time, we were not doing anything sensible with tags on the server projects :)18:57
fungipretty sure that predated pbr18:58
*** infra-re_ has quit IRC18:58
jeblairand since then, we started doing something "sensible" and we have run into the problem we anticipated :)18:58
AJaegerjeblair: ;)18:58
dougwigclarkb: awesome.18:58
jeblairclarkb: we kicked that can down the road a LONG TIME.18:58
*** mrmartin has quit IRC18:58
clarkbseptember 2012 practically another lifetime18:59
fungiif we're only just _now_ getting annoyed by it, then successful punt!18:59
AJaegeras I said before: It's self-healing ;)18:59
*** amotoki has quit IRC18:59
fungiusually when i punt on solving something thoroughly, it comes back to bite me the next week not 3.5 years later18:59
*** kzaitsev_mb has joined #openstack-infra18:59
fungiso anyway, yes i think the sane solution is two jobs. we can still have both use the same script with just a little logic tweak19:00
Sam-I-Am^ this19:00
*** jsavak has quit IRC19:00
*** _nadya_ has quit IRC19:01
jeblairso my next question is, why does return [] ?19:01
*** FallenPegasus has quit IRC19:01
zaroanteaya, fungi : i added the info on cleaning up the registered users group.  please look to see if it makes sense.19:01
AJaegerI can give it a try but like to trade in four reviews for project-config, please: 278051, 278061, 278641, 27862719:01
anteayazaro: thank you19:01
* anteaya looks19:02
*** FallenPegasus has joined #openstack-infra19:02
clarkbfungi: dougwig also my d-g change did not merge and after a first recheck it tripped on something there too :/19:02
clarkbthe gate fail was cinder related but it didn't look like disk space problems so I dunno19:03
clarkbrechecked again for more data gathering19:03
dougwigclarkb: yeah i saw the oslo messaging timeout error from nova, on a floating ip. just a few projects in the mix in that failure.  :)19:03
*** jsavak has joined #openstack-infra19:03
dimsdougwig : was it grenade?19:03
anteayazaro: in step 2 of your added steps, can that patch merge prior to gerrit and puppet being stopped?19:03
dougwigdims: yes it was19:04
*** harlowja has joined #openstack-infra19:04
*** sabeen3 has joined #openstack-infra19:04
dimsdougwig : got a url to the log?19:04
fungizaro: 270537 only looks like it handles the database part of the work (cleaning up references in 4 tables). that helps, but what about the groups files? is it safe to ignore them or do we need to clean them up in all the affected repos?19:04
*** arthus has quit IRC19:05
*** _nadya_ has joined #openstack-infra19:06
zarofungi: it's safe to ignore. i tested that.19:06
*** infra-red has joined #openstack-infra19:06
zarofungi: but would be nice to cleanup :)19:06
fungizaro: okay, so if there is a group mentioned in a groups file with a uuid that is not referenced in the database, gerrit just ignores it?19:06
jeblairzaro: many of the older system-config changes are not searchable in gerrit19:06
zarofungi: yes.19:07
jeblairzaro: example:
*** sabeen3 has quit IRC19:07
*** jsavak has quit IRC19:07
*** sabeen1 has quit IRC19:07
fungizaro: that's very simple then--thanks! i was worried this would get a lot messier19:07
dimsdougwig : i'll take a quick peek will report back if i spot something19:07
jeblairzaro: or by it's change-id:,n,z19:07
*** mknawabi has joined #openstack-infra19:07
*** infra-re_ has joined #openstack-infra19:08
*** yamahata has quit IRC19:08
jeblairzaro: do you have any idea why or how we could fix that?19:08
zarojeblair: do you happen to know how old?  like from 2.2 or 2.4?19:09
jeblairzaro: it's worth noting, though i do not know if it's related, that was during the time where we were cherry-picking as a merge strategy19:09
jeblairzaro: i do not know the version (though changes to other repos from the same time period are searchable)19:09
*** rhallisey has joined #openstack-infra19:10
*** infra-red has quit IRC19:11
jeblairzaro: i think it was 2.419:11
*** annegentle_ has quit IRC19:11
zarojeblair: how are you finding these non-searchable changes?19:11
jeblairzaro: git history19:11
zarojeblair: does it work with direct url?19:11
jeblairzaro: yes19:11
Shrewsmtreinish: that's unfortunate (re: shade test_results) and i've never noticed. patch would be welcomed19:12
*** fawadkhaliq has quit IRC19:12
anteayait would be great to get another jeepyb +2 on this prior to tomorrow's gerrit downtime, it helps to clean up the errand 'registered users' group when never should have been created:
*** nelsnels_ has joined #openstack-infra19:12
zarojeblair: ok.  i have no idea what's going on off the top of my head but will dig deeper to see.19:12
jeblairzaro: i happened upon 13725, then just started incrementing the number from there; 13726, 7, 8 all work; 13729 is another system-config change which does not... etc.19:13
jeblair(6,7,8 are not system-config changes)19:13
*** akshai has quit IRC19:13
*** akshai has joined #openstack-infra19:13
*** infra-re_ has quit IRC19:14
*** nelsnelson has quit IRC19:15
*** annegentle_ has joined #openstack-infra19:18
*** moravec has joined #openstack-infra19:18
*** flepied has joined #openstack-infra19:22
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix tag publishing of server projects
AJaegerjeblair, clarkb, fungi: here's the docs change, please review carefully ^19:22
*** Sukhdev has joined #openstack-infra19:23
*** sabeen has joined #openstack-infra19:23
*** jsavak has joined #openstack-infra19:23
*** notnownikki has quit IRC19:24
*** jsavak has joined #openstack-infra19:24
*** sabeen3 has joined #openstack-infra19:25
*** boris-42 has joined #openstack-infra19:26
dimsanteaya : one last job for this round of running stuff against oslo.* master -
funginibalizer: any reason for me not to approve 275477 (Add OmfraCloud to puppetmaster_clouds) yet?19:26
*** electrofelix has quit IRC19:27
*** moravec has quit IRC19:27
nibalizeronly if we wanted to flatten auth19:27
*** moravec has joined #openstack-infra19:28
*** sabeen has quit IRC19:28
anteayadims: this adds a job to devstack, please get mtreinish to review the patch19:28
dimsack thanks19:28
fungianteaya: that change doesn't clean up anything, just prevents manage-projects from making new messes19:29
anteayafungi: I beg your pardon19:29
funginibalizer: oh, right, that's why i held off. you said something last night about a new patchset to do that19:29
anteaya248126 prevents manage-projects from making new messes19:30
nibalizerya i didn't get around to that last night19:30
nibalizerI can do today19:30
fungianteaya: it's the other change (270537) which has the cleanup script for the present mess19:30
anteayafungi: very good19:30
anteayaI'm having such a hard time keeping anything straight in my head today19:30
funginibalizer: no worries, i think my memory is just going19:30
anteayasorry to add to the confusion19:30
fungianteaya: me too!19:30
clarkbdougwig: fungi of course as soon as I hold a node to debug the test decides to pass19:30
dougwigclarkb: you just need to login to every rally job.19:31
*** jsavak has quit IRC19:31
*** moravec1 has joined #openstack-infra19:32
*** jsavak has joined #openstack-infra19:32
*** markus_z has quit IRC19:32
*** peterlisak has quit IRC19:32
*** dkaigarodsev_ has quit IRC19:32
*** moravec has quit IRC19:32
*** annegentle_ has quit IRC19:33
clarkbya I will have to be more aggressive19:33
mriedemwhen we hit something like this, the job is just automatically restarted isn't it?
*** peterlisak has joined #openstack-infra19:33
clarkbmriedem: yes19:34
fungiAJaeger: starting to look at 279238, what happens if a stable point release is tagged on a client while there's not yet a newer tag on master?19:34
*** FallenPe_ has joined #openstack-infra19:34
*** dkaigarodsev has joined #openstack-infra19:34
mriedemclarkb: ok, someone was trying to classify that in e-r but i don't see a point19:34
fungiAJaeger: i think in that case you'll end up with the master documentation replaced by the stable branch documentation?19:34
clarkbmriedem: I am trying to debug that with neutron rally tests. Theory is it is related to the sparse swapfile but my change to switch that to preallocated swapfile hasn't been able to merge yet19:34
fungiAJaeger: wait, nevermind. if there's no newer tag on master then there's no newer documentation already published for "master"19:35
fungiAJaeger: since those projects only run this job on release and not in post19:35
clarkbgrr just missed one19:36
*** FallenPegasus has quit IRC19:37
*** markusz has joined #openstack-infra19:38
crinkleyolanda: i wrote up things i went through to get the non-ca certs to work, one of these might help
clarkbdougwig: fungi ok watching two more now19:39
*** annegentle_ has joined #openstack-infra19:39
*** shardy has quit IRC19:41
anteayazaro fungi I have incorporated your manage-projects clean up steps in the rename workflow on the etherpad19:42
fungithanks anteaya19:42
AJaegerfungi: I'm not getting your problem. The extra parameter is only used for *server* repos, and there's always content published19:43
clarkbnow two in a row passing on me :)19:44
AJaegerfungi: ah, you figured it out, I think...19:44
*** ajmiller_ has joined #openstack-infra19:44
clarkbneutron just needs people to stare at these tests to make them pass19:44
clarkb100% effective so far19:44
mtreinishShrews: sure, I'll push a fix for that in a bit19:44
mtreinishShrews: also do you know why: is failing the ansible job19:45
*** ccrouch has joined #openstack-infra19:45
Shrewsmtreinish: yeah. ansible broke us19:45
Shrewsmtreinish: fix put up for review19:46
mtreinishah, ok19:46
Shrewsthough now i have to fix ansible19:46
fungiAJaeger: spotted one obvious error (see inline comment) but i'm also still thinking through the logic19:46
AJaegerfungi: thanks for spotting - but yes, please think through it properly...19:46
*** ajmiller has quit IRC19:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix tag publishing of server projects
*** tzn has joined #openstack-infra19:51
*** Sukhdev has quit IRC19:54
*** Sukhdev has joined #openstack-infra19:54
*** dimtruck is now known as zz_dimtruck19:55
*** zz_dimtruck is now known as dimtruck19:55
*** sridhar_ram1 has joined #openstack-infra19:55
*** tzn has quit IRC19:55
*** eliqiao_ has quit IRC19:56
*** amotoki has joined #openstack-infra19:56
*** sridhar_ram has quit IRC19:56
Sam-I-AmAJaeger: ^^^ thanks!19:57
*** Swami has quit IRC19:57
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Add openstack/broadview-lib project
*** eliqiao_ has joined #openstack-infra19:58
openstackgerritDmitriy Ukhlov proposed openstack-infra/project-config: Adds neutron tempest tests for oslo.messaging drivers
*** ociuhandu has quit IRC19:59
*** amotoki has quit IRC20:00
*** fawadkhaliq has joined #openstack-infra20:02
*** baoli has quit IRC20:03
clarkb did fail but not due to the thing we are trying to catch20:04
*** tzn has joined #openstack-infra20:05
*** ayoung has quit IRC20:06
*** aysyd has quit IRC20:06
*** redixin has joined #openstack-infra20:06
*** vgridnev has joined #openstack-infra20:07
clarkbI have 4 in progress rally neutron neutron jobs I am watching now. eventually we should get one20:08
*** esker has joined #openstack-infra20:09
*** austin81 has joined #openstack-infra20:11
openstackgerritMatthew Treinish proposed openstack-infra/shade: Prepare functional test subunit stream for collection
*** nwkarsten has joined #openstack-infra20:12
*** ddieterly has quit IRC20:13
mtreinishShrews: ^^^ I'm not sure if that'll work or not though20:13
*** hashar has joined #openstack-infra20:14
*** ldnunes has quit IRC20:14
clarkbjenkins 05 is no longer running jenkins20:16
clarkbI am restarting the service now20:16
*** esikachev has quit IRC20:16
anteayaclarkb: thank you20:17
samueldmqpabelanger: looks like stackalytics.o.o has been fixed20:19
fungiheh. it wasn't much of a jenkins server i guess20:19
samueldmqpabelanger: thanks for looking at it20:19
*** kushal has quit IRC20:19
*** sbalukoff has quit IRC20:19
openstackgerritAustin Clark proposed openstack-infra/project-config: Add javascript jobs to stackviz
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Make propose_translation_update more robust
clarkbsomething seems very broken though20:21
clarkbso much red on zuul status page20:21
clarkbI guess I should stop trying to catch the break20:22
mordredclarkb: that does seem to be an issue20:22
*** _dpaterson has joined #openstack-infra20:22
*** ayoung has joined #openstack-infra20:22
*** andreykurilin__ has quit IRC20:22
AJaegerclarkb: and lots of jobs in post and check - and few in gate20:23
clarkbI guess it is a good time to get lunch20:23
mtreinishdims: ^^^ that seems like your territory20:23
clarkband then maybe not lookt at a compute rfor a while20:23
austin81timothyb89, mtreinish, clarkb, jeblair:
*** rcernin has quit IRC20:24
*** annegentle_ has quit IRC20:26
flaper87Is there a way (or would it make sense) to not run CI jobs for patches that "just" fix docstrings/comments ?20:26
flaper87Like, just run pep8 and move on20:26
*** jlk has quit IRC20:26
clarkbflaper87: not really since zuul doesn't introspect change diffs20:26
*** annegentle_ has joined #openstack-infra20:26
flaper87What am I missing? :D20:26
clarkbwell you have to know if a change only touches docstrings20:27
*** jlk has joined #openstack-infra20:27
*** jlk has quit IRC20:27
*** jlk has joined #openstack-infra20:27
clarkbthat requires parsing the diff and the source to determine if the string is a dosctring20:27
clarkbmtreinish: dims I think everything is fialing on that now, I am digging in20:27
clarkbbut yall should definitely too because ya20:27
*** esikachev has joined #openstack-infra20:28
openstackgerritAustin Clark proposed openstack-infra/project-config: Add javascript jobs to stackviz
*** slagle has quit IRC20:28
flaper87clarkb: right, if there's no way to do that now, would it make sense (or useful) to have it ? I'm happy to help with that, TBH.20:28
clarkbmtreinish: dims actually I may have conflated the no logger with the error when that is just output from the command, the failure is the traceback and is squarely in neutron-db-manage20:28
clarkbflaper87: my initial opinion is no, we shouldn't put significant effort into turning zuul into a python CI scheduler20:29
*** ddieterly has joined #openstack-infra20:29
*** slagle has joined #openstack-infra20:29
clarkbbut maybe that is something that could be made pluggable in zuulv3 if you want to attach internal checks to short circuit a run20:30
clarkbI dunno20:30
flaper87clarkb: roger. I just feel it's a waste of CI resources to have jobs running for docstring changes20:30
AJaegerflaper87: but how often does it happen?20:30
pabelangersamueldmq: np20:30
flaper87I wonder if there's a generic enough way to do this. Generic as in multi-lingual. I'd assume there is20:30
*** fawadkhaliq has quit IRC20:30
flaper87AJaeger: TBH, it doesn't matter how often it happens. It's still a waste and OpenStack is getting bigger. We should sum all the cases in every project using CI resources20:31
anteayafungi: for the possible requirements patches for tomorrow's rename, specs would not be in requirements, nor would tests, and I don't think anything fuel related is in requirements20:31
clarkbflaper87: it actually isnt a waste if the results are still valuable20:31
*** jtomasek has quit IRC20:31
flaper87Also, the frequency increases every time we're 2 or 3 months away from the summit ;)20:31
clarkbflaper87: and they definitely are from a trending standpoint20:31
fungianteaya: yeah, that's just there as a double-check20:31
clarkbimo fixing bugs in our software is much more valuable20:31
fungianteaya: it often won't be needed20:32
clarkbsee the current db manage thing that is making every job fail20:32
*** kzaitsev_mb has quit IRC20:32
anteayafungi: awesome, just double checking my addled thinking, I'll move on, thank you20:32
openstackgerritMerged openstack-infra/elastic-recheck: Add query for horizon connection refused test bug 1544619
openstackbug 1544619 in OpenStack Dashboard (Horizon) "test_create_delete_instance race fails in gate-horizon-dsvm-integration with "error: [Errno 111] Connection refused"" [Undecided,Confirmed]
flaper87clarkb: I'm volunteering to help here, not asking anyone to do it20:32
flaper87just wondering if it makes sense from an infra/ci perspective20:32
flaper87if you don't think it does, then fine20:32
fungiflaper87: i have to agree with clarkb on this one, random openstack bugs cause far more collateral damage in wasted workers than changes which fix a docstring20:33
*** Swami has joined #openstack-infra20:33
fungiflaper87: also how many languages to you want to write a parser for to identify comment formats?20:33
clarkbflaper87: and I am sayin your volunteered help is better directed at improving our software20:33
flaper87sure, not saying that's not true. there will always be bugs, fwiw. This is just a micro optimization in our process20:33
fungialso spelled premature optimization20:34
clarkboh derp this is a change that is broken in a giant stack20:34
flaper87ok, I get it. not useful20:34
clarkbso that giant stack is all going to fail but that explains why the gate isn't dying quicker20:34
anteayafungi: I'm wrong sahara-scenario is in there20:34
anteayaI'll make a patch20:34
openstackgerritLingxian Kong proposed openstack-infra/project-config: Add distil jobs back
clarkbmtreinish: sorry for the alarm, I just started seeing many many neutron tests all fail the same way without realizing they are part of the same stack20:35
*** |-paul-| has joined #openstack-infra20:35
mtreinishclarkb: ah ok, no worries20:35
*** dpaterson has quit IRC20:35
lane_kongAJaeger: hi, i want to know if there is any guidance for the configurations in gate jobs20:36
AJaegerlane_kong: did you read the Infra Manual?
fungiflaper87: basically it would be useful at least to first do some math on the percentage of changes which only touch comments in the languages you're interested in doing this for, and then the average number of patchsets those types of changes incur (compared to our overall average). i'd wager it ends up being a reasonably small percentage as to be best ignored20:37
AJaegerlane_kong: for official projects, there's the PTI as well:
AJaegerlane_kong: for a project like distil: Start with one job and evolve ;)20:38
flaper87fungi: fair enough. I'll do this and report back either way. You guys know the gate and CI usage way better than me for sure :)20:38
lane_kongAJaeger: ok, we wil20:38
lane_kongAJaeger: thanks!20:38
fungiflaper87: the relative patchset count is also almost certainly further amplified by people who make code-altering changes which are broken rechecking their change a few times before deciding they should actually look at the failing job logs20:39
openstackgerritAnita Kuno proposed openstack/requirements: Change project name from sahara-scenario to sahara-tests
anteayafungi: does that have to wait for the repo rename or can it go in anytime?20:40
*** julim has quit IRC20:40
clarkbdougwig: that job looks to be hung and my ssh connection is not responsive20:40
clarkbdougwig: the last thing in syslog is some ovs-vsctl thing20:41
dougwigclarkb: ovs shouldn't mess up your ssh connection.  kernel panic?  or did eth0 get messed with somehow?20:41
fungianteaya: needs to happen during or soon after the maintenance20:41
clarkbdougwig: I am going to check nova console-log shortly20:42
anteayavery good20:42
anteayafungi: I'll workflow -1 it20:42
clarkbdougwig: that should tlel us if kernel panic or similar hapapned20:42
fungianteaya: otherwise we just end up breaking the list before the maintenance rather than having it get broken after the maintenance20:42
clarkbdougwig: this again20:44
clarkbwhich is what showed up in the magnum jobs20:44
clarkbso something is really making the kernel unhappy, if we can get that d-g change in we can rule out the swap sparse file20:44
dougwigclarkb: or really slow.  how does memory pressure look?  lots of swapping to a slow vm disk?20:46
nibalizerflaper87: we already have some stuff so that changes that only touch unit tests don't run tempest , or if you changed only docs under /doc20:46
*** ybathia has joined #openstack-infra20:46
bkeroclarkb: Looks like memory or disk problems20:46
nibalizeranyways if someone did modify python code Id want the test to validate that its still python20:46
clarkbdougwig: no idea, I can't get on the box20:46
clarkbbkero: right thats the theory with the sparse swap file, you can get into a situation where you have no disk and no memory and bam you lose20:46
bkerojbd = journaling block device = kernel thread responsible for the maintenance of20:47
dougwigclarkb: that's an internap node, not ovh like last night.20:47
clarkbdougwig: yes but it also uses sparse swapfile20:47
bkeroclarkb: Doesn't the linux kernel explicitly prevent you from using a sprarse swapfile?20:47
clarkbbkero: nope, you loopback mount it20:47
bkeroSo you have to loopback-mount to create a level of indirection20:47
clarkbI have a change up to stop doing that and fallocate a file instead20:47
clarkbbut that has broken on other things20:48
austin81greghaynes: I am in need of your nodepool expertise. Should I create an entirely new element for stackviz in /nodepool/elements/ ... or just add my npm/tar/gulp steps to an existing element (probably  node-devstack?)20:48
bkeroclarkb: Is disk space that tight that you can't just pre-allocate it?20:48
bkeroOr is this ironic bring-up time?20:48
fungidangerous in production obviously, but was a workaround for platforms where we have no second partition to use for swap and not enough space on the rootfs for some jobs to take up a lot of it with a preallocated swapfile20:48
clarkbbkero: that was the concern but it may be biting us hard here20:48
bkeroMaybe ramzswap at that point?20:48
clarkbbkero: that likely won't help at all :)20:49
clarkbits jobs like magnum with 1.2GB of head room booting many 1gb ram or bigger VMs20:49
bkeroJust trading CPU for memory :)20:49
clarkb(I don't recommend this but hey)20:49
fungithe theory was we have some jobs which need pretty much all of our smallest root filesystems, but other jobs which need swap on providers that don't give us a non-swapfile option, so sparse should allow each of them to use the system in the way they need. however this ignores the possibility of some jobs wanting to use all disk and swap20:49
clarkbbkero: that is up to them to do20:50
clarkbbut that would possibly be an option for them20:50
bkeroSo...if you need the sparsefile for bring-up time, and KVM isn't running as root, why not set your reserved blockcount on FS to match your swapfile size?20:50
dougwigpreallocate should help.  but vm disk swap; fine for page outs, but if a nova vm has to actively use it, man that job is doomed either way.20:50
clarkbbkero: or just preallocate... it is what we are doing20:50
bkeroYeah, that works if you can afford the time to preallocate20:51
clarkbbkero: its dirt cheap on ext420:51
bkeroalthough depending on the filesystem that might be really quick20:51
fungibkero: yeah, so much of these impacted jobs run as root anyway that reserved block count isn't useful20:51
zaroanteaya, fungi : here's my ranming projects test:
clarkb* it is what we are trying to do20:52
zaroit didn't work right for me. so could you please verify that my testing is valid before we continue to debug?20:52
clarkbdougwig: they apparently had great success until we started adding these new clouds without swap in their flavors20:52
fungii think probably the ultimate solution is to stop providing swap, but perhaps provide a hint as to the most effective place on a given system to add swap, and then let jobs which need to mkswap something at an appropriate size before they run their main payload20:52
clarkbdougwig: for rally I wonder if rally is using a larger flavor than tempest20:52
clarkbdougwig: since tempest uses a 64MB ram flavor iirc20:52
*** daemontool_ has joined #openstack-infra20:53
*** e0ne has quit IRC20:53
clarkbboris-42: any idea what flavor rally will use on the rally neutron jobs?20:53
dougwigclarkb: i know octavia uses 512, one at a time.  let me check on rally20:53
zarofungi, anteaya : it doesn't look like projects are saved to account_project_watches table anymore.20:54
anteayazaro: thanks for testing, looking at your paste20:54
anteayazaro: thats good we found out now, thank you20:54
fungizaro: they only appeared in account_project_watches if people set their accounts to watch that project20:54
anteayazaro: sorry it is an etherpad, looking at your etherpad20:54
* dims reading scroll back20:54
clarkbdougwig: fwiw looks very similar to the internap one20:55
zarofungi: ok. that's good.20:55
clarkbdougwig: so Ithink this is the same issue with different symptoms20:55
fungizaro: but that looks like our process to me. is it possible the project name is now tracked in a second place besides the changes table?20:55
dimsclarkb : ack on the "failure is the traceback and is squarely in neutron-db-manage"20:55
*** daemontool has quit IRC20:55
zarofungi: i took a look but didn't find any other projects tables or columns in the db.20:55
anteayazaro fungi well except the part where zaro manually created the project that it will be renamed to20:55
*** Sukhdev has quit IRC20:56
clarkbdims: you can ignore that20:56
fungianteaya: oh! good eye20:56
clarkbdims: I didn't realize that all of the failing jobs belonged to a large related stack that was broken20:56
anteayaas our process doesn't have any existance of the project that has the new name prior to the process commencing20:56
dimsclarkb : no worries :)20:56
*** [1]Thelo has joined #openstack-infra20:56
zaroanteaya: ahh, ok.  i thought we actually created new project first?20:57
*** ociuhandu has joined #openstack-infra20:57
anteayathe old project exists20:57
fungizaro: yeah, for this test you should have only created yprojects/ytest1 and then after the process is complete yprojects/ytest1 will no longer exist and yprojects/ytest2 will be there instead20:57
dougwigclarkb: rally-neutron looks to be using the "tiny" flavor from devstack.20:57
anteayathe existing project is _renamed_20:57
anteayano project is created20:57
clarkbdougwig: it should use nano iirc20:57
dimsanteaya : mtreinish has blessed
anteayaand test2 should have exactly the same git log as test120:58
zaroanteaya, fungi : ok.  i thought it was just moving changes from one project to another.20:58
clarkbdougwig: so that may be a simple fix if that is why we have memory pressure there20:58
zaroi'll try without creating ytest2 first20:58
*** Thelo has quit IRC20:58
*** [1]Thelo is now known as Thelo20:58
anteayait is taking one existing project with history and giving the repo a new name20:58
anteayaand maintaining the history20:58
*** moravec1 has quit IRC20:59
anteayazaro: thank you for good clear notes20:59
anteayazaro: glad we caught this now, thank you20:59
*** baoli has joined #openstack-infra21:00
anteayadims: does jenkins like it yet?21:01
anteayadims: jobs are running should finish in a few minutes21:02
*** notmorgan is now known as morganfainberg21:02
therveIf someone can review that would unblock kilo requirements (and heat kilo gate)21:03
*** dkranz has quit IRC21:03
*** jsavak has quit IRC21:03
mordredyay my patch was useful21:04
mordredclarkb: ^^ that requirements patch is the type of patch infra folks can do21:04
*** jsavak has joined #openstack-infra21:04
mordredas it relates to our afs mirrors21:04
mordredfungi: ^^21:04
dimsanteaya : that's just periodic, so we'll only know when the first run runs21:04
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Remove neutron grenade from lbaas jobs lists
dimsanteaya : it's a cut-n-paste from the standard template with extra PROJECTS and LIBS_FROM_GIT, so fairly safe21:05
*** Gordoni has joined #openstack-infra21:05
dimsand won't break anything else :)21:05
*** jsavak has quit IRC21:06
*** jsavak has joined #openstack-infra21:06
fungimordred: indeed, no apt-get build-dep with our mirrors, unless we add deb-src lines21:06
*** yamahata has joined #openstack-infra21:06
mordredfungi: yup21:06
fungier, rather, unless we start mirroring source packages _and_ add deb-src lines to the sources.list21:06
* morganfainberg waits to see who will be the first to fall prey to mor<tab>21:06
mordredfungi: TIL that build-dep requires deb-src lines21:06
dougwigclarkb: not finding the rally folks i know, so i filed a bug on the tiny/nano issue.21:07
fungimorganfainberg: you're not not morgan any more again?21:07
*** ddieterly has quit IRC21:07
morganfainbergfungi: only for a bit :P21:07
olaphnow he's just plain not mordred21:07
fungimorganfainberg is indeed notmordred21:08
* morganfainberg resists urge to make it even more confusing21:08
*** mfalatic has joined #openstack-infra21:08
anteayadims: I was waiting for the xml and config-layout job to finish21:09
morganfainbergok. so.. i missed a ping earlier someone was maybe saying swiftclient things?21:09
morganfainbergit's out of my scrollback buffer... maybe clarkb ?21:09
* morganfainberg is trying to be good at responding.21:10
dimsanteaya : thanks!21:10
clarkbwas mordred21:10
anteayadims: thank you21:10
morganfainbergclarkb: ooh21:11
*** ihrachys has joined #openstack-infra21:11
morganfainbergmordred: oh hai, about what you said here earlier when i was asleep...or getting teeth drilled21:11
anteayamorganfainberg: was this it?
morganfainbergoh that21:12
openstackmorganfainberg: Error: "!" is not a valid command.21:12
morganfainbergthat was it.21:12
morganfainberganteaya: ok not important21:12
anteayamorganfainberg: awesome21:12
morganfainbergmordred: nvm. figured it out thanks to anteaya21:12
*** morganfainberg is now known as notmorgan21:12
openstackgerritMerged openstack/os-testr: Enable testr run passthrough arguments
*** _nadya_ has quit IRC21:12
anteayafungi zaro I'm offline in about 45 minutes for about an hour or so, will pick up when I get back, depending on what zaro's next round of testing reveals21:13
anteayafungi: I'm at the point of generating the db commands but will hold off until I hear from zaro21:14
*** kzaitsev_mb has joined #openstack-infra21:14
zaroanteaya, fungi : ok. revisted testing worked like a charm.  i have updated the etherpad for reference. so looks like we are good.21:15
openstackgerritMerged openstack-infra/project-config: Periodic Devstack job against oslo.* from master
*** kgiusti has left #openstack-infra21:19
fungianteaya: zaro: thanks for all the planning and testing/validation21:21
anteayazaro: thank you!21:23
*** flip214 has quit IRC21:23
*** esikachev has quit IRC21:23
*** esker has quit IRC21:23
*** flip214 has joined #openstack-infra21:24
*** flip214 has quit IRC21:24
*** flip214 has joined #openstack-infra21:24
*** ybathia has quit IRC21:24
mriedemis it possible to disable people from voting on changes? -- apparently people can't vote on that21:26
*** ajmiller_ is now known as ajmiller21:26
*** ihrachys has quit IRC21:26
*** esker has joined #openstack-infra21:26
*** ihrachys has joined #openstack-infra21:27
yolandahi crinkle, so i read your notes . The update of the certs, with update-ca-certificates is running fine for me,  i'm thinking that the problems is the self-signed cert missing some setting, to act as CA21:27
fungimriedem: we're fixing that tomorrow during a gerrit maintenance21:27
yolandaeven if i do a verify cert using the same cert as path, fails21:28
fungiyolanda: there's a constraint you can set. let me find it21:28
yolandafungi some flag in openssl.cnf?21:28
mriedemfungi: ah ok21:28
fungiyolanda: yeah, you need to set a basic constraint to allow signing, which is how a cert is identified as an authority by some software21:29
yolandafungi i think the problem may be around these flags yes21:29
*** dprince has quit IRC21:31
fungiyolanda: the corresponding rfc section is
*** esker has quit IRC21:31
yolandabasicConstraints = CA:FALSE21:32
yolandalooks as it may be that21:32
yolandafungi, key usage looks fine keyUsage = nonRepudiation, digitalSignature, keyEncipherment21:32
fungiright, i think you may need to flip that to true. still digging in openssl docs21:33
yolandabut i bet ca needs to be true21:33
yolandagoing to try that21:33
crinkleyolanda: I didn't need to do anything special in openssl.cnf to generate self-signed certs21:36
*** _amrith_ is now known as amrith21:37
yolandabut crinkle you are not using pure self-signed certs, but a root ca, and intermediate cert right?21:37
zaromtreinish: yes, you can deny groups of from voting. it's done with gerrit acls.21:37
yolandai was following same tutorial as you i think21:37
crinkleyolanda: no I've reproduced your issue using just the self-signed certs21:37
mtreinishzaro: ?21:37
mtreinishzaro: oh, you meant mriedem, nm21:37
crinkleyolanda: without the CA21:37
clarkbhrm not getting any of the stuff done today that I wanted to get done, but fixing gate is important21:38
clarkbnibalizer: when is a good time to do mirror stuff?21:38
*** MarkAtwood has joined #openstack-infra21:39
yolandacrinkle if you have self-signed certs working in openstack, can you create one with alt names for east and west and share? if that works we could use that in our hiera, and test on east cloud21:39
yolandai may have something wrong on my local openssl.cnf, don't know21:40
clarkbcacti says we are well below bw thresholds
crinkleyolanda: sure21:40
*** jsavak has quit IRC21:41
clarkbbhs is this side of the atlantic at least21:41
yolandacrinkle, fungi, it works now21:41
yolandait was the flag21:41
crinkleyolanda: oh yay21:42
yolandai still have the funny neutron bug, but that's something known, and i need to test a patch that degorenko created21:42
yolandacrinkle, mine contain the alt names, do you want to test?21:42
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Adjust Ops Guide Specialty Team meeting
clarkbI guess I should check haproxy logs for thta IP21:42
crinkleyolanda: sure21:42
yolandagoing to paste you in private21:43
yolandaas you need the key21:43
nibalizerclarkb: um21:43
clarkbgrep /var/log/haproxy.log returns nothing21:43
nibalizerclarkb: i haave an appointment until 321:43
nibalizeryou should maybe just go without me21:43
clarkbnibalizer: no rush I am pulling on the tests are failing thread and being distrated21:43
nibalizeri still haven't had time to redo the auth21:43
fungiyolanda: awesome work!21:44
clarkbfungi: that implies packets never reached git.o.o21:44
clarkbfungi: which means it could be anything between git.o.o and that ip ?21:44
clarkbfirewall, asymettric route, cosmic rays21:44
yolandafungi, thanks! i hope we have it 100% functional tomorrow, with the neutron fix21:44
yolandacrinkle, we may need to update the puppet code, to don't use chain file in apache, and add the certs to global trust21:45
crinkleyolanda: i'm working on that now21:45
clarkbI will try running some tests from bhs1 to git.o.o21:45
yolandaalso we can set only one cert for all components, as they are wildcard certs21:45
fungiclarkb: yeah, no ipv6 on that worker, so would have to be something on the network between them dropping packets21:45
*** moravec has joined #openstack-infra21:46
*** esker has joined #openstack-infra21:46
clarkbmotd on bhs1 mirror is fun21:46
fungiclarkb: back in my isp days, i was frequently plagued by too-small flow tables and state tables on various sorts of routing devices21:47
fungiwhich would cause exactly these sorts of symptoms21:47
*** |-paul-| has quit IRC21:47
nibalizerclarkb: ok21:47
*** jsavak has joined #openstack-infra21:48
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Set PS4 to provide additional debug info
*** alivigni has quit IRC21:48
*** jsavak has quit IRC21:48
yolandaleaving for today, thanks for help with the certs!21:49
clarkb is another case of this21:49
fungiyolanda: thanks for working on it!21:49
clarkbI should run a logstash queruy to see how bad it is21:49
*** jsavak has joined #openstack-infra21:49
fungialso ovh-bhs121:50
fungiwonder if they have a routing issue? except git.o.o and jenkins*.o.o are all in the same rax region21:50
*** bpokorny has quit IRC21:51
fungithough entirely different netblocks so i guess could be a problem with a particular prefix in bgp21:51
clarkbthere are ~22 instance in the last week all from today21:51
clarkband ovh-gra1 has some occureances as well21:51
fungibut all in ovh?21:52
*** eharney has quit IRC21:52
*** ociuhandu has quit IRC21:52
fungiwe don't have any ovh'ers hanging out in channel i guess21:52
*** dpaterson has joined #openstack-infra21:52
*** ayoung has quit IRC21:52
*** weshay has quit IRC21:53
*** ociuhandu has joined #openstack-infra21:53
clarkbmtr is unhappy about a level3 hop in dallas21:53
clarkbfrom ovh bhs mirror21:53
openstackgerritgreghaynes proposed openstack/diskimage-builder: Install docker for tests
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests
clarkbbut no loss behind it so that may just be a device that doesn't like us21:54
fungiyeah, it may just throttle/filter icmp ttl exceeded messages21:54
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Update SSL configuration
fungior it may be filtering udp to its loopback (does mtr use udp by default like traditional traceroute?)21:55
fungier, no, not to its loopback21:55
dimsfungi : clarkb : a zuul irc reporter is easy to do? (wanted to see if the oslo-master periodic jobs can report back on channel)21:55
fungiits loopback may not be routable21:55
fungifrom the internet21:55
greghaynesianw: that is a really wierd fail...21:55
fungicommon cause21:56
greghaynesianw: yum related so wondering if you might have any insigt21:56
fungiif you're getting 100% loss on an intermediate hop21:56
clarkbdims: shouldn't be too hard, reporters are pretty pluggable and self contained21:56
*** daemontool_ has quit IRC21:56
*** amotoki has joined #openstack-infra21:56
dimsclarkb : hmmm :)21:57
clarkbdims: we do already have an email reporter which we use to report periodic job results21:57
*** moravec has quit IRC21:57
clarkbdims: not sure if that is useulf to you though21:57
*** rcarrillocruz has joined #openstack-infra21:57
*** rcarrillocruz has left #openstack-infra21:58
dimsclarkb : right i saw that, may be better to start with that21:58
clarkbfungi: 98.8% so ya it likely isn't a real problem there21:58
clarkbfungi: no loss behind it21:58
*** nwkarsten has quit IRC21:58
*** sdake has quit IRC21:58
*** nwkarsten has joined #openstack-infra21:59
greghaynesIs there somewhere I can look to figure out the URI's for the new apt-mirrors?21:59
greghaynesI am hoping to switch DIB over to using them..22:00
mtreinishclarkb: we've actually turned off the email reporting for a bunch of the periodic jobs recently22:00
*** sridhar_ram1 is now known as sridhar_ram22:00
*** ayoung has joined #openstack-infra22:00
clarkbgreghaynes: I think22:01
greghaynesclarkb: ah, cool22:01
*** amotoki has quit IRC22:01
*** sdake has joined #openstack-infra22:01
clarkbgreghaynes: er s/apt/ubuntu/22:01
clarkbbecause we can't colocate ubuntu and debian22:01
austin81mtreinish: fixed
greghaynesclarkb: successs:
clarkbgreghaynes: for example22:01
clarkbgreghaynes: and the region/provider info is in /etc/nodepool on the test slaves22:02
*** esker has quit IRC22:02
*** rossella_s has quit IRC22:03
*** chlong has quit IRC22:03
*** nwkarsten has quit IRC22:03
*** burgerk has quit IRC22:03
*** esker has joined #openstack-infra22:03
*** rossella_s has joined #openstack-infra22:03
fungigreghaynes: switch dib over how?22:03
greghaynesfungi: in our CI to use those mirrors when we build images22:04
greghaynes(we build a bunch of images as part of our gate)22:04
fungigreghaynes: there was already a dib-related bug reported earlier today where an element was copying the sources.list from the host but not the apt.conf to go along with it22:04
fungiso i guess try not to make that same mistake22:04
greghaynesfungi: oh, that sounds like something we would do22:05
fungiour mirrors are not signed, so you need the apt configuration we're using to disable signing22:05
SlickNikfungi: do you know where the apt-conf is on the host node?22:05
greghaynesfungi: ah! makes sense22:05
fungiSlickNik: in /etc/apt22:05
clarkbSlickNik: /etc/apt/22:05
*** ajmiller_ has joined #openstack-infra22:05
fungigreghaynes: er, to disable signature verification i mean22:06
clarkbjhesketh: good day22:06
fungiSlickNik: greghaynes: specifically, we're using run-parts type snippets in the *.d directories under /etc/apt22:06
clarkbfungi: I am thinking the only thing we can really do here is collect VM uuids and IP addrs then apss that along to ovh?22:07
*** _dpaterson has quit IRC22:07
clarkbfungi: I will work on collecting that info unless you think there is something more direct we can do22:07
greghaynesfungi: cool, thanks22:07
jheskethclarkb: "I said, good day!"22:07
fungiclarkb: likely. if we see a pattern on some netblocks but not others that might be useful for them22:07
fungiclarkb: though also possible it's just certain nova hosts which are impacted and not others22:08
*** Sukhdev has joined #openstack-infra22:08
fungiin which case i wouldn't expect any pattern in the ip addresses22:08
*** ajmiller has quit IRC22:08
ianwgreghaynes: checking...22:08
*** tiswanso has quit IRC22:10
*** thorst_ has quit IRC22:11
crinkleyolanda: your cert doesn't work for me, I see instead of the wildcard22:11
*** thorst_ has joined #openstack-infra22:11
*** dimtruck is now known as zz_dimtruck22:11
*** nwkarsten has joined #openstack-infra22:11
SlickNikclarkb / fungi: strange we're using the apt-conf element from dib to override the apt-conf while image-building, but it seems to fail:
*** ajmiller_ is now known as ajmiller22:14
SlickNikFull output at if it's helpful at all22:14
mtreinishdims: what's wrong with using openstack-health to track the job you're adding?22:14
ianwgreghaynes: hmm, so i'm not sure about the error, but yeah yum actually got hung up22:15
mtreinishdims: because if there's a gap there I'd like to figure it out so we can fix it22:15
greghaynesianw: seems like another potentially bad package somewhere in the mix22:15
ianwgreghaynes: firstly, i don't think it's worth our time debugging the centos6 builds, because they're just not used22:15
mtreinishthat's why I stopped the email reporting on periodic-qa22:15
SlickNikAh, maybe I think we might need to copy the file to the chroot before we can use it as an override?22:15
fungiSlickNik: i wonder if it doesn't handle /etc/apt/apt.conf.d/* files?22:15
ianwgreghaynes: we should switch it to 722:15
greghaynesianw: I think that patch is to switch on centos 722:16
greghaynesianw: so, yes :)22:16
mtreinishpleia2: that reminds me want to help with locking up the openstack-qa list on mailman?22:16
greghaynesianw: we can remove the 6 tests if we want22:16
*** thorst_ has quit IRC22:16
ianwgreghaynes: it's building *on* 7, but testing building 622:16
pleia2mtreinish: yep!22:16
pleia2mtreinish: did you track down the list admins?22:16
*** tpsilva has quit IRC22:16
*** araji has joined #openstack-infra22:17
ianwgreghaynes: yeah, i want us to functional test centos-minimal and fedora-minimal latest22:17
SlickNikwill look into it some more — thanks for confirming that the file is indeed at /etc/apt/apt.conf.22:17
ianwgreghaynes: we are *so* close!22:17
greghaynesianw: Centos is just odd here because there are separate elements for 6 and 7, so we have to explicitly turn them on/off22:17
pleia2mtreinish: as discussed, we'll just shut it down administratively through the UI instead of deleting on the infra side22:17
*** esker has quit IRC22:17
mtreinishpleia2: well it's sdague, jaypipes, and dkranz22:17
mtreinishpleia2: yeah, that's fine22:18
ianwgreghaynes: so this is why i wrote ->
pleia2mtreinish: just have one of them give you the admin password and I can tell you what to change22:18
ianwgreghaynes: it gives you a nice interface to the functional tests, and we can change defaults for ci in a list22:18
greghaynesianw: cool, I'll have a look22:18
*** annegentle_ has quit IRC22:18
greghaynesianw: perfect22:18
mtreinishpleia2: sdague probably has called it a day already, dkranz isn't on irc, and dunno if jaypipes is still online22:19
*** annegentle_ has joined #openstack-infra22:19
ianwgreghaynes: i saw your change to install docker / do those image output tests ... i think that's fine but i think we can run it separately22:19
*** vgridnev has quit IRC22:19
*** ganso has quit IRC22:19
mtreinishpleia2: I also doubt any of them remember the admin password, we haven't used that list for anything real in at least a couple years22:20
*** zz_dimtruck is now known as dimtruck22:20
pleia2mtreinish: ah, I doubt I can retrieve the password, but I can probably set a new one and give that to you, I'll look into it now22:21
clarkbfungi: is what I have so far22:21
clarkbif someone is able to grep haproxy logs to see if any of those show up that would be awesome22:21
clarkbI need t ofind the remaining IPs and their uuids22:21
*** amitgandhinz has quit IRC22:21
fungiSlickNik: /etc/apt/apt.conf.d/* actually, i think22:22
*** dims has quit IRC22:23
mtreinishpleia2: ok, thanks22:23
*** hdd has quit IRC22:23
fungiclarkb: will check them in just a sec22:23
*** annegentle_ has quit IRC22:23
clarkbfungi: there is also at least one suspect net block22:23
*** araji has quit IRC22:24
*** markvan_ has quit IRC22:24
*** markvan has joined #openstack-infra22:25
*** markvan has quit IRC22:25
*** Swami has quit IRC22:26
*** dims has joined #openstack-infra22:26
jeblairmtreinish, dims: ++ using the dashboard for that22:27
*** markvan has joined #openstack-infra22:27
dimsmtreinish : +1 to openstack-health from me as well, currently the jobs just don't show up because of the restriction you mentioned mtreinish (
*** pradk has quit IRC22:28
dimsmtreinish : only additional thing that would be useful is to be able to specify the list of specific jobs somehow to show up on one page (if we don't have that already)22:29
dimsmtreinish : jeblair : the reporter seems useful as a push mechanism to get in front of folks (just like the elastic recheck bot)22:30
mtreinishdims: well right now it does metadata grouping22:31
mtreinishso you get:
mtreinishwhich is all the periodic jobs22:31
*** thorst_ has joined #openstack-infra22:31
mtreinishjeblair: is working on adding arbitrary tags to jobs so we can build custom groups22:32
jeblairdims: we're adding job tags to zuul so you'll be able to group them in the zuul config and health will be able to group on that22:32
*** annegentle_ has joined #openstack-infra22:32
jeblairjob tags patch ^22:32
clarkbfungi: that is a complete list22:33
*** thorst_ has quit IRC22:33
clarkb158.69.64.0/24 seems very suspect22:33
*** thorst_ has joined #openstack-infra22:33
dimsjeblair : ah nice22:33
openstackgerritSean M. Collins proposed openstack-infra/devstack-gate: [WIP] Switch to new Neutron devstack
dimsmtreinish : y i need just the oslo.* master jobs, hopefully we can do that with the zuul tags22:34
clarkbgrep says we have used more IPs in that range that presumeably did not exhibit this issue22:34
*** daemontool has joined #openstack-infra22:34
jeblairdims: i like the irc reporter in principle, but i also don't want that to cause us to want to make lots of little pipelines so that certain jobs get reported to certain channels, etc (which is what prompted the job tags thing but for email)22:34
*** mfalatic has quit IRC22:34
jeblairdims: if we did do an irc reporter, we'd probably want to do something where we specified which channels were used in the jobs22:35
clarkbjeblair: fyi are ovh nodes that exhibited trouble connecting to git.o.o22:35
dimsjeblair : right.22:35
jeblairdims: but at any rate, health is probably better for this specific use case22:35
jeblairclarkb: thanks; do you have an email to send that to?22:36
jeblair(i'm wondering if we should record contacts in our secrets file too...)22:36
clarkbjeblair: I do, but still in the collecting data phase22:36
jeblairclarkb: cool22:37
dimsjeblair : agree. if we can get past the "subunit2sql data pruning" problem mtreinish mentioned22:37
anteayazaro: can you check what I have on to incorporate the cleanup registered users work? can we get rid of that section on the end now?22:37
clarkbhaproxy says the first one on that list did a cD which I think is client side disconnect?22:37
fungiclarkb: i found these hits
*** ayoung has quit IRC22:37
mtreinishdims: I just need to figure out why the cron job I added doesn't do anything22:37
fungiclarkb: those are the addresses which actually show up in the current haproxy log and how many times22:37
dimsmtreinish : does it run? :)22:37
*** dtardivel has quit IRC22:38
anteayaSergeyLukjanov: can you look over in preperation for the rename tomorrow, please?22:38
fungiclarkb: start time on the file is Feb 11 03:18:01 so presumably all the entries in your list are since then?22:38
mtreinishdims: well fungi and I were looking at it a while ago and IIRC there wasn't any indication one way or the other22:38
mtreinishdims: I've got: up to try and add more verbosity when it's doing something22:38
dimsmtreinish and where do the logs go?22:39
clarkbfungi: 0802 for the first one22:39
clarkbfungi: so yup22:39
mtreinishdims: on logstash.o.o22:39
mtreinishdims: so infra-root needs to check for us :)22:39
*** thorst_ has quit IRC22:39
jeblairmtreinish: what do i need to check?22:40
clarkbfungi: I will add the timestamps for each occurance to the list as well as the haproxy lines for each one that hit22:40
clarkbfungi: that should be a fairly complete set of data to hand to ovh22:40
jeblairmumble mumble local afs git mirror mumble22:41
mtreinishjeblair: if this cron job is working:
dimsmtreinish : ah. ok22:41
mtreinishthe log file is /var/log/subunit2sql_migration.log22:41
jeblairsubunit2sql-db-manage expire: error: argument --expire-age/-e: expected one argument22:41
jeblair0 3 * * 7 subunit2sql-db-manage --config-file /etc/subunit2sql.conf expire --expire-age $expire_age >> /var/log/subunit2sql_migration.log 2>&1 &22:42
jeblairmtreinish: i think maybe change ' to " in puppet?22:42
*** salv-orlando has joined #openstack-infra22:42
mtreinishjeblair: that probably would do it :)22:42
fungii wonder what the vos release time would be like if we were calling it on every gerrit ref-updated event22:43
*** thorst_ has joined #openstack-infra22:43
jeblairfungi: well, for that we could just use a rw volume; though then we lose fault tolerance22:43
fungialso warming the cache in our nodepool images would probably be the biggest challenge22:43
jeblairgood thing that was just a mumble and not a real suggestion :)22:43
*** salv-orl_ has quit IRC22:44
*** redixin has quit IRC22:45
openstackgerritTim Burke proposed openstack/requirements: Update flake8 constraints
openstackgerritTim Burke proposed openstack-dev/hacking: Run all flake8 checks
anteayafungi: also when you get a minute can you give a once over, I filled in the commands22:47
*** dingyichen has joined #openstack-infra22:47
*** thorst_ has quit IRC22:48
anteayaI won't be around for a good chunk of tomorrow as I have to clean my house since I'll be traveling for 2 weeks22:48
fungijeblair: yep. and musings on my part, not actual design ;)22:48
anteayabut I will be here for the renaming22:48
jeblairmtreinish: are you going to fix the '?22:48
jeblairor should i?22:48
anteayajust want to ensure my prep is accurate, if we have time tonight to do so22:48
mtreinishjeblair: I'm writing the commit msg right now22:48
jeblairmtreinish: cool22:49
zaroanteaya: etherpad LGTM and yes, you can get rid of section at the end.22:49
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix the expiration cron job
mtreinishjeblair: ^^^22:49
anteayazaro: thank you22:50
anteayafungi: you made a spelling correction in the fuel plugin one, what did I have there?22:51
anteayaoh I didn't have anything at all22:51
*** jsavak has quit IRC22:52
*** jpr has quit IRC22:52
anteayasorry about that, and thanks for fixing it22:53
fungianteaya: yeah, missing openstack/ prefix throughout for that repo22:53
clarkbfungi: jeblair how does that look? any other information that you think we should add?22:53
*** mriedem has quit IRC22:53
anteayafungi: yeah, sorry, looks like you caught them all though, thank you22:53
clarkbnote there have been 8 more since I started collecting info22:53
clarkbI probably won't add them to the list because etime22:54
*** sdake_ has joined #openstack-infra22:54
jeblairclarkb: you added the haproxy lines to indicate that it worked a little bit in close temporal proximity?22:55
bknudson_this job failed: -- says the main setup script failed... how do I find the setup script log?22:56
clarkbjeblair: yes22:56
jeblairclarkb: though the first one is cD22:56
clarkbjeblair: for that subset of instances they did manage to connect to git.o.o to some degree22:56
jeblairwhich i guess is still interesting22:56
clarkbjeblair: but only those with the log lines22:56
jeblairclarkb: yeah.  makes sense.22:56
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Remove job filters and add missing paths to subunit collection
bknudson_hmm, maybe it's this: 2016-02-11 01:26:33.429 | E: You must put some 'source' URIs in your sources.list22:57
*** sdake has quit IRC22:57
clarkbjeblair: and grepping for that suspect /24 in nodepool log sows we use quite a few IPs in that range22:58
clarkbjeblair: so other instances with those IPs are working fine22:58
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Remove job filters and add missing paths to subunit collection
*** sigmavirus24 is now known as sigmavirus24_awa22:58
mtreinishdims, jeblair: part 2 ^^^22:58
*** xyang1 has quit IRC22:58
jeblairclarkb: yeah, so maybe they can localize to a  switch/compute node or something22:58
anteayabknudson_: this may help:
dimsmtreinish : cool thanks22:59
*** bpokorny has joined #openstack-infra22:59
mtreinishdims: hopefully that won't blow up in our face23:00
anteayabknudson_: this doesn't look good:
mtreinishI've never tested any of this stack with more than a couple of test suites23:00
*** sridhar_ram has quit IRC23:00
mtreinishdunno how well the db is gonna scale when we throw all of them at it23:00
mtreinishI want to say perfectly, but who knows :)23:00
fungianteaya: i don't see anything else obviously wrong, looks great!23:01
bknudson_anteaya: fatal: Couldn't find remote ref refs/zuul/stable/kilo/Za987afe0f7504815a9a17535e4429a92 seems to happen pretty often23:01
*** thorst_ has joined #openstack-infra23:03
bknudson_so I don't know if that's good or bad... seems to be ignored23:03
*** gildub has joined #openstack-infra23:03
*** dpaterson has quit IRC23:03
bknudson_I'm going to go with "recheck"23:04
anteayafungi: wonderful, thank you23:04
anteayabknudson_: yeah I get thrown when fatal things aren't really fatal23:05
fungiclarkb: yeah, that looks like the basic useful detail i'd send them23:05
fungialong with some explanatory prose of course23:05
anteayazaro: long live the blueman group23:06
*** regXboi has quit IRC23:07
clarkbjeblair: fungi sent with prose. You are cc'd23:10
*** ihrachys has quit IRC23:11
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Allow zuul-release to push merge commits
jheskethjeblair: if you have a moment ^23:12
jheskethI'd like to push up my merge of master -> feature/zuulv323:12
austin81clarkb, fungi, jeblair, greghaynes: Adding stackviz config to nodepool/elements and nodepool/scripts. Can someone inform me on what nodepool/scripts/ does? I'm not sure if I should be making a script that calls it.23:13
clarkbaustin81: is used by nodepools snapshot image builds to update the image with the contents we want23:13
clarkbaustin81: you will likely want to add an element to nodepool/elements that installs stackviz too23:14
anteayatonyb: so if you look in #openstack-meeting-3 the ops guide folks need some meetbot 10123:14
clarkbwe are moving everything to dib which uses the dib elements and moving away from snapshot builds but still use them until we have reliably image uploads to rax23:14
*** dizquierdo has joined #openstack-infra23:14
anteayatonyb: at least for adding addiional chairs, ending a meeting, and popping into -infra for help if meetbot isn't don't as they expect23:14
anteayatonyb: I'm not sure how to convey that to them23:15
clarkbspeaking of reliable builds is going once going twice review it soon or I will self approve23:15
austin81clarkb: I added an element to elements/node-devstack/install.d/99-prepare-stackviz23:15
tonybanteaya: Thanks, I'll check the logs23:15
austin81clarkb: Seemed like the right place for it, not sure where to add the corressponding script in nodepool/scripts23:15
anteayatonyb: awesome23:15
anteayatonyb: oh they aslo don't know how to start a meeting either it seems, :(23:16
clarkbaustin81: is where I would add it for the snapshots23:16
*** dizquierdo has quit IRC23:16
anteayanot with confidence23:16
clarkbaustin81: have a or whatever and call it from prepare_devstack.sh23:16
austin81clarkb: okay, great thanks!23:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Enable windmill-jobs for all ansible-roles
mtreinishaustin81: probably take a look at what I did for the tempest testr preseeding for an example. I had to go through basically the same process for doing that23:17
austin81mtreinish: That's been my main guide to figuring out nodepool :)23:18
*** ybathia has joined #openstack-infra23:18
*** thorst_ has quit IRC23:20
clarkbeventually we will trim that scripts dir to only things that need to run on the VM after they boot before testing23:20
clarkbbut we haven't completely switched to dib yet23:20
*** thorst_ has joined #openstack-infra23:21
*** ybathia_ has joined #openstack-infra23:21
austin81mtreinish: Mind if I add our code into
mtreinishaustin81: you probably want to isolate the stackviz specific stuff into a seperate script, and just call that script from prepare_node_devstack.sh23:22
austin81mtreinish: that works too. I'll call it after the subunit2sql stuff23:22
*** ybathia has quit IRC23:23
*** ybathia has joined #openstack-infra23:24
*** ayoung has joined #openstack-infra23:25
tonybanteaya: So I'm assuming that the #endmeeting failed to work because it wasn't issueed by a meeting chair? and you have ops so that's why it worked for you?23:26
*** ybathia_ has quit IRC23:26
clarkbtonyb: there is a 60 minute timeout on it23:27
clarkbafter that anyone can end it23:27
tonybclarkb: okay thanks.  That explains it.23:28
*** thorst_ has quit IRC23:29
anteayayeah the timeout23:29
anteayaand yeah the chair thought she made other people chair but she didn't23:30
anteayaand the first attempt to start the meeting failed due to a whitespace before the #23:30
*** chlong has joined #openstack-infra23:30
anteayabut they thought it was because the successful person who did start the meeting had a registered nick23:30
anteayaI don't think they will figure this out on their own23:31
anteayaalso the abandoned the meeting without asking for direction or support from infra23:31
anteayaI just noticed it in my channel flipping23:31
*** baoli has quit IRC23:33
*** moravec has joined #openstack-infra23:34
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)
austin81mtreinish: ^23:36
*** baoli has joined #openstack-infra23:36
fungiclarkb: jeblair had some test suggestions on 277941 that could be added at a later date, fyi. approved23:37
clarkbfungi: thanks, I meant to play with that more in a followup. The tricky thing is going to be races between first delete image being in  a delete state and second image being in build state23:38
clarkbdefinitely doable but as an optimization I figured it could wait (see host of other things I have been battling the last couple days)23:38
clarkbI still havne't done the thing :/23:39
openstackgerritMerged openstack-infra/nodepool: Handle image upload job failures
lifelessclarkb: for home?23:41
fungilifeless: for pbx.openstack.org23:42
fungiclarkb: in your correlated list of connection failures, i also note that the successful hits in the haproxy log are not correlated to the failures on the client side, suggesting they are not from the connections which failed23:43
clarkbfungi: agreed23:43
*** wolverin_ has joined #openstack-infra23:43
fungiimplying initial tcp/syn never arriving presumably23:43
*** wolverin_ is now known as wolverineav23:43
*** annegentle_ has quit IRC23:44
fungiclarkb: oh! i take that back23:44
*** MarkAtwood has quit IRC23:44
fungiin at least a couple instances, the hit logged to the haproxy is ~120 seconds before the failure on the client23:44
*** FallenPe_ has quit IRC23:45
fungithat smells like a connection timeout duration23:45
*** wolverineav has quit IRC23:45
fungikeeping in mind that haproxy is logging only successful establishment of sockets (hence full handshakes) we likely can't rule out that some of these connected initially and then timed out23:46
fungii have to remind myself that log is not a packet capture and assume some application timing behaviors23:46
openstackgerritMerged openstack/requirements: Stop using build-dep for numpy/pyyaml depends
*** marcusvrn_ has quit IRC23:47
openstackgerritMerged openstack-infra/devstack-gate: Preallocate swapfile
fungiclarkb: i take it no grenade jobs ran out of space on / with that ^ in your rechecks23:48
clarkbfungi: not that I saw23:48
fungiso hopefully we can just consider swap a sunk space cost on those machines and call it a day23:49
clarkbfungi: I even logged into one at one point an it had 50GB of free space23:49
fungiodd, because i thought that we were running close to max in ovh with grenade23:49
fungibut maybe that was early on and they upped our disk size?23:49
*** ayoung has quit IRC23:50
clarkbit is 80GB now I want ot say it was more problematic when it was 50GB?23:51
clarkbat the very least if there are real disk constraints we should shake them out with that d-g change in23:51
clarkbright now we ar ein this weird place where things break without being debuggable23:51
clarkbwhich I would prefer to transition to things break and are debuggable23:51
*** dimtruck is now known as zz_dimtruck23:52
nibalizerinfra-root im gonna take the hiera lock to move aroud the omfra credentials some more23:52
fungigo for it23:52
*** nwkarsten has quit IRC23:52
*** nwkarsten has joined #openstack-infra23:53
nibalizeralso i resolved the issue (with crinkle's help) about the infracloud compute hosts not running puppet23:53
nibalizerit was because i had group/infracloud.yml instead of group/infracloud.yaml23:53
clarkbnibalizer: ingy says you gotta use the yaml23:54
clarkb(he actually didn't seem to care but said that perosnally uses yaml)23:54
nibalizerour automation cares23:55
nibalizerin that case23:55
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements
*** ggillies has quit IRC23:56
*** amotoki has joined #openstack-infra23:57
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add gate-grenade-dsvm-neutron to tempest failure rate dashboard
anteayatonyb: the original chair showed up again so I shared some explaination of meetbot commands23:58
anteayano acknowledgement yet though23:58
anteayain meeting-323:58
*** nwkarsten has quit IRC23:58
tonybanteaya: okay.  I'm also going to reach out to darrenc as I know him ....23:58
anteayagood idea23:58
funginibalizer: worth noting, cronspam from all of those has been bouncing in a bunch of places because they're sending with unqualified hostnames as the domain part of the sender address rather than having an fqdn23:59
Shrewsfungi: fwiw, i seem to have stopped receiving email notifications from gerrit, and i have not changed any settings23:59

Generated by 2.14.0 by Marius Gedminas - find it at!