Tuesday, 2016-03-15

*** ZZelle has quit IRC00:00
clarkbthe way openstack has been built to be a distributed system shouldn't rely on consistent gids. Definitely not keystone. Cinder and swift maybe00:00
*** HeOS has quit IRC00:01
*** HeOS has joined #openstack-infra00:01
fungijhesketh: i'll defer to jeblair on interpreting the tc's "feedback" given they reviewed it at his urging, but i find the particular choice of domain to host it at more or less an implementation detail. have it publish to a directory on static.o.o and we can either have a separate vhost or we can add a line to the governance.o.o vhost. in the tc meeting they seemed to support the latter00:01
*** davideagnello has quit IRC00:02
*** dizquierdo has quit IRC00:02
*** burgerk has joined #openstack-infra00:02
jheskethfungi: sure. In that case it sounds like it's ready for voting and we'll figure out the where in a bit00:03
fungijhesketh: yep. i'd rather consider the choice of url a nonblocking detail and just put it up for vote as is00:03
*** Sukhdev has quit IRC00:05
*** Sukhdev has joined #openstack-infra00:05
*** Sukhdev has quit IRC00:06
openstackgerritIan Wienand proposed openstack-infra/system-config: Really reinstall requests on Fedora  https://review.openstack.org/29265100:07
ianwurgh, an hour of building and debugging to realise this stupid issue ^00:07
*** Sukhdev has joined #openstack-infra00:07
*** angdraug has quit IRC00:08
fungiianw: just --upgrade is insufficient00:08
fungii can't recall having to use --force-reinstall unless it thought the latest version was already installed00:09
openstackgerritamrith proposed openstack/requirements: Update upper-constraints for python-troveclient v2.2.0  https://review.openstack.org/29265200:09
ianwfungi: it thinks the latest version is installed (at this point, since the fedora packaged version == latest version)00:09
ianwit's all very silly, but see the comment above it00:09
lifelessif you're replacing a system package, you will need force-reinstall, *or* to remove the old package first00:10
lifelessthe actual bug is on line 11800:10
lifelessyou're not removing the package metadata00:10
lifelessrm -rf .../requests-*.{egg,dist}-info00:11
lifelessianw: ^00:11
*** kencjohnston has joined #openstack-infra00:12
mordredyah. I recommend removing the system package first - less pain00:12
*** kencjohnston has quit IRC00:12
fungimordred: in this case i think there are base components of fedora which depend on python-requests00:12
fungiso removing the system package can be tricky00:12
mordredfungi: funny how this hole keeps resurfacing00:12
ianwyeah, we need the package installed so rpm doesn't re-install it and try to "unvendor" requests' sub-packages00:13
mordredof course, distro requests does the thing I want00:13
ianwi can remove the egg-info too, i guess.  i think it's a different way to get to the same point00:14
mordredsince it respects system CA00:14
fungii only expect it to get worse over time as more and more basic components of distros get (re)written in python and don't want to reinvent certain wheels addressed by existing python libraries outside stdlib00:14
mordredtoo bad I have an installation script somewhere that actually copies the patched file from distro-packaged reuqests into place in a virtualenv00:14
mordredI mean00:14
* mordred cries00:14
*** davideagnello has joined #openstack-infra00:14
morganmordred: *comfort* painful stuff.00:15
* anteaya hands mordred a dry kitten00:15
morganfungi: it makes me sad everytime this comes up00:15
anteayawith a box of tissues00:15
morgananteaya: i was wondering who had access to mordred's endless supply of <insert animal here>.00:15
morgananteaya: ;)00:15
ianwmordred: well, it's probably really fedora-minimal's fault00:15
fungimorgan: he gets them from an open source00:16
anteayaha ha ha00:16
morganfungi: i see what you did there.00:16
openstackgerritTony Breeds proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes  https://review.openstack.org/27027300:16
openstackgerritTony Breeds proposed openstack-infra/release-tools: Actually use the URL specific in --project-list  https://review.openstack.org/29265700:16
fungiianw: what is fedora-minimal doing to drag in requests?00:16
lifelessmordred: as fungi says, can't remove the system package, but we can remove the metadata more fully00:16
openstackgerritKen Johnston proposed openstack-infra/system-config: Add #osic to meetbot  https://review.openstack.org/29179800:17
ianwmodred: the rpm'd packaged requests is fine, but it's not part of the minimal image.  so puppet drags in the pip version at some point for something we install that wants it.  later, devstack comes along and tries to install the packaged version00:17
mordredlifeless: yes indeed00:17
fungiianw: a big part of why ubuntu-minimal exists is to not use cloud-init so we don't wind up with system packaged python-requests00:17
* morgan waves at lifeless and goes back to lurking.00:17
mordredianw: why is devstack trying to install the _PACKAGED verions?00:17
* lifeless waves morgan 00:17
*** Qiming has joined #openstack-infra00:17
clarkbfungi: and to handle rax network config00:17
mordredianw: that sounds like a bug in devstack to me - distro requests is not going to be respecting upper-constraints :)00:17
lifelessmordred: because devstack does that on some platforms AIUI.00:17
fungiianw: does fedora have a tool to tell you what the installed reverse-dependencies of a package are?00:17
lifelessmordred: which I think is nuts00:17
mordredlifeless: we should fix that00:17
clarkbwe could cloud init in a venv for the first thing00:18
*** eharney has quit IRC00:18
fungiclarkb: good point. rackspace's nova-agent had similarly terrible dependencies we needed to avoid, one of which may also have been python-requests00:18
mordredclarkb: kindof. there are problems with cloud-init in a venv, but I believe we could figure it out00:18
mordredalso, nova-agent likes to re-write files on running servers00:19
ianwmordred: cause something in http://logs.openstack.org/45/290445/2/check/gate-tempest-dsvm-platform-fedora23-nv/2c8150d/logs/devstacklog.txt.gz#_2016-03-14_18_34_50_919 pulls it in00:19
clarkbya nova agent is pretty silly00:19
*** burgerk has quit IRC00:19
fungiianw: if we can get at the reverse-depends we can track it back more easily to see what's actually dragging it in. that's how i tracked down a bunch of this on our ubuntu images over the years00:20
ianwmordred: i think the constraints will be correct, because we're "pip-ing" requests really.  we just need to fool rpm that the package is installed so it doesn't try again00:20
*** Kennan has quit IRC00:20
mordredianw: I disagree. we should stop installing the rpm00:21
*** Kennan has joined #openstack-infra00:21
ianwmordred: but what about when we need a package that depends on the rpm?00:21
jeblairfungi, jhesketh: in the reviews and moreso in the tc meeting, i think there was generally positive reaction to hosting at governance.o.o, and no opposition that i recall.  so i think we should aim to implement it there.  i leave the mechanics of that to fungi, but will be happy to draw up a quick amendment change now if we want to vote on them concurrently; or would be happy to push a new patchset over top if that is preferred.00:22
*** Swami has quit IRC00:22
mordredianw: (my airplane wifi is being slow in loading that logfile - one sec while I read it)00:22
fungijhesketh: i'll leave the choice between new patchset or follow-on amendment to you. for me they're basically the same outcome00:22
jeblairmordred is a spinning cursor00:22
*** dims has joined #openstack-infra00:23
ianwhttp://logs.openstack.org/20/289120/1/check/gate-tempest-dsvm-platform-fedora23-nv/429ff47/logs/devstacklog.txt.gz#_2016-03-07_03_12_09_191 is an actual log of python-requests being pulled in00:23
mordredianw: do you have access to a box in the state after such a thing has happened?00:24
fungiianw: i think for now the question is whether we _do_ need a package which depends on python-requests. usually those will be other python-based tools/libraries which can also be had from pypi00:24
mordredianw: such that you can tell which of the packages depended on python-requests?00:24
fungithis is also why i was asking what tool on fedora tells me the installed reverse-dependencies of a given package. i know how to do it on debian derivatives but am pretty useless on rh-based distros these days00:25
mordredfungi: you could try "yum remove python-requests" and see what it suggests you need to uninstall too00:26
fungioh, great point00:26
ianwi can get a system into this state, my cloud is just being a bit silly00:26
fungithough answering my own question, it looks like `repoquery --whatrequires ...` may be what i was looking for00:27
fungisomething like `repoquery --whatrequires --installed --recursive python-requests`00:28
openstackgerritTony Breeds proposed openstack-infra/release-tools: DNM: Add a release tool to find bugs tagged for backport  https://review.openstack.org/29266000:28
ianweven still, i think *not* have python-requests leaves the system in a very precarious state that all sorts of devstack-plugins, etc, will find they screw themselves up by installing the packaged version over the pip version00:28
*** thorst_ has quit IRC00:28
ianwit usually doesn't matter that much -- it's just in this case with the vendoring rpm/cpio can't handle overwriting the symlinks to the vendored bits00:29
fungiianw: why would they do that? we don't have that problem on ubuntu do we?00:29
*** thorst_ has joined #openstack-infra00:29
ianwmaybe apt doesn't half-install the package?00:29
ianwor does the debian pkg not unvendor?  or maybe it hard-links instead of symlinks and that works better?00:30
fungiianw: i mean do we have a situation where devstack demands system packages which are depending on python-requests rather than pip installing anything it needs which uses requests?00:30
*** Shrews has quit IRC00:31
*** docaedo has quit IRC00:31
fungii'm curious what is dragging the python-requests package into the fedora-minimal image in the hopes we might be able to avoid it00:31
*** mnaser has quit IRC00:31
*** mordred has quit IRC00:31
*** phschwartz has quit IRC00:31
*** afazekas_ has quit IRC00:32
*** afazekas has joined #openstack-infra00:32
*** mordred has joined #openstack-infra00:32
*** phschwartz has joined #openstack-infra00:32
*** dims has quit IRC00:32
*** docaedo has joined #openstack-infra00:32
*** Shrews has joined #openstack-infra00:32
*** yamamoto_ has quit IRC00:34
ianwfungi: the package, or the pip version?  nothing drags the package in00:34
jheskethif anybody has time to review https://review.openstack.org/#/c/289125/6 that'd be helpful :-)00:35
fungiianw: you were saying the fedora-minimal element was probably responsible for causing the python-requests package to be installed on the system00:35
fungiianw: so i'm curious how/why00:36
ianwfungi: no, it's responsible for it *not* being installed on the base system.  which means that as part of system-config puppet, we pip install it.  and then later devstack comes along and installs the rpm version, which then "half-installs" because it can't handle that it's trying to put symlinks where directories are00:36
*** thorst_ has quit IRC00:37
fungiianw: i thought devstack was only using pypi for python libraries00:37
fungididn't we make that switch a while back?00:38
*** mnaser has joined #openstack-infra00:38
fungialso we may want to fix whatever part of system-config is causing arbitrary python libraries to be globally preinstalled on our images00:38
ianwok: http://paste.openstack.org/show/490445/00:39
fungii'm personally surprised we pip install requests while puppeting the images00:39
fungiexcept into virtualenvs00:39
*** markvoelker has joined #openstack-infra00:40
*** ddieterly has joined #openstack-infra00:40
fungiianw: so euca2ools and/or gdisk are using python-requests i guess00:40
fungigross, we're ending up with system-installed python-lxml too00:41
*** Guest53749 has quit IRC00:41
ianwfungi: yep.  the python-lxml doesn't matter so much, i guess, because rpm can install it.  if pip wants a different version, it would just re-install it over the system version00:43
ianwpip being whatever is in requirements00:43
anteayajhesketh: +200:44
jheskethanteaya: thank you :-)00:44
anteayawelcome :)00:44
fungimordred: btw, your routine for fixing the ca trusts with pip installed requests is a function in devstack's lib/tls00:44
mordredfungi: :)00:44
mordredit's euca2ool00:46
fungiianw: it looks like euca2ools is only being installed from system packages on rpm-based platforms?00:46
fungii see it in files/rpms and files/rpms-suse but not for debian derivatives00:47
fungiare we not using it on ubuntu, or are we getting it there some other way?00:47
ianwfungi: i'll certainly investigate getting rid of that; i have some memory of us having old unit tests.  i still want to make the change though, to avoid the system being in this precarious state and anyone else (plugins, in particular, installing essentially random stuff) having to debug this00:47
ianwnot unit tests, exercises -- functional tests i guess00:48
jheskethjeblair: ping00:48
mordredianw: I think the design of desvstakc-gate nodes is currently suck that installing python from distro packages is unadvised and you get to keep both pieces when it breaks00:50
mordredianw: now, I do think that us installing requests from pip not in a virtualenv in puppeting things is a bug and we should fix that00:50
fungiianw: mordred looks like it was missed as a part of https://review.openstack.org/15659400:50
lifelessI wonder if we can write a linter00:50
amrithheya folks in infra, anyone here got admin privileges to the ML server? I'm wondering if you could take a look at something for me ... ttx maybe?00:50
lifelessto catch things00:50
jheskethjeblair: (sorry, just noticed your response above)... with using governance.o.o was there any opinion from the TC on how/where to link the elections from00:50
anteayaamrith: ttx is asleep00:50
anteayaamrith: just share the details you shared in -dev00:51
anteayaif someone has permissions and the time to help, they will say so00:51
fungiamrith: depends on which admin privs you need. list configuration or systems administration00:51
mordredianw: but I agree that puppet installing requests with pip is a bug00:51
mordredZOMG. My internet is so bad it's like watching computers in the 80s try to keep00:51
mordredup with touch typing00:51
lifelessmordred: they kept up fine00:51
mordredThe duplex is killing it. I'm typing lines in another buffer and pasting them no00:52
lifelessmordred: until that newfangled windows thing came along and messed it up00:52
amrithfungi, here's my issue. I didn't get one (just one as far as I can tell) email that went o the ML today. It was this one http://markmail.org/message/43pfbo3oagjvo3lo00:52
mordredlifeless: actually - I remember my dad WAY overtyping the dos-based word processor00:52
fungii do recall being able to type faster than my 110 baud acoustic coupled modem could send00:52
amrithso, others in my company got a copy, just one person didn't.00:52
mordredand also the old modems :)00:52
lifelessmordred: hmm, possible I guess. certainly modems had to buffer00:52
amrithso I checked on our mail server (I'm the admin there) and the mail trace shows that I didn't get a copy.00:52
amrithso I'm wondering why, I'm still on openstack-dev00:53
amrithI got other mails, before and after.00:53
fungiamrith: i'm checking the mta log on our end now00:53
amrithso was wondering if someone could look at the log.00:53
amriththanks fungi00:53
mordredlifeless: my friend nathan's dad also overtyped the computer's ability to keep up. we were both in awe of the typing speeds of our respective dads at the time00:54
anteayayour dad's could type?00:55
fungii wish i knew an easier way to get the message-id of a ml post out of pipermail than to download mailman's mbox archive00:55
*** sdake has joined #openstack-infra00:55
clarkbI recall doing this once00:55
clarkblet me check my command history00:55
lifelessmordred: apple ]['s? or trs-80s? or something else?00:55
openstackgerritIan Wienand proposed openstack-infra/system-config: Really reinstall requests on Fedora  https://review.openstack.org/29265100:56
clarkboh wait don't have my key loaded anymore00:56
clarkbfungi: ^ check my command history on lists as I seem to recall doing this when we were getting the spams00:56
ianwlifeless: ^ thanks, that version clears metadata f00:56
mordredanteaya: very well00:56
*** thorst_ has joined #openstack-infra00:56
*** dims has joined #openstack-infra00:56
fungiclarkb: for finding the full message headers of a ml post?00:57
clarkbfungi: just the id00:57
*** Qiming has quit IRC00:57
*** ddieterly has quit IRC00:57
anteayamy head is not dealing with the concept of a typing father very well at all00:57
clarkbfungi: its logged somewhere iirc00:57
amrithI can get you a msg-id00:57
amrithone second00:57
*** esikachev has joined #openstack-infra00:58
*** dimtruck is now known as zz_dimtruck00:59
*** noorul has joined #openstack-infra00:59
ianwfungi: so euca2ools isn't available via pypi, and i think it was probably really mistakenly removed as part of that purge.  devstacks tools/euca.sh test would be broken there ... but it probably shows that it's irrelevant00:59
*** Shrews has quit IRC00:59
*** ddieterly has joined #openstack-infra00:59
noorulI am using gerrit customized by openstack ci team00:59
noorulBut I don't see the workflow step00:59
mordredanteaya: well, my dad was a telco engineer, his dad was a doctor - both could program01:00
anteayamordred: oh makes sense01:00
anteayamy dad could not program01:00
fungiianw: since we removed it from being installed on ubuntu, i think we would have noticed if that were still being exercised01:00
anteayahe could build houses and chop wood though01:00
*** bpokorny_ has quit IRC01:00
mordredianw: since ec2 is a plugin now, I do not think there is any reason for base devstack to install anything related to AWS01:00
noorulAm I missing something?01:01
anteayanoorul: I don't understand what you mean by the workflow step?01:01
fungiamrith: ftr, Message-ID: <56E7117E.9050608@linux.vnet.ibm.com>01:01
amrithfungi, but I did get this one ... <56E706FD.5070402@linux.vnet.ibm.com>01:01
noorulanteaya: Merge step01:01
amrithfungi, I sent you a screen shot of the message in question01:01
fungiamrith: ftr, Message-ID: <56E7117E.9050608@linux.vnet.ibm.com>01:01
noorulanteaya: Where in we put Workflow label +!01:01
fungiamrith: er, sorry for echoing twice01:02
anteayanoorul: commenting on patches01:02
ianwfungi: right, we don't run the exercises in the gate.  but from a devstack point of view, you should be able to run the exercises.  which i guess means the euca test is basically irrelevant because nobody is reporting issues with this01:02
anteayanoorul: you are at the step where you want to leave a comment on a patch in the ci-sandbox repo in gerrit?01:02
ianwmordred: yes, i guess it doesn't even work with the tools01:03
anteayanoorul: is this for a third party ci to comment on our gerrit's patches?01:03
noorulanteaya: I am in step where I want to merge the patch01:03
ianwbecause no plugin is installed.  i can clean this up01:03
amrithyes, I posted them above as well ;)01:03
noorulanteaya: No, I am running this for non-openstack projects01:03
amrithI've sent you some screen shots by PM01:03
anteayanoorul: oh okay01:03
amrithnotice that I get one, but not the other01:03
noorulanteaya: Trying to get a demo setup up01:03
*** esikachev has quit IRC01:03
openstackgerritJoshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o  https://review.openstack.org/29266601:03
jheskethfungi: went with ^ to avoid losing existing votes01:03
clarkbnoorul: you have to configure it in your all-projects iirc01:04
anteayaand I do believe I should be off to bed01:04
anteayaas I have to be up in several hours01:04
anteayanight all01:05
noorulclarkb: I see, http://docs.openstack.org/infra/system-config/gerrit.html01:06
clarkbnoorul: towards the bottom of http://docs.openstack.org/infra/system-config/gerrit.html#access-controls you will see where workflow is configured01:07
*** sdake has quit IRC01:10
openstackgerritJoshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o  https://review.openstack.org/29266601:10
fungiamrith: it will take me a while to track this down since i need to correlate the message ids (which only seem to be logged with the sender details) to the deliveries. they're keyed by exim id values i just need to work up a map to find which copy of that message was destined to your address01:10
amriththanks fungi ... I'm also opening a support request with our mail provider with the message traces that I have.01:11
*** thorst_ has quit IRC01:12
*** zz_dimtruck is now known as dimtruck01:12
*** thorst_ has joined #openstack-infra01:13
*** markvoelker has quit IRC01:14
*** Sukhdev has quit IRC01:17
fungiamrith: from what i can find in our mta logs, it doesn't appear mailman attempted to deliver a copy of that particular message to you01:17
fungiamrith: is it possible you have your account set to only subscribe to a limited set of tags on that list?01:18
*** pvaneck has quit IRC01:18
amrithfungi, I have my account set to send me stuff even if it is not tags that I've subscribed to01:18
amrithfungi, "Do you want to receive messages that do not match any topic filter?": Yes01:20
*** thorst_ has quit IRC01:21
*** zhurong_ has joined #openstack-infra01:21
fungiamrith: i believe that does not also send you things which do match a topic filter for a topic to which you are not subscribed01:22
amrithBut this line (below it) does. "If no topics of interest are selected above, then you will receive every message sent to the mailing list. "01:22
amrithI chose no topics01:22
fungiokay, so you don't have any of them checked01:22
amrithin any event, stable isn't a valid topic on that list.01:22
amrithyes, none are checked.01:22
*** gongysh has quit IRC01:24
*** tzn has joined #openstack-infra01:24
fungiamrith: i agree, it looks like stable is not an official topic01:25
fungii definitely see exim delivering messages to you when mailman asks it to do so, and i don't see any of them rejected01:25
fungii also don't see mailman receiving any bounces for your address01:25
fungii'm trying to figure out where and how it would log a decision not to send you a particular message though01:26
*** chlong has quit IRC01:26
amrithfungi, thanks for looking. I'm not sure it makes sense to take more time about this, I'll think about it again tomorrow and see what I can do.01:27
amriththx for your time01:28
*** tzn has quit IRC01:29
fungiamrith: yeah, i'm coming up dry but maybe when jeblair's around i can get his eyes on it too. i think he's spent more time staring into the abyss of mailman logs than i have01:29
* amrith shudders ... I was mail manager for a while, sendmail and I were on first name terms. much scotch was consumed ;)01:30
fungiyeah, our mta (exim) i'm quite at home with, but the listserv (mailman) is a bit more opaque to me01:31
Clintsendmail was fun before spam becamse so prevalent01:31
amrithClint, yes, those were the good days ;)01:31
clarkbthink on the bright side, your experience with sendmail probably makes you a Make ninja01:31
fungisendmail was fun back before people started using m4 macros to autogenerate the c object file they configured it with ;)01:32
amrithyes, but today people have tools to generate makefiles and sendmail.cf ;) I used to do it by hand01:32
*** wei-wang has joined #openstack-infra01:33
amrithah, what fungi said ...01:33
amrithtime to go. have some more reviews to do before I sleep.01:34
amriththanks fungi Clint ... g'night.01:34
funginight amrith01:34
*** tongli has quit IRC01:34
*** yamamoto_ has joined #openstack-infra01:34
*** kencjohnston_ has joined #openstack-infra01:36
*** woodster_ has quit IRC01:37
*** sridhar_ram has quit IRC01:39
*** zhurong_ has quit IRC01:39
*** burgerk has joined #openstack-infra01:39
*** burgerk has quit IRC01:39
*** rfolco has joined #openstack-infra01:40
*** zhurong_ has joined #openstack-infra01:41
*** kencjohnston_ has quit IRC01:42
*** yamamoto_ has quit IRC01:43
*** gildub has joined #openstack-infra01:43
*** tsg has quit IRC01:46
*** Qiming has joined #openstack-infra01:47
*** amrith is now known as _amrith_01:49
openstackgerritJoshua Hesketh proposed openstack-infra/puppet-httpd: Add support for alias and directory variables  https://review.openstack.org/29267401:50
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Add election alias to governance.openstack.org  https://review.openstack.org/29267501:50
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Publish election data to static server  https://review.openstack.org/29267601:50
*** sbalukoff has quit IRC01:52
jheskethfungi, jeblair: ^ there's my first iteration of the election stuff (granted prematurely before the spec)01:54
*** rockyg has joined #openstack-infra01:54
*** IlyaG has quit IRC01:57
*** woodster_ has joined #openstack-infra01:57
*** apoorvad has quit IRC01:58
*** piet has joined #openstack-infra02:01
*** salv-orlando has joined #openstack-infra02:02
*** SumitNaiksatam has joined #openstack-infra02:03
*** salv-orl_ has quit IRC02:04
*** ddieterly has quit IRC02:05
*** rhallisey has quit IRC02:05
*** piet has quit IRC02:06
*** piet has joined #openstack-infra02:06
*** yamamoto has joined #openstack-infra02:07
*** rhallisey has joined #openstack-infra02:08
jeblairjhesketh: i don't recall a preference from the TC as to path02:08
*** gongysh has joined #openstack-infra02:09
jeblairfungi, _amrith_: i don't have a lot of experience debugging mailman topics, sorry.  happy to help as i can, but no magic bullets here02:10
*** fawadkhaliq has quit IRC02:12
fungijeblair: sounds like it wasn't topic-related02:12
jeblairfungi: oh, i must have mis-skimmed?02:13
fungijeblair: just that mailman (from what i can tell correlating the message id in the exim logs) never submitted a delivery to him for that particular message02:13
fungii went scrounging around in mailman's various logs but couldn't find anything which would elucidate the matter at hand02:13
jeblairfungi: so, reason unknown, but no reason to suspect it was topics?02:13
fungijeblair: according to amrith he had no topics selected in his mailman prefs for openstack-dev at all, so it should default to sending all messages02:14
fungiand also, the one subject tag which was used in the message wasn't an official subscribable list tag anyway02:14
fungiso shouldn't have been handled any differently than tagless messages afaik02:15
fungiso i'm fairly stumped02:15
*** tphummel has quit IRC02:17
jheskethjeblair: ah okay. fungi said (above) that the tc seemed to prefer having it on g.o.o (noting that isn't a decision)02:21
jhesketheither way, I think it is indeed a better fit02:21
jheskethbut will update the commit messages to not lie02:21
*** noorul has quit IRC02:21
*** gildub has quit IRC02:22
openstackgerritJoshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o  https://review.openstack.org/29266602:22
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Add election alias to governance.openstack.org  https://review.openstack.org/29267502:26
*** yamamoto has quit IRC02:27
*** rockyg has quit IRC02:31
*** rfolco has quit IRC02:35
*** cody-somerville has joined #openstack-infra02:35
*** rhallisey has quit IRC02:35
*** dims has quit IRC02:38
*** armax has quit IRC02:38
*** cj has joined #openstack-infra02:39
*** dimtruck is now known as zz_dimtruck02:39
*** gongysh has quit IRC02:41
*** bryan_att has quit IRC02:42
*** tqtran has quit IRC02:42
*** kencjohnston_ has joined #openstack-infra02:48
*** kencjohnston_ has quit IRC02:49
*** IlyaG has joined #openstack-infra02:50
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29268702:51
openstackgerritgreghaynes proposed openstack/diskimage-builder: WIP add element to install glean.rs  https://review.openstack.org/29268902:54
mordredfungi: https://review.openstack.org/29269002:55
mordredfungi: just for you02:55
mordredjeblair: (although you might enjoy it too)02:55
*** baoli_ has joined #openstack-infra02:57
*** asselin_ has joined #openstack-infra02:57
*** armax has joined #openstack-infra02:58
*** esikachev has joined #openstack-infra02:59
*** gildub has joined #openstack-infra03:01
*** HeOS has quit IRC03:02
*** HeOS has joined #openstack-infra03:02
*** yamahata has joined #openstack-infra03:02
*** doug-fish has joined #openstack-infra03:03
*** IlyaG has quit IRC03:03
*** esikachev has quit IRC03:03
*** fawadkhaliq has joined #openstack-infra03:06
*** yamamoto has joined #openstack-infra03:09
*** kushal has joined #openstack-infra03:11
*** markvoelker has joined #openstack-infra03:11
*** esker has quit IRC03:13
*** mtanino has quit IRC03:17
*** Qiming_ has joined #openstack-infra03:20
*** yuanying has quit IRC03:22
*** Kennan has quit IRC03:23
*** asettle has quit IRC03:23
*** Qiming has quit IRC03:24
*** Qiming__ has joined #openstack-infra03:25
*** Kennan has joined #openstack-infra03:26
*** erlon has quit IRC03:27
*** Qiming_ has quit IRC03:29
*** chlong has joined #openstack-infra03:30
fungimordred: an amusing opportunity to recount some relevant early history of our development!03:30
*** kaisers has quit IRC03:36
*** links has joined #openstack-infra03:38
*** yuanying has joined #openstack-infra03:38
*** markvoelker has quit IRC03:44
*** bpokorny has joined #openstack-infra03:44
*** yuanying has quit IRC03:45
*** tphummel has joined #openstack-infra03:46
*** kaisers has joined #openstack-infra03:50
*** tphummel has quit IRC03:50
mordredfungi: :)03:53
mordredfungi: that's kinda how I was looking at it03:54
*** ayoung has quit IRC03:57
*** HeOS has quit IRC04:02
tonybmordred: ;p04:02
*** HeOS has joined #openstack-infra04:03
*** IlyaG has joined #openstack-infra04:04
*** pahuang has quit IRC04:04
*** keedya has quit IRC04:05
*** sdake has joined #openstack-infra04:06
*** yuanying has joined #openstack-infra04:06
*** rguillebert has quit IRC04:07
*** ajmiller has joined #openstack-infra04:08
*** sdake_ has joined #openstack-infra04:09
*** IlyaG has quit IRC04:09
*** ajmiller_ has joined #openstack-infra04:09
*** sdake has quit IRC04:11
*** ajmiller has quit IRC04:13
*** bpokorny has quit IRC04:14
*** HeOS has quit IRC04:15
lifelessmordred: -lol-04:17
*** amotoki has joined #openstack-infra04:17
*** piet has quit IRC04:19
*** sdake_ has quit IRC04:19
*** piet has joined #openstack-infra04:19
*** armax has quit IRC04:20
*** pahuang has joined #openstack-infra04:21
*** ajmiller_ has quit IRC04:21
ianwfungi / mordred : so, https://review.openstack.org/#/c/292651 (re-install requests really) i'd appreciate it if we could fix quickly, as we are currently shipping a completely broken requests due to not re-installing it.  when the ec2 stuff goes through, i can think about removing this work-around04:24
*** armax has joined #openstack-infra04:25
mordredianw: wfm04:26
ianwthanks, if you can +A it, i can feel good that it will be in tomorrows builds :)04:30
*** piet has quit IRC04:30
ianwwhich hopefully will upload, but that's another issue :)04:30
*** piet has joined #openstack-infra04:31
*** treaki_ has joined #openstack-infra04:31
*** treaki__ has quit IRC04:35
*** sbalukoff has joined #openstack-infra04:42
*** wei-wang has quit IRC04:43
*** piet has quit IRC04:44
*** Sukhdev has joined #openstack-infra04:45
*** hichihara has joined #openstack-infra04:47
*** baoli_ has quit IRC04:50
ianwfungi / mordred : ^04:53
*** piet has joined #openstack-infra04:53
*** woodster_ has quit IRC04:57
*** wei-wang has joined #openstack-infra04:59
*** tzn has joined #openstack-infra05:01
*** tzn has quit IRC05:01
openstackgerritMerged openstack-infra/system-config: Really reinstall requests on Fedora  https://review.openstack.org/29265105:04
*** pahuang has quit IRC05:12
*** sdake has joined #openstack-infra05:15
*** amotoki has quit IRC05:21
*** roxanagh_ has quit IRC05:23
*** pahuang has joined #openstack-infra05:24
*** roxanagh_ has joined #openstack-infra05:24
*** armax has quit IRC05:29
*** HeOS has joined #openstack-infra05:30
*** amotoki has joined #openstack-infra05:31
*** sdake has quit IRC05:33
*** amotoki has quit IRC05:35
*** _nadya_ has joined #openstack-infra05:37
*** roxanagh_ has quit IRC05:40
*** roxanagh_ has joined #openstack-infra05:41
*** markvoelker has joined #openstack-infra05:41
*** Sukhdev has quit IRC05:43
*** piet has quit IRC05:44
*** roxanagh_ has quit IRC05:45
*** amotoki has joined #openstack-infra05:47
*** armax has joined #openstack-infra05:47
*** zzzeek has quit IRC05:49
*** abregman has joined #openstack-infra05:49
*** zzzeek has joined #openstack-infra05:50
*** salv-orl_ has joined #openstack-infra05:54
*** amotoki has quit IRC05:55
*** jaosorior has joined #openstack-infra05:57
*** salv-orlando has quit IRC05:57
*** kushal has quit IRC05:58
*** rlandy has quit IRC05:58
*** esikachev has joined #openstack-infra05:59
*** _nadya_ has quit IRC06:03
*** esikachev has quit IRC06:04
*** Daisy has joined #openstack-infra06:07
*** Daisy has quit IRC06:10
*** Daisy has joined #openstack-infra06:11
*** kushal has joined #openstack-infra06:14
*** markvoelker has quit IRC06:14
*** Daisy_ has joined #openstack-infra06:16
*** Daisy has quit IRC06:19
*** IlyaG has joined #openstack-infra06:22
*** IlyaG has quit IRC06:27
*** MarkAtwood has quit IRC06:33
*** jcooley has quit IRC06:35
*** asettle has joined #openstack-infra06:38
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/28884906:41
*** roxanagh_ has joined #openstack-infra06:42
*** asettle has quit IRC06:43
*** placko_ has joined #openstack-infra06:46
*** Daisy_ has quit IRC06:49
*** Daisy has joined #openstack-infra06:50
*** roxanagh_ has quit IRC06:51
*** Daisy_ has joined #openstack-infra06:52
*** Daisy has quit IRC06:52
*** ildikov_ has quit IRC06:53
*** asettle has joined #openstack-infra06:54
*** HeOS has quit IRC07:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Don't take into account commit message for skip-if filter  https://review.openstack.org/24738307:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Split pipeline description by double newlines on status page  https://review.openstack.org/23994307:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add possibility to use SSH instead of HTTP to get ref list  https://review.openstack.org/24643307:01
*** HeOS has joined #openstack-infra07:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Report job status 'QUEUED' for queued jobs  https://review.openstack.org/27296707:02
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names  https://review.openstack.org/28266007:02
*** oanson has quit IRC07:02
*** baoli has joined #openstack-infra07:02
*** baoli has quit IRC07:07
*** mikelk has joined #openstack-infra07:10
*** tesseract has joined #openstack-infra07:13
*** tesseract is now known as Guest9751907:13
*** Daisy_ has quit IRC07:15
*** ihrachys has joined #openstack-infra07:17
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: move documents and python 2.7 checks to community  https://review.openstack.org/29274607:19
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: move documents and python 2.7 checks to community  https://review.openstack.org/29274607:21
lennybasselin: regarding firewall you are correct. I've checked this issue with our IT department and we suppose to be fine with the IP change. Thank you.07:22
*** dingyichen has quit IRC07:27
*** maishsk has joined #openstack-infra07:29
*** fawadkhaliq has quit IRC07:30
*** scheuran has joined #openstack-infra07:33
yolandagood morning07:41
*** Daisy has joined #openstack-infra07:43
*** kingia_ has joined #openstack-infra07:46
*** placko_ has quit IRC07:47
*** kingia has quit IRC07:48
*** yamahata has quit IRC07:52
*** nmagnezi has joined #openstack-infra07:53
*** kingia has joined #openstack-infra07:53
*** penguinolog has joined #openstack-infra07:54
penguinologGood morning07:55
*** asettle has quit IRC07:55
*** jlanoux has joined #openstack-infra07:55
*** kingia_ has quit IRC07:56
penguinologGuys, could anybody explain for me: is tox env docs enough for job template gate-xxx-docs (template python-jobs)?07:57
penguinologchangeset: https://review.openstack.org/#/c/292746/08:00
yolandaAJaeger, question about https://review.openstack.org/#/c/289589/5/jenkins/jobs/projects.yaml . Why is the job definition duplicated here, one with job suffix nv and the other without?08:00
*** Guest97519 has quit IRC08:00
AJaegerpenguinolog: we run "tox -e venv python setup.py build_sphinx" for the docs gate as explained in the PTI http://governance.openstack.org/reference/cti/python_cti.html08:01
*** matrohon has joined #openstack-infra08:04
*** tesseract has joined #openstack-infra08:05
AJaegeryolanda: let me check...08:05
*** tesseract is now known as Guest9179408:06
AJaegerin this case: The -nv is not needed, let'S ask to remove it...08:06
AJaegerIn general: -nv for non-voting jobs, the other as voting08:06
*** Qiming__ is now known as Qiming08:06
yolandayep, but i was seeing that dupe...normally just one definition needed right?08:06
*** _nadya_ has joined #openstack-infra08:06
AJaegernormally just one - unless they really need both - and they might.08:07
AJaegerAlso, looking closer now: They don't use that job at all;( Ask them to add it to their check queue...08:07
* AJaeger completely missed that - that was a immature +2 ;(08:07
yolandaAJaeger, tsk tsk :)08:07
yolandaso you will add the comments yourself?08:08
AJaegerI can...08:08
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community  https://review.openstack.org/29274608:09
AJaegerdone, yolanda08:10
yolandaAJaeger, i was thinking in approving https://review.openstack.org/#/c/29257208:10
yolandado you think we will be fine? i can monitor08:10
*** flepied has quit IRC08:12
AJaegerit should be fine, yes...08:12
*** markvoelker has joined #openstack-infra08:12
yolandaok here it goes08:12
AJaeger32 jobs only - so hope that goes through just fine...08:12
openstackgerritMerged openstack-infra/project-config: zuul/layout: save TripleO CI resources  https://review.openstack.org/29233908:15
AJaegeryolanda: The followup will be more fun - and we should wait a day or two before we merge that08:15
yolandai haven't looked. more jobs involved?08:15
AJaegerthe remaining 414 devstack-trusty ones ...08:17
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community  https://review.openstack.org/29274608:17
*** rossella_ has joined #openstack-infra08:17
AJaegeryolanda: I already -1 any jobs that add new devstack-trusty (and also check for bare-trusty).08:18
AJaegerNew trusty jobs all have to use ubuntu-trusty08:18
yolanda++ i'll do the same then08:18
*** salv-orl_ has quit IRC08:18
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29268708:18
AJaegeryolanda: https://review.openstack.org/#/c/292573 is the followup08:18
yolandayay, it looks much more complex yes08:19
*** fhubik has joined #openstack-infra08:19
openstackgerritMerged openstack-infra/project-config: Start devstack-trusty to ubuntu-trusty switch  https://review.openstack.org/29257208:20
openstackgerritMerged openstack-infra/project-config: Add Designate Role to OpenStack-Ansible  https://review.openstack.org/29232608:21
openstackgerritMerged openstack-infra/project-config: Do not run gate jobs when doing docs-only patch  https://review.openstack.org/29218708:21
openstackgerritMerged openstack-infra/project-config: ec2-api job: remove tempest from rally job  https://review.openstack.org/29241808:22
*** arxcruz has joined #openstack-infra08:23
openstackgerritMerged openstack-infra/project-config: Enable Ceilo plugin for Sahara dsvm jobs  https://review.openstack.org/21775108:23
penguinologAJaeger, sorry, couild you check https://review.openstack.org/#/c/292746/ for consistency? (tox envs added: pylint, pep8, py27)08:24
openstackgerritMerged openstack-infra/project-config: New project for TripleO UI  https://review.openstack.org/29122708:27
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project  https://review.openstack.org/29276008:31
*** maishsk_ has joined #openstack-infra08:31
*** fawadkhaliq has joined #openstack-infra08:31
*** maishsk has quit IRC08:32
*** maishsk_ is now known as maishsk08:32
*** chlong has quit IRC08:32
*** vincentll has joined #openstack-infra08:33
*** [1]Thelo has joined #openstack-infra08:33
*** Thelo has quit IRC08:36
*** jlanoux has quit IRC08:36
*** [1]Thelo is now known as Thelo08:36
*** fhubik has quit IRC08:37
*** Hal has joined #openstack-infra08:38
*** Hal is now known as Guest686108:39
*** kzaitsev_mb_ has joined #openstack-infra08:40
*** markvoelker has quit IRC08:44
*** Guest91794 has quit IRC08:45
*** fhubik has joined #openstack-infra08:46
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community  https://review.openstack.org/29274608:47
tobiashjhesketh: I just tried out this keepalive patch: https://review.openstack.org/#/c/238988/08:47
AJaegerpenguinolog: why do you +1 your own changes?08:47
tobiashjhesketh: It doesn't work for me because of a missing conversion of the config parameter. I added a -1 but the patch has already +2 from you so.08:48
AJaegertobiash: other reviewers will see your -108:48
*** kingia_ has joined #openstack-infra08:49
*** zeih has joined #openstack-infra08:50
*** kingia has quit IRC08:51
*** dizquierdo has joined #openstack-infra08:51
*** esikachev has joined #openstack-infra08:51
*** kingia has joined #openstack-infra08:53
*** kingia_ has quit IRC08:53
*** _degorenko|afk is now known as degorenko08:54
*** AJaeger has quit IRC08:54
*** AJaeger has joined #openstack-infra08:54
*** e0ne has joined #openstack-infra08:57
*** dizquierdo has quit IRC08:57
*** dizquierdo has joined #openstack-infra08:58
openstackgerritEvgeniy L proposed openstack-infra/project-config: Create bareon-allocator repository  https://review.openstack.org/29276508:58
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29276608:58
*** Guest91794 has joined #openstack-infra09:00
*** tzn has joined #openstack-infra09:01
*** zz_dimtruck is now known as dimtruck09:02
*** dtardivel has joined #openstack-infra09:03
*** HeOS has quit IRC09:03
*** HeOS has joined #openstack-infra09:04
*** flepied has joined #openstack-infra09:05
*** jordanP has joined #openstack-infra09:05
*** hashar has joined #openstack-infra09:05
*** HeOS has quit IRC09:05
*** HeOS has joined #openstack-infra09:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible  https://review.openstack.org/29277109:11
*** sc68cal has quit IRC09:11
openstackgerritAlexey Ovchinnikov proposed openstack-infra/project-config: Add publishing capability to buildimage-lxd job  https://review.openstack.org/29041809:12
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project  https://review.openstack.org/29276009:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible  https://review.openstack.org/29277109:18
*** zeih has quit IRC09:18
penguinologAJaeger, just follows standard rules inside team: until self +1, it's read as "forgot to set -W"09:19
*** Daisy has quit IRC09:19
*** jistr has joined #openstack-infra09:20
AJaegerpenguinolog: that's not a standard OpenStack rule... Your +1 does not harm, it's just unusual in most (nearly all?) projects.09:20
*** salv-orlando has joined #openstack-infra09:21
penguinologAJaeger, fixed: set to 009:21
penguinologcould anybody else review https://review.openstack.org/#/c/292746/5 ?09:22
odyssey4meheh, agreed AJaeger - never heard of that convention myself09:22
*** hashar has quit IRC09:22
odyssey4meif you're still working on it, mark it -w or at least set the subject to some form of 'WIP'09:23
*** andymaier has joined #openstack-infra09:23
*** Guest6861 has quit IRC09:24
*** hashar has joined #openstack-infra09:25
*** jlanoux has joined #openstack-infra09:25
*** Hal has joined #openstack-infra09:27
*** Hal is now known as Guest8992009:28
*** denisra has quit IRC09:28
*** denisra has joined #openstack-infra09:29
*** salv-orlando has quit IRC09:29
*** salv-orlando has joined #openstack-infra09:30
*** rcernin has joined #openstack-infra09:31
*** tzn has quit IRC09:33
*** lucas-dinner is now known as lucasagomes09:33
*** kushal has quit IRC09:35
openstackgerritThierry Carrez proposed openstack-infra/project-config: Pre-release stable/mitaka ACLs  https://review.openstack.org/29278109:36
*** zhurong_ has quit IRC09:37
*** maishsk has quit IRC09:40
*** watersoul has joined #openstack-infra09:44
*** watersoul_ has quit IRC09:47
*** derekh has joined #openstack-infra09:47
*** hichihara has quit IRC09:52
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891509:53
*** mrmartin has joined #openstack-infra09:53
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29268709:58
*** Qiming has quit IRC09:59
*** claudiub has joined #openstack-infra09:59
*** sdague has joined #openstack-infra10:01
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TEST: overcloud SSL with netiso  https://review.openstack.org/29278710:01
*** ociuhandu has quit IRC10:05
*** gildub has quit IRC10:07
*** rossella_ has quit IRC10:07
*** rossella_s has joined #openstack-infra10:08
*** sc68cal has joined #openstack-infra10:08
*** e0ne has quit IRC10:08
*** e0ne has joined #openstack-infra10:08
*** jcoufal has joined #openstack-infra10:10
*** Guest89920 has quit IRC10:11
*** pcrews_ has quit IRC10:11
*** tpsilva has joined #openstack-infra10:11
*** sc68cal has quit IRC10:13
tobiashHi, I've setup a new zuul (current master) + jenkins (1.642.2) environment and somehow no build urls get passed back to zuul.10:14
*** pcrews_ has joined #openstack-infra10:14
tobiashZuul seems to get update messages when a build starts, but they don't contain the build url10:15
*** sambetts|afk is now known as sambetts10:15
tobiashIs there something I have to configure in jenkins (it knows it's base url)?10:16
openstackgerrityolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce  https://review.openstack.org/29279210:16
yolandaAJaeger, fix for devstack-gate and reproduce ^10:17
*** mrmartin has quit IRC10:17
eliqiaohi AJaeger10:18
eliqiaoAJaeger: I have a question about https://review.openstack.org/#/c/292687/3/zuul/layout.yaml10:18
*** mkj has quit IRC10:19
*** electrofelix has joined #openstack-infra10:21
*** AJaeger has quit IRC10:21
*** notnownikki has joined #openstack-infra10:23
jheskethtobiash: did casting it to an int work for you? What did you have in your config file (eg was it a string)10:24
*** oanson has joined #openstack-infra10:24
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281410:26
*** rguillebert has joined #openstack-infra10:26
tobiashjhesketh: yepp, in the config file I had "keepalive=60"10:27
tobiashwithout the casting paramiko threw errors10:27
tobiashwith casting it works for me10:27
jheskethhmm, I wonder why gerrit's port works... maybe nobody has tried actually setting it10:28
*** flwang1 has joined #openstack-infra10:28
yolandajhesketh, care reviewing https://review.openstack.org/292521 ? has a +2 from AJaeger, is to enable a nv job in bifrost10:29
igorbelikovhey folks, is zuul stuck? no changes in queues at http://status.openstack.org/zuul/ and the graphs seem flatlining10:29
jheskethtobiash: oh, nevermind, it has an int10:29
tobiashjhesketh: zuul itself seemed to work, I saw the errors in the log (something like operator + not supported for float and string)10:29
flwang1hi guys, how can i create a new group on gerrit for a project? thanks10:29
jheskethtobiash: interesting that there's a + operator on a port number... got a copy of that log per hcnace?10:30
tobiashjhesketh: have to look10:31
*** jlanoux_ has joined #openstack-infra10:32
*** jlanoux has quit IRC10:32
tobiashjhesketh: http://paste.openstack.org/show/490466/10:33
tobiashwith casting this error goes away10:33
jheskethtobiash: oh right, for the keepalive10:34
jheskethtobiash: yeah, looks correct. Nice catch :-)10:34
jheskethyolanda: lgtm10:36
yolandajhesketh, thx10:36
*** vgridnev has joined #openstack-infra10:36
jheskethyolanda: if you have time for https://review.openstack.org/#/c/289125/ that'd be appreciated10:37
*** esikachev has quit IRC10:37
*** sfinucan has joined #openstack-infra10:37
*** shardy has joined #openstack-infra10:39
jheskeththanks :-)10:40
*** dimtruck is now known as zz_dimtruck10:40
*** zz_dimtruck is now known as dimtruck10:42
*** esikachev has joined #openstack-infra10:42
*** markvoelker has joined #openstack-infra10:42
*** x00350071 has quit IRC10:42
igorbelikovyolanda: zuul seems stuck for the past 2 hours, no new changes in queues, is this a known situation?10:43
yolandaigorbelikov, i was not aware10:43
*** x00350071 has joined #openstack-infra10:43
*** esikachev has quit IRC10:44
*** tzn has joined #openstack-infra10:44
igorbelikovyolanda: thanks!10:44
*** esikachev has joined #openstack-infra10:44
yolandajhesketh, still around? i see lots of10:46
yolanda2016-03-15 10:45:57,094 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent ref-replicated None>10:46
yolanda2016-03-15 10:45:57,097 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.10:46
yolandain zuul logs10:46
yolandait seems to get trigger events, but slowly10:46
jheskethyolanda: those errors have been there since the gerrit upgrade (afaik)10:46
yolandamm, red herring then10:47
jheskeththe no changes thingy is less cool10:47
yolandanew changes are not queued10:47
jheskethyolanda: maybe check an older log just to confirm10:47
*** andymaier has quit IRC10:47
jheskethyolanda: did it reload recently? it might be recalcaulting dependencies etc10:47
jheskeththat usually holds it up for a little (although probably not hours)10:47
yolandai'm seeing several "Done adding trigger event" on latest entries10:47
jheskeththen again, I had a 500 error from gerrit, so maybe that's slowing things down10:48
yolandamm, i tried telnet to port 4730, and STATUS command is not responding10:50
yolandajhesketh, do you think there can be another alternative, apart from zuul restart?10:54
jheskethgearman isn't responding? :-s10:54
yolandai can telnet but doesn't respond to any command10:55
jheskethyolanda: did you do 'status' lower case?10:55
* jhesketh isn't sure if it's case sensitive10:55
yolandaouch, uppercase10:55
yolandastatus works10:56
jheskethright, the load and memory look okay on the server, so I'm not sure we need to restart zuul just yet10:56
yolandalog is strange, not collecting any new change, not interacting with jenkins...10:56
igorbelikovyolanda jhesketh looks like about the same time our fuel ci stopped posting comments on gerrit via gerrit-trigger. There’s a queue of unprocessed gerrit events in jenkins logs, so maybe this is on gerrit side?10:57
*** AJaeger has joined #openstack-infra10:57
yolandai cannot see errors on gerrit log10:58
*** gildub has joined #openstack-infra10:59
yolandaand i did a recheck of my change, and zuul picked that properly10:59
yolanda2016-03-15 10:45:57,094 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent ref-replicated None>10:59
yolanda2016-03-15 10:45:57,097 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.10:59
yolandamm, not that10:59
yolanda2016-03-15 10:58:37,602 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent comment-added openstack-infra/devstack-gate master 292792,1>10:59
yolandaso it can get events from gerrit but is not adding them to the queue10:59
jheskethif zuul is stuck posting results back it may not process the event queue11:00
*** Qiming has joined #openstack-infra11:00
yolandaonly entries in zuul right now, are the adding trigger event ones11:00
igorbelikovhttp://paste.openstack.org/show/490471/ - I see a lot of this in our logs. Our jenkins picks up changes fine, but has problems posting back results, and this is via gerrit-trigger, so zuul is not involved in the process11:01
*** maishsk has joined #openstack-infra11:02
jheskethyolanda: do you see any "Processing result event" in the log?11:02
*** mizar_ has joined #openstack-infra11:03
*** mizar_ is now known as flongo11:03
*** fabio_ has joined #openstack-infra11:03
jheskethyolanda: and is there a "Processing trigger event" afterwards11:03
*** fabio_ has quit IRC11:03
jhesketh(ie has it gotten stuck on the result)11:03
yolandajhesketh, no Processing result since 08:34, 2 hours and a half before11:04
*** fverbs has joined #openstack-infra11:04
jheskethyolanda: but any trigger event afterwards?11:04
yolandajhesketh, yes, it continues posting logs about Adding trigger event11:05
yolandathe Adding trigger/Done adding trigger pair11:05
jheskethyolanda: that's in another thread... are there any "Processing trigger event" ?11:05
*** AJaeger_ has joined #openstack-infra11:06
odyssey4meyolanda odd, it seems that https://review.openstack.org/292326 has merged and executed, but http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate is empty11:06
*** kushal has joined #openstack-infra11:07
jheskethyolanda: okay so it's stuck posting results back11:07
*** AJaeger__ has joined #openstack-infra11:08
yolandajhesketh, looks like. In status page there are several jobs with all entries in SUCCESS, but they are still there and not reported11:08
*** x00350071 has quit IRC11:08
yolandawhat should be best way to unlock it?11:08
jheskethyolanda: well there are 317 results waiting to be posted back11:08
jheskethyolanda: not sure... (without losing the work anyway)11:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible  https://review.openstack.org/29277111:09
yolandayep, a restart will do it, but it will cause that people need to recheck their jobs11:09
*** AJaeger has quit IRC11:10
jheskethyolanda: I'm also not sure if gerrit is having trouble11:10
yolandayes, it can be gerrit... but i don't see errors on logs, and queue is empty11:10
yolandahow can verify it?11:10
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281411:10
yolandaalso a gerrit restart is less harmful than zuul11:11
jheskethsince igorbelikov's CI stopped processing events and it hasn't repliacted odyssey4me's changes11:11
*** AJaeger_ has quit IRC11:11
jheskethyolanda: I'm actually not sure what effect it'll have on a zuul that's stuck posting back results11:11
*** moravec has joined #openstack-infra11:11
jheskethbut it might be safer11:11
yolandaok let's try first a gerrit restart...11:12
*** dims has joined #openstack-infra11:12
*** rhallisey has joined #openstack-infra11:12
yolanda#status notice Gerrit is going to be restarted11:12
openstackstatusyolanda: sending notice11:12
* jhesketh doesn't like restarting things randomly as guesses but doesn't have a better suggestion11:12
yolandayes, at this point we don't have better options11:12
yolandamm, it unlocked zuul actually11:13
-openstackstatus- NOTICE: Gerrit is going to be restarted11:13
sdaguewell, I don't think gerrit is having any more 500 errors than normal11:14
yolandai start to see zuul submiting changes11:14
jheskethyolanda: right, that particular result may have been lost and it should continue through the result queue11:14
sdaguethe even queue length is still super high11:14
openstackstatusyolanda: finished sending notice11:15
*** wei-wang has quit IRC11:15
*** markvoelker has quit IRC11:15
yolandai sent a recheck and i still don't see in queue11:15
sdaguewell with the event queue over 3000, that's not totally surprising, right?11:16
yolandazuul log looks much better now11:16
yolandait's weird that gerrit fails this way, the logs were not really showing a problem11:16
yolandaapart from the normal gerrit problems11:17
jheskethyolanda: the result queue doesn't appear to be going down11:17
jheskethbut all the check jobs have disappeared11:17
jhesketh(that's possible because I think they were all done since the other changes are waiting in the event queue)11:18
yolandai see some entries for merged jobs in zuul, and posting events as well11:18
jheskethyolanda: oh, there it goes11:18
yolandayes, check is there now11:18
jheskethand the event queue is being processed11:18
jheskethlooks like nodepool is responding to the load appropriately11:20
*** dims has quit IRC11:20
yolandamm, there are lots of old entries in gerrit.log about loosing database connection11:21
*** dims has joined #openstack-infra11:21
*** dizquierdo is now known as dizquierdo_afk11:21
*** dims has quit IRC11:21
*** dims has joined #openstack-infra11:21
AJaeger__jhesketh: https://review.openstack.org/#/c/289125/6/jenkins/jobs/xstatic.yaml does not install any distro packages11:22
AJaeger__Is that correct?11:22
jheskethAJaeger__: it was copied from the tarball job... should it install any?11:23
AJaeger__jhesketh: ah, you just do tarballs - should be fine...11:23
AJaeger__I think gerrit did not record any events from those jobs that were finished ;(, so I rechecked a few ...11:26
yolandaAJaeger yes, seems that several job postings were lost11:26
jheskethoh, that sucks :-s11:27
*** mrunge has quit IRC11:28
AJaeger__yolanda: what about a notice to ask for rechecks?11:28
yolandamakes sense, i'll send11:28
*** ldnunes has joined #openstack-infra11:29
AJaeger__yolanda: give a timeframe if possible11:29
*** ihrachys has quit IRC11:31
*** ihrachys has joined #openstack-infra11:31
yolanda#status notice Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 08:30 UTC to 10:30 UTC approximately. Please recheck any affected jobs by this problem.11:32
openstackstatusyolanda: sending notice11:32
*** mrmartin has joined #openstack-infra11:32
yolandamm, i think i messed with timezone... is 11:30 utc :(11:32
*** tzn has quit IRC11:32
AJaeger__yolanda: approximately ;)11:32
flongohi all, I do not know if my review is affected by the problem: https://review.openstack.org/#/c/292801/11:33
-openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 08:30 UTC to 10:30 UTC approximately. Please recheck any affected jobs by this problem.11:33
yolanda#status notice Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 09:30 UTC to 11:30 UTC approximately. Please recheck any affected jobs by this problem.11:33
flongohow long does it take to jenkins before starting to test a review that has been approved?11:33
*** tzn has joined #openstack-infra11:33
yolandaflongo, it depends on the load. I suggest you check http://status.openstack.org/zuul. If you don't see your change there, and you don't have any results posted on gerrit, please recheck11:34
openstackstatusyolanda: finished sending notice11:34
openstackstatusyolanda: sending notice11:34
-openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 09:30 UTC to 11:30 UTC approximately. Please recheck any affected jobs by this problem.11:35
*** ihrachys has quit IRC11:36
AJaeger__flongo, your change is in the check queue.11:36
AJaeger__And once it passes that one it goes into gate queue...11:36
*** kushal has quit IRC11:37
openstackstatusyolanda: finished sending notice11:37
flongoyolanda: yes I saw it in the queue, thanks11:37
flongoAjaeger__ thanks!11:38
*** weshay has joined #openstack-infra11:39
*** mrunge has joined #openstack-infra11:40
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Update Email Configuration file  https://review.openstack.org/29260511:41
*** thorst_ has joined #openstack-infra11:41
*** dizquierdo_afk is now known as dizquierdo11:42
*** jed56 has quit IRC11:43
*** rfolco has joined #openstack-infra11:45
*** Hal has joined #openstack-infra11:45
*** Hal is now known as Guest2771911:46
*** nmagnezi has quit IRC11:46
*** gildub has quit IRC11:48
*** erlon has joined #openstack-infra11:49
*** nmagnezi has joined #openstack-infra11:49
*** jlanoux has joined #openstack-infra11:53
*** _amrith_ is now known as amrith11:54
*** salv-orl_ has joined #openstack-infra11:54
*** jlanoux_ has quit IRC11:56
*** salv-orlando has quit IRC11:57
*** kgiusti has joined #openstack-infra11:58
*** trown has joined #openstack-infra12:00
*** HeOS has quit IRC12:00
*** HeOS has joined #openstack-infra12:01
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo  https://review.openstack.org/27221012:03
*** mrmartin has quit IRC12:04
*** ociuhandu has joined #openstack-infra12:05
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Update Zuul Layout for Puppet OpenStack  https://review.openstack.org/29255512:05
*** zhurong_ has joined #openstack-infra12:07
*** ihrachys has joined #openstack-infra12:07
*** dtardivel has quit IRC12:08
*** salv-orl_ has quit IRC12:08
*** jaosorior has quit IRC12:08
*** aysyd has joined #openstack-infra12:08
*** jaosorior has joined #openstack-infra12:09
*** jaosorior has quit IRC12:14
*** shardy has quit IRC12:16
*** thiagop has joined #openstack-infra12:16
thiagopHello folks, I'm having some trouble setting up a CI server with the puppet modules wrt a RSA key being mistaken as ECDSA. Does anybody know how to solve this?12:18
*** zhurong_ has quit IRC12:18
thiagopquick google showed how to solve this direclty on paramiko, but I don't think it should need to change code to do that...12:19
rcarrillocruzthiagop: when you say set up CI server, what do you mean?12:19
rcarrillocruzis it a gerrit12:19
rcarrillocruza zuul12:20
rcarrillocruzwhich puppet module are you seeing that failure12:20
thiagoprcarrillocruz: yes, all of that. Using the single_node_ci.pp12:20
*** bdemers has quit IRC12:20
rcarrillocruzso you mean puppet-openstackci12:20
thiagoprcarrillocruz: yep12:20
*** _david_ has joined #openstack-infra12:20
rcarrillocruzpaste the site.pp you are using12:21
rcarrillocruzthe command12:21
rcarrillocruzand the full error12:21
rcarrillocruzon paste.openstack.org12:21
thiagoprcarrillocruz: in fact, the error is on zuul.log after the installation completes12:21
*** sfinucan has quit IRC12:22
thiagoprcarrillocruz: http://paste.openstack.org/show/490495/12:22
_david_fungi, jeblair, mordred We need a feedback if AccountPatchReview thing is important for you to stay, or can be removed12:22
*** sfinucan has joined #openstack-infra12:22
_david_^^^ https://groups.google.com/d/topic/repo-discuss/KhXKMKTJNMs/discussion12:22
*** ayoung has joined #openstack-infra12:23
rcarrillocruzdo you have this zuul pointing at your own gerrit12:25
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box'  https://review.openstack.org/28613712:25
rcarrillocruzor to infra gerrit , i.e. review.openstack.org12:25
*** dims has quit IRC12:25
AJaeger__fungi, I updated https://etherpad.openstack.org/p/bindep with some findings about database setup including some questions. Could you extend/comment, please?12:25
*** dims has joined #openstack-infra12:26
thiagoprcarrillocruz: no, it's pointed to openstack's. And the key is already on our user's account (I tested the key reading the stream with SSH)12:26
rcarrillocruzare you sure your zuul.conf points to the correct ssh private file?12:28
*** piet has joined #openstack-infra12:29
*** Guest27719 has quit IRC12:30
thiagoprcarrillocruz: yeah, the file is generated using the key we paste on common.yaml and I already tested the key for format issues using it on SSH12:30
thiagoprcarrillocruz: the key path on zuul.conf is correct too12:30
rcarrillocruzand if you do12:32
rcarrillocruzsudo su - zuul12:32
rcarrillocruzyou can cat just fine the sshkey you have on zuul.conf12:32
*** baoli has joined #openstack-infra12:32
*** edmondsw has joined #openstack-infra12:32
rcarrillocruzi'm asking all these questions cos that error is a pretty generic one12:33
rcarrillocruzand i've encountered it because of missing param12:33
rcarrillocruzor lack of perms on zuul user to access it12:33
*** baoli_ has joined #openstack-infra12:33
rcarrillocruzi never encountered an issue about ecdsa12:33
*** armax has quit IRC12:33
thiagoprcarrillocruz: cat works fine12:33
*** hichihara has joined #openstack-infra12:35
thiagoprcarrillocruz: ssh -i using key and from zuul user works fine too12:36
*** jaosorior has joined #openstack-infra12:36
*** baoli has quit IRC12:37
*** rlandy has joined #openstack-infra12:39
*** trown has quit IRC12:43
*** shardy has joined #openstack-infra12:44
*** sulo has quit IRC12:44
*** lucasagomes is now known as lucas-hungry12:45
*** sulo has joined #openstack-infra12:45
*** sulo has quit IRC12:46
*** asettle has joined #openstack-infra12:46
*** sulo has joined #openstack-infra12:46
*** shardy has quit IRC12:47
*** shardy has joined #openstack-infra12:48
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TEST: overcloud SSL with netiso  https://review.openstack.org/29278712:49
*** gordc has joined #openstack-infra12:50
*** amrith is now known as _amrith_12:50
AJaeger__yolanda: 292521 is not entering teh gate queue - is zuul buggy as well and refusing to add anythign to gate?12:51
*** asettle has quit IRC12:51
yolandagoing to check logs, yes... is weird that it doesn't get added12:51
*** AJaeger__ is now known as AJaeger12:52
*** dprince has joined #openstack-infra12:52
igorbelikovyolanda: same for 291762 and 29172412:52
*** trown has joined #openstack-infra12:53
*** markvoelker has joined #openstack-infra12:53
yolanda2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Considering adding change <Change 0x7f50847e78d0 292521,1>12:53
yolanda2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f50847e78d0 292521,1> can not merge, ignoring12:53
yolanda2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f50847e78d0 292521,1> is not ready to be enqueued, ignoring12:53
AJaegeryolanda: and repo openstack-ansible-os_designate seems to exist but is not setup completely ;(12:53
*** ddieterly has joined #openstack-infra12:54
openstackgerrityolanda.robla proposed openstack-infra/project-config: Start adding test for bifrost dhcp  https://review.openstack.org/29252112:54
yolandai rebased, let's see12:54
AJaegeryolanda: other jobs are not entering either like 9207412:54
AJaegerI meant: 29207412:54
AJaegerOh, 286756,6 just entered gate!12:55
yolandai don't see errors for 29207412:55
yolandaigorbelikov, i saw the same "not ready to be enqueued" error for 29176212:56
openstackgerrityolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce  https://review.openstack.org/29279212:58
*** ddieterly is now known as ddieterly[away]13:01
*** ddieterly[away] is now known as ddieterly13:01
*** eharney has joined #openstack-infra13:01
*** trown is now known as trown|ooobootcam13:02
*** armax has joined #openstack-infra13:05
openstackgerritThierry Carrez proposed openstack-infra/project-config: Pre-release stable/mitaka ACLs  https://review.openstack.org/29278113:05
*** dizquierdo has quit IRC13:05
*** yamamoto has quit IRC13:05
*** sdake has joined #openstack-infra13:08
*** maishsk has quit IRC13:08
*** maishsk has joined #openstack-infra13:08
SamYaplelol mordred "First and foremost - I am the only ATC in the project currently."13:09
SamYaplethats a pretty solid reason13:09
*** jlanoux has quit IRC13:10
pc_mHi! I added coverage testing to a repo (similar to what Rally project, where it compares coverage of change set to latest). Getting failures frequently, which are due to "build timed out". Is there anything that can be done?13:10
dhellmannwe have a stuck review that we can't make merge: https://review.openstack.org/#/c/292539/  We've tried rechecking and adding another workflow. Is there anything else we can do to get it to merge?13:10
*** jlanoux has joined #openstack-infra13:10
*** ddieterly has quit IRC13:11
dhellmannAJaeger, fungi : do either of you have any suggestions? ^^13:11
sdaguedhellmann: there was a gerrit / zuul outage overnight13:11
AJaegerdhellmann: zuul/gerrit is somehow ill ;( yolanda has tried to heal it but that was only a band aid ;(13:11
dhellmannsdague : yeah, we've been trying to land it since then13:12
*** yamahata has joined #openstack-infra13:12
dhellmannah, ok, so ongoing issues13:12
AJaegerdhellmann: I fear we need the full fire brigade to wake up ;(13:12
*** ddieterly has joined #openstack-infra13:12
dhellmannok, we've done the release for that patch so it can wait, thanks AJaeger and yolanda13:13
AJaegerdhellmann: do you have some time for a chat? Then I join you in #openstack-releases13:13
sdaguedhellmann: are you 100% it's cleanly rebased on master? I've found that there seems to be times where things wedge for this reason and it doesn't tell you13:13
dhellmannsdague : I'm not, so we could try that, too13:14
sdagueI would try that as a work around, hit the rebase button, and see if you get different results13:14
*** akshai has joined #openstack-infra13:14
*** amitgandhinz has joined #openstack-infra13:15
AJaegersdague: I really doubt it based on what I've seen. But let's try...13:16
yolandanot sure what's going on now13:16
*** salv-orlando has joined #openstack-infra13:16
dimssdague : AJaeger : trying that trick now13:16
*** ddieterly has quit IRC13:17
SamYaplehello, can someone help me with this skip-if that just merged https://review.openstack.org/#/c/292187/13:18
yolandasame error again, not arriving to gate13:18
SamYapleit does not appear to be working https://review.openstack.org/#/c/292869/13:18
*** ig0r__ has joined #openstack-infra13:19
SamYaplethat patch is running all gates and it should not based on that skip-if above13:19
igorbelikovyolanda: I have at least 7 changes with the same issue, so yeah, something is definitely wrong:(13:20
yolandagate is just empty13:20
yolandaso something is quite wrong yes13:20
*** piet has quit IRC13:20
*** tonytan4ever has joined #openstack-infra13:21
*** armax has quit IRC13:21
*** asettle has joined #openstack-infra13:21
*** ig0r_ has quit IRC13:22
*** greghaynes has quit IRC13:22
*** jlanoux has quit IRC13:25
*** chlong has joined #openstack-infra13:25
*** asettle has quit IRC13:25
yolandaany other infra-root can help with the issue? right now i cannot see more options than a zuul restart13:26
*** ayoung has quit IRC13:26
*** jlanoux has joined #openstack-infra13:26
yolandai just keep seeing that "not ready to be enqueued, ignoring" error13:27
dimsyolanda: that's why the gate queue is empty. right?13:29
*** dimtruck is now known as zz_dimtruck13:29
fungiyolanda: jhesketh: it's possible if gerrit lost its network socket to trove for some reason that it could have been blocking zuul on a report call from that timeframe13:29
yolandayes, all changes going to gate seem to hit that problem13:29
*** mriedem_away is now known as mriedem13:30
yolandafungi but what seems strange is that check queue looks fine now, and gate fails13:31
fungithiagop: if memory serves, openssh will try several configured key types, so if it doesn't recognize the rsa key as valid it'll look for a dsa one, then an ecdsa one, then it'll emit a misleading error about not finding a valid ecdsa key even though the underlying cause is a corrupt/malformed rsa key13:31
yolandaif there was some block, zuul won't be operating...13:31
yolandafungi, also.. zuul-server -t should be validating syntax properly? is that reilable? i ran it, to ensure that layout.yaml is valid, and i got a failure13:32
*** rfolco has quit IRC13:33
*** larsbutler has quit IRC13:33
*** rfolco has joined #openstack-infra13:33
thiagopfungi: do you think "ssh" is fixing the key somehow? I thought if I could connect using the same key, that should work with zuul too..13:33
*** piet has joined #openstack-infra13:34
*** piet has quit IRC13:34
*** infra-red has joined #openstack-infra13:35
*** greghaynes has joined #openstack-infra13:38
*** sdake has quit IRC13:38
*** Shrews has joined #openstack-infra13:38
yolandafungi, any ideas apart from trying to restart zuul?13:39
*** pradk has joined #openstack-infra13:40
*** _ari_ has joined #openstack-infra13:40
*** ayoung has joined #openstack-infra13:40
*** akshai has quit IRC13:42
AJaegerfungi, yolanda : The repos that have been created today like tripleo-ui and openstack-ansible-os_designate are not setup on git.openstack.org ;(13:43
*** akshai has joined #openstack-infra13:44
*** sigmavirus24_awa is now known as sigmavirus2413:45
*** akshai has quit IRC13:45
yolandaAJaeger, are they created on gerrit properly?13:45
*** akshai has joined #openstack-infra13:45
yolandalast entry in logs for manage_projects is from 10 Mar, that's not good13:46
fungi_david_: i'm not aware of any of our existing tooling querying the account_patch_reviews table in mysql if that's what you're asking. perhaps jeblair is aware of something, but http://codesearch.openstack.org/ isn't turning up any direct use13:47
*** mgrosz has joined #openstack-infra13:47
*** lucas-hungry is now known as lucasagomes13:48
*** nwkarsten has joined #openstack-infra13:48
*** nwkarsten has quit IRC13:48
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Only publish API guides from master  https://review.openstack.org/29288813:48
yolandafungi, AJaeger, also puppetdb doesn't seem to be good for review.openstack.org13:48
_david_fungi, thanks.13:48
AJaegeryolanda: I didn't check further13:48
*** automagically_ has joined #openstack-infra13:48
_david_fungi, This is something you can use on the change screen: check box if a path was reviewed already or not13:48
*** nwkarsten has joined #openstack-infra13:48
_david_(and diff side by side screen)13:48
*** mgrosz has quit IRC13:48
*** automagically_ has left #openstack-infra13:49
*** yamamoto has joined #openstack-infra13:49
kzaitsev_mb_hi, I think, that our patches in murano are not scheduled for gate checs and are not merged as result — is this a known issue?13:49
*** kzaitsev_mb_ is now known as kzaitsev_mb13:49
*** yamamoto_ has joined #openstack-infra13:49
AJaegerkzaitsev_mb: yes, a known issue. We have a generic problem with our CI13:50
kzaitsev_mbk, thanks not bugging you then =)13:50
*** Eyal_ has joined #openstack-infra13:50
yolandadoesn't look good13:50
yolandafatal: [87741fa9-92cb-4d1e-8195-6841e2468fb2]: UNREACHABLE! => {"changed": false, "msg": "SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue", "unreachable": true}13:51
fungiAJaeger: thanks for writing that up, a quick skim looks promising but i'll dive into it after coffee13:51
odyssey4meAJaeger yolanda I take it that we're still working through the aftermath of some failures? I'm specifically concerned with how https://review.openstack.org/292326 merged, the sync got executed and we have an empty repo in http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate as a result. :/13:51
AJaegerfungi, I have that git grep oslo_db |grep tests automated now and can give more - just wanted some first ideas...13:52
yolandaodyssey4me, so several issues seem to be happening. I'm taking a look at puppet failures on gerrit13:52
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box'  https://review.openstack.org/28613713:52
yolandai'd prefer if another infra-root can help with zuul, i only can see the solution of restarting the service13:52
odyssey4meyolanda thanks, I'll leave you to it - ping me if you come to a resolution or need anything from me13:53
*** Jeffrey4l has joined #openstack-infra13:53
*** yamamoto has quit IRC13:53
AJaegerodyssey4me: coffee, chocolate, ...? ;)13:53
fungipc_m: coverage jobs are generally very slow. i think it's because they run unit tests serially to accommodate the coverage validation piece. this is a big reason we've previously mostly run coverage post-merge rather than in check/gate (aside from the fact that gating on a specific percentage of coverage is sort of pathological)13:53
* odyssey4me hands AJaeger, yolanda, gerrit and zuul a cup of strong coffee :)13:54
thiagopfungi rcarrillocruz found it! The gerrit user on the common.yaml was with e-mail suffix13:54
AJaegeryummy, odyssey4me ! Thanks13:54
*** bdemers has joined #openstack-infra13:55
thiagoprcarrillocruz: some idiot (aka me) configured it wrong13:55
*** bdemers has quit IRC13:55
rcarrillocruzthat message is pretty generic as i said, i saw that with various misconf issues13:55
rcarrillocruzglad you fixed it13:55
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting  https://review.openstack.org/29289413:55
thiagoprcarrillocruz: thank you, and you too fungi13:56
pc_mfungi: We wanted to do comparison, so that people had an idea of whether their commit was improving coverage or not.13:56
* pc_m coverage in the 60% range currently13:57
*** keedya has joined #openstack-infra13:58
yolandafungi, checking puppet runs on review.openstack.org, i get fatal: [87741fa9-92cb-4d1e-8195-6841e2468fb2]: FAILED! => {"failed": true, "msg": "'dict object' has no attribute 'gitinfo'"}13:59
openstackgerritZara proposed openstack-infra/storyboard-webclient: Edit task notes from board  https://review.openstack.org/29175514:01
pc_mfungi: on a somewhat related note: is there a way to have py27 run serial fashion in a local repo?14:01
openstackgerritRossella Sblendido proposed openstack-infra/system-config: Add vhost for Neutron reviews dashboard  https://review.openstack.org/28428114:03
Eyal_Is there a problem with the zuul gateway there are no jobs running ?14:03
*** Guest41345 has quit IRC14:03
fungidhellmann: odd... 2016-03-15 13:27:19,936 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f4f8643a350 292539,3> does not match pipeline requirement <ChangeishFilter open: True current-patchset: True required_approvals: [{'username': 'jenkins', 'verified': [1, 2]}, {'workflow': 1}]>14:03
fungii disagree with the log, it looks to me at least like those conditions are met14:03
AJaegerEyal_: yes, we're on it...14:04
*** mrmartin has joined #openstack-infra14:05
Eyal_AJaeger: thanks14:05
SamYapleX/win 6614:07
*** Eyal_ has quit IRC14:07
*** bdemers has joined #openstack-infra14:07
*** bryan_att has joined #openstack-infra14:07
fungiokay, i'm now caught up with scrollback and absorbing the additional details of the zuul/gerrit situation14:08
*** piet has joined #openstack-infra14:08
*** infra-red has quit IRC14:08
*** armax has joined #openstack-infra14:10
*** smarcet has joined #openstack-infra14:10
*** armax has left #openstack-infra14:10
*** infra-red has joined #openstack-infra14:11
*** salv-orlando has quit IRC14:12
*** salv-orlando has joined #openstack-infra14:13
*** infra-red has quit IRC14:13
*** infra-red has joined #openstack-infra14:13
fungii don't see any recent changes to zuul's configuration to account for the current behavior14:15
*** ajmiller has joined #openstack-infra14:17
yolandafungi so this morning zuul was totally stuck, publishing results. A gerrit restart seemed to unlock it, but then the problems on gate started14:17
yolandacheck started to work fine but then no changes were entering gate, and the error was that the change was not ready to be queued14:17
fungiyeah, i read the scrollback14:17
*** jlanoux has quit IRC14:18
fungizuul's been running since the 11th, so if it's a recently introduced bug then it's been lying dormant for 4 days14:18
*** rbrndt has joined #openstack-infra14:19
yolandafungi is zuul -t reliable to validate syntax in layout.yaml?14:19
yolandathat's one of the things i thought, that some change introduced in layout.yaml, could have broken it14:20
*** oanson has quit IRC14:20
*** esikachev has quit IRC14:21
*** esikachev has joined #openstack-infra14:22
fungiwell, the behavior i posted above looks pretty self-contradictory, so i don't suspect it's something a malformed config could cause14:22
AJaegeryolanda: we run zuul -t as part of our project-config gate, see tools/run-layout.sh14:22
*** sshnaidm has quit IRC14:23
fungifinding a lot of "AttributeError: 'str' object has no attribute '_ZuulGearmanClient__gearman_job'" in the log, but that's been going on since well before the last restart14:24
*** sshnaidm has joined #openstack-infra14:25
yolandaAJaeger, i see. I miss the job-list in that command14:25
*** sdake has joined #openstack-infra14:25
*** akshai has quit IRC14:25
yolandaso that will be the problem i had when trying to test14:25
*** ihrachys has quit IRC14:25
fungiwe have reconfiguration failures in the logs, just found one but the traceback is lengthy so will put in a paste in a moment14:25
*** mrmartin has quit IRC14:26
*** akshai has joined #openstack-infra14:26
fungiyolanda: exception raised during zuul reconfiguration http://paste.openstack.org/show/490518/14:27
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add playbook to manage power state of ironic servers  https://review.openstack.org/28498114:27
fungilooks like maybe we merged a bad regular expression?14:27
*** infra-red has quit IRC14:27
yolandamm, that's error i got when running the zuul-server -t as well14:27
*** infra-red has joined #openstack-infra14:28
fungihave we maybe broken the exit code for our zuul configuration check jobs?14:28
*** fhubik has quit IRC14:29
*** fhubik has joined #openstack-infra14:29
yolandai'm checking the changes that landed this morning14:29
fungifirst occurrence was at 08:23:21 utc today14:29
fungiso it would have been something merging just prior to that14:30
fungithere were a couple this morning with regex modifications14:30
fungione is fairly straightforward, the other less so14:30
yolandado you have some candidates?14:30
*** Guest91794 has quit IRC14:31
yolandathat regex looks fine...14:31
smarcetfungi: yolanda: morning guys, if u have time could u please review https://review.openstack.org/#/c/292605/? i need this to add error email log capabilities14:31
yolandathe doc not ending with $ ?14:31
*** fhubik has quit IRC14:31
*** fhubik_real has joined #openstack-infra14:32
fungii'm having a look at the job results to see what our layout check said14:32
fungibut i haven't moved to my main computer for the day, and loading the console log for that job is taking forever on this very under-powered netbook14:33
AJaegersmarcet: right now nothing can merge, we first need to debug and fix our CI system, please have some patience14:33
yolandayep, i was loading the check for layout.yaml as well, but is long to see14:33
*** kingia_ has joined #openstack-infra14:33
*** xyang1 has joined #openstack-infra14:33
yolandathere is a traceback on apscheduler,  <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'now' . But that looks unrelated. And test reports no errors found14:34
fungi"No errors found in layout.yaml!"14:34
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291214:34
*** salv-orlando has quit IRC14:35
*** salv-orl_ has joined #openstack-infra14:35
yolandawell, all these tests have been positive.. so if there is an error, is hidden somehow14:35
*** kingia has quit IRC14:35
fungiis your reproduction with zuul -t on your local system?14:36
yolandano, not local14:36
fungioh, not as easy to bisect then14:36
*** bdemers has quit IRC14:37
*** ihrachys has joined #openstack-infra14:37
yolandafungi, i can test locally, copying the layout.yaml14:37
yolandaand i can remove the suspectingo nes14:38
*** kingia has joined #openstack-infra14:38
fungiyeah, if you can get it to fail on the current state of layout.yaml, then try before today's changes merged for a baseline14:39
fungijeblair: summary is that zuul is claiming changes which should be ready to merge are not matching the required criteria14:39
fungijeblair: we also have reconfiguration exceptions in the log starting at 08:23:21 utc today and the traceback at http://paste.openstack.org/show/490518/ makes me suspect we merged a bad regex14:40
fungii don't know if the two issues are related, but have a feeling they might be14:41
yolandafungi, we have a winner14:41
*** mtanino has joined #openstack-infra14:41
fungiyolanda: was it 292187?14:41
fungiso i guess that needs an emergency fix or revert14:41
fungifollowed by a manual submit in gerrit14:42
*** kingia_ has quit IRC14:42
yolandai'm checking the problem, i spotted a missing $ in docs, but not sure if that's it14:42
*** ddieterly has joined #openstack-infra14:42
fungialso we need to understand why zuul -t fails for you on it but doesn't fail in the layout job which passed that change with flying colors14:42
yolandaproblem is on the name of the project, that regex fails14:44
yolandabut yes, is risky that the check job for that passes14:44
*** yamahata has quit IRC14:44
*** mrmartin has joined #openstack-infra14:44
yolandafungi, how do you operate on these cases? in downstream i would have stopped puppet, fix layout.yaml manually14:44
*** keedya has quit IRC14:44
yolandaso zuul starts running, then do the revert14:45
*** mrmartin has quit IRC14:45
yolandaand restore puppet14:45
*** zz_dimtruck is now known as dimtruck14:45
cloudnullo/ all -- question, did our gerrit sign-in change to a new openID solution instead of the Launchpad one?14:45
* cloudnull is likely out of the loop14:45
mriedemAJaeger: question in here for you on the different nodes https://review.openstack.org/#/c/289255/2/jenkins/jobs/projects.yaml14:45
fungiyolanda: in cases like this it's almost as quick to just shove it through gerrit and update puppet out of cycle14:45
fungiyolanda: but if you want to stop puppet and fix it by hand instead go ahead14:46
yolandaso revert and force push? do we have the right perms for that normally, or is something that we need to add?14:46
cloudnullim seeing http://snag.gy/AzqwE.jpg right now.14:46
fungijust don't forget to unblock puppet for it again afterward14:46
yolandaok going to do it with stopping puppet...14:47
*** regXboi has joined #openstack-infra14:47
jeblairyolanda: don't force push; there's a group called "Project Bootstrappers"; you can add yourself to it, then use the gerrit ui to submit the change manually14:47
fungiyolanda: no, you can put it into gerrit as a normal review, add yourself to the project-bootstrappers group in gerrit, code review +2 verify +2 workflow +1 it and then submit it from the change screen14:47
*** Eyal_ has joined #openstack-infra14:48
fungicloudnull: we haven't changed the gerrit openid provider, no. should still be login.launchpad.net14:48
AJaegermriedem: answered briefly. Does that help?14:48
jeblairfungi, cloudnull: i get that error too14:48
mriedemAJaeger: so should he use ubuntu-trusty now?14:48
mordredI also get the error14:48
AJaegermriedem: Yes!14:48
mordredhas login.launchpad.net been sunsetted for login.ubuntu.com finally?14:48
mriedemAJaeger: thanks14:49
openstackgerrityolanda.robla proposed openstack-infra/project-config: Revert "Do not run gate jobs when doing docs-only patch"  https://review.openstack.org/29292414:49
mordredlogin.lp.net/+openid is not loading for me14:49
yolandainfra-root so can you review that?14:49
*** Eyal_ has quit IRC14:49
cloudnullseems LP is having issues http://cdn.pasteraw.com/e5kxzqy1enz9q0a0qe5dy6wug25629814:49
yolandajeblair ++14:50
mordredlogin.ubuntu.com is also not loading, fwiw14:50
jeblairclicking on 'log in' on launchpad is hanging as well14:51
fungiand yeah, i suspect the line ending inside the negative lookahead on that project name regex is the issue14:51
fungii can probably add myself to project-bootstrappers via the gerrit api14:51
cloudnullfrom #launchpad <cjwatson> frederickjh: Yes, there are issues with one of our internal infrastructure clouds; it's being worked on.14:52
yolandaouch.. Provider is not supported, or was incorrectly entered.14:52
mordredcloudnull: awesome. thanks14:52
yolandawhen trying to login to gerrit14:52
mordredjeblair, fungi: seems like a good topic for an alert14:52
jeblairmordred: ++; you typing it?14:53
fungiagreed, go for it. i'm messing with gerrit's set-group-members function14:53
*** dosaboy has quit IRC14:53
mordredjeblair: yah - is it #alert or #status (I'm still coffeeing)_14:53
openstackgerritZara proposed openstack-infra/storyboard-webclient: Edit task notes from board  https://review.openstack.org/29175514:53
jeblairmordred: "#status alert <text>"14:54
*** maishsk has quit IRC14:54
*** dosaboy has joined #openstack-infra14:54
*** dizquierdo has joined #openstack-infra14:54
*** hichihara has quit IRC14:54
mordredhow about: #status alert Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue14:54
mordred#status alert Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue14:55
openstackstatusmordred: sending alert14:55
openstackgerritMerged openstack-infra/project-config: Revert "Do not run gate jobs when doing docs-only patch"  https://review.openstack.org/29292414:55
*** dosaboy has quit IRC14:55
jeblairfungi: nicely done!14:55
funginow to take myself back out of the p-b group again14:56
mfischhey infra folks, openid signin is broken/looks really weird14:56
*** dosaboy has joined #openstack-infra14:56
mfisch"Provider is not supported, or was incorrectly entered."14:56
*** maishsk has joined #openstack-infra14:56
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891514:56
*** dosaboy has quit IRC14:57
AJaegermfisch: see status alert that is just send around14:57
fungimfisch: irc channel topics are updating as we speak14:57
-openstackstatus- NOTICE: Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue14:57
mfischim always missing the hot trends14:57
*** ChanServ changes topic to "Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue"14:57
fungithere you go14:57
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: The puppet-cookiecutter need gems to works.  https://review.openstack.org/29293314:58
yolandainfra-root, is there any way to force a zuul reconfig? or just wait?14:58
fungiyolanda: you can manually puppet apply if you want to just exercise the machinery faster14:59
fungiinstructions are in our system-config docs14:59
*** links has quit IRC14:59
SamYaplehello, recently merged this jobs filter for kolla with skip-if https://review.openstack.org/#/c/292187/, it _should_ be filtering all gates except docs when a docs patch is detected but it is not doing so see https://review.openstack.org/#/c/292869/14:59
*** jpr1 has quit IRC14:59
SamYapledoes this filter take a little while to take affect or is it incorrect14:59
yolandafungi but the update of layout.yaml is triggering the reload of zuul automatically?14:59
yolandaSamYaple, we just reverted the change14:59
fungiSamYaple: reverted in https://review.openstack.org/29292415:00
*** markvoelker has quit IRC15:00
*** jpr has joined #openstack-infra15:00
clarkbyolanda: no, you have to update the file then appl a signal to zuul. Puppet does both15:00
fungiproblem with the regex somehow isn't spotted by the check job but blew up zuul all morning15:00
openstackstatusmordred: finished sending alert15:00
yolandaclarkb ok good to know15:00
*** mrmartin has joined #openstack-infra15:01
tobiashwhere does zuul fill in the reporter_config?15:01
*** jaosorior has quit IRC15:01
pabelangeranteaya: clarkb: Sorry, was on PTO yesterday.15:01
SamYapleso are you saying i broke things yolanda fungi ?15:01
jeblaircan we move to incident?15:01
tobiashmy gerrit rejects "--verified 2" in the gate pipeline where "--label verified=2" seems to work somehow15:02
*** roxanagh_ has joined #openstack-infra15:02
pabelangerclarkb: I'd have to look into grafana and see how to cache the data.  I know you can have graphite render the image a static over grafana using css15:02
SamYapleah man. thats awful! I feel bad now15:02
*** rhedlind has joined #openstack-infra15:03
SamYaplewhats wrong with that regex, it validates for me what I would expect it to do15:03
fungiSamYaple: don't feel bad. we obviously have a test coverage issue this exposed (as well as people approving regular expressions without trying them out)15:03
pabelangeranteaya: re: stackalytics.o.o no, not yet. We needed to enable logs first to see what is happening.   It would be good to compare to stackalytics.com too, and see if they are pinning the CPU15:03
SamYaplefungi: well it does work for me with POSIX grep, but is it evaluated differently in the gate?15:04
fungiSamYaple: http://paste.openstack.org/show/490518/ is the traceback15:04
SamYapleesh. ok let me test this out in python15:05
SamYaplei didnt know how it was being eval'd, but re makes sense15:05
SamYapleso the gate job isn't processing the regex. that seems odd15:06
SamYaplei think it failed on my first patchset15:06
*** tesseract has joined #openstack-infra15:06
*** bdemers has joined #openstack-infra15:06
*** adrian_otto has joined #openstack-infra15:07
*** dosaboy has joined #openstack-infra15:07
*** roxanagh_ has quit IRC15:07
*** tesseract is now known as Guest7146715:07
adrian_ottois there a known problem with SSO for review.openstack.org?15:07
SamYapleadrian_otto: cloudnull was just talking about that15:07
SamYaple14:53:20 < cloudnull> from #launchpad <cjwatson> frederickjh: Yes, there are issues with one of our internal infrastructure clouds; it's being worked on.15:08
SamYapleadrian_otto: ^15:08
adrian_ottothanks SamYaple15:08
*** IlyaG has joined #openstack-infra15:09
clarkbwe also updated all the channel topics with info on that15:09
*** dosaboy has quit IRC15:09
adrian_ottothanks clarkb15:09
SamYaplefungi: regex seems valid.... http://paste.openstack.org/show/490527/15:10
dmsimardpabelanger: how are you setting up/collecting the various roles you need to run playbooks right now ? Galaxy or setup.py ?15:10
dmsimardi.e, your dib or zuul roles15:10
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add Ubuntu binary build and deploy gate for kolla  https://review.openstack.org/29294215:12
*** IlyaG has quit IRC15:13
*** krtaylor has quit IRC15:13
pabelangerdmsimard: zuul-cloner right now15:14
pabelangerdmsimard: https://github.com/openstack/windmill/blob/master/tools/install_roles.sh15:15
SamYaplefungi: is there more info we can get on that traceback? As far as i can tell that regex is valid (and local python re agrees with me) so i wonder what happens in translation from yaml to it being compiled15:16
pabelangerI've started working on a script to parse from ansible requirment.yml file to zuul-cloner but still need to finish it up15:16
dmsimardpabelanger: I've never published anything ansible-galaxy ready, trying to look into it right now15:17
fungiSamYaple: yeah, we're troubleshooting over in #openstack-infra-incident so as not to get drowned out by the general support noise in here15:17
fungiSamYaple: i'm trying to recreate the failure which generates that traceback15:17
pabelangerdmsimard: well, ansible-galaxy publish, is just them reading your github.com repo and cloning it locally15:17
pabelangerI tried to use it a few weeks ago, but it was forcing a README.md file15:18
pabelangerdmsimard: pretty sure they fixed it to use README.rst but haven't tested yet15:18
dmsimardpabelanger: yeah, I've been meaning to try and use the requirements.yml file with galaxy15:18
dmsimardnot necessarily publishing15:18
dmsimardI thought you needed something like a metadata.json file (akin to puppet) but nothing fancy like that, no.15:19
*** _amrith_ is now known as amrith15:19
pabelangerdmsimard: Ya, i've tested it locally and it works great15:19
pabelangerjust like zuul-cloner better :)15:19
dmsimardwell, in your context it makes sense15:19
*** dosaboy has joined #openstack-infra15:19
*** piet has quit IRC15:20
openstackgerritMerged openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo  https://review.openstack.org/27221015:23
*** yamahata has joined #openstack-infra15:24
yolandaok zuul seems to be back to life15:24
*** infra-red has quit IRC15:24
*** krtaylor has joined #openstack-infra15:25
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make task notes visible for logged-out users  https://review.openstack.org/29295715:26
*** salv-orl_ has quit IRC15:27
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891515:27
*** pvaneck has joined #openstack-infra15:27
mordredinfra-root: launchpad is back- I'm going to send the ok15:27
jeblairmordred: thanks; remember you can add text after the ok "#status ok <text>"15:28
mordred#status ok Launchpad SSO is back to normal - happy hacking15:28
openstackstatusmordred: sending ok15:28
mordredjeblair: thanks! I'd forgotten that15:28
*** infra-red has joined #openstack-infra15:28
*** armax has joined #openstack-infra15:29
*** MarkAtwood has joined #openstack-infra15:29
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"15:30
-openstackstatus- NOTICE: Launchpad SSO is back to normal - happy hacking15:30
yolandazuul happy, launchpad happy, we are happy now :)15:31
yolandaAJaeger, have the projects been created? i ran manage-projects on gerrit15:31
*** xyang1 has quit IRC15:31
pabelangerAnd, I'm offline again for the next hour or so. Relocating back to relatives farm. Then I'm back for the rest of the day15:32
openstackstatusmordred: finished sending ok15:33
*** chlong has quit IRC15:33
*** jlanoux has joined #openstack-infra15:34
*** nmagnezi has quit IRC15:35
*** infra-red has quit IRC15:37
openstackgerritMerged openstack-infra/project-config: Start adding test for bifrost dhcp  https://review.openstack.org/29252115:37
*** Qiming has quit IRC15:38
*** fawadkhaliq has quit IRC15:40
*** fawadkhaliq has joined #openstack-infra15:40
openstackgerritMerged openstack-infra/tripleo-ci: Rename the ceph to upgrades  https://review.openstack.org/29235915:41
*** Swami has joined #openstack-infra15:41
*** kushal has joined #openstack-infra15:42
*** sridhar_ram has joined #openstack-infra15:43
*** ddieterly is now known as ddieterly[away]15:43
*** infra-red has joined #openstack-infra15:43
*** vgridnev has quit IRC15:43
*** maishsk has quit IRC15:43
*** fawadkhaliq has quit IRC15:45
*** yamahata has quit IRC15:45
*** ddieterly[away] is now known as ddieterly15:46
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks  https://review.openstack.org/29139315:46
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics.  https://review.openstack.org/29139215:46
*** vgridnev has joined #openstack-infra15:46
*** jpr has quit IRC15:48
*** amrith is now known as _amrith_15:49
*** vgridnev has quit IRC15:49
*** Jeffrey4l has quit IRC15:50
*** infra-red has quit IRC15:50
*** tongli has joined #openstack-infra15:51
*** gyee has joined #openstack-infra15:51
*** acruz has joined #openstack-infra15:51
*** daemontool has joined #openstack-infra15:51
bartashhi all15:52
*** arxcruz has quit IRC15:52
clarkbbartash: hello15:52
*** maishsk has joined #openstack-infra15:53
bartashcould anyone help with the issue on this review? https://review.openstack.org/#/c/292302/  jenkins fails to execute py27 job15:54
clarkbbartash: looks like it executes, it just fails to succeed. See http://logs.openstack.org/02/292302/1/check/gate-shaker-python27/9b8f5d0/console.html#_2016-03-14_11_38_21_828 for the error15:55
clarkbbartash: looks like a missing dependency15:55
clarkbbartash: I want to say the the pyzmq lib would bundle the C and compile it on install if it had to. You will probably want to double check on that, maybe they broke their packaging15:56
*** bpokorny has joined #openstack-infra15:56
*** nwkarsten has quit IRC15:57
*** asselin__ has joined #openstack-infra15:57
*** salv-orlando has joined #openstack-infra15:57
*** asselin_ has quit IRC15:57
*** links has joined #openstack-infra15:58
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu  https://review.openstack.org/29301215:58
bartashclarkb: yes, I saw that log file too. But how to solve this problem and whom should I ask about it?15:59
*** IlyaG has joined #openstack-infra15:59
clarkbbartash: the first thing I would do is try to reproduce it locally. Make sure you rebuild your virtualenv when you do `tox -repy27`15:59
clarkbbartash: then go from there, chase it up the pyzmq tree and see why that is breaking15:59
*** acruz has quit IRC16:00
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move layout job to bare-precise  https://review.openstack.org/29301516:00
*** akshai has quit IRC16:00
*** akshai has joined #openstack-infra16:01
*** mrmartin has quit IRC16:02
bartashclarkb: it succeeded on my local virtualenv16:03
*** maishsk has quit IRC16:03
*** asettle has joined #openstack-infra16:03
bartashclarkb: you mean go through the console log with that failure?16:03
clarkbbartash: did you rebuild with tox -r?16:03
*** _amrith_ is now known as amrith16:04
clarkbbartash: I think it is possibly a new packaging issue16:04
clarkbso you want to rebuild16:04
*** tphummel has joined #openstack-infra16:06
morganclarkb, seeing something weird in gerrit - i have a change that is +2/+A but the list view isn't reflecting it.16:06
*** _david_ has quit IRC16:06
clarkbbartash: it is using an older pyzmq version according to the freeze16:06
morganclarkb: not sure if it's something out of whack or a weird cache thing.16:07
morganor something wider spread.16:07
fungimorgan: which change?16:07
clarkbbartash: possibly due to the constraints?16:07
fungiwe're trying to hunt down one for further testing16:07
morganfungi: so looking at https://review.openstack.org/#/q/project:openstack/stackalytics+status:open https://review.openstack.org/#/c/292653/16:07
morganfungi: happy to help find one :)16:07
*** asettle has quit IRC16:08
*** ramishra has quit IRC16:08
openstackgerritSam Yaple proposed openstack-infra/project-config: Kolla docs-only patch -- take 2  https://review.openstack.org/29302116:08
*** bpokorny has quit IRC16:09
*** bpokorny has joined #openstack-infra16:10
*** ramishra has joined #openstack-infra16:10
bartashclarkb: I'm quite noob in openstack gerrit and don't know about how this all mechanism works inside. We just depend on the patch shown above, and I wanted to find someone who can solve it, as it looks like a problem is not connected to Shaker project itself, isn't it?16:11
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the container CI job  https://review.openstack.org/29302216:11
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate  https://review.openstack.org/29302316:11
clarkbbartash: it is a problem in the shaker test suite16:11
*** ddieterly is now known as ddieterly[away]16:11
clarkbbartash: it relies on pyzmq but it isn't getting installed properly. You can find the install logs at http://logs.openstack.org/02/292302/1/check/gate-shaker-python27/9b8f5d0/tox/ for that job16:12
bartashclarkb: ok, I understand. thank you a lot for help.16:13
clarkbbartash: to clarify, gerrit is just a code review tool. Connected to gerrit we have zuul which runs test jobs on changes. This is one of those tests jobs and it runs `tox -repy27` and that was the result16:13
bartashclarkb: so the problem is just in shaker-job, right?16:13
*** cbader has quit IRC16:13
*** cbader has joined #openstack-infra16:14
clarkbbartash: yes that is where I would start. tox -repy27 is returning an error so that is what needs to be debugged16:14
*** sc68cal has joined #openstack-infra16:14
*** mrmartin has joined #openstack-infra16:14
bartashclarkb: ok, thanks, I'll continue looking in this direction16:15
fungimorgan: the approval event for that change seems to have been missed by zuul during an event stream issue earlier today. it'll need to be reapproved16:16
*** armax has quit IRC16:16
morganfungi: right. is that related to why the list view doesn't reflect the detail view?16:16
morganfungi: i was less concerned about it needing reapproval vs something wrong with gerrit :)16:16
fungimorgan: oh, i was looking at it in gertty and didn't see anything odd. i'll check with the webui16:17
morganfungi: yeah webui is inconsistent even with hard refreshes.16:18
*** kzaitsev_mb has quit IRC16:18
fungimorgan: i don't spend enough time looking at the gerrit webui... what specifically is missing? what are list view and detail view?16:19
morganfungi: the +2 / +A isn't showing in the list view16:19
morganfungi: where it shows in the detail view16:19
*** amrith is now known as _amrith_16:19
fungiahh! on your query16:19
fungithe query result list view. got it16:19
*** ramishra has quit IRC16:21
*** hashar has quit IRC16:21
fungimorgan: that is indeed a strange inconsistency. i wonder if we have something out of sync in gerrit's database16:22
fungior the lucene index16:22
*** roxanagh_ has joined #openstack-infra16:22
morganfungi: yah16:22
*** ramishra has joined #openstack-infra16:23
*** tzn has quit IRC16:23
*** tzn has joined #openstack-infra16:24
*** markvoelker has joined #openstack-infra16:25
*** jed56 has joined #openstack-infra16:27
fungimorgan: aha, the approval was at 10:26 utc and the start time on our gerrit process (from this morning's restart) was also 10:26 utc. i wonder if there's a race in gerrit which could account for that specific change16:28
morganfungi: maybe. happy to get it a re-freshed approval16:28
morganfungi: if you're content no issue(s) i'll go do that/recheck/whatever16:29
morganfungi: but it looked weird so wanted to pass on first :)16:29
sc`i had to -/+ workflow a change to get it picked up after i had already approved it16:29
*** tzn has quit IRC16:29
*** ramishra has quit IRC16:30
sc`reverify was just kicking off recheck it appeared16:30
*** mikelk has quit IRC16:31
jeblairsc`: yes, that's a known quirk of the system used to ensure that patches pass check before being enqueued in gate16:31
*** ramishra has joined #openstack-infra16:31
*** sdake_ has joined #openstack-infra16:31
*** roxanagh_ has quit IRC16:32
sc`good to know16:32
clarkbdepending on how the periodic jobs go/went https://review.openstack.org/#/c/292573/ would be a good one to get in to finish up the devstack-trusty to ubuntu-trusty migration16:32
* clarkb looks for evidence of goodness on ubuntu-trusty16:34
*** sdake has quit IRC16:34
*** esikachev has quit IRC16:35
*** rcernin has quit IRC16:35
*** skolekonov is now known as _skolekonov16:36
clarkbhttp://logs.openstack.org/41/285541/5/check/gate-tempest-dsvm-full-liberty/93061ef/console.html fails but looks like a legit tempest fail not host related16:36
*** flongo has quit IRC16:36
*** eil397 has joined #openstack-infra16:38
*** andrearosa has quit IRC16:38
*** tmorin has joined #openstack-infra16:39
clarkbjenkins07 has almost 7k threads I am going to run the restart playbook again it16:42
*** Guest71467 has quit IRC16:42
*** piet has joined #openstack-infra16:43
*** scheuran has quit IRC16:44
*** kzaitsev_mb has joined #openstack-infra16:44
*** jpr has joined #openstack-infra16:45
*** links has quit IRC16:47
*** tzn has joined #openstack-infra16:47
*** mrmartin has quit IRC16:50
*** apoorvad has joined #openstack-infra16:51
*** vincentll has quit IRC16:51
*** flongo has joined #openstack-infra16:52
*** SumitNaiksatam has quit IRC16:52
anteayapabelanger: okay happy relocating16:53
*** tzn has quit IRC16:54
*** dprince has quit IRC16:54
*** MCoLo__ has quit IRC16:54
tmorinhi sdague: would you have some time to look at https://review.openstack.org/#/c/291103/16:55
*** akshai has quit IRC16:55
tmorintmorin: it's a simple job change to move job parameters from the job yaml file to an rc file in the project repo16:55
tmorinAJaeger wanted you to have a look16:56
tmorinand it was already blessed by dougwig as Neutron liaison16:56
*** nwkarsten has joined #openstack-infra16:57
*** akshai has joined #openstack-infra16:57
*** vgridnev has joined #openstack-infra16:58
*** akshai has quit IRC16:58
*** e0ne has quit IRC16:58
*** sc68cal has quit IRC16:59
jeblairtmorin: is that file in every branch of the repo?  what's the plan for managing changes to that across multiple branches?17:00
*** akshai has joined #openstack-infra17:00
*** fawadkhaliq has joined #openstack-infra17:00
tmorinjeblair: no, its not in every branch17:01
tmorinjeblair: the jobs are experimental only, we plan to enable them only in recent branches17:01
*** nwkarsten has quit IRC17:01
tmorinjeblair: so we've no plan to put the rcfiles in past branches17:01
*** esikachev has joined #openstack-infra17:02
jeblairfungi, clarkb, yolanda: upon further inspection, a failed reconfiguration should not cause a problem with timer triggers.  the code i was looking at is just a bunch of unimplemented methods added by the connections change.  i think i have already removed them in zuulv3.17:05
*** esikachev has quit IRC17:07
jeblairfungi, clarkb, yolanda: a failed reconfiguration will cause some kind of strange state with the scheduler losing track of the triggers it has defined, though i can't actually figure out what the result of that would be.  i think they will continue to work.17:07
yolandajeblair, i wonder if there is some way to prevent these reconfiguration failures or make them more visible on logs17:08
jeblairfungi, clarkb, yolanda: so, as best as i can tell, the code has latent bugs, but they should not actually cause things to stop working.17:08
jeblairyolanda: the reconfiguration failure was prevented -- it never loaded the bad configuration17:08
clarkbjeblair: so the regex not being old python stdlib compatible was the only real problem?17:08
*** tqtran has joined #openstack-infra17:09
jeblairclarkb: the only thing i can prove at this point is that the regex caused the reconfiguration to abort, and zuul continued to use the old config as designed17:09
yolandainfra-root, can there be a problem with gerritbot? i sent a change right now and didn't see anything posted17:10
*** nwkarsten has joined #openstack-infra17:10
*** fawadkhaliq has quit IRC17:10
clarkbyolanda: there are two openstackgerrit's here17:10
clarkblikely something is unhappy17:10
*** MarkAtwood has quit IRC17:11
*** asselin__ has quit IRC17:11
*** vgridnev has quit IRC17:12
*** bdemers has quit IRC17:12
*** sambetts is now known as sambetts|afk17:12
yolandai see some errors on gerritbot log17:13
*** bdemers has joined #openstack-infra17:13
bknudsonAJaeger: does devstack support opensuse? I tried downloading opensuse and it failed with an error17:14
anteayaAJaeger: I figured you would probably have some experience with bknudson's question17:14
*** openstackgerrit has quit IRC17:14
*** openstackgerrit_ is now known as openstackgerrit17:14
*** tmorin has left #openstack-infra17:14
*** openstackgerrit has quit IRC17:15
yolandarestarted gerritbot17:15
anteayayolanda: thank you17:15
*** openstackgerrit_ has joined #openstack-infra17:15
*** openstackgerrit_ is now known as openstackgerrit17:16
*** esikachev has joined #openstack-infra17:16
*** openstackgerrit_ has joined #openstack-infra17:16
fungiyolanda: clarkb: the extra gerritbot is coming from the replacement gerrit server. could stand to shut that one down for now17:17
yolandaoh i can stop it yes17:17
*** fawadkhaliq has joined #openstack-infra17:18
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291217:20
anteayawell it works17:20
anteayawe can check that off the list for April 11th17:20
rbradforI'm getting ERROR: missing Change-Id in commit message footer when I'm trying to submit a patchset (3 reviews). I've done this before without any issues.http://paste.openstack.org/show/490559/17:21
*** asettle has joined #openstack-infra17:21
clarkbrbradfor: what does `git log -1` show?17:22
clarkber might need -3 based on that short log17:22
rbradforclarkb, sure -3 show my 3 commits, but no Change-Id17:22
clarkbrbradfor: ok, then you need to force the commit hook to run17:23
fungiyeah, git-review had never been used in that clone before according to the output17:23
clarkbgit rebase -i HEAD~317:23
fungiso it downloaded the commit hook which adds the change-id after you committed your changes17:23
clarkbchange each pick to reword, save each one and that should add teh change ids17:23
*** mrmartin has joined #openstack-infra17:24
*** pcaruana has quit IRC17:24
rbradforclarkb, fungi thanks, reword has given me change-ids17:24
*** SumitNaiksatam has joined #openstack-infra17:25
*** fawadkhaliq has quit IRC17:25
fungiand it asks nothing in return!17:25
clarkbgit-review does its best to add the change id to the last commit if it is missing but doesn't know how to add them to a whole stack17:25
*** ddieterly[away] is now known as ddieterly17:25
*** fawadkhaliq has joined #openstack-infra17:25
*** asettle has quit IRC17:26
*** jlanoux has quit IRC17:27
*** jlanoux has joined #openstack-infra17:28
*** jlanoux has quit IRC17:28
*** jlanoux has joined #openstack-infra17:29
*** esikachev has quit IRC17:31
*** matrohon has quit IRC17:31
mriedemi need some help with what i thought was a simple change17:33
mriedemeasy peasy17:33
mriedembut fails with http://logs.openstack.org/17/292617/1/check/gate-project-config-layout/2be8620/console.html#_2016-03-14_21_59_01_54617:33
clarkbmriedem: looking. Also we should just merge that es status page change in e-r if it isn't already17:33
*** dprince has joined #openstack-infra17:33
mriedem^ follows the same pattern as the other jobs using that template from what i can tell17:33
*** Sukhdev has joined #openstack-infra17:34
mriedemclarkb: i'll poke sdague17:34
mriedemsdague: poke! https://review.openstack.org/#/c/290832/17:34
clarkbmriedem: I think you need to add the stable python jobs template in jenkins/jobs/projects.yaml too17:34
clarkbmriedem: let me find an example17:35
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL  https://review.openstack.org/28198817:35
*** sfinucan has quit IRC17:35
clarkbmriedem: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n3818 the two bitrot lines there need to be added to designate17:35
clarkbmriedem: which is at https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n101517:36
mriedemclarkb: ah, ok17:36
*** pcaruana has joined #openstack-infra17:36
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Add periodic-kilo/liberty jobs to openstack/designate  https://review.openstack.org/29261717:37
*** sc68cal has joined #openstack-infra17:39
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quotas support  https://review.openstack.org/28511017:40
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quota ansible tests  https://review.openstack.org/29307217:40
*** jlanoux has quit IRC17:41
*** mrmartin has quit IRC17:42
*** mriedem1 has joined #openstack-infra17:43
*** mriedem has quit IRC17:44
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq  https://review.openstack.org/29307417:45
*** mriedem1 has quit IRC17:46
*** abregman has quit IRC17:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq  https://review.openstack.org/29307417:46
*** rossella_s has quit IRC17:47
Zarahi, we believe all storyboard email patches have landed (other instances send emails, and infra's mail config was merged last week), but emails still aren't arriving from storyboard.openstack.org . Anyone know what we might have missed?17:48
*** sputnik13 has joined #openstack-infra17:48
*** derekh has quit IRC17:49
*** mrmartin has joined #openstack-infra17:51
krotscheckZara: It could be that the cloud which storyboard is running on does not permit outgoing mail. I seem to remember something like that.17:51
krotscheckfungi would know more.17:51
krotscheckOr clarkb17:52
clarkbkrotscheck: it should be running in rax which does permit email (it is a big reason for why our control plane is in rax)17:52
krotscheckclarkb: Oh good.17:52
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test  https://review.openstack.org/28944517:52
krotscheckON an unrelated note, I'm concerned about something the UX team is doing with their repo- they're proposing and committing binary files.17:52
krotscheckgit... doesn't diff those. It just stores them as blobs.17:52
clarkbZara: I can look at exim logs and see what there is to see17:52
clarkbZara: but fungi and jeblair know far more about mail than I do17:53
*** mriedem has joined #openstack-infra17:53
Zaraclarkb: brilliant, thank you. yeah, I know very little about it; annoying position to be in...17:53
krotscheckI anticipate that repository is going to balloooooooon.17:53
clarkbkrotscheck: what kind of binary files? but generally yes that isn't considered the greatest of ideas17:53
*** salv-orl_ has joined #openstack-infra17:54
*** ybathia has joined #openstack-infra17:54
krotscheckclarkb: Turns out I may have spoken too soon. There's a PDF up for reveiw right now, but in the future they'll only accept svg17:54
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test  https://review.openstack.org/27921817:55
*** ybathia has left #openstack-infra17:55
clarkbZara: 2016-03-15 16:44:12 H=localhost (storyboard.openstack.org) [] sender verify fail for <no-reply@storyboard.openstack.org>: Unknown user17:55
clarkbI think exim is saying it cna't use the no-reply user to send mail because it doesn't know who that user is?17:55
clarkbit is trying to send mail though17:55
Zarasounds like it, nice to know it's trying! :)17:56
*** salv-orlando has quit IRC17:56
Zaraso I guess we want it to either know who that user is, or send from a different user...17:57
openstackgerritTimur Sufiev proposed openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job  https://review.openstack.org/29307717:58
*** ihrachys has quit IRC17:58
fungiclarkb: Zara: yes, i believe this came up before when i checked logs for SotK a month or more ago. exim is performing sender verification on locally-injected messages there. we should be able to configure it to not do that, or we could configure it to use a valid from address17:59
clarkbstoryboard is a valid user so we could use storyboard@storyboard.openstack.org17:59
*** bdemers has quit IRC17:59
anteayathat sounds like an easy way forward17:59
fungiright, it mainly wants to know that there's somewhere it can send the non-delivery reports18:00
Zaraif we send it to storyboard@storyboard.openstack.org, what happens if someone replies?18:00
anteayadoes anyone recieve the mail sent to storyboard@storyboard.openstack.org?18:00
*** Sukhdev has quit IRC18:00
anteayawhat Zara said18:00
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL  https://review.openstack.org/28198818:00
fungias an example, gerrit is sending as review@openstack.org18:01
fungiand setting the reply-to header to someone/somewhere it would make sense to reply18:01
clarkbexim is listening on port 25 so it should get delivered to storyboard18:01
*** bdemers has joined #openstack-infra18:02
fungi(in the case of gerrit review comments, it sets reply-to to the preferred e-mail address of the comment author)18:02
funginot to imply that gerrit is an ideal model for dealing with e-mail notification, but it is at least _a_ model we currently handle and have for many years18:03
*** _nadya_ has quit IRC18:03
*** dims_ has joined #openstack-infra18:03
clarkbin storyboard's case it probably does make sense to deliver it to storyboard itself and have it do updates to tasks based on emails18:03
*** dims has quit IRC18:03
clarkbbut that may be an eventual goal rather than an immediate worry18:04
*** flongo has quit IRC18:04
pabelangerianw: centos7 dib is failing on https://rdoproject.org/repos/rdo-release.rpm18:04
pabelangerany ideas what we're getting from rdo?18:05
persiaHaving storyboard receive the mail for now, and ignore it is probably the easiest way forward.  There have been discussions in the patches about having Storyboard parse email in the future.18:05
fungiyes, as a down-the-road feature opportunity, having the option for storyboard to receive and process replies nakes sense, so in anticipation of that having a bit bucket we can send replies into for now (storyboard@storyboard.openstack.org) seems time18:05
fungier, seems fine18:05
fungihah, what persia said more eloquently18:05
jeblairyeah, i wrote a router to do verp to so it can handle bounces18:05
*** ddieterly is now known as ddieterly[away]18:05
jeblairi think it should also do that; preferably sooner rather than later :)18:06
Zara:) okay, so first step, changing the sender and reply-to to storyboard@storyboard.openstack.org18:07
*** tonytan4ever has quit IRC18:07
*** kzaitsev_mb has quit IRC18:07
pedroalvarezZara: I was clonning the repo to do that, are you doing it already?18:08
jeblairthat should work but will cause bounces to be delivered to the storyboard user's mail spool (so that's probably best considered a temporary measure)18:08
Zarapedroalvarez: ah, you're ahead of me; I'd just found the right file18:08
Zarago for it18:08
*** mrmartin has quit IRC18:10
*** lucasagomes is now known as lucas-dinner18:10
Zarajeblair: okay, I'll bear it in mind. they need a lot of work in every direction... but it's a start! :)18:10
fungiright, we have until the rootfs fills up or runs out of inodes to get storyboard handling inbound e-mail and discarding or tracking bounces ;)18:10
openstackgerritPedro Alvarez Piedehierro proposed openstack-infra/puppet-storyboard: Change default $sender_email_address  https://review.openstack.org/29308018:11
thiagopbumping my head against nodepool, image-build is getting stuck. How can I see logs or something to identify what happened?18:11
*** terriyu has joined #openstack-infra18:11
anteayapedroalvarez: thank you18:12
pabelangerthiagop: you should see the output in image.log18:12
thiagop(I didn't run the service yet)18:12
thiagoppabelanger: in the same folder I ran image-build?18:12
*** adrian_otto1 has joined #openstack-infra18:12
pedroalvarezanteaya: :)18:12
anteayathere's that smile18:13
pedroalvareztomorrow could be the day that storyboard sends emails :)18:13
anteayasuch a wonderful smile, lights up a room it does18:13
anteayapedroalvarez: yay tomorrow18:13
pabelangerthiagop: depends on how you installed it, and if logging.conf is setup18:13
pabelangercheck /var/log/nodepool18:14
*** sridhar_ram1 has joined #openstack-infra18:14
*** sridhar_ram has quit IRC18:14
thiagoppabelanger: no logs on this folder. I installed it using puppet-openstackci18:14
*** adrian_otto has quit IRC18:14
pabelangerthiagop: check /etc/nodepool/logging.conf18:14
pabelangerit should point to /var/log/nodepool/image.log18:15
pabelanger/var/www/nodepool I think18:15
*** jistr has quit IRC18:15
Zara(:D once it works, the emails are off by default, so users can choose to trial them. because they're really annoyi-- er, they could do with being more configurable, but we'd like to know what to prioritize)18:15
*** mrmartin has joined #openstack-infra18:16
AJaegerbknudson: devstack should support openSUSE, there were two weeks ago 3 patches to fix some problems. Hope those merged now...18:16
openstackgerritMerged openstack-infra/project-config: Skip magnum functional test jobs on docs changes  https://review.openstack.org/27789218:17
thiagopon builder-logging.conf it points to /var/log/nodepool/image/image.log, but this file isn't generated18:17
anteayaZara: yup, thanks for the optional annoyance18:17
*** _nadya_ has joined #openstack-infra18:17
thiagoppabelanger: ^18:17
anteayaand for jeblair for making it possible18:17
sc68calAJaeger: ping me if those patches haven't18:17
*** jordanP has quit IRC18:17
pabelangerthiagop: is nodepool-builder configured to use it?18:17
pabelangerbeen a while since I used puppet-openstackci18:18
pabelangerthiagop: check debug.log too18:18
thiagoppabelanger: no debug.log or nodepool.log either.18:18
* thiagop feels like something is missing on the docs...18:18
pabelangerYour the 2nd person this week to say logging wasn't working18:19
pabelangerI wonder if some step is missing18:19
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate  https://review.openstack.org/29302318:19
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891518:19
*** infra-red has joined #openstack-infra18:19
thiagoppabelanger: http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool18:19
thiagopI'm on the image-build step of this^18:20
*** dizquierdo has quit IRC18:20
pabelangerthat's because you are not using service nodepool-builder start18:20
thiagopI found it strange that we run this thing without nodepool service is actually running18:20
pabelangerno, my mistake18:20
pabelangerthe server should be running18:20
*** flongo has joined #openstack-infra18:21
pabelangerps -ef | grep nodepool18:21
*** flongo has quit IRC18:21
*** terriyu has quit IRC18:22
thiagoppabelanger: no mention of nodepool-builder, but the service is here (stopped)18:22
thiagopno mention *on the docs*18:22
AJaegersc68cal: let me check and get back to you...18:22
thiagoppabelanger: nothing special on ps: http://paste.openstack.org/show/490575/18:23
openstackgerritMerged openstack-infra/project-config: Add periodic-kilo/liberty jobs to openstack/designate  https://review.openstack.org/29261718:23
openstackgerritMerged openstack-infra/project-config: Add experimental job to test xstatic packaging  https://review.openstack.org/28912518:23
*** _nadya_ has quit IRC18:23
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291218:24
*** vgridnev has joined #openstack-infra18:24
pabelangerthiagop: Ya, looks like nodepool service is not running18:25
pabelangerthiagop: sudo service nodepool status18:25
pabelangerthiagop: sudo service nodepool start18:25
pabelangerthen, check ps again, ensure -l /etc/nodepool/logging.conf is passed18:25
*** sridhar_ram1 is now known as sridhar_ram18:26
pabelangerthen, try image-build again18:26
*** flepied has quit IRC18:26
pabelangerthiagop: you'll see something like the following when working: http://nodepool.openstack.org/dib.devstack-centos7.log18:26
fungithe instructions should probably say to manually start the service. our puppet modules generally don't ensure services are started because this can have the nasty effect of puppet automatically restarting the service when you've intentionally stopped it, or restarting it over and over when you've done something that makes it crash18:26
pabelangerYa, that could be the issue18:27
*** daemontool has quit IRC18:27
*** baoli_ has quit IRC18:27
*** IlyaG has quit IRC18:27
*** piet has quit IRC18:28
*** terriyu has joined #openstack-infra18:28
*** fhubik_real has quit IRC18:28
*** e0ne has joined #openstack-infra18:28
*** piet has joined #openstack-infra18:28
*** mrmartin has quit IRC18:29
*** dims_ has quit IRC18:29
*** esikachev has joined #openstack-infra18:29
thiagopfungi: now I have debug.log, but no image.log yet18:30
*** tonytan4ever has joined #openstack-infra18:30
thiagoppabelanger: ^18:30
thiagopdebug log keeps repeating this: http://paste.openstack.org/show/490576/18:31
thiagopshould I start the nodepool-builder too?18:31
openstackgerritShiv Haris proposed openstack-infra/project-config: Adding branch creation priv to group networking-brocade-release  https://review.openstack.org/29263518:33
thiagopthis is the debug of "nodepool image-build"18:33
jeblairthiagop: yes, the builder service is responsible for building and uploading images18:33
*** [1]Thelo has joined #openstack-infra18:33
jeblairthiagop: that paste indicates there is a job queued for the builder, and it will run as soon as you start the builder18:34
*** sdake_ has quit IRC18:34
thiagopjeblair: so something is missing in that doc...18:34
jeblairthiagop: yep18:34
*** degorenko is now known as _degorenko|afk18:34
thiagopjeblair: I'll send a patch as soon as I discover which file to update18:35
fungithiagop: which document were you following? i'm happy to point you to the source for whatever it is18:35
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate  https://review.openstack.org/29302318:35
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891518:35
*** Thelo has quit IRC18:35
*** [1]Thelo is now known as Thelo18:35
*** esikachev has quit IRC18:36
*** vgridnev has quit IRC18:37
*** vgridnev has joined #openstack-infra18:39
AJaegersc68cal, bknudson: The suse changes for devstack have all merged. If anything is not ok, please ask toabctl who's more active in this area than me.18:39
fungithiagop: the source rts file for that document is http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/doc/source/third_party_ci.rst18:40
sc68calAJaeger: ack. thanks18:40
fungithiagop: changes should be submitted through gerrit just like for the rest of our tooling18:40
thiagopfungi: thanks! will do next18:40
anteayapleia2: when did we stop paginating git.o.o, do you know?18:40
fungier, s/rts/rst/18:40
fungianteaya: there was a recent change i saw fly by to up the page length again to something we're unlikely to hit for a very long time18:40
anteayait makes searching faster, I do find but I was caught by surprise by the change18:40
anteayafungi: ah okay, so intentional18:41
*** dalgaaf has joined #openstack-infra18:42
anteayawell the change wasn't to puppet-cgit: http://git.openstack.org/cgit/openstack-infra/puppet-cgit/log/18:42
anteayawould it have been in system-config?18:42
*** terriyu has quit IRC18:42
fungianteaya: yep, https://review.openstack.org/28874518:43
fungiso that changed about 5-6 days ago18:43
anteayacool, had just found that myself18:44
anteayaalways so slow compared to you :)18:44
fungithough that was 1500 and we're already well over 1000, so maybe "a very long time" is not so long, depending on how you measure18:44
anteayameasureing in fuel units?18:44
anteayamight have to up it by the end of the month18:44
*** BobBall has quit IRC18:44
fungihow many xstatics does it take to equal one fuel?18:45
anteayaI haven't done the converstaion18:45
anteayamy eyes blur when I see either word18:45
fungii think the rates fluctuate daily18:45
anteayalike the stock market18:45
fungibetter to invest in mordred's spirit animal fund18:45
fungiit seems to pay hefty dividends on a consistent schedule18:46
anteayait does at that18:46
anteayanever seems to run low18:46
* sc68cal chuckles18:48
*** asettle has joined #openstack-infra18:48
AJaegerinfra-root, this morning yolanda approved new repos http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ and http://git.openstack.org/cgit/openstack/tripleo-ui/ - but those are not imported in git.o.o yet "Repository seems to be empty". Also they are not setup in gerrit18:49
AJaegercould somebody look at these, please?18:49
*** _amrith_ is now known as amrith18:49
fungiAJaeger: yes, yolanda was previously investigating why puppet doesn't seem to have successfully updated review.o.o since the 10th but we all got so wrapped up in the zuul config issue i think we forgot to follow up18:49
fungias such, our image updates for today are probably all shot18:50
*** e0ne_ has joined #openstack-infra18:50
*** ddieterly[away] is now known as ddieterly18:51
*** e0ne has quit IRC18:51
jeblairfungi, pleia2: we're going to have an election!  i think that's wonderful and exciting!  i hope you both win!18:51
nibalizerhrm syslog doesn't have any puppet runs on review.o.o18:51
openstackgerritThiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst  https://review.openstack.org/29308718:51
thiagopfungi jeblair ^18:51
fungijeblair: i need a volunteer to run the electorate generation script for me on the day before the election and get the results to tristanC and tonyb18:52
fungijeblair: are you free to do that?18:52
jeblairfungi: what day is that?18:52
fungii'll make sure the script is retested and working as intended in the next day or so18:52
tonybjeblair: March 17th (UTC)18:52
fungithough it was working fine for me as of a couple weeks ago18:52
*** PsionTheory has joined #openstack-infra18:52
fungiso i don't expect any surprises18:53
*** asettle has quit IRC18:53
*** electrofelix has quit IRC18:53
openstackgerritRob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack  https://review.openstack.org/29309018:53
*** oanson has joined #openstack-infra18:54
*** vgridnev has quit IRC18:55
openstackgerritThiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst  https://review.openstack.org/29308718:55
*** adrian_otto1 has quit IRC18:56
pabelangerfungi: does new channels for accessbot cause the bot to drop from IRC?18:57
pabelangeror is that meetbot?18:57
fungichanges to the meetbot channel list will cause the "openstack" meetbot to leave and rejoin due to restarting, and so lose its state on running meetings18:57
AJaegerpabelanger: system-config IRC changes are the problem ;)18:57
AJaegerpabelanger: project-config is fine18:58
*** adrian_otto has joined #openstack-infra18:58
AJaegerpabelanger: that's my rule of thumb ;)18:58
jeblairfungi: thursday pacific time is not great for me actually; i'm traveling/meeting.  of course, i could do it just after midnight utc (wednesday evening pacific) but perhaps it would be better to find someone else who is more available thursday in case there are issues...18:58
pabelangerAJaeger: okay, I'll wait for osic change until later tonight18:58
fungijeblair: sure, i'll see who's available with gerrit admin rights18:58
fungisince we haven't yet switched to a build of gerrit with zaro's backport which would make this runnable by everyone18:58
fungiit's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour19:00
jeblairfungi: k.  i'm on board if you can't find someone with better availability.19:00
*** adrian_otto has quit IRC19:00
*** flepied has joined #openstack-infra19:01
openstackgerritMerged openstack-infra/project-config: Move layout job to bare-precise  https://review.openstack.org/29301519:01
openstackgerritMerged openstack-infra/project-config: Add CentOS & Fedora dashboards to grafana  https://review.openstack.org/29212319:02
yolandafungi i wanted to check with mordred, about puppet not running on review. I was worried about some strange situation with having 2 sites called review.openstack.org19:02
*** flwang1 has quit IRC19:03
ianwpabelanger: i think that was the problem yesterday.  the timestamp suggests we haven't tried building again...19:03
*** kushal has quit IRC19:03
anteayayolanda: oh good point19:03
*** bnemec has quit IRC19:03
*** vgridnev has joined #openstack-infra19:04
pabelangerianw: okay, I need to look into system-config and see what RDO we use19:04
ianwpabelanger: the reason we need it is for ovs19:06
openstackgerritMerged openstack-infra/system-config: Make sure wheel-mirror works on centos7  https://review.openstack.org/29014619:06
pabelangerianw: Ah, thank makes sense19:07
*** sdake has joined #openstack-infra19:07
ianwpabelanger: the bigger question is, why didn't it retry the build today.  i haven't read scrollback but i guess that was not expected19:07
*** pvaneck has quit IRC19:10
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291219:10
*** dims has joined #openstack-infra19:12
*** bnemec has joined #openstack-infra19:13
*** krotscheck is now known as kro_death_by_sin19:14
*** fverbs has quit IRC19:14
*** kro_death_by_sin is now known as kro_sinus_hell19:14
*** bnemec has quit IRC19:16
*** dprince has quit IRC19:18
*** bnemec has joined #openstack-infra19:18
*** dims has quit IRC19:18
*** e0ne_ has quit IRC19:19
*** xyang1 has joined #openstack-infra19:19
*** ybathia has joined #openstack-infra19:21
*** sdake_ has joined #openstack-infra19:21
*** sdake has quit IRC19:23
*** dims has joined #openstack-infra19:26
*** sridhar_ram has quit IRC19:28
pabelangerianw: Ya, I am not sure.  Would need to check the logs myself too. I can do that after meetings19:29
ddieterlystack.sh is hanging at this point19:30
ddieterlysudo grep -q "^#includedir.*/etc/sudoers.d" /etc/sudoers ||19:30
ddieterly    echo "#includedir /etc/sudoers.d" | sudo tee -a /etc/sudoers19:30
ddieterlyany ideas why?19:30
ddieterlyi'm following the instructions at https://github.com/openstack-infra/devstack-gate19:31
*** dprince has joined #openstack-infra19:31
*** ociuhandu has quit IRC19:32
*** ybathia has quit IRC19:33
*** fawadkhaliq has quit IRC19:35
*** PsionTheory has quit IRC19:36
*** aysyd_ has joined #openstack-infra19:37
*** aysyd has quit IRC19:37
*** aysyd_ is now known as aysyd19:37
*** hashar has joined #openstack-infra19:39
*** nwkarsten has quit IRC19:40
ianwddieterly: which bit is hanging?  might be useful to put logs in paste.openstack.org .  also, more a #openstack-qa type question19:41
ddieterlyianw seems like every sudo command is hanging in stack.sh19:42
ddieterlyianw which log file is it? i can't seem to find anything useful in the log19:42
*** cody-somerville has quit IRC19:42
*** tsg has joined #openstack-infra19:43
*** cody-somerville has joined #openstack-infra19:43
*** salv-orl_ has quit IRC19:43
ianwddieterly: have you set NOPASSWD for your user?19:43
ddieterlymmm... i don't think so19:44
ddieterlythat sounds like the problem19:44
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate  https://review.openstack.org/29310819:44
ddieterlyoh, yes i did19:44
*** _nadya_ has joined #openstack-infra19:45
*** dims has quit IRC19:45
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891519:46
*** dprince has quit IRC19:48
*** abregman has joined #openstack-infra19:48
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891519:50
*** sdake_ has quit IRC19:51
*** pvaneck has joined #openstack-infra19:52
asselinthiagop, jeblair fungi pabelanger fyi those instructions were accurate as of nodepool 0.2.0. They need to be updated to the new version of nodepool, or 3rd party ci requested to pin to 0.2.0 (latest tag)19:52
*** ociuhandu has joined #openstack-infra19:53
*** sridhar_ram has joined #openstack-infra19:54
pabelangerasselin: should we add a nice fat warning to the page?19:56
ddieterlylooks like the stack user also needs the NOPASSWD setting19:56
*** rockyg has joined #openstack-infra19:58
*** MarkAtwood has joined #openstack-infra19:58
anteayadavid-lyle: my goodness, you are stilling getting accolades19:59
anteayadavid-lyle: can we make the david lyle going away parade into an annual event?19:59
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Pin nodepool to latest tag 0.2.0 for 3rd party CI  https://review.openstack.org/29311219:59
david-lyleanteaya: stockholm syndrome is real20:00
asselinpabelanger, ^^ I can also add to the docs saying this is assuming version 0.2.020:00
anteayadavid-lyle: ha ha ha20:00
*** annegentle has joined #openstack-infra20:01
* craige reminds -infra folk that he needs more eyes on his dev phabricator instance: https://os01.mcwhirter.com.au/ - it's very close to ready and I need more eyes identifying issues. Thanks!20:02
anteayapiet: ^^20:02
craigeand txt :-)20:03
craigeugh ttx20:03
pabelangerasselin: might be worth it20:03
*** tonytan4ever has quit IRC20:04
pietcraige Does it include Pholio?20:04
*** ig0r_ has joined #openstack-infra20:04
*** tonytan4ever has joined #openstack-infra20:04
craigeIIRC yes (airport) piet . if not, I'm happy to add it.20:05
*** ig0r__ has quit IRC20:06
pietcraige The OpenStack UX team is looking to move away from Invision (closed source mock review tool) to Pholio20:06
*** erikwilson has joined #openstack-infra20:06
*** vgridnev has quit IRC20:07
pietcraige We are getting a lot of complaints about Invision being too slow, so would like to move sooner than later20:08
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL  https://review.openstack.org/28198820:08
*** IlyaG has joined #openstack-infra20:09
thiagopI think DIB is breaking due to this commit: https://github.com/openstack-infra/project-config/commit/b1a806767ea4e7b8fd280d8189f4c0dfd685984520:10
*** vgridnev has joined #openstack-infra20:10
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test  https://review.openstack.org/27921820:10
*** dims has joined #openstack-infra20:11
*** sdake has joined #openstack-infra20:11
thiagopos_designate isn't on git.openstack.org, it has an 'upstream' directive that should have been considered on the openstack-repos element20:11
*** erikwilson has quit IRC20:11
*** adrian_otto has joined #openstack-infra20:12
*** abregman is now known as abregman|afk20:12
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Clarify nodepool image build instructions post 0.2.0  https://review.openstack.org/29312220:12
asselinpabelanger, thiagop ^^20:12
*** vgridnev has quit IRC20:13
dmsimardpabelanger: turns out there *is* an equivalent to the puppet metadata.json for galaxy20:13
ianwpabelanger / fungi : can someone check the nodepool builder logs and see why it seems to have stopped?  it would also be nice to restart it, because it hasn't picked up centos-minimal jobs20:13
thiagopasselin: the line ahead explains how to start nodepool, not nodepool-builder20:13
*** esker has joined #openstack-infra20:14
thiagopasselin: am I ignoring something?20:14
asselinthiagop, I haven't used nodepool after tag 0.2.0 yet so I cannot update the instructions. You can switch back to tag 0.2.0 or look at the nodepool documentation whch is hoepfully up-to-date20:15
asselinthiagop, http://docs.openstack.org/infra/nodepool/20:15
smarcetfungi: not sure what happens here https://review.openstack.org/#/c/292912/20:15
smarcetfungi: but its failling, i tried same commands locally and works just fine20:16
asselinthiagop, if you want to use the 0.2.0 tag: https://review.openstack.org/29311220:16
*** kgiusti has left #openstack-infra20:16
thiagopasselin: the nodepool docs were surprisingly not helpful for me... :/20:18
mgagnewhat's nodepool-builder? o_O20:19
pabelangerdmsimard: Oh, I knew that20:19
pabelangerdmsimard: just didn't register that's what you were looking for20:19
pabelangerianw: fungi: I can take a peak20:20
dmsimardpabelanger: no worries :p20:20
fungismarcet: could it be something to do with the default php pear search path?20:20
smarcetnot sure20:21
fungismarcet: my php hacking days are so far behind me that it's a somewhat opaque topic for me these days20:21
fungismarcet: but i'll stare at it a little more and see if i come up with anything20:21
smarceti mean didnt change anything related to that20:21
pabelangerfungi: ianw: last image to build was ubuntu-trusty at 2016-03-14 15:45:45,436 but was a failure20:22
*** tonytan4ever has quit IRC20:22
*** tonytan4ever has joined #openstack-infra20:23
pabelangersorry, it retried at 2016-03-14 16:36:08,054 and was successful20:23
fungipabelanger: ianw: as i mentioned earlier, puppet not applying on review.o.o means several new repos broken on git.o.o means all nodepool image builds are going to bomb today20:23
*** kzaitsev_mb has joined #openstack-infra20:23
*** rfolco has quit IRC20:23
*** matrohon has joined #openstack-infra20:23
*** rfolco has joined #openstack-infra20:24
ianwfungi: ok, well what's a bit more concerning is they don't seem to have run at all, rather than failed.  or at least logs on nodepool.openstack.org don't seem updated20:24
nibalizerfungi: review.openstack.org doesn't exist in ansible all --list-hosts20:25
ianwunless, possibly, dib is stuck on a git hang20:25
thiagopmgagne: appears to be a service to listen to image-build requests and pass it/monitors DIB execution20:25
nibalizeri wonder if its going to uuids since we have two review.o.o ?20:25
pabelangerfungi: ianw: Ya, I don't see any image builds for the 15th20:25
thiagopmgagne: but I may be wrong, first day digging into it20:25
pabelangerdon't see any errors right now20:25
*** nwkarsten has joined #openstack-infra20:25
*** nwkarsten has quit IRC20:25
*** nwkarsten has joined #openstack-infra20:26
funginibalizer: yolanda theorized that her by-uuid blacklist for the replacement review.o.o is somehow responsible20:26
funginibalizer: but agreed if the inventory switches to only uuid for both of those, then ansible/puppet may get interesting?20:27
ianwthiagop / mgagne :  nodepool-builder is what builds the images.  it was separated into a separate process from the main nodepool daemon a little while ago.20:27
mgagneianw: still optional?20:27
*** roxanagh_ has joined #openstack-infra20:28
*** oanson has quit IRC20:28
*** sridhar_ram1 has joined #openstack-infra20:29
*** bswartz has quit IRC20:29
nibalizerhow is the by-uuid blacklist implemented?20:30
*** dimtruck is now known as zz_dimtruck20:30
pabelangerfungi: which syntax should I be using to send a nodepool command?  I believe you use sudo over su nodpool?20:30
*** piet has quit IRC20:30
*** sridhar_ram has quit IRC20:30
*** bswartz has joined #openstack-infra20:30
*** piet has joined #openstack-infra20:30
*** keedya has joined #openstack-infra20:31
*** flwang1 has joined #openstack-infra20:31
pabelangerianw: fungi: I think they are queued to be built, about to confirm20:31
ianwmgagne: having it as a separate daemon is still optional20:32
*** keedya has quit IRC20:32
ianwmgagne: see --no-builder option20:32
smarcetfungi: no worries, i was able to reproduce it locally20:32
smarcetworkiung on it20:32
*** roxanagh_ has quit IRC20:32
fungismarcet: oh, good, because i'm also testing several other things and trying to follow the tc meeting right now, so my attention was a little divided. sorry!20:33
fungipabelanger: sudo -H -u nodepool nodepool list20:33
fungipabelanger: et cetera20:33
mgagneianw: thanks for the info. the doc is not very explicit about it tbh20:34
*** regXboi has quit IRC20:34
*** fitoduarte has joined #openstack-infra20:34
*** gordc has quit IRC20:34
fungipabelanger: it's more crucial for dib operations since those create files on the filesystem in a predictable location, so not doing that as root is more sanitary, and also doing it as root leaves files that the nodepool user will later fail to be able to remove/replace20:34
*** esikachev has joined #openstack-infra20:34
*** oanson has joined #openstack-infra20:34
pabelangerfungi: roger20:35
pabelanger| 176408 | centos-7         | /opt/nodepool_dib/centos-7-1458051067         | 1458051067 | building | 00:06:23:26 |20:35
pabelangerfungi: ianw ^20:35
pabelangerlooks like dib is queued to build20:35
pabelangerbut nodepool-builder didn't do it20:35
thiagopthat was me20:35
thiagopI started the builder and it worked20:35
*** MarkAtwood has quit IRC20:36
ianwpabelanger: yeah, we know about that one, becaues the builder hasn't reloaded and thus hasn't registered to consume the job20:36
thiagopI followed the process by tail -f on /var/log/nodepool/images/images.log20:36
*** maishsk has joined #openstack-infra20:36
*** e0ne has joined #openstack-infra20:37
*** flwang1 has quit IRC20:37
ianwpabelanger: does ps show any stuck dib processes?20:37
*** salv-orlando has joined #openstack-infra20:37
ianwalthough that doesn't make sense, we should still get logs20:37
*** rockyg has quit IRC20:37
*** esker has quit IRC20:38
*** rockyg has joined #openstack-infra20:38
thiagopthe builder has a log of its own too20:38
pabelangerianw: Ya, so besides that, I don't see a reason why new dibs were not created today.  I only see centos7 being added to the queue at 14:00 UTC20:38
*** esker has joined #openstack-infra20:38
pabelangerother dibs were skipped20:38
openstackgerritMerged openstack-infra/elastic-recheck: Add a cluster health indicator to the graph output  https://review.openstack.org/29083220:39
*** esikachev has quit IRC20:39
pabelangerthiagop: good to know you got it working20:39
thiagoppabelanger: I'm now fixing the openstack-repos element and testing20:39
thiagoppabelanger: appears that a commit yesterday had an out-of-the-box approach20:40
*** baoli has joined #openstack-infra20:41
pabelangerianw: so, we could kick off a build manually, but like fungi said, going to fail to clone repos.  Guess we hold off until gerrit is fixed20:41
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job  https://review.openstack.org/29313020:41
mriedemsdague: dansmith: jbernard_: flaper87: smcginnis_away: ^20:41
*** fawadkhaliq has joined #openstack-infra20:42
*** IlyaG has quit IRC20:43
anteayamriedem: you do realize that the -nv at the end of the job means non-voting?20:43
mriedemanteaya: i do20:43
anteayamriedem: so you would make a copy of the -nv job and remove the -nv bit?20:43
AJaegermriedem: see also https://review.openstack.org/27332620:43
*** MarkAtwood has joined #openstack-infra20:44
*** zz_dimtruck is now known as dimtruck20:44
mriedemAJaeger: we aren't making it voting yet20:44
mriedemthe plugin job that is20:44
AJaegermriedem: ah...20:44
mriedemjbernard_ said this morning that was waiting on some mirror updates20:45
jbernard_mriedem: yeah, as soon as clarkb gives me the ok, ill upde the plugin to fetch from the mirror20:45
*** tongli has quit IRC20:46
clarkbI mean its been low on my priority list20:46
clarkb(been trying to get things that affect the release done like logstash 2.0 and new clouds and so on)20:47
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job  https://review.openstack.org/29313020:47
smarcetfungi: i think that problem its the latest composer.phar version20:47
smarcetwith older version does not happens20:47
clarkbI can start poking at fixing the puppet module though20:47
mriedemclarkb: re: ceph, we understand, was just pointing out why we're not making the plugin job voting yet20:47
mriedemi really just don't want to run redundant jobs on 3 projects when they are all nonvoting20:48
*** IlyaG has joined #openstack-infra20:49
*** ihrachys has joined #openstack-infra20:50
openstackgerritlifeless proposed openstack/requirements: Ignore more build temp dirs  https://review.openstack.org/25160420:50
*** sdake has quit IRC20:50
*** vgridnev has joined #openstack-infra20:51
clarkbwow new puppet is typed?20:51
zigoIn my tempest tests, I'm running Tempest with 2 testr threads. Am I right that this may cause some problems?20:52
zigoShould I remove the --parallel thing?20:52
*** SamYaple is now known as NotSamYaple20:52
nibalizer  arst20:52
clarkbzigo: we run it with 4 threads on most of our jobs20:53
clarkbzigo: the value we use is numprocs/220:53
zigoclarkb: How much RAM do you have available?20:53
*** sdake has joined #openstack-infra20:53
clarkbzigo: this allows other processes like openstack itself and mysql to run on the other half of the cus20:53
clarkbzigo: 8GB20:53
*** NotSamYaple is now known as SamYaple20:53
zigoI noticed, because I'm running it on a 12 core machines, and I had 24 threads running, which failed some tests ... :)20:53
*** _nadya_ has quit IRC20:53
AJaegercould some project-config cores, please review a rework the of the api-guide jobs as templates so that we can use it for more projects? Change is https://review.openstack.org/#/c/292135 - and needed by https://review.openstack.org/#/c/285519.  And https://review.openstack.org/#/c/292888 is part of the series20:53
*** ybathia has joined #openstack-infra20:55
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291220:55
*** keedya has joined #openstack-infra20:56
*** jpr has quit IRC20:57
*** moravec has left #openstack-infra20:58
*** baoli has quit IRC20:58
*** flwang1 has joined #openstack-infra20:58
*** ddieterly is now known as ddieterly[away]20:59
*** baoli has joined #openstack-infra20:59
*** sridhar_ram1 has quit IRC21:01
*** moravec has joined #openstack-infra21:01
*** _ari_ has quit IRC21:01
*** annegentle has quit IRC21:02
*** sdake_ has joined #openstack-infra21:02
*** gordc has joined #openstack-infra21:03
*** thorst_ has quit IRC21:03
*** angdraug has joined #openstack-infra21:03
*** f1ller is now known as filler21:03
*** infra-red has quit IRC21:04
openstackgerritlifeless proposed openstack/requirements: Note that MySQL-python is more than just GPL.  https://review.openstack.org/29313821:04
*** sdake has quit IRC21:05
*** MarkAtwood has quit IRC21:05
*** keedya has quit IRC21:05
*** rfolco has quit IRC21:06
*** aysyd has quit IRC21:06
*** smarcet has quit IRC21:07
bswartzanyone know where to find logs for jobs that run in release pipe?21:07
*** bdemers has quit IRC21:08
*** thorst_ has joined #openstack-infra21:09
AJaegerAnd use the SHA - firsttwochars/complete-sha21:09
*** sdague has quit IRC21:09
*** eharney has quit IRC21:09
bswartzAJaeger: http://logs.openstack.org/8a/8a7e5d27cbc6e5345a1eb86789482defbfbdb934/21:09
bswartzI see only "post" here21:09
bswartznot release21:09
AJaegerbswartz: what's the tag and project?21:09
*** shardy has quit IRC21:10
bswartzAJaeger: ^21:10
AJaegerbswartz: run "git showref 1.1.0"21:10
AJaegerto get the sha of the release21:10
openstackgerritMerged openstack-infra/openstackid: Refactoring  https://review.openstack.org/29291221:10
bswartzty that worked21:11
AJaegerbswartz: looking at your URL, it's d09924e72e292d8ee321e7e9a8b189885f2e521821:11
bswartznow to find why it failed....21:11
*** keedya has joined #openstack-infra21:11
*** baoli has quit IRC21:13
jeblairjesusaur: let's drop gear debug logs for now21:13
*** thorst_ has quit IRC21:13
*** baoli has joined #openstack-infra21:13
bswartzbtw I never knew that tags get their own commit hash21:14
* bswartz learned something new about git21:14
*** maishsk has quit IRC21:15
clarkbmriedem: jbernard_ http://download.ceph.com/debian-hammer/ is what we want mirrored? its a bit annoying that its a mirror per ceph release x distro release21:15
*** maishsk_ has joined #openstack-infra21:15
*** ldnunes has quit IRC21:15
clarkbbut I guess that allows ceph to get stuff out to users in a very controlled fashion21:15
*** thorst_ has joined #openstack-infra21:15
*** jtomasek has quit IRC21:16
lifelessAJaeger: are you waiting on me for anything?21:17
*** Guest93664 is now known as med_21:17
*** med_ has joined #openstack-infra21:17
*** jtomasek has joined #openstack-infra21:18
AJaegerlifeless: no, I'm not. Note that we discussed finally constraints in the infra meeting...21:18
jbernard_clarkb: hammer should be fine for now21:19
AJaegerlifeless, http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-03-15-19.03.html21:19
*** thorst_ has quit IRC21:20
*** oanson has quit IRC21:20
*** esker has quit IRC21:20
*** esker has joined #openstack-infra21:20
*** ddieterly[away] is now known as ddieterly21:20
lifelessAJaeger: cool - Nakato ^21:21
*** IlyaG has quit IRC21:21
AJaegerNakato, lifeless : Hope we get this done early in Newton...21:21
lifelessAJaeger: bwahahahahahah21:22
*** yamahata has joined #openstack-infra21:22
AJaegerthat's a nice ending for the day ;)21:22
* AJaeger waves good-bye21:23
clarkbfungi: given http://docs.ceph.com/docs/hammer/install/get-packages/#apt how do I take that release.asc and get a fingerprint I can pass to the apt::key puppet module?21:23
*** keedya has quit IRC21:23
clarkbits possible that apt::key will take that file directly too I can check that21:23
*** sdake_ is now known as sdake21:23
clarkbya looks like that is an option, I suppose we can bikeshed over how silly that is21:25
fungiright, in fact i think we should be using full fingerprints. seems like i already suggested that in another change somewhere?21:25
clarkboh wait reprepro uses a different module21:25
clarkboh right becuase root not apt need to trust the key21:25
fungiusing too-short key ids is unsafe, and since we don't need to type this over and over we can just put in a fingerprint and call it a day21:26
*** esker has quit IRC21:26
*** amitgandhinz has quit IRC21:26
clarkbthat asc is not the fingerprint though its the actual key?21:26
fungioh, indeed i didn't look at the file--sorry21:27
*** baoli has quit IRC21:27
fungigpg can render that to a fingerprint for you. so can openssl rsa21:27
*** oanson has joined #openstack-infra21:27
*** baoli has joined #openstack-infra21:27
clarkbgpg --fingerprint $path fails though21:27
*** trown|ooobootcam has quit IRC21:27
*** jtomasek has quit IRC21:28
clarkblooks like I can shove that in public hiera and read it in as a whole though21:29
fungiit may need to be imported into a keyring if you want to use --fingerprint. i'm looking through the manpage for a workflow which doesn't imply an import step21:29
Clintfungi: --no-default-keyring --keyring whateverkey.file --fingerprint or so?21:30
clarkbheh key_id is required21:30
clarkbeven though I have the entire key here for consumption21:31
fungiclarkb: if this is the puppet apt module, it at least treats the fingerprint as an id21:31
openstackgerritMerged openstack-infra/tripleo-ci: Archive all of the delorean logs  https://review.openstack.org/27141621:31
fungireally the id is just a truncated fingerprint21:31
clarkbfungi: no its https://github.com/n1tr0g/golja-gnupg because we are updating roots keyring not the apt keyring21:31
clarkband the fingerprint apepars to work, its just funny that if I provide the entire key contents it still wants a fingerprint/id21:32
clarkbnow to figure out what the fingerprint is21:32
*** baoli has quit IRC21:32
fungiopenssl rsa -in release.asc -pubout -outform DER | openssl md5 -c21:33
fungithat gives me d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e21:33
*** thorst_ has joined #openstack-infra21:33
clarkbd4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e is what I get21:33
clarkbok they match21:33
clarkbnow to see if thats on a public keyserver I can just fetch from or if I need to vendor the key21:34
*** thorst_ has quit IRC21:34
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks  https://review.openstack.org/29139321:34
*** thorst_ has joined #openstack-infra21:34
*** asettle has joined #openstack-infra21:34
fungii'm still working on the equivalent gpg incantation based on Clint's suggestions21:35
*** amrith has quit IRC21:35
*** amrith has joined #openstack-infra21:35
Clintfungi: --with-colons might make that easier21:36
*** eharney has joined #openstack-infra21:36
*** pcaruana has quit IRC21:38
clarkbthat key does not apear to be known to mit21:38
clarkbso I think I have to vendor it21:38
fungiClint: it seems to want to create the keyfile at the path i give it rather than using that21:38
*** keedya has joined #openstack-infra21:38
*** jpr has joined #openstack-infra21:38
*** pradk has quit IRC21:38
fungiclarkb: oh, is this the insane puppet module which insists that you give it a key id so it can retrieve the key from a keyserver network before importing it? we should not use that thing21:39
*** sc68cal has quit IRC21:39
fungithere os zero benefit i can think of to forcing puppet to go download a key when you can provide a key21:39
clarkbfungi: I don't know, it just says key_id is required even when providing the key string directly21:39
clarkbfungi: I am just reusing what mordred has already done and refactoring it to be useable for more than one mirror21:40
fungiyes, i audited this when someone was suggesting using it in our puppet somewhere and said we should look for something that doesn't blindly depend on a keyserver network even though it gets no benefit from using one21:40
*** e0ne has quit IRC21:40
fungiit's as if whoever wrote that module either had an entirely different (and odd) use case in mind, or was blindly copying magic instructions from the internet on how to use openpgp keys21:41
*** ZZelle has joined #openstack-infra21:41
lifelessfungi: ... or both?21:41
fungilifeless: inclusive or!21:41
lifelessfungi: english!21:42
* fungi revises the oed to include xor vs or disambiguation21:42
jeblairfungi, lifeless: thank you!  i wonder why no one thought to do that before!21:43
fungiif people can get "blog" added to the oed, surely we can get "xor"21:43
*** baoli has joined #openstack-infra21:43
lifelesssadly, oed is subscription21:44
lifelessso much for information wants to be free21:44
* anteaya goes for a walk21:44
*** maishsk_ has quit IRC21:45
*** baoli has quit IRC21:45
*** baoli has joined #openstack-infra21:46
*** matrohon has quit IRC21:47
*** adrian_otto has quit IRC21:49
*** vgridnev has quit IRC21:49
*** fawadkhaliq has quit IRC21:51
*** fawadkhaliq has joined #openstack-infra21:51
*** oanson has quit IRC21:52
*** pcaruana has joined #openstack-infra21:52
*** mriedem has quit IRC21:55
*** abregman|afk has quit IRC21:56
openstackgerritClark Boylan proposed openstack-infra/system-config: Make reprepro useable on multiple mirrors  https://review.openstack.org/29315421:58
openstackgerritClark Boylan proposed openstack-infra/system-config: Apt mirror for ceph hammer release  https://review.openstack.org/29315521:58
clarkbsomeone that understands apt and reprepro better than me should review that21:58
*** baoli has quit IRC21:59
*** pvaneck has quit IRC21:59
*** ybathia has quit IRC22:00
*** xyang1 has quit IRC22:00
*** yamahata has quit IRC22:00
*** fawadkhaliq has quit IRC22:00
*** woodster_ has joined #openstack-infra22:01
*** fawadkhaliq has joined #openstack-infra22:01
*** harlowja has joined #openstack-infra22:01
*** MarkAtwood has joined #openstack-infra22:01
fungilifeless: yeah, i meant wordnet!22:01
fungithough then the suggestion loses a lot of its punch22:01
*** baoli has joined #openstack-infra22:02
craigeheya piet. Pholio is certainly there. PLease log in and test it :-)22:04
*** tpsilva has quit IRC22:07
lifelessfungi: :)22:07
*** sdake has quit IRC22:09
*** baoli has quit IRC22:10
*** rockyg has quit IRC22:10
*** baoli has joined #openstack-infra22:10
*** dims_ has joined #openstack-infra22:11
*** sigmavirus24 is now known as sigmavirus24_awa22:12
*** flwang1 has quit IRC22:12
*** flwang1 has joined #openstack-infra22:12
*** jpr has quit IRC22:13
*** david-lyle has quit IRC22:13
*** dims has quit IRC22:14
*** gordc has quit IRC22:15
*** piet has quit IRC22:16
*** dims has joined #openstack-infra22:16
*** dims_ has quit IRC22:17
*** claudiub has quit IRC22:17
*** dimtruck is now known as zz_dimtruck22:18
dhellmannjeblair, fungi, clarkb : can one of you bless this ACL change when you have a few minutes? I'd like to go ahead and start tagging and branching tomorrow, but want the safeguards in place first. https://review.openstack.org/#/c/292781/22:19
*** _david_ has joined #openstack-infra22:19
*** gmurphy has left #openstack-infra22:19
clarkbdhellmann: would it be possible to let release managers just have those perms always and not use them?22:21
clarkbthen we don't have to do this every single release?22:22
flwang1AJaeger: excuse me, how can i create a new people-group on gerrit for zaqar-ui which is sharing the same core team of zaqar? thanks22:22
thiagopcalling it a day, thanks for your help fungi jeblair asselin ianw pabelanger22:22
*** keedya has quit IRC22:22
clarkbflwang1: when you udpate the acl files and put a group in them that does not already exist our tooling will create that group22:23
*** thiagop has quit IRC22:23
ianwpabelanger: so, we got distracted ... we know the builds won't work, but we didn't establish why nodepool didn't even try?22:23
clarkbflwang1: however if the intent is to make the groups the same just reuse the old group22:23
fungiclarkb: in the past it was that we were granting exclusive permissions to release management until the release, and then granting permission to stable management or core teams22:23
clarkbfungi: in this change the perms appear shared22:23
flwang1clarkb: not same, that's why i asked, since i want to add some horizon guys into the team, thanks22:23
clarkbboth $project-release and Release managers get +2 and approve22:24
fungiclarkb: in that case it probably missed the exclusiveGroupPermissions line or forgot the purpose of the change to the steady march of time22:24
flwang1clarkb: will the tool add me as the 'ptl' so that i can add more people?22:24
clarkbflwang1: not automatically, once the group is created you will need to request that a gerrit admin add the first user who can then add others22:24
flwang1who is the gerrit admin i should bug?22:25
clarkbflwang1: they answer to infra-root in this channel, you can also just send mail for the request on the infra mailing list22:25
flwang1clarkb: got, thanks a lot22:25
docaedohowdy - any infra-root folk have a minute to look at something on the app-catalog server for me?22:28
jeblairdocaedo: yes22:28
docaedosweet! There's this script (http://git.openstack.org/cgit/openstack/app-catalog/tree/tools/asset_history.sh) that figures out when any asset was last modified based on git blame22:28
*** ddieterly has quit IRC22:28
*** tonytan4ever has quit IRC22:28
docaedoon production, that script does not seem to find the dates. To confirm, I would like if someone could look for the file:22:29
*** piet has joined #openstack-infra22:29
docaedoand confirm that assets_merge.yaml has lots of asset names with no deats22:29
jeblairdocaedo: where in the filesystem should i find that tree?22:30
docaedoum .. where the repo is checked out, give me a sec and I can find the right path22:31
jeblairwell, it's checked out in /opt22:31
jeblairbut we usually don't write things to checked out repos22:31
jeblairbecause that can make git sad in an automated environment22:31
*** gnuoy_ has joined #openstack-infra22:32
SamYaplegit reset --hard HEAD  # burn it all!22:32
jeblairbut apparently we do on the app server!22:32
docaedonormally anything that changes in there from the scripts also cleans up after itself, but I merged a change that would leave that assets_merge.yaml file sitting around22:32
*** gnuoy_ has quit IRC22:32
jeblairdocaedo: a todo item might be to leave /opt/apps_site as a pristine git repo, and then install/copy that to some other location22:32
*** asettle has quit IRC22:33
docaedojeblair: agreed :)22:33
jeblairdocaedo: ls /opt/apps_site/openstack_catalog/web/static: assets_dead.yaml  assets.schema.yaml  assets.yaml  blank.json  CACHE  css  images  js  lib22:33
*** MarkAtwood has quit IRC22:34
openstackgerritIan Wienand proposed openstack-infra/project-config: Add Centos7 wheel cache jobs  https://review.openstack.org/29027222:34
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add a core group for zaqar-ui  https://review.openstack.org/29316622:35
*** pvaneck has joined #openstack-infra22:35
*** abregman|afk has joined #openstack-infra22:35
*** asettle has joined #openstack-infra22:35
docaedojeblair: ok thanks - that's a little surprising but helps me at least think of some other things to change around22:35
jeblairdocaedo: what runs that script?22:36
* docaedo find the PDX wifi is not so useful today...22:37
bkerodocaedo: have they fixed their captive portal cert yet?22:38
*** _david_ has quit IRC22:40
pabelangerianw: no, we don't. We can confirm with clarkb why images didn't seem to queue for building today22:42
SamYaplepabelanger: did my fstrim patch merge?22:43
SamYaplei seem to be breaking infra lately, you should check22:43
pabelangerSamYaple: I am not sure. ianw would know more22:43
docaedobkero: Maybe? I did not get a warning, but at the same time I seem to be hopping between at least four different APs, so I get connectivity for about a minute, then nothing for another minute... so fun!22:43
pabelangerassuming it is for diskimage-builder22:43
clarkbI don't think dib has released since then22:43
SamYaplepabelanger: sorry i read your statement wrong. i thought you meant you couldnt build images22:44
pabelangeror nodepool-builder hasn't been restarted too22:44
*** baoli has quit IRC22:44
pabelangerSamYaple: last I tested we had no issues22:44
bkerodocaedo: yuck :(22:44
clarkbpabelanger: ianw if I had to guess it is because centos-7 is in a busy loop failing to build?22:44
*** edmondsw has quit IRC22:44
pabelangerclarkb: Ya, that is what I think too22:45
clarkbto fix that I think we just need to restart the image build so that it will reread its config22:45
pabelangerif safe, we can try nodepool-builder restart22:45
pabelangerclarkb: good with me doing that then?22:46
clarkbpabelanger: just make sure there aren't any in progress uploads, nodepool image-list | grep building22:46
pabelangerclarkb: sure, 1 sec22:46
docaedojeblair: just realized no new assets have been merged since the change to keep assets_merge.yaml merged, so that should not have surprised me. But the glamor of air travel calls so I'll try this again tomorrow - thanks for your help!22:46
pabelangerclarkb: Ya, currently centos-722:47
pabelangerwhich is expected22:47
jeblairdocaedo: bon voyage22:47
clarkbif that is the only one then ya restart the builder22:47
pabelangerokay, restarted22:48
*** weshay has quit IRC22:48
pabelangerclarkb: ianw: centos-7 building22:49
*** hashar has quit IRC22:49
pabelangerianw: also, piping to image.log only22:49
dhellmannclarkb : we could, but the point isn't to add permissions, it's to make it so the core team doesn't have them22:53
clarkbdhellmann: right but we don't need to add and remove perms to do that22:53
pabelangerianw: going to fail because of stale git repos22:53
dhellmannclarkb : how would we do that?22:54
clarkbdhellmann: the existing stable branch acls already exclude the core group22:54
pabelangerfungi: is there anything I can do to fix the project import for gerrit?22:54
clarkbdhellmann: just add your group to that branch prefix and don't bother removing it22:54
fungiclarkb: they want to also exclude the stable branch reviewers though22:54
pabelangerfungi: ianw Oh, maybe not22:54
pabelangerseems to pass22:54
clarkbfungi: but this change does not do that22:55
clarkbat least it doesn't seem to22:55
fungiright, i think the change is probably wrong but i haven't reviewed it yet22:55
clarkbat least for nova22:55
dhellmannclarkb : doesn't it?22:55
fungistill fiddling with election bits22:55
dhellmannclarkb : these also exclude the stable-maint groups, don't they?22:56
clarkboh I see you are using release/milestone there22:56
*** piet has quit IRC22:56
pabelangerhttps://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ is the failure for dib22:57
pabelangerclarkb: fungi: so, is that fixed my manually rerunning manage_project.py on gerrit?22:58
clarkbpabelanger: uh I think we needed to figure out why puppet waasn't running22:59
clarkbthen get it to run22:59
clarkbdhellmann: fungi my concern with this is its multile changes per release that have to flow through infra which seems far from ideal. So trying to determine if there is a way we can just set the perms to be what we want and forget about them22:59
fungiyeah, skipping the hard problem to just update the repos isn't a great workaround. we'll just be back to broken as soon as someone merges another new project creation change which is about twice a day on average lately seems like22:59
fungipabelanger: ^23:00
clarkbdhellmann: fungi one option would be to use groups that change membership23:00
anteayaclarkb: I think yolanda has said earlier that she was hesitant to run puppet on gerrit since we have two gerrit nodes right now23:00
*** rbrndt has quit IRC23:00
clarkbanteaya: that shouldn't be an issue especially if ansible can only ssh to one of them23:00
anteayaclarkb: and she is uncertain what will happen to the gerrits if puppet runs23:00
dhellmannclarkb, fungi : I agree. I would love to do make this simpler. Can we do that next cycle where we have time to think through the solution?23:00
clarkbdhellmann: sure23:00
anteayaclarkb: I don't know the status of what ansible can ssh into23:00
anteayaclarkb: I just remember her saying that comment while we were in the meeting23:01
clarkbanteaya: neither do I, depends on how yolanda configured the new machine23:01
dhellmannclarkb : groups sounds like a good answer, actually, since that's self-serve for the release team and may even be something we could automate23:01
anteayaclarkb: so that is why puppet is not running, I do believe23:01
fungiclarkb: dhellmann: yes, a gerrit admin manually changing numerous group memberships twice was discussed as the alternative to a project-config core reviewing and approving changes twice23:01
fungiclarkb: dhellmann: but if we put all those groups under the control of the release team instead, then i guess it's up to them to expend that effort23:02
dhellmannif it's scriptable, it's fine with me23:02
dhellmannthe question is, how would we do that? create stable-$series groups? or lock all stable branches at the same time?23:02
dhellmannclarkb , fungi : added as a todo in https://etherpad.openstack.org/p/newton-relmgt-plan, maybe we can discuss at the summit23:03
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq  https://review.openstack.org/29307423:04
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test  https://review.openstack.org/28944523:04
*** yamamoto_ has quit IRC23:05
ianwpabelanger: 2016-03-15 22:53:29,483 INFO nodepool.image.build.centos-7: fatal: Couldn't find remote ref master <- it fails on git, as expected23:05
pabelangerfungi: clarkb: okay, so I assume no body stopped it. I can look into the logs and find out why23:06
*** dims has quit IRC23:06
clarkbpabelanger: yolanda dd according to anteaya23:06
pabelangerclarkb: anteaya Ya, I would expect 2 review.o.o to be fine. Especially since ansible would only SSH into 123:07
pabelangerso, guess we hold off until yolanda in the morning. Or look to reenable puppet ourself23:08
clarkbso review.o.o is not in the emergency file23:10
clarkbmaybe its in the regular thing but I need to loo kthat up23:10
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472623:10
*** bryan_att has quit IRC23:12
clarkbansible is definitely not running on review.o.o if I am reading /var/log/puppet_run_all.log correctly23:13
clarkbso its not going and then failing its just not going23:14
*** dingyichen has joined #openstack-infra23:14
pabelangerya, I haven't looked at puppet logs yet23:14
clarkbreview.o.o is in /etc/ansible/hostfile23:14
clarkband ansible --list-hosts isn't a vlaid thing despite what our docs say23:15
fungiclarkb: the concern is that maybe adding the uuid of the duplicate review.o.o, or perhaps the mere existence of a second review.o.o causing them to both be known only by uuid in the inventory, is causing ansible to skip or break applying to them23:16
clarkbyou apparently have to provide a hostslist ? isn't that redudnant?23:16
clarkbfungi: what implies there is a bug in our ansible stuff23:16
clarkbwe should be able to ansible one, both, or neither23:16
jeblairthere are strange log entries in syslog with out of order dates23:17
pabelangerI would expect that too23:17
jeblair(on review.o.o)23:17
clarkb2016-03-15 23:03:27,321 p=5429 u=root |  87741fa9-92cb-4d1e-8195-6841e2468fb2 : ok=13   changed=6    unreachable=0    failed=023:18
clarkbthats running on one of the two review.o.os23:18
*** roxanagh_ has joined #openstack-infra23:18
fungiclarkb: right, that's what i'm saying: maybe there's a bug. we haven't done much with duplicate hosts in the inventory up to this point so it's possible we didn't notice before now that it doesn't work as expected23:18
pabelangerhttp://puppetdb.openstack.org/node/review.openstack.org seems to confirm that23:18
clarkbnow to figure out which of the two that is23:19
fungialso it's possible yolanda's assertion that puppet isn't running on review.o.o is based on puppetboard23:19
pabelangeris that puppetdb is reporting23:19
funginote that the reports get submitted with whatever identity the inventory has for review.o.o23:19
jeblairit's definitely not running on the prod server23:19
clarkbthat uuid belongs to the current prod server23:19
jeblairno entries in syslog since the 10th23:19
clarkbaccording to nova show23:19
EmilienMcrinkle: what are the plans in opnestack-infra to move to puppet4? I'm wondering if Puppet OpenStack CI should do the move in our scenario jobs or not23:20
clarkbcrazy theory time23:20
* fungi pulls up a chair23:20
clarkbmaybe its puppeting as something other than review.o.o because that uuid doesn't match the site.pp entry?23:20
clarkbeg its just applying the server manifest? though that should break the firewall so maybe not23:21
jeblairclarkb: it should also leave log entries23:21
fungiwouldn't puppet apply still be getting called and log?23:21
clarkbjeblair: good point23:21
fungiyeah, tat23:21
crinkleEmilienM: we haven't made a solid plan yet, but we've inched toward it by adding puppet4 linters23:21
jeblairand login entries that show up in last23:21
pabelangerEmilienM: you could do testing with fedora-2323:21
clarkbso that uuid isn't in dns at all23:21
clarkbits possible we could be talking to the wrong ip23:21
crinklepabelanger: well we could also easily do testing with ubuntu-trusty/centos7 using the puppetlabs repos23:22
EmilienMcrinkle: I know _ody was about to patch openstack/puppet-openstack-integration to deploy puppet4 instead of 3, but I'm wondering if we need to keep functional testing for 3 or syntax/lint would be enough.23:22
clarkb104.130.159.134 2001:4800:7818:102:be76:4eff:fe05:9b12 is what it shoud talk to, time to work backwards and see if it is doing that23:22
*** roxanagh_ has quit IRC23:22
fungiwe do have that uuid in puppetboard, but unreported http://puppetboard.openstack.org/node/87741fa9-92cb-4d1e-8195-6841e2468fb223:22
EmilienMpabelanger: do you mmean puppet4 is in fedora23 ? our jobs are running centos7 and until now, we use puppetlabs packaging. What do you suggest?23:23
fungiipaddress6 2001:4800:7819:103:be76:4eff:fe05:852523:23
clarkbthats what ansible cache says23:23
fungiclaims facts were updated 30 minutes ago23:23
clarkbbut nova show disagrees?23:23
jeblairfungi: those are the addresses of the new server23:23
clarkbya I think this is the bug23:24
crinkleEmilienM: i would vote for functional testing with 3 at least for a while longer while it's still fully supported23:24
clarkbansible inventory is mixing them up and we are puppeting the new server while reporting the uuid of the old server23:24
fungieither that or the way we thought configuring it to skip a host by uuid does the opposite of what we expected23:24
EmilienMcrinkle: ok. Then we could create experimental jobs in openstack-infra that would deploy puppet4 at demand23:24
clarkbfungi: I don't think we are skipping but let me double check on that23:24
fungioh, though it's definitely confusing the uuid of one with the addresses of the other23:24
crinkleEmilienM: ++23:25
fungiso, yeah, switcheroo. i don't see how incorrect config expectations could account for this at all23:25
clarkbya that uuid doesn't show up in the disabled stuff anywhere so I think it is just mixing them up23:25
*** nwkarsten has quit IRC23:25
pabelangerEmilienM: right, we default to puppet 4 on fedora, since it is native puppet package23:27
EmilienMpabelanger: awesome.23:27
clarkbI think I found the bug23:27
clarkbpatch in a couple23:28
fungiclarkb: shade or in our inventory tooling?23:28
*** chlong has joined #openstack-infra23:28
clarkbour inventory tooling23:28
*** fverbs has joined #openstack-infra23:28
clarkbits a bad list traversal23:28
fungioh, that's faster to fix at least23:28
clarkbI just have to find out where this file lives in config :)23:28
fungii know i found it not long ago23:29
clarkbis the file I need to edit23:29
fungino, that's part of ansible i think23:30
fungimordred: ^23:30
fungii believe that's an ansible-openstack component23:30
clarkbfound it23:31
clarkband yes it is in ansible23:31
* clarkb puts on the hazard suit and figures out github things23:31
*** adrian_otto has joined #openstack-infra23:32
clarkbhttps://github.com/ansible/ansible/blob/devel/contrib/inventory/openstack.py#L152 is the bug fwiw23:32
clarkbI will work on a PR23:32
*** IlyaG has joined #openstack-infra23:37
clarkbmy ansible clone is incredibly slow :/23:37
*** abregman|afk has quit IRC23:39
*** Qiming has joined #openstack-infra23:39
*** ihrachys has quit IRC23:45
*** sdake has joined #openstack-infra23:45
*** kzaitsev_mb has quit IRC23:47
clarkbmordred: ^ for when you have a moment23:47
*** harlowja has quit IRC23:47
*** eil397 has quit IRC23:48
*** salv-orlando has quit IRC23:49
anteayaclarkb: nice one23:50
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472623:50
*** yamamoto_ has joined #openstack-infra23:51
*** bpokorny has quit IRC23:53
clarkbI will admit to not know how to test that easily23:53
clarkbwe can just give it a go and see what happens :)23:53
*** tqtran is now known as tqtran-afk23:53
anteayaha ha ha23:53
anteayathat is a way23:53
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test  https://review.openstack.org/28944523:58
*** dalgaaf has quit IRC23:58
*** sdake_ has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!