Tuesday, 2014-01-14

*** harlowja has joined #openstack-infra00:00
zarofungi: odd, why do i need to do that?  i see the gerrit stuff right after i clone jjb.00:00
*** melwitt has joined #openstack-infra00:00
*** Loquacity has joined #openstack-infra00:00
*** loq_mac has quit IRC00:00
zarofungi: yeah, that fixed it.  but i don't know why i have to do that.00:00
*** dolphm has joined #openstack-infra00:01
fungizaro: git won't have a remote named "gerrit" until the first time git-review runs. when you push a tag you're not running git-review, so if you've *never* run git-review in that copy of the repository you'd be missing the remote00:01
arosenfungi:  hrm passlib  is in keystone's requirements.txt so it should get installed but i don't see that happening here:00:01
jamielennoxfungi/arosen: that doesn't look like something that would come from keystoneclient tagging00:02
*** sarob has quit IRC00:02
dolphmarosen: maybe it's getting a version of passlib without passlib.hash?00:02
zarofungi: thanks.00:03
dolphmpython-keystoneclient requirements.txt diff from 0.4.2 to 0.4.1 http://pasteraw.com/rnxcju8hjq049m3y870uda5pbdcjl100:04
jamielennoxpasslib should have nothing to do with keystoneclient00:04
dolphmi was about to say -- where's passlib? lol00:04
fungiarosen: jamielennox: dolphm: yeah, i'm stumped, but honestly haven't had time to look closely at the log00:04
*** yamahata has joined #openstack-infra00:06
zaromgagne: would the fix be to be more open to the types of acceptable parameters?00:07
zaromgagne: do you want to propose something?00:07
mgagnezaro: not sure how to handle it as I'm not sure how we are supposed to handle deprecation00:07
zarois that jenkins deprecsted or only jjb deprecated?00:08
*** sarob has joined #openstack-infra00:09
mgagnezaro: not sure if the plugin got updated and now accepts additional parameters as implemented by the committer or not. I would have to check. I'm probably running an old version of the pipeline plugin00:09
* zaro takes a look on latest ver00:10
mgagnezaro: I'm running 1.3.3. 1.4.2 looks to be the latest.00:11
*** sarob has quit IRC00:13
*** sarob has joined #openstack-infra00:13
*** sandywalsh has quit IRC00:15
*** bknudson has joined #openstack-infra00:16
openstackgerritSabari Murugesan proposed a change to openstack/requirements: nova api validation fw requires jsonschema >= 2.0.0  https://review.openstack.org/6646400:17
zaromgagne: 1.4.2 allows additional params: boolean, current params, params from properties file, and predefined params00:18
*** sarob has quit IRC00:18
*** thuc has joined #openstack-infra00:19
*** thuc has quit IRC00:19
*** thuc has joined #openstack-infra00:20
*** thuc has quit IRC00:20
*** hogepodge has quit IRC00:20
*** salv-orlando has joined #openstack-infra00:21
*** thuc has joined #openstack-infra00:21
*** thuc has quit IRC00:22
*** mozawa has joined #openstack-infra00:22
*** thuc has joined #openstack-infra00:22
*** tian has quit IRC00:23
*** salv-orlando has quit IRC00:26
*** gyee_ has quit IRC00:28
*** boris-42 has quit IRC00:32
*** praneshp has joined #openstack-infra00:33
*** sdake has quit IRC00:34
jerryzfungi: https://bugs.launchpad.net/openstack-ci/+bug/1266603  please add the initial reviewers if you have time. thanks00:36
uvirtbotLaunchpad bug 1266603 in openstack-ci "Please add initial core reviewers for stackforge/compass project" [Undecided,New]00:36
*** erfanian has joined #openstack-infra00:40
*** yamahata has quit IRC00:44
*** yamahata has joined #openstack-infra00:44
*** dkranz has joined #openstack-infra00:49
zarofungi: build failed on tag.. https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/00:49
zarofungi: I can't repro on my machine and it hasn't failed before.  would you be able to re-run?00:50
fungizaro: weird. i can retrigger it and see if it continues to happen00:51
*** CaptTofu has quit IRC00:51
*** sandywalsh has joined #openstack-infra00:51
*** dcramer_ has joined #openstack-infra00:53
fungizaro: https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/5/console00:54
fungi"ERROR: No artifacts found that match the file pattern "target/gearman-plugin-${PROJECT_VER}.hpi". Configuration error?"00:55
zarofungi: actually it's a test that fails which causes no artifact to be created.00:56
fungioh, also i see it says "cp ./target/gearman-plugin.hpi ./target/gearman-plugin-.hpi" so it's missing a version value00:56
fungigot it00:56
fungizaro: it ran on precise5 both times. do you think there's any benefit to clearing the workspace for the job on that slave and trying again?00:57
zarofungi: is it possible to run on another machine?00:57
fungisure, i can mark that slave temporarily offline and retrigger it again00:57
zarofungi: we could try but chances are low that will fix it.00:58
*** nati_uen_ has quit IRC00:58
zarofungi: more likely a timing issue is my guess.00:58
fungizaro: running on precise25 this time... https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/6/console00:59
fungiand i've brought precise5 back online now00:59
*** wenlock has quit IRC00:59
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: refactor bot to be based on argparse  https://review.openstack.org/6640101:00
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: moving readiness checks into stream  https://review.openstack.org/6596101:00
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: fix bot to actually use super()  https://review.openstack.org/6640501:00
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot  https://review.openstack.org/6643501:00
*** melwitt has quit IRC01:01
*** ryanpetrello has joined #openstack-infra01:02
*** nati_ueno has joined #openstack-infra01:02
*** nati_ueno has quit IRC01:02
*** nati_ueno has joined #openstack-infra01:04
zarofungi: it passed on precise2501:04
fungizaro: all set for me to trigger gearman-plugin-jenkinsci-upload for tat tag then?01:05
zarofungi: that would be great.01:05
*** nati_uen_ has joined #openstack-infra01:05
zarofungi: what should i do in this case?  if anything.01:06
fungizaro: success https://jenkins.openstack.org/job/gearman-plugin-jenkinsci-upload/4/console01:06
fungizaro: well, it would be great if you had a theory, from the logs, as to why it failed consistently on precise5 (2 out of 2 times, for whatever that sample size is worth)01:07
*** melwitt has joined #openstack-infra01:07
zarofungi: i could formulate a theory, but not much more.  i can't repo so it would be difficult to prove and fix.01:08
*** nati_ueno has quit IRC01:08
fungizaro: mainly wondering whether i should be suspecting the workspace, or the installed packages/files on the server01:09
*** rnirmal has joined #openstack-infra01:10
*** senk has quit IRC01:12
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot  https://review.openstack.org/6643501:12
*** ^d has quit IRC01:13
*** nosnos has joined #openstack-infra01:13
zarofungi: my theory is that precise5 is maybe signficantly faster or slower than the other machines.  i think it's timing related rather than bad file or something.01:13
*** mrda has quit IRC01:14
clarkbsdague fungi it is easy to determine a lag value for es with public data01:14
fungioh, too weird01:14
clarkbsdague fungi every event has a received at field take the difference of that and the timestamp value01:15
clarkbnote that since files are batch processed you only want that delta for the end of files01:15
zarofungi: is there a history of gearman-plugin builds on precise5?01:15
*** zhiwei has joined #openstack-infra01:15
zaroclarkb: _david_ fixed the javamelody security flaw.01:16
*** senk has joined #openstack-infra01:16
zaroclarkb: new build is up on review-dev.o.o01:16
clarkbfungi you can also connect to localhost:4730 on logstash.o.o via tcp and run the 'status' command to see the queue depth which we should get into graphite soon01:16
fungiclarkb: oh, good point01:16
clarkbzaro: cool01:16
*** rnirmal_ has joined #openstack-infra01:16
fungiclarkb: and i probably knew about the 4730/tcp interface but should have taken better notes to remind myself01:17
fungiclarkb: i've turned up four worker processes each on 05 and 06 now, since the first four servers were spending a lot of time with cpu pegged01:17
*** senk has quit IRC01:17
*** rnirmal has quit IRC01:17
*** rnirmal_ is now known as rnirmal01:17
*** vkozhukalov has joined #openstack-infra01:18
clarkbfungi awesome make sure you restart logstash-indexer when you do that01:18
clarkbotherwise y2k01:18
fungiahh, good to know. also adding to my notes01:18
fungiclarkb: which host is logstash-indexer on?01:20
clarkbfungi its on the individual worker hosts01:20
clarkbthe 4 gearman processes talk to a single local logstash-indexer01:21
*** jerryz has left #openstack-infra01:21
*** jerryz has joined #openstack-infra01:21
fungii'll keep hunting for the initscript then01:21
clarkbits an upstart job >_>01:21
*** melwitt has quit IRC01:21
*** harlowja has quit IRC01:22
fungithanks! restarted01:22
*** nati_uen_ has quit IRC01:22
*** marcoEmbrane has left #openstack-infra01:22
*** nati_ueno has joined #openstack-infra01:23
clarkbnp and thank you01:24
*** sarob has joined #openstack-infra01:24
*** sarob_ has joined #openstack-infra01:24
fungipush-log        16      16      2401:25
fungii'm sure that means something01:26
* fungi reads01:26
*** sarob_ has quit IRC01:26
clarkbfirst number is queue length second is active workers last is total workers01:26
clarkbso that looks good01:26
*** sarob_ has joined #openstack-infra01:26
fungithat makes sense, and those numbers look right01:27
*** mrda has joined #openstack-infra01:27
fungiat least the total workers number looks correct01:27
fungiwith the addition of the 8 more earlier01:27
fungizaro: we don't keep much build history on our jenkins masters, so all you get is https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/buildTimeTrend unless you know somewhere else i should look01:28
*** sarob has quit IRC01:28
*** praneshp has quit IRC01:29
fungiclarkb: i tried a couple more new projects today and got the same symptoms, after confirming that the globally-installed jeepyb is new enough to include the hoped fix, so back to the drawing board01:29
clarkbfungi: for adding projects?01:30
*** kraman has quit IRC01:30
fungiclarkb: one of them used an existing acl, so no new groups were needed01:30
fungiclarkb: when puppet ran it, never even got as far as cloning into /var/lib/jeepyb01:30
*** sarob_ has quit IRC01:30
fungiclarkb: when i reran it myself, worked right off the bat01:31
clarkbmaybe fixing each consequitive problem exposes a new bug?01:31
fungisame for one which was adding two new groups. failed under puppet, worked when i reran01:31
fungiit's possible01:31
fungithe one which had new groups to add never got as far as creating either group the first time around01:32
fungialso, i'm trying to beat pypi-mirror into submission for the requirements integration runs, but am running into some weird behavior with how it's populating the local mirror in /var/www on the slave01:33
*** harlowja has joined #openstack-infra01:33
zarofungi: since that failing test has been there for a long time, i was hoping to see if any gearman-plugin builds have passed on precise5.  that history didn't help and i don't know where else to look.01:34
fungizaro: yeah, sorry. me either01:34
clarkbit is tempting to rewrite from scratch ... throw away the cruft and start again with what we have learned01:34
*** gokrokve has quit IRC01:35
*** rakhmerov has quit IRC01:35
fungion the pypi-mirror integration runs, it hit an issue where a project wanted oslo.messaging, which doesn't have a release on public pypi yet. that should still be in the local mirror it created from all the git checkouts, but the mirror is only getting populated with release versions from pypi01:36
*** gokrokve has joined #openstack-infra01:36
fungiit's strange01:36
*** pballand has quit IRC01:36
fungiit's building wheels from the same versions too01:36
*** gokrokve_ has joined #openstack-infra01:37
fungiand md5sums of the tarballs match what's hosted on pypi, so it's downloading those, not building them from scratch01:38
*** praneshp has joined #openstack-infra01:38
*** yaguang has joined #openstack-infra01:39
fungithat may be the intended behavior, i have to read back through, but if so it's not working with the -f link to prerelease oslo.messaging on tarballs.o.o01:39
clarkbis the dev requirements list being used properly?01:39
fungiit's possible that's the issue01:40
dstufftAre y'all using -f in a requirements.txt that you're shuttling through a setup.py using pbr?01:40
*** ryanpetrello has quit IRC01:40
*** senk has joined #openstack-infra01:40
*** gokrokve has quit IRC01:41
fungidstufft: until we get in good enough shape to upload prerelease wheels to pypi, i think01:41
dstufftfungi: are you using pip 1.5 or did you rollback to 1.4?01:41
fungidstufft: in this specific case, i think it's using 1.5 but i need to dissect it a bit more to be able to tell01:41
dstufftif I recall pbr shuttled -f <tarball link> through a setup.py using dependency links01:42
dstufftwhich 1.5 disabled by default01:42
dstufft--process-dependency-links will turn them back on01:42
*** gokrokve_ has quit IRC01:42
dstufftif that's the case anyways01:42
fungidstufft: oh, cool--thanks! i'll give that a shot01:42
dstufftor the envvar would be something like PIP_PROCESS_DEPENDENCY_LINKS=101:42
fungidstufft: what version of pip added --process-dependency-links?01:43
dstufftfungi: 1.5 I think, it was a quick dprecation because of the security concern and the fact that most projects just didn't use them01:43
fungidstufft: makes sense. envvar will be a saner solution in this case. thanks again01:44
dstufftpip 1.5, breaking all the things :[01:44
dstufftfungi: np :)01:44
dstufftI think the last thing we have left that pip doesn't have locked down about as good as it can get, is setup_requires, need to figure out how to wrest control of those from setuptools still :/01:45
fungiclarkb: a quick git grep through pypi-mirror says it has no reference at all to dev-requirements.txt01:45
fungiclarkb: which makes sense in the normal case... we don't want to be pulling those into our actual mirror01:46
fungiso i'm beginning to suspect it's as dstufft says, we're probably getting pip 1.5 there, the -f is being ignored in the individual projects' requirements.txt files, and so i may be able to patch up the calling environment in the integration test script to work around that01:48
*** hogepodge has joined #openstack-infra01:48
*** dkliban has joined #openstack-infra01:48
sdaguefungi: honestly, we should shoot dev-requirements in the head01:48
sdagueI really think it's the wrong approach01:48
sdagueif people want prerelease, they go in the integrated gate01:49
fungisdague: well, that's a big worry i have. we've manufactured another "solution" which now breaks until pip 1.5, so even if i work around it in the integration job, it's still broken for the rest of the world01:49
fungier, under pip 1.501:49
sdagueso if deleting the contents of dev requirements fixes things, we should do that01:49
fungisdague: in this case, i can't install glance with pip 1.5 without passing an extra option so that the -f link to oslo.messaging on tarballs.o.o will be respected01:50
sdaguefungi: so there is no real reason for that right01:50
sdagueit's just that they have a totally old requirement01:51
fungisdague: so we're talking about not just deleting dev-requirements.txt, but also all the -f links in the various projects requirements files01:51
sdaguethis is just because glance was lazy and never updated it01:51
sdaguethat line predates havana release01:51
fungiokay, i'll get a patch proposed. how is anyone else using oslo.messaging though if it's not released yet? (and why have we not even reserved that one on pypi?)01:52
sdaguefungi: actually, I'm working ont he glance patch01:53
sdaguefungi: because it's in /opt/stack01:53
fungisdague: okay, awesome. i'll leave you to it01:53
dstufftfungi: for the record I grumped about the use of dependency links in pbr back when Ifirst noticed it ;P01:54
* dstufft will get all the packages to work how he wants them to :D01:54
fungidstufft: i believe it was added to support a use case we had and wanted to divest ourselves of. i don't think it was ever intended to be kept supported in there long-term (though i could be mistaken)01:54
*** fifieldt has joined #openstack-infra01:55
sdaguefungi: though I think you might need to do some ninja merging01:56
sdaguebecause I think that will fail on the requirements test01:56
sdagueoh, wait...01:57
fungisdague: what i'm working on is the requirements integration job which runs against the requirements project changes as well as pypi-mirror and pbr01:57
sdagueyeh, but we won't be able to change a glance requirement01:58
fungichanges to glance should only just get checked that they use sane requirements entries01:58
sdaguebecause it will fail the integration job01:58
sdagueoh... maybe01:58
fungiunless we also need to change the openstack/requirements repository, it shouldn't be an issue01:58
fungisdague: my bigger concern is that i may need to fix this simultaneously in openstack/requirements, openstack-infra/pypi-mirror and possibly also openstack-dev/pbr, depending on what else i hit while testing it01:58
openstackgerritSean Dague proposed a change to openstack/requirements: remove oslo.messaging -f urls  https://review.openstack.org/6647402:00
fungisdague: on that earlier item though, do you know of any reason i should *not* register oslo.messaging on pypi? it is intended to get released there, right? would be a shame for someone else to squat it02:00
openstackgerritSean Dague proposed a change to openstack/requirements: remove oslo.messaging -f urls  https://review.openstack.org/6647402:02
sdaguehmmm... yeh, and oslo.messaging was never released?02:03
sdaguethat seems wrong02:03
fungiseems wrong to me too. no harm in uploading prereleases to pypi until there's an actual initial release02:04
sdagueany library that is olso.* should be release on pypi02:04
fungiit's not like you're going to get a pip 1.3 user accidentally "upgrading to a prerelease from a release version if there never was a release version before02:04
*** ryanpetrello has joined #openstack-infra02:04
sdagueI'm so seriously confused that they had 12 1.2.0a releases, and are now doing 1.3.0a tags02:05
sdagueand nothing to pypi02:05
sdagueso we probably need dhellmann in the midst of all of this02:05
sdaguebecause I incorrectly assumed we actually saw a 1.2.0 at havana02:05
clarkbya no release02:06
fungiwell, i've gone ahead and reserved it on pypi with a 0 version02:06
sdagueand that we were not having core openstack things rely on unreleased libraries02:06
fungiso that we don't have to try to take it over from someone or rename later02:06
fungibut correct, the tags here never had a non-prerelease version http://git.openstack.org/cgit/openstack/oslo.messaging/refs/02:07
fungistarted at 1.2.0a1 and then skipped from 1.2.0a11 to 1.3.0a1 with no 1.2.0 ever existing02:08
*** jhesketh has joined #openstack-infra02:08
sdagueyeh, so I actually don't think the rest of this is solvable until they release02:08
sdaguenot in any sane way02:08
sdagueenough so, that I'd say it's worth an email thread02:09
sdaguethe -f pre release urls never sat well with me anyway02:09
sdagueso I'm glad to see them gone02:09
fungiand yes, your proposed change to glance will simply break for everyone, because there is no release of oslo.messaging02:09
fungiuntil they tag a non-prerelease version02:10
zhiweifungi: hi02:10
fungizhiwei: hi there02:10
sdaguefungi: yep02:10
sdagueit will actually pass requirements and tempest though :)02:11
zhiweiclarkb told me I should contact you about renaming stackforge project issue.02:11
*** nati_uen_ has joined #openstack-infra02:11
fungizhiwei: right, i saw in scrollback, he said he'd be around on the 18th. i'll bring it up in tomorrow's team meeting and see whether there are any objections to doing it then02:11
clarkbfungi I can help with rename on saturday02:11
*** thuc_ has joined #openstack-infra02:12
zhiweiok, thanks clark and fungi.02:12
fungiclarkb: works for me. i'll be here and i can do it. just don't want to risk it if i'm the only one around. appreciated!02:12
sdagueclarkb: so any idea what the average lag time is on the logstashing? because I'm running about 26 minutes and not finding logs for jobs most of the time02:12
*** sdake has joined #openstack-infra02:13
*** sdake has quit IRC02:13
*** sdake has joined #openstack-infra02:13
zhiweibecause I will have my Chinese New Year holiday from next Friday, so just want to get it done before holiday.02:13
*** yamahata has quit IRC02:13
clarkbsdague: I dont have an idea have you oooked at the received at field and compared to timestamp?02:14
clarkbsdague we will be graphiting queue length soon which will give us hard numbers02:14
sdagueclarkb: so we honestly might be loosing things02:15
*** nati_ueno has quit IRC02:15
fungisdague: http://logstash.openstack.org/#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6OTAwLCJncmFwaG1vZGUiOiJjb3VudCJ902:15
sdagueclarkb: like I'm pretty sure - http://logstash.openstack.org/#eyJzZWFyY2giOiJmaWxlbmFtZTpcImNvbnNvbGUuaHRtbFwiIEFORCBtZXNzYWdlOlwiW1NDUF0gQ29weWluZyBjb25zb2xlIGxvZ1wiIEFORCBidWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIGJ1aWxkX2NoYW5nZTo2NjM3NiBBTkQgYnVpbGRfcGF0Y2hzZXQ6MiBBTkQgYnVpbGRfbmFtZTpcImdhdGUtbm92YS1wZXA4XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzO02:15
sdagueshould exist02:15
*** thuc has quit IRC02:15
clarkbyou are going to make me boot a proper machine :P02:15
fungiclarkb: filename:"console.html" AND message:"[SCP] Copying console log" AND build_status:"FAILURE" AND build_change:66376 AND build_patchset:2 AND build_name:"gate-nova-pep8"02:16
*** yamahata has joined #openstack-infra02:16
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add mailing list for OpenStack Ambassadors  https://review.openstack.org/6647802:16
*** thuc_ has quit IRC02:17
fungiclarkb: sdague do we load non-dsvm jobs into logstash?02:17
sdaguesome times02:17
fungiwhen did we start putting unit test console logs in there?02:17
*** esker has joined #openstack-infra02:17
sdagueI assumed we were doing it all the time02:17
fungiyeah, looks like we do02:18
sdagueso if you tell me we only do it for gate and not check, that would be useful info to have02:18
sdaguebecause I am assuming it's always around, and if it's not, that would explain some other things02:19
clarkbwe have always put all console.html files in there02:19
sdaguelike right now I'm waiting for check-tempest-dsvm-full 66371,1 to show up02:19
*** pballand has joined #openstack-infra02:19
sdagueclarkb: ok, great02:19
sdagueso that pep8 console.html above should have gone in a couple hours ago02:19
sdagueand it didn't02:20
sdagueand there is a gate-horizon-pep8 I'm waiting on that the job ended 40 minutes ago02:20
sdagueand the last gate-horizon-pep8 jobs are from yesterday02:21
clarkbsdague: I am not follwing? which one?02:21
lifelessfungi: just ssling up the new cloud now02:21
fungilifeless: exciting!02:21
sdagueclarkb: 2014-01-13 20:52:19  WARNING  [recheckwatchbot] Console logs not available after 0:26:49.363097s for gate-horizon-pep8 66465,102:21
sdaguethat was the fail that never showed up02:21
sdagueor a fail that never showed up02:21
sdaguewhen I see gate-horizon-pep8 jobs in logstash, they are indexed in 1s02:22
*** jishaom has joined #openstack-infra02:22
sdaguecomparing received_at vs. @timestamp02:22
fungilifeless: is the auth-url going to need a change at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n199 ?02:23
sdagueanyway, I'm being called out for the evening02:23
lifelessfungi: yah02:23
sdagueso enjoy the rest of it02:23
lifelessci-overcloud.tripleo.org , on https02:23
clarkbsdague: so first hunch is that that file isn't making it into the log archive quick enough and the logstash machinery is giving up on it02:23
fungilifeless: okay, submitting a review for that. you said the username, password and project-id will stay the same though?02:23
*** bknudson has quit IRC02:24
lifelessfungi: I *think* you have a set in hiera or somewhere for cd-overcloud...02:24
clarkbsdague: or, when did the logstash client get restarted. its possible we missed things during that process restart. we can definitely make both cases more robust02:24
fungilifeless: we do (the ruby variables on those lines above it are coming from hiera)02:24
lifelessfungi: ah, history - there are three clouds in question02:24
lifelessfungi: the poc, which had one-off creds02:25
lifelessfungi: cd-overcloud, which has automated creds02:25
clarkbcan do a reload that fork and execs itself for new code while keeping sockets open02:25
lifelessfungi: and ci-overcloud which has those from cd-overcloud, I hope02:25
lifelessfungi: though right now, I'm failing to talk to it, which is worrying02:25
clarkband can do more of an exponential backoff for logs if they don't become available02:25
clarkbfungi: ^02:25
fungilifeless: oh, okay. yeah we just have poc.tripleo.org in teh nodepool config, but i believe i remember adding the other creds of which you speak02:26
lifelessfungi: so the other creds, are the ones to duplicate02:26
*** hogepodge has quit IRC02:26
fungilifeless: noted02:26
fungiclarkb: i'll get the timestamps02:26
*** jasondotstar has joined #openstack-infra02:28
fungiclarkb: sdague: Jan 13 20:56:11 logstash sudo:    fungi : TTY=pts/1 ; PWD=/home/fungi ; USER=root ; COMMAND=/usr/sbin/invoke-rc.d jenkins-log-client restart02:29
clarkbhmm that is about half an hour after the test finished02:30
clarkbwhich makes the other thing a possibility02:30
clarkbthe console log isn't being available before the timeout02:31
clarkbfungi: sdague: we can grep for te change number in all of the logstash worker logs to see if it timed out02:31
clarkbI need to start packing up soon. need to checkout in a bit. I can poke at this more when back in seattle02:32
clarkbfungi: sdague: but I do agree that things seem to be missing rather than late which is helpful because it narrows where we need to look02:32
*** dkliban has quit IRC02:33
*** ArxCruz has quit IRC02:34
clarkbthe logs are quite verbose so we should be able to see what is going on. I am going to afk now for a bit02:39
*** jhesketh has quit IRC02:39
*** pballand has quit IRC02:40
*** dkranz has quit IRC02:42
clarkboh quick thought before I go, the new scp plugin may have changed behavior in a subtle way that tickles a logstash bug or vice versa02:43
*** dkranz has joined #openstack-infra02:43
clarkbnew jenkins masters...02:43
fungialso possibilities02:45
fungiwe do now have 5 production jenkins masters instead of 3, and 3 of those 5 are running a newer version of jenkins too02:46
clarkblooks like log archive returned a 40402:50
* clarkb will paste logs in a minute02:50
*** nati_uen_ has quit IRC02:50
clarkbfungi: I think it is a new race exposed by new scp plugin02:51
*** nati_ueno has joined #openstack-infra02:51
clarkbfungi: before the file would always exist thoguh often in a 0 byte form02:51
clarkbnow the file may not exist when logstash processes the events02:51
fungiso the processes need to wait and recheck i guess02:52
clarkbI lean toweards making the scp plugin a little less async02:52
*** coolsvap has quit IRC02:53
clarkbfungi: but we can also tackle this in the worker if the retry flag is sset02:53
clarkbin any case, I think that isolates the problem02:53
clarkband I am really going to afk now02:53
openstackgerritAaron Rosen proposed a change to openstack/requirements: Require httplib2>=0.7.5  https://review.openstack.org/6648602:54
clarkbto solve in scp plugin we can have a sync point across both threads after the file has first been touched02:54
clarkbthen allow the two threads to run completely independently02:54
clarkbzaro: ^02:54
*** krotscheck has quit IRC02:59
*** vipul is now known as vipul-away02:59
*** thomasem has joined #openstack-infra02:59
*** thomasem has quit IRC02:59
*** nati_ueno has quit IRC03:00
*** AaronGr is now known as AaronGr_Zzz03:02
*** jerryz has quit IRC03:09
*** pballand has joined #openstack-infra03:10
*** emagana has quit IRC03:14
*** dcramer_ has quit IRC03:14
*** tian has joined #openstack-infra03:16
*** vipul-away is now known as vipul03:19
lifelessfungi: ok so I found a machine with dead fan03:19
lifelessfungi: redeploying now03:19
fungilifeless: sounds fun. should we continue launching nodepool nodes in the poc cloud, or switch to using this one going forward?03:20
lifelessfungi: I deleted the POC cloud today03:20
lifelessfungi: it didn't have the right network setup at all03:20
fungiguess my question is answered03:20
lifelessfungi: the new cloud API endpoint is https://ci-overcloud.tripleo.org:13000/v2.003:21
lifelessfungi: it's not up yet, but your EOD is approaching03:21
lifelessfungi: so if you can copy the cd-overcloud creds to ci-overcloud, and that API endpoint03:21
lifelessfungi: it should all come together smoothly03:21
lifelessfungi: I have to run now to pick up C, I'll be back in a few hours when you should be asleep :)03:21
lifelessfungi: once it's up, I'm going to want to get you to allocate a floating IP in the cloud for the broker03:22
*** senk has quit IRC03:22
fungilifeless: okay03:23
fungiand yeah, eod was more like 5 hours ago, but that doesn't seem to be stopping me03:27
*** ^d has joined #openstack-infra03:28
*** pballand has quit IRC03:32
*** rnirmal has quit IRC03:33
*** jasondotstar has quit IRC03:34
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Update TripleO Cloud API endpoint for Nodepool  https://review.openstack.org/6649103:39
fungilifeless: ^ (whenever you're back)03:42
*** ^d is now known as ^demon|away03:43
*** jasondotstar has joined #openstack-infra03:46
*** ^demon|away has quit IRC03:51
*** senk has joined #openstack-infra03:57
*** slong has quit IRC03:57
*** jasondotstar has quit IRC04:00
*** weshay has quit IRC04:02
*** mrda has quit IRC04:07
*** mrda has joined #openstack-infra04:08
*** nati_ueno has joined #openstack-infra04:10
*** praneshp has quit IRC04:10
*** reed has quit IRC04:13
*** esker has quit IRC04:13
*** reed has joined #openstack-infra04:14
*** nati_ueno has quit IRC04:15
openstackgerritJun Jie Nan proposed a change to openstack/requirements: Update python-keystoneclient to 0.4.2  https://review.openstack.org/6649404:25
*** noorul has quit IRC04:29
*** dkliban has joined #openstack-infra04:35
*** coolsvap has joined #openstack-infra04:39
*** slong has joined #openstack-infra04:49
*** SergeyLukjanov has joined #openstack-infra04:52
*** slong has quit IRC04:53
*** nati_ueno has joined #openstack-infra04:53
*** nati_ueno has quit IRC04:55
*** nati_ueno has joined #openstack-infra04:55
*** praneshp has joined #openstack-infra04:56
*** dstufft has quit IRC05:00
*** slong has joined #openstack-infra05:05
*** dcramer_ has joined #openstack-infra05:05
*** dcramer__ has joined #openstack-infra05:10
*** dcramer_ has quit IRC05:12
*** praneshp_ has joined #openstack-infra05:18
*** pballand has joined #openstack-infra05:19
*** praneshp has quit IRC05:20
*** praneshp_ is now known as praneshp05:20
*** chandankumar has joined #openstack-infra05:24
*** erfanian has quit IRC05:24
*** pballand has quit IRC05:39
*** nati_uen_ has joined #openstack-infra05:49
*** nati_uen_ has quit IRC05:50
*** nati_uen_ has joined #openstack-infra05:50
*** nati_ueno has quit IRC05:53
*** mrda has quit IRC06:00
*** dkranz has quit IRC06:03
*** dkranz has joined #openstack-infra06:03
*** denis_makogon has joined #openstack-infra06:06
*** coolsvap has quit IRC06:07
*** dcramer__ has quit IRC06:08
*** dkliban has quit IRC06:09
*** SergeyLukjanov has quit IRC06:12
*** dstanek has quit IRC06:13
*** dkliban has joined #openstack-infra06:14
*** pelix has left #openstack-infra06:17
*** boris-42 has joined #openstack-infra06:18
*** qs201 has joined #openstack-infra06:22
*** qs201 has quit IRC06:23
*** senk has quit IRC06:26
*** senk has joined #openstack-infra06:27
mordredsdague, fungi: re: oslo.messaging - once we start publishing wheels, we're ok and can deal with it better06:29
mordredsdague, fungi: it's myu fault that we're not yet06:29
*** dkliban has quit IRC06:29
clarkbmordred you are up late06:31
*** coolsvap has joined #openstack-infra06:32
mordredclarkb: it's only 10:3006:34
mordredclarkb: where are you?06:34
mordredwow. this is backwards06:34
clarkboh right you are west coast now06:34
clarkbmordred still in perth flying back tonight06:34
clarkbvisiting the museum of western australia currently06:35
mordredclarkb: nice. oz good?06:35
clarkbya the weather finally cooled06:35
*** reed has quit IRC06:37
*** nati_ueno has joined #openstack-infra06:38
*** nati_uen_ has quit IRC06:41
*** zhiwei has left #openstack-infra06:45
*** gokrokve has joined #openstack-infra06:47
*** dkliban has joined #openstack-infra06:51
*** gokrokve has quit IRC06:53
*** vkozhukalov has quit IRC06:53
lifelessfungi: ok way to late, but the API is up on SSL06:55
*** slong has quit IRC06:55
*** marun has quit IRC07:01
*** boris-42 has quit IRC07:04
*** DinaBelova has joined #openstack-infra07:04
*** senk has quit IRC07:05
*** dkliban has quit IRC07:05
lifelessclarkb: are you perchance around ?07:06
*** senk has joined #openstack-infra07:06
*** jamielennox is now known as jamielennox|away07:07
*** yolanda has joined #openstack-infra07:12
*** tian has quit IRC07:20
*** DinaBelova has quit IRC07:22
clarkblifeless: sorry no. museuming07:22
*** nati_uen_ has joined #openstack-infra07:26
*** nati_uen_ has quit IRC07:26
*** nati_uen_ has joined #openstack-infra07:27
*** tian has joined #openstack-infra07:28
*** nati_ueno has quit IRC07:30
*** DinaBelova has joined #openstack-infra07:34
*** SergeyLukjanov has joined #openstack-infra07:35
*** praneshp has quit IRC07:38
*** jcoufal has quit IRC07:48
*** gokrokve has joined #openstack-infra07:48
*** gokrokve has quit IRC07:52
*** harlowja is now known as harlowja_away08:00
*** jishaom has quit IRC08:04
*** jcoufal has joined #openstack-infra08:08
fifieldtrussellbryant.net appears to be down!08:09
* fifieldt craves reviewstats08:09
*** bauzas has joined #openstack-infra08:12
*** fbo_away is now known as fbo08:20
*** nati_ueno has joined #openstack-infra08:21
*** nati_uen_ has quit IRC08:24
*** nati_ueno has quit IRC08:25
*** nati_ueno has joined #openstack-infra08:25
*** fbo is now known as fbo_away08:26
*** jishaom has joined #openstack-infra08:27
*** vkozhukalov has joined #openstack-infra08:27
*** flaper87|afk is now known as flaper8708:36
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API  https://review.openstack.org/6311808:42
*** denis_makogon has quit IRC08:42
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API  https://review.openstack.org/6311808:43
*** pblaho has joined #openstack-infra08:46
*** gokrokve has joined #openstack-infra08:48
*** ruhe has joined #openstack-infra08:49
*** _ruhe has joined #openstack-infra08:53
*** gokrokve has quit IRC08:53
*** jasondotstar has joined #openstack-infra08:53
*** ruhe has quit IRC08:53
*** che-arne has quit IRC08:53
*** _ruhe is now known as ruhe08:54
*** jpich has joined #openstack-infra09:00
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add manuals-upstream-translation-update to api-site  https://review.openstack.org/6651909:04
*** coolsvap has quit IRC09:05
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API  https://review.openstack.org/6311809:07
*** NikitaKonovalov has joined #openstack-infra09:07
*** jishaom has quit IRC09:07
*** emagana has joined #openstack-infra09:08
*** mkerrin has joined #openstack-infra09:08
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add project for savanna-ci configs  https://review.openstack.org/6652109:10
*** yassine has joined #openstack-infra09:11
*** dizquierdo has joined #openstack-infra09:12
*** emagana has quit IRC09:13
*** mozawa has quit IRC09:13
*** fbo_away is now known as fbo09:18
*** nati_uen_ has joined #openstack-infra09:20
*** mancdaz_away is now known as mancdaz09:20
*** mancdaz is now known as mancdaz_away09:21
*** nati_ueno has quit IRC09:23
*** dpyzhov has joined #openstack-infra09:24
*** afazekas has joined #openstack-infra09:24
BobBallAwaylifeless: great - thanks.  That's what I wanted to know :)09:29
*** BobBallAway is now known as BobBall09:29
*** dpyzhov has quit IRC09:29
*** derekh has joined #openstack-infra09:36
*** fbo is now known as fbo_away09:36
*** coolsvap has joined #openstack-infra09:38
*** dpyzhov has joined #openstack-infra09:39
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API  https://review.openstack.org/6311809:39
*** ruhe is now known as ruhe_away09:41
*** ruhe_away is now known as ruhe09:41
*** fbo_away is now known as fbo09:41
*** ruhe is now known as ruhe_away09:42
*** mancdaz_away is now known as mancdaz09:45
*** gokrokve has joined #openstack-infra09:48
*** markmc has joined #openstack-infra09:48
*** ruhe_away is now known as ruhe09:49
*** gokrokve has quit IRC09:53
*** ruhe is now known as ruhe_away10:00
*** ruhe_away is now known as ruhe10:01
*** dpyzhov has quit IRC10:06
*** flaper87 is now known as flaper87|afk10:06
*** dpyzhov has joined #openstack-infra10:06
*** nati_ueno has joined #openstack-infra10:07
*** nati_uen_ has quit IRC10:11
*** nati_ueno has quit IRC10:13
*** nati_ueno has joined #openstack-infra10:13
*** flaper87|afk is now known as flaper8710:13
*** NikitaKonovalov has quit IRC10:24
*** NikitaKonovalov has joined #openstack-infra10:27
*** coolsvap has quit IRC10:27
*** ruhe is now known as ruhe_away10:31
*** lcestari has joined #openstack-infra10:31
*** mrda has joined #openstack-infra10:35
*** ruhe_away is now known as _ruhe10:41
*** gokrokve has joined #openstack-infra10:48
*** gokrokve has quit IRC10:52
*** dpyzhov has quit IRC10:52
*** dpyzhov has joined #openstack-infra10:59
*** nati_uen_ has joined #openstack-infra11:01
mkodererlifeless: ping11:03
*** nati_ueno has quit IRC11:05
*** boris-42 has joined #openstack-infra11:07
*** mrda has quit IRC11:07
*** _ruhe is now known as ruhe11:11
*** flaper87 is now known as flaper87|afk11:16
*** andreaf has joined #openstack-infra11:19
*** ArxCruz has joined #openstack-infra11:23
*** flaper87|afk is now known as flaper8711:26
*** yaguang has quit IRC11:29
*** nprivalova has quit IRC11:35
*** ildikov_ has joined #openstack-infra11:41
*** amotoki_ has quit IRC11:46
*** jcoufal has quit IRC11:47
*** gokrokve has joined #openstack-infra11:48
*** hashar has joined #openstack-infra11:52
*** gokrokve has quit IRC11:53
*** jcoufal has joined #openstack-infra11:54
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file  https://review.openstack.org/6628011:55
*** hashar has quit IRC11:58
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Get rid of Django  https://review.openstack.org/6601611:59
*** rfolco has joined #openstack-infra11:59
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: [WIP] Add tests for Alembic migrations  https://review.openstack.org/6641412:00
*** yamahata has quit IRC12:01
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file  https://review.openstack.org/6628012:07
*** nati_uen_ has quit IRC12:12
*** dstufft has joined #openstack-infra12:12
*** HenryG_ has quit IRC12:13
openstackgerritAlexandre Levine proposed a change to openstack-infra/config: Adding empty gce-api project to stackforge  https://review.openstack.org/6195412:13
*** mfink has quit IRC12:15
*** nati_ueno has joined #openstack-infra12:17
*** SergeyLukjanov has quit IRC12:18
openstackgerritEric Guo proposed a change to openstack/requirements: Whitelist external netifaces requirement  https://review.openstack.org/6654912:19
*** SergeyLukjanov has joined #openstack-infra12:19
ttxfungi: gate apparently blocked on queueing12:20
uvirtbotLaunchpad bug 1268892 in openstack-ci "Zuul is stuck - not starting a lot of jobs in check/gate" [Undecided,New]12:20
ttxlacking slaves12:21
*** iv_m_ has joined #openstack-infra12:21
sdaguettx: yeh, I think node pool exploded12:23
*** coolsvap has joined #openstack-infra12:24
sdagueI never gathered if clarkb was back in the states today12:24
sdagueI think jeblair was flying today12:24
*** dstanek has joined #openstack-infra12:25
*** dstanek is now known as dstanekafk12:27
*** dstanekafk is now known as dstanek12:28
ttx#status notice Gate currently blocked due to slave node exhaustion12:29
openstackstatusNOTICE: Gate currently blocked due to slave node exhaustion12:29
ttxsdague: don't think there is much we can do except praying fungi will get up early12:31
sdaguettx: you need to make sure that jonathan realizes how much fungi has been holding down the fort this week.12:31
ttxsdague: I'll do my best to convey that.12:31
ttxtoo bad we got so low on Monday and now we've piled up a queue again12:32
* fifieldt will also convey this12:32
ttxsdague: given when the starvation started, it looks like we could use some alerting12:35
ttxbecause i'm pretty sure we had infra-core around when it actually started12:35
sdaguecould be12:35
ttx(looking at the test nodes graph at the bottom of zuul status)12:35
EmilienMsdague: Hi Sean, I need to go ahead with Tempest / Neutron tests that I'm doing https://review.openstack.org/#/c/66143/, but I face to a blocking patch for devstack-gate. Could have a look when you can of course ? https://review.openstack.org/#/c/66142/12:35
sdagueEmilienM: so I don't yet have d-g authority12:37
EmilienMsdague: oh, my bad.12:37
sdaguettx: yeh, could be.12:37
*** sandywalsh has quit IRC12:38
*** heyongli has joined #openstack-infra12:39
*** dpyzhov has quit IRC12:39
*** jasondotstar has quit IRC12:42
*** ruhe is now known as ruhe_away12:42
*** mozawa has joined #openstack-infra12:45
*** coolsvap has quit IRC12:46
*** portante has quit IRC12:46
*** gokrokve has joined #openstack-infra12:48
*** yamahata has joined #openstack-infra12:48
*** sandywalsh has joined #openstack-infra12:51
*** dpyzhov has joined #openstack-infra12:51
*** ruhe_away is now known as ruhe12:51
*** SergeyLukjanov has quit IRC12:53
*** gokrokve has quit IRC12:53
*** SergeyLukjanov has joined #openstack-infra12:54
*** nati_ueno has quit IRC12:54
*** yaguang has joined #openstack-infra12:57
*** smarcet has joined #openstack-infra13:00
*** bingbu has joined #openstack-infra13:06
*** mkerrin has quit IRC13:07
*** afazekas has quit IRC13:07
*** yaguang has quit IRC13:09
*** yaguang has joined #openstack-infra13:09
*** bingbu has quit IRC13:12
*** afazekas has joined #openstack-infra13:19
*** esker has joined #openstack-infra13:20
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API  https://review.openstack.org/6311813:20
*** ruhe is now known as ruhe_away13:23
*** yamahata has quit IRC13:23
*** yamahata has joined #openstack-infra13:23
*** jishaom has joined #openstack-infra13:24
*** esker has quit IRC13:24
*** ruhe_away is now known as _ruhe13:31
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: move to static LOG  https://review.openstack.org/6656413:32
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs  https://review.openstack.org/6656513:32
*** dstanek has quit IRC13:32
*** _ruhe is now known as ruhe13:32
*** jishaom has quit IRC13:34
vkozhukalovhey guys, is it ok to use libvirt python API inside tempest scenarios? I ask because here  http://docs.openstack.org/developer/tempest/overview.html is written that I am not allowed to hit hypervisors directly, but using libvirt is not hitting hypervisors directly. I need to create several tiny VMs in my testing scenario (ironic functional testing).13:38
*** SergeyLukjanov is now known as _SergeyLukjanov13:40
*** pblaho has quit IRC13:40
*** _SergeyLukjanov has quit IRC13:40
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations  https://review.openstack.org/6641413:41
*** mfink has joined #openstack-infra13:42
*** hashar has joined #openstack-infra13:43
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations  https://review.openstack.org/6641413:46
*** gokrokve has joined #openstack-infra13:48
ttxfungi: for when you get up, uregtn action needed on slave pool exhaution blocking gate13:48
*** weshay has joined #openstack-infra13:51
*** miqui has joined #openstack-infra13:53
*** gokrokve has quit IRC13:53
*** afazekas has quit IRC13:57
*** jasondotstar has joined #openstack-infra13:58
*** dkliban has joined #openstack-infra14:01
*** flaper87 has quit IRC14:01
*** iv_m_ has quit IRC14:01
*** thomasem has joined #openstack-infra14:03
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: [Don not review] Add tests for Alembic migrations  https://review.openstack.org/6641414:03
*** dprince has joined #openstack-infra14:04
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: [Don not review] Add tests for Alembic migrations  https://review.openstack.org/6641414:05
*** heyongli has quit IRC14:05
*** SergeyLukjanov has joined #openstack-infra14:06
*** dkranz has quit IRC14:08
*** changbl has quit IRC14:09
*** thuc has joined #openstack-infra14:10
fungittx: looking14:11
fungiwow, that's some graph14:12
sdaguefungi: yep14:12
ttxfun heh14:12
*** dkliban has quit IRC14:14
*** afazekas has joined #openstack-infra14:15
*** max_lobur has joined #openstack-infra14:15
*** ArxCruz has quit IRC14:15
fungilooks like it's a bug in nodepool14:17
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations  https://review.openstack.org/6641414:17
fungii'm collecting some state information before i shut it down, comment out the affected provider and start it back again14:17
SergeyLukjanovfungi, morning14:17
*** ArxCruz has joined #openstack-infra14:19
*** dims has quit IRC14:22
*** julim has joined #openstack-infra14:22
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: webapp: set cache-control headers to prevent caching  https://review.openstack.org/6658314:22
fungiokay, nodepool thread dump captured, puppet agent stopped, config file altered and daemon started cleanly again. it claims to be launching new nodes now, so should hopefully have some picking up jobs in 10-15 minutes14:23
*** esker has joined #openstack-infra14:23
fungigood afternoon, SergeyLukjanov14:23
*** dims has joined #openstack-infra14:23
hashargood morning14:24
fungigood afternoon, hashar14:24
hasharif you feel like reviewing some HTTP headers for Zuul web app : https://review.openstack.org/#/c/66583/ :-D14:24
*** tma996 has joined #openstack-infra14:25
fungihashar: i would love to do that. we'll see how far i get dealing with broken things first14:25
hasharit doesn't send cache-controls which would cause the status.json to be cached by Varnish :-]14:25
hasharah take your time, that is definitely not urgent14:25
*** saschpe has quit IRC14:25
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Use venv to build documentation  https://review.openstack.org/6598714:27
*** afazekas has quit IRC14:28
*** saschpe has joined #openstack-infra14:29
*** thuc has quit IRC14:29
*** flaper87|afk has joined #openstack-infra14:29
*** thuc has joined #openstack-infra14:30
*** flaper87|afk is now known as flaper8714:30
*** mrodden has joined #openstack-infra14:30
*** oubiwann has joined #openstack-infra14:31
*** saschpe has quit IRC14:31
*** saschpe has joined #openstack-infra14:31
*** flaper87 has quit IRC14:32
*** flaper87 has joined #openstack-infra14:32
*** thuc has quit IRC14:34
fungithe nodepool issue is being tracked in bug 126900114:35
uvirtbotLaunchpad bug 1269001 in nodepool "Nodepool stops building any new nodes when one provider is down" [High,Triaged] https://launchpad.net/bugs/126900114:35
sdaguefungi: man, that nodepool graph is a thing :)14:38
fungithing and a half14:39
*** bknudson has joined #openstack-infra14:39
sdagueso, unrelated, we are definitely completely loosing logs going into ES14:39
sdaguewhich I'll hound clarkb about when he's back14:39
*** burt has joined #openstack-infra14:40
fungisdague: yes, did you catch clarkb's comments on that? he found tracebacks suggesting that the jenkins master signalled the log was ready, but returned a 404 when an attempt was made to retrieve it. most likely a race in the jenkins plugin14:41
sdagueoh, interesting14:41
sdagueno, missed that14:41
fungiprobably introduced/exposed with the threading fixes we added to the plugin so it would work on newer jenkins14:41
*** mfer has joined #openstack-infra14:42
sdaguewell that explains some things14:42
fungiso it's likely been happening since the upgrade to jenkins 1.5.43 a couple weeks back14:42
fungier, 1.543, whatever14:42
openstackgerritAntoine Musso proposed a change to openstack-infra/gear: inline helps for interactive commands  https://review.openstack.org/6659114:43
fungihe was going to get zaro to look into adding a sync point between the threads for that14:43
sdaguewell at least having the problem uncovered means we can address it14:44
*** vogxn has joined #openstack-infra14:44
sdaguethe newest code I wrote for the bot will also log those timeouts in channel14:44
*** afazekas has joined #openstack-infra14:44
*** dkliban has joined #openstack-infra14:44
sdagueI need to get jog0 and mtreinish to review through all that code14:45
sdaguebut it should make things a ton more reliable14:45
*** senk has quit IRC14:46
* hashar curses Mac OS python for not including poll() :-(14:48
*** gokrokve has joined #openstack-infra14:48
dhellmannsdague, fungi : yes, PLEASE reserve oslo.messaging on pypi14:49
fungihashar: curse apple's disdain for some free software licenses14:49
hasharwell it is BSD based so we might use kqueue() :D14:50
fungidhellmann: i did last night, so it won't get squatted (or rather, openstack is now squatting it)14:50
dhellmannfungi: thanks14:50
*** eharney has joined #openstack-infra14:50
*** nosnos has quit IRC14:50
dhellmannfungi: I think we were assuming the "oslo" prefix would protect us14:50
fungidhellmann: mordred: also, the lack of an oslo.messaging release seems to be my current blocker for getting the requirements integration job working, since pip 1.5 no longer follows -f urls without additional prompting14:51
*** vogxn has quit IRC14:51
fungiwell, without passing an additional option/envvar14:52
markmc<markmc> dhellmann, do we have the publish-beta-releases-to-pypi thing yet?14:52
fungiso anyone installing, for example, glance from source using 'pip install .' is currently going to break on being unable to find oslo.messaging on pypi unless they know to add the right magic incantation14:53
dhellmannmarkmc: do you still consider messaging beta?14:53
markmcdhellmann, yes14:53
dhellmanncan we just call it 0.1?14:53
*** gokrokve has quit IRC14:53
*** senk has joined #openstack-infra14:53
markmcdhellmann, I wouldn't be very keen on going backwards in release numbers14:54
* dhellmann looks at the current version #14:54
dhellmannoh, right14:54
fungii'm going to continue trying to work around it in the pypi-mirror script for now, but just be aware that pip 1.5 is unfond of -f urls14:55
markmcfungi, ok, thanks14:55
markmcfungi, mordred had a plan for this, just don't know what the status is14:55
sdaguerealistically, it's going to really impact other users as well because of pip 1.5 behavior14:55
dhellmannwe were going to start building wheels14:55
fungimy pleasure. wanted to be sure someone knew the urgency crept upward on that14:55
*** chandankumar has quit IRC14:55
markmcthe wheels thing was just to avoid affecting <1.4 users14:56
markmcdoesn't apply to oslo.messaging, really14:56
dhellmannyeah, but that was how we were going to do alpha and beta releases14:56
dhellmannso everything was consistent14:56
dhellmannwhat's standing in the way of calling 1.3 done?14:56
markmcall the outstanding reviews and time for things to settle down14:57
sdagueor release 1.2.0 off the last 1.2.0a14:57
sdaguethen let out 1.3 when it's ready14:57
markmcsdague, 1.2.0a wasn't api stable14:57
*** dkliban has quit IRC14:57
dhellmannyeah, I don't think we want people using that one14:57
sdaguedhellmann: well the already are14:58
sdagueglance users are using it in the field today if they are CDing14:58
*** senk has quit IRC14:58
sdagueso the cats out of the ba14:58
markmcicehouse only14:58
sdaguemarkmc: yeh, so HP & Rax clouds14:58
sdagueno one important :)14:58
*** senk has joined #openstack-infra14:59
markmcI don't want API unstable stuff on pypi if it isn't marked as beta14:59
markmcwe can do that, even if we can't publish wheels yet14:59
*** dolphm has left #openstack-infra14:59
markmci.e. wheel vs beta-pypi-publishing is mostly orthogonal15:00
*** sarob has joined #openstack-infra15:00
*** sarob has quit IRC15:01
fungiyes, we can for example add a pypi-upload job for it in the pre-release pipeline for now, then strip that out at or immediately following the first stable release15:02
*** sarob has joined #openstack-infra15:02
markmcfungi, can pypi-upload mark it as beta15:02
markmcfungi, I forget how that works in pypi ?15:02
dstufftdhellmann: you know i've debated making it possible for a project to request registration of a namespace like oslo.*15:02
fungithe risk is in pip-1.3 and earlier unexpectedly upgrading someone from a stable release to a newer pre-release, but if there's been no first stable release it's moot15:02
markmcfungi, or maybe it's not a pypi metadata thing, but just how pip interprets version numbers ?15:02
sdaguemarkmc: so we're going to need another work around for that as well, right15:02
sdaguebecause pip 1.4 doesn't install beta levels by default now15:03
*** sdake has quit IRC15:03
*** sandywalsh has quit IRC15:03
fungimarkmc: use a beta version string (like it's doing now) instead of a release one. there's also a trove classifier for project maturity, though it's probably completely ignored for any real purpose15:03
*** coolsvap has joined #openstack-infra15:04
markmcsdague, it does if you ask for e.g. >=1.3.0a2 AFAIR15:04
markmcsdague, i.e. you explicitly ask for a beta version15:04
fungimarkmc: right15:04
*** vkozhukalov has quit IRC15:04
markmcfungi, ok, I'm up for that15:04
*** prad has joined #openstack-infra15:04
*** ruhe is now known as ruhe_away15:05
fungiso really, it's just as simple as us adding a few more lines to zuul's layout.yaml so that for now, the alpha/beta releases of oslo.messaging end up on pypi and then in the future we remove that upload job for pre-release tags when it's time for the first real release15:05
fungigetting mordred's buy off on that as a stop-gap while he's working on the last bits of the wheels stuff would be good before i go forward with it, but i don't see any reason why it wouldn't work15:06
markmcfungi, yep15:06
markmcfungi, I'll take a look at the infra config change15:06
*** dkranz has joined #openstack-infra15:07
fungiawesome. thanks markmc15:07
markmcfungi, thank you15:07
dstufftfungi: yes the classifer is completely ignored except for display in the ui15:08
dhellmanndstufft: that would be a nice feature15:08
*** ruhe_away is now known as _ruhe15:11
*** _ruhe is now known as ruhe15:12
*** dpyzhov has quit IRC15:12
*** rakhmerov has joined #openstack-infra15:14
*** sdake has joined #openstack-infra15:15
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs  https://review.openstack.org/6656515:15
*** sarob has quit IRC15:17
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs  https://review.openstack.org/6656515:18
*** herndon_ has joined #openstack-infra15:19
*** sandywalsh has joined #openstack-infra15:20
*** dcramer__ has joined #openstack-infra15:20
*** dpyzhov has joined #openstack-infra15:21
*** praneshp has joined #openstack-infra15:22
mferfungi did you hear back from the foundation on using openstack in a stackforge name?15:23
mordredfungi: what did I do?15:23
mordredmfer: tell me the openstack/stackforge question15:23
fungimfer: staff meeting is in ~30 minutes so i plan to get some confirmation15:23
fungibut with mordred around he can probably also help on that15:23
mferI'm curious because I'm thinking of changing the launchpad project for the go binding to be something more sane15:23
fungiboard member and all15:23
*** gokrokve has joined #openstack-infra15:24
*** dkliban has joined #openstack-infra15:24
mfermordred on https://review.openstack.org/#/c/62069/ I proposed a name for the php sdk as openstack-sdk-php. is it ok to use the name openstack in a stackforge project like this?15:24
*** praneshp has quit IRC15:25
mferthis is in the same pattern as aws, azure, and others use for their sdks. for DX and findability it's the best name15:25
mordredyeah - actually, let fungi get confirm from jonathan15:25
fungiproblem being, we've already let several stackforge projects in like that (puppet-openstack, cl-openstack, openstack-chef, et cetera)15:25
mferthat's why i'm trying15:25
mordredI _think_ it's fine15:25
mordredthe main issue is more in usage15:25
mordred"PHP SDK for OpenStack" vs. "OpenStack PHP SDK"15:26
fungii think it's fine too, but if for some reason it's not fine, then i'd like to make sure we start correcting it. i know there have been recent stirrings about "pollution" of the trademark in light of the "what is core" spider/defcore discussions15:26
mordredone indicates that it's an SDK in PHP for OpenStack - the second could be seen to indicate that it is a php sdk that is _part_ of OpenStack15:26
BobBallIs there a way to get a change's dependency tree from gerrit's ssh interface?  The json that I'm getting back doesn't seem to have the dependency information in it... which is somewhat surprising15:26
mordredBobBall: hrm. you know, I'm not sure. perhaps not15:27
*** rakhmerov has quit IRC15:27
fungiBobBall: the dependencies are actually the git tree itself. so if you can identify the parent commit sha, then you can query it for a change number as well15:27
mfermordred this is a PHP SDK for OpenStack. It's to communitcate with APIs and such. the naming pattern for the project is based around other expectations. For example aws does aws-sdk-lang15:27
*** gokrokve_ has joined #openstack-infra15:27
mfermordred i'm trying to just follow the common patterns to make it findable for devs who want to write php apps against openstack apis15:28
mferwhether you find it sad or beautiful... php is the most widely use language on the web by a long long long shot.15:28
mordredmfer: I'm 95% sure what you're doing is fine15:28
*** gokrokve has quit IRC15:28
BobBallNot what I was hoping to hear fungi ... :)15:28
BobBallbut thank15:28
mordredbut we _should_ for sure get jonathan's buy in15:28
fungimordred: in other news, and why i invoked the summoning ritual earlier, any objections to adjusting oslo.messaging's pre-release pipeline to upload tarballs to pypi for now, until they're ready for the first stable release?15:29
*** rakhmerov has joined #openstack-infra15:29
fungithere's no risk of pip 1.3 "upgrading" a user from a stable release to a pre-release if there never was a stable release15:29
mordredI personally don't mind that. the key will be fixing it to not upload tarballs to pypi on pre-releases any more once they do make a release15:29
mordredbut hopefully we'll have wheel uploads finished by then15:29
*** yaguang has quit IRC15:30
fungiright, i made sure to highlight the need to make that adjustment come release time15:30
mordredI'm largely in favor of removing support for external -f urls myself15:31
*** krotscheck has joined #openstack-infra15:31
*** rakhmerov has left #openstack-infra15:32
*** rakhmerov has joined #openstack-infra15:32
*** sandywalsh has quit IRC15:32
fungiit's currently flumoxing the pypi-mirror run inside the requirements integration job, since the mirror is getting built from upstream pypi downloads, but it can't find a release of oslo.messaging on pypi. then later when it tries to satisfy glance install dependencies from that mirror, and pip 1.5 refuses to follow the -f url in the requirements.txt, looks in the local mirror it built instead and15:33
fungisees no oslo.messaging, then dies15:33
fungiat least, this is the behavior i'm currently able to get out of it once i whitelist netifaces, psutil, et cetera15:34
*** ildikov_ has quit IRC15:35
mordredfungi: have you figured out how to land the whitelist into the mirror build?15:41
mordredfungi: is it purely a patch to run-mirror?15:41
*** kraman has joined #openstack-infra15:41
fungimordred: so far it looks like https://review.openstack.org/6636415:42
mordredfungi: wow. that's gross15:43
fungithe most recent logs there show the behavior i'm running into with glance's -f requirement on unreleased oslo.messaging15:43
fungiwell, that's a hack to see what else is broken15:43
fungiit could certainly be refined15:43
mordredwell, at least we KNOW we're not going to land additional broken reqs now :)15:43
mordredfungi: I'd +2 that patch, but my team access is all gone15:44
funginote, some of those pip commands might not need the whitelists. i just splatted them on all four so i could keep moving, since it takes so long to run15:44
fungibut so far it appears it's just those four requirements, now that netaddr is getting uploaded15:44
*** sandywalsh has joined #openstack-infra15:44
kramanmordred: ping15:45
fungimordred: as for gerrit group memberships, jeblair queried and bulk-removed you in the db. i can add you back now, but need to hunt down where he saved the list15:46
*** jgrimm has joined #openstack-infra15:46
fungiso probably not until after the conference call i'm on for the next hour15:46
*** wenlock has joined #openstack-infra15:46
fifieldtyay, conference call15:46
mordredfungi: it's no rush - just letting you know, I can't +2 that patch15:47
fungimordred: i wouldn't want you to +2 that patch right now anyway. not until i can get the job to pass (that's why i have it marked wip)15:47
kramanmordred: can you take a look at https://github.com/kraman/zuul/compare/solum_hacks . would like to discuss some of the changes there to see if there are better ways to do them15:48
fungififieldt: btw, finally got around to peeking at gerrit logs... "[patchset-created] output: welcome-message: error: too few arguments"15:49
*** thedodd has joined #openstack-infra15:49
fifieldtthanks fungi, I really appreciate you took the time to look15:49
fifieldtthings have been busy for you of late :)15:49
fungififieldt: i suspect the script is looking for arguments which the patchset-created hook does not normally pass15:49
fungififieldt: yes, yes they have15:50
fungiscattered flurry of insane15:50
fifieldtand not the tasty, sugary kind of flurry, either15:50
funginot that anybody's told me anyway15:50
*** NikitaKonovalov has quit IRC15:53
*** SergeyLukjanov has quit IRC15:53
fifieldtargh, I'm terrible15:55
fifieldtI copied ssh client methodology from one script, and the arguments for a different method15:55
*** ^d has joined #openstack-infra15:56
*** pballand has joined #openstack-infra15:56
*** senk has quit IRC15:57
fungiit happens15:58
*** reed has joined #openstack-infra15:59
*** ruhe is now known as ruhe_away16:00
*** eharney_ has joined #openstack-infra16:01
*** eharney has quit IRC16:02
*** eharney_ is now known as eharney16:02
*** senk has joined #openstack-infra16:02
*** rcleere has joined #openstack-infra16:02
fifieldtbtw fungi, should we create a role id similar to     trivial_rebase_role_id          => 'trivial-rebase@review.openstack.org' for welcome message ?16:04
fifieldtat the moment it doesn't have one16:05
fifieldtI guess it's needed so gerrit gets the labels right16:05
*** DinaBelova has quit IRC16:05
fifieldtor maybe it isn't used16:06
* fifieldt shrugs16:06
*** sarob has joined #openstack-infra16:06
*** afazekas has quit IRC16:09
*** ruhe_away is now known as _ruhe16:09
fungififieldt: hrm... yeah we probably do need a service account created for it so that it can have its own name in gerrit comments16:12
fifieldtshould I send a message to the mailing list?16:12
fifieldtdoing the request16:12
fifieldtargh, will also need to know where the SSH key will live16:13
fungififieldt: either ml post or a openstack-ci bug to help keep me from forgetting, sure16:13
fifieldtsince the private key needs to live on gerrit, I guess I'll just file a bug16:14
fungiyeah, i can generate and add it to hiera16:15
*** AaronGr_Zzz is now known as AaronGr16:16
uvirtbotLaunchpad bug 1269057 in openstack-ci "Make a service account for welcome-message" [Undecided,New]16:16
*** jcoufal has quit IRC16:16
*** changbl has joined #openstack-infra16:18
*** dpyzhov has quit IRC16:21
*** boris-42 has quit IRC16:21
*** markmcclain has joined #openstack-infra16:23
fungififieldt: thanks!16:24
*** thuc has joined #openstack-infra16:24
*** praneshp has joined #openstack-infra16:25
*** markmcclain has quit IRC16:25
*** thuc has quit IRC16:25
openstackgerritTom Fifield proposed a change to openstack-infra/config: Fix CLI args for welcome-message  https://review.openstack.org/6662316:25
*** thuc has joined #openstack-infra16:26
*** moted has quit IRC16:26
*** thuc has quit IRC16:26
*** thuc has joined #openstack-infra16:27
*** markmcclain has joined #openstack-infra16:28
*** rakhmerov has quit IRC16:28
*** ArxCruz has quit IRC16:29
*** rfolco has quit IRC16:32
*** bauzas has left #openstack-infra16:32
*** ArxCruz has joined #openstack-infra16:33
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Fix up `zuul --version`  https://review.openstack.org/6662916:34
*** MarkAtwood has joined #openstack-infra16:35
*** rfolco has joined #openstack-infra16:36
anteayasorry for coming in blind, I have not read any backscroll :/ can gerrit talk to the foundation website yet, or do we still get 404? https://bugs.launchpad.net/openstack-org/+bug/126825416:38
uvirtbotLaunchpad bug 1268254 in openstack-org "no communication with gerrit" [Undecided,New]16:38
*** sarob has quit IRC16:38
anteayaI'm just going through my email16:38
*** SergeyLukjanov has joined #openstack-infra16:41
*** markmcclain has quit IRC16:41
anteayamikal: we missing one, tail-f has a -1 on this one: https://review.openstack.org/#/c/56184/17 can you give me a hand clearing, or shall I take care of this one?16:43
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: change # of tries back to 20  https://review.openstack.org/6663116:45
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs  https://review.openstack.org/6656516:45
SergeyLukjanovdoes anyone use bip and can share config file? looks like I have a bad karma on setting up bouncers ;)16:45
fungianteaya: i have no idea--did the user you talked to retry?16:45
*** jbryce has joined #openstack-infra16:45
*** gyee_ has joined #openstack-infra16:45
fungii have been too busy to check e-mail bugs16:46
sdaguejog0: if you have plane review time, check out the elastic recheck queue I've got going16:46
sdaguethe bot will be tons more reliable after all of this16:46
*** ^demon has joined #openstack-infra16:47
*** ^d has quit IRC16:49
*** beekneemech is now known as bnemec16:50
*** _ruhe is now known as ruhe16:51
*** blamar has quit IRC16:53
anteayafungi: understood16:53
*** gothicmindfood has joined #openstack-infra16:54
anteayafungi: I had advised him to wait until at least Monday to check in, I will see if I can find him16:54
anteayaand if so, I will ask him16:54
*** sandywalsh_ has joined #openstack-infra16:54
*** fifieldt has quit IRC16:55
*** blamar has joined #openstack-infra16:55
*** krotscheck has quit IRC16:55
fungiahh, originally going to ask him to retry, but then i never heard the outcome16:56
fungier, you originally said you were going to, i mean16:56
fungianyway, i mentioned it to toddmorey and jbryce and if we can get some detailed timestamps of the incidents it will help16:57
anteayafungi: he did retry and got the same error16:57
anteayasince we were face to face he may have told me on the walk to lunch and we never shared with you16:57
*** mrodden has quit IRC16:57
anteayasorry about that, the problem of face to face16:58
jbryceare there full error logs from the client side?16:58
jbrycewith timestamps, full stack trace, etc?16:58
*** mrodden has joined #openstack-infra16:58
jbrycei know the first one was about 3:50 utc sunday. what about the second?16:59
anteayajbryce: shortly after that one, within the hour I believe16:59
fungijbryce: i will dig on the gerrit end and see if it logged anything16:59
*** tma996 has quit IRC16:59
anteayabut I don't have a specific time frame for it, sorry16:59
jbrycefungi: thanks17:00
* fungi needs to switch computers real quick and will dig for more detail17:00
jbrycehas anyone signed up successfully since then?17:00
*** esker has quit IRC17:01
*** gothicmindfood_ has joined #openstack-infra17:01
*** esker has joined #openstack-infra17:01
*** gothicmindfood has quit IRC17:02
fungithat's a good question, i should be able to do a date range query on that field in the gerrit db17:05
*** portante has joined #openstack-infra17:05
*** sarob has joined #openstack-infra17:05
*** esker has quit IRC17:06
*** vkozhukalov has joined #openstack-infra17:06
*** ruhe is now known as ruhe_away17:07
*** rakhmerov has joined #openstack-infra17:08
fungijbryce: here's what one of the exceptions looks like... http://paste.openstack.org/show/61199/17:09
fungijava likes to get very wordy with its backtraces17:09
*** woodspa has joined #openstack-infra17:09
fungimorning zaro... did you see clarkb's comments in scrollback about the scp plugin?17:10
jbrycefungi: looking17:10
sdaguezaro: I can definitely confirm we are dropping log files going to ES17:11
*** gokrokve_ has quit IRC17:11
*** woodspa has quit IRC17:11
*** herndon_ has quit IRC17:11
*** gokrokve has joined #openstack-infra17:11
*** Craig has joined #openstack-infra17:12
fungijbryce: it must be working at least some of the time, because i see 8 accounts which have added/updated contact info via that api successfully since midnight utc17:13
*** mozawa has quit IRC17:13
*** Craig is now known as cp16net17:16
*** gokrokve has quit IRC17:16
*** sandywalsh_ has quit IRC17:17
*** ruhe_away is now known as _ruhe17:17
anteayajeblair: there is another 3rd party testing meeting scheduled: http://lists.openstack.org/pipermail/openstack-dev/2014-January/024297.html17:17
zarosdague: what is ES?17:17
fungicentos6-3 workspace for gate-ceilometer-python26 seems to be broken... https://jenkins01.openstack.org/job/gate-ceilometer-python26/3135/console (offlining and cleaning up now)17:17
sdaguezaro: elastic search17:17
fungizaro: elasticsearch17:17
anteayaI bet it conflicts but wanted to share with you17:18
*** DinaBelova has joined #openstack-infra17:18
*** herndon has joined #openstack-infra17:18
zarofungi: yeah, i noticed the comment.  but not sure how i could repro.17:18
*** ildikov_ has joined #openstack-infra17:18
fungizaro: i'm guessing it happens intermittently so probably not reproducible 100% of the time17:19
*** thomasem has quit IRC17:19
zarofungi: i mean what do i need to setup to even attempt to repro?17:20
sdagueI think it's probably around 5 - 10% of the time based on rough estimates of what I'm seeing17:20
zaroi'm not real familiar with ES under the hood.17:20
fungizaro: probably something to listen for the ready event and then try immediately to access the url on the master17:20
*** yassine has quit IRC17:20
fungiprobably don't need the whole logstash-jenkins-client worker to reproduce it17:21
zarothought i would just wait for clarkb17:21
fungizaro: sure. mainly just making sure you saw his comments17:21
zarodidn't he fly back yesterday?17:21
fungizaro: no idea what yesterday means in this context. he was sightseeing in au while we were asleep17:22
fungiso may be on a plane now, but i don't really know17:22
zarofungi: ohh i heard something about packing and checking in..  maybe just flying to another part of au.17:23
*** thomasem has joined #openstack-infra17:23
zarowell i'll catch him next time he's in channel.17:23
sdaguefungi: I think he's in the air now17:23
sdaguefrom what I remember of his comments17:24
*** Ryan_Lane has quit IRC17:24
*** boris-42 has joined #openstack-infra17:26
SpamapSok I have another missing commit from git.openstack.org17:27
* SpamapS saves local repo in a tarball17:27
SpamapSclarkb: want that tarball?17:27
fungiSpamapS: what't the project name and commit id?17:27
fungione of the servers in the farm is probably lagging behind on replication for some reason17:28
SpamapSfungi: heat, 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a2617:28
* fungi checks17:28
*** max_lobur is now known as max_lobur_afk17:28
mordredSpamapS: you're better at doing that than anyone else17:30
fungiSpamapS: all four of the git servers behind git.o.o have that ref in their openstack/heat repo mirrors17:30
*** pcrews_ has quit IRC17:31
fungiSpamapS: is it missing on some specific branch?17:31
* fungi checks master17:32
fungii see it in the history of the master branch on each of them17:33
*** hashar has quit IRC17:33
SpamapSfungi: let me try pulling again17:33
SpamapSerror: Unable to find 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26 under https://git.openstack.org/cgit/openstack/heat17:33
SpamapSCannot obtain needed commit 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a2617:33
fungiweird. i'll try cloning from them17:34
SpamapSfungi: I'll push the tarball of my repo up17:34
SpamapS-rw-r--r-- 1 clint clint 259M Jan 14 09:29 heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.gz17:34
mordredSpamapS: use the cloud17:35
BobBallWould someone be able to review our nodepool changes?  They are needed to get XenAPI tested by jenkins which in turn is needed ASAP (pre I-2 if possible) to stop a nova patch to deprecate XenAPI driver.  Three nodepool changes at https://review.openstack.org/#/dashboard/504417:36
fungiable to clone from the filesystem on all of them too17:36
SpamapSmordred: I'm using the xz at this point17:38
jbrycefungi, anteaya: no errors at that timestamp either17:40
fungijbryce: what are the chances that server sits behind a proxy at the provider?17:40
jbrycefungi: 1,000,000%17:40
fungiwondering whether the 404 is coming from something in front of where you have logs17:41
SpamapS-rw-r--r-- 1 clint clint 171M Jan 14 09:29 heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz17:44
SpamapSnot bad xz.. not bad17:44
SpamapSfungi: I'm uploading that to HP Cloud's us east object storage. Perhaps you have a VM there you can inspect it from?17:45
*** markmcclain has joined #openstack-infra17:46
fungiSpamapS: perhaps. is the url only reachable from within that cloud region?17:46
*** jerryz has joined #openstack-infra17:46
SpamapSfungi: no it is public17:46
SpamapSfungi: but it will be 10Gb in that region. ;)17:47
*** NikitaKonovalov has joined #openstack-infra17:47
SpamapSfungi: but it will be 10Gb in that region. ;)17:47
SpamapStcp        0  52308        ESTABLISHED 6097/firefox17:47
SpamapShealthy Send-Q there :)17:47
fungiit'll be 50mbps at my house, and that's plenty for a 171mib file17:47
jbrycefungi: the http logs are actually sourced from the proxy because it's load-balancing multiple servers behind it. it's also possible there's some fault that's not being logged because it's dying in a different part of the response. if that's the case, there's actually a different error page than the standard 404 that gets served, so if we can replicate in real-time or capture the body of the 404 response we could nail that dow17:47
jbrycehowever that is usually caused by a failure of one of the load-balanced web nodes to respond and then there's usually an app error logged...17:47
SpamapSfungi: cool17:47
jbryceif anyone runs into an error with the gerrit interface to www.openstack.org you can feel free to contact me directly asap (210-317-2438)17:49
sdagueSpamapS: do you have a corp firewall between you and the git server?17:50
fungijbryce: okay, thanks. i'm getting in touch with the user fifield reported having issues at samsung, to confirm whether they're getting an error on contact form submission or not17:50
sdaguewe found some git ops were getting disrupted by a layer 7 attempted injection by the IBM corp firewall17:51
sdagueback when I had to deal with such things17:51
*** NikitaKonovalov has quit IRC17:51
SpamapSsdague: no17:51
SpamapSit is https direct to git.o.o17:51
sdagueSpamapS: yes, this was over https17:51
sdaguethere was some ssl issue17:52
SpamapSwell I'm also at home. but.. wow.. that is _eeeeevil_17:52
*** NikitaKonovalov has joined #openstack-infra17:52
sdagueyeh, agreed17:52
*** markmcclain1 has joined #openstack-infra17:52
sdaguejust throwing it out there as a possibility :)17:52
*** jcoufal-m has joined #openstack-infra17:53
*** derekh has quit IRC17:54
*** markmcclain has quit IRC17:55
SpamapSfungi: https://region-b.geo-1.objects.hpcloudsvc.com/v1/42841271810042/clint-fewbar/heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz?temp_url_sig=42841271810042:G6JETLFJDYYT771NJWPB:ffda56fc309016c67433e361fc85b843a2431f53&temp_url_expires=138972302417:55
fungiSpamapS: thanks, i'll have a look17:56
SpamapSfungi: https://region-b.geo-1.objects.hpcloudsvc.com/v1/42841271810042/clint-fewbar/heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz   should work too (horizon just giving me tempurls for the hell of it)17:57
*** NikitaKonovalov has quit IRC17:57
fungii'll try that one. the first was giving me "17:57
fungiHTTP request sent, awaiting response... 401 Unauthorized"17:57
*** mancdaz is now known as mancdaz_away17:57
*** NikitaKonovalov has joined #openstack-infra17:57
fungithat worked17:57
jog0sdague: I'm taking a redeye and taking today off so will review that stuff in canada18:00
sdagueok, cool18:00
*** NikitaKonovalov has quit IRC18:01
*** harlowja_away is now known as harlowja18:03
*** changbl has quit IRC18:05
*** changbl has joined #openstack-infra18:05
*** gokrokve has joined #openstack-infra18:06
*** NikitaKonovalov has joined #openstack-infra18:07
*** jcoufal has joined #openstack-infra18:09
*** NikitaKonovalov has quit IRC18:10
*** mrodden has quit IRC18:10
harlowjaclarkb yt, or others that know more about the 'openstack_citest' database on the slaves, just a few questions releated to it18:13
fungiSpamapS: preliminary confirmation... i can't remote update your broken local copy (same error you reported, same commit at fault), but i can make a fresh clone from the same exact remote and those commits are included intact18:13
fungiharlowja: i may be able to answer questions about that18:14
harlowjaah, sweet fungi18:14
*** NikitaKonovalov has joined #openstack-infra18:15
harlowjaso intersting question, testr runs in parallel right, and say each test uses 'openstack_citest' as its database, then basically each test needs to be serialized to avoid messing up that database, is that expected, does the ''openstack_citest' have access to create other temp databases to avoid that serialization?18:15
*** ianw has quit IRC18:15
*** dcramer__ has quit IRC18:15
harlowjaif each test creates/drops it, then they could collide :-/18:15
*** gyee_ has quit IRC18:16
harlowjabut if each test made ''openstack_citest-$XYZ' then they could avoid that, but it doesn't appear that the ''openstack_citest' has those capabilties18:16
*** markmcclain1 has quit IRC18:16
fungiharlowja: it's possible with postgres, but not with mysql right now (there's been talk of switching from mysql to drizzle to overcome that)18:17
*** Ajaeger1 has joined #openstack-infra18:17
fungiharlowja: basically, drop isn't grantable without making the user a full mysql admin18:17
fungiharlowja: so if we're trying to reign in the privs of unit test scope activities, that presents an issue18:18
fungibut it's mostly a shortcoming of mysql18:19
*** krotscheck has joined #openstack-infra18:19
harlowjai'm fine with serializing it to avoid screwing up each test, if thats how it has to go :)18:19
SpamapSfungi: that seems like "software bug" to me.18:19
SpamapSfungi: as in, either local client is doing someting wonky on fetch, or reporting the wrong problem, or cgit is not responding properly.18:20
fungiharlowja: i think that's the route oslo was talking about, but there has been some concern raised that it flies in the face of parallelizing these things. i'd recommend ambushing mordred and clarkb to get into the weeds on that decision18:20
harlowjafungi although i'm unsure how nova is doing it, they seem to be dropping and recreating :-/18:20
*** jpich has quit IRC18:20
harlowjaif no drop permissions wonder how thats worked for them :-/18:20
* mordred reads scrollback18:20
fungiSpamapS: i did a git gc and git fsck on it and am getting a persistently dangling commit 0c83ac25a912843cfdee3ecd8bfb70921d326974 which is unreadab;e18:20
fungiharlowja: entirely possible i'm mistaken. if so mordred will correct me18:21
harlowjafungi np18:21
sdagueharlowja: you could also declare locks around the tests18:21
harlowjanova also doesn't seem to be ensuring that each test locks a file to avoid colliding, so sorta weird, ha18:21
harlowjasdague ya, if i have to :)18:21
sdaguefortunately we have this nice oslo lock infrastruture :)18:22
sdagueI think in the nova case if you back it out far enough, there really is only one test18:22
harlowjasdague ah :)18:22
sdaguewhich is why the parallel isn't the problem18:22
sdagueit's the migrations test18:22
sdagueit's huge, but only actually one test18:22
harlowjasdague ya, taskflow uses part of the oslo locking stuff, the one problem with the olso locking code is https://github.com/openstack/oslo-incubator/blob/master/openstack/common/lockutils.py#L34 (the sucking in of oslo.logging, which brings in oslo.config)18:23
*** mrodden has joined #openstack-infra18:23
harlowjaalso lockutils itself seems to suck in oslo.config18:23
harlowjaanyways, if have to do process lock per test, thats fine18:23
harlowjasdague the other part that confuses me is if fungi is right, then nova shouldn't be able to drop/create the openstack_citest db :-/18:24
sdaguewhy not?18:24
*** atiwari has joined #openstack-infra18:25
harlowjaah, nm, i think its fine18:25
fungiharlowja: i believe we create that db using puppet18:25
*** esker has joined #openstack-infra18:25
*** andreaf has quit IRC18:25
fungiharlowja: if you git grep inside openstack-infra/config i believe you'll find it there18:25
harlowjafungi ya, the nova code then must be legacy18:25
harlowja*the nova code that tries to create it18:25
sdagueI actually think it's created both ways18:26
sdagueand that's allowed18:26
harlowjak, nm then :)18:26
mferfungi we're you able to get feedback on using the openstack name?18:26
sdagueonce the permissions are set in mysql, nova can drop and recreate to it's hearts content18:26
mordredsdague: that is correct18:26
harlowjasdague ya, good point18:26
harlowjak, this answered my question, will add some shared process locks on taskflow tests that are gonna touch that db18:27
fungimfer: the initial answer was that it's a case-by-case basis. i'm sending mark collier the details right now and he's going to determine whether it needs to go before foundation legal counsel18:27
*** jamielennox|away is now known as jamielennox18:27
mferfungi thanks. if we could know a general answer for SDKs that would be useful for any others as well18:27
*** markmcclain has joined #openstack-infra18:27
*** NikitaKonovalov has quit IRC18:28
fungimfer: i agree. it's my understanding that if it's a generalized approach allowing open participation and isn't vendor-focused or otherwise supporting a specific commercial endeavor then it's non-infringing18:29
fungiso i suspect sdks fall into that category for the most part, but i am not a lawyer ;)18:29
mferfungi for my stuff it won't be vendor focused but will work cleanly for vendors18:29
fungiyeah, i think as long as it tries to work for everyone equally and not one particular company, that's the crux of the requirement18:30
mferand i agree whole heartedly with that18:30
*** _ruhe is now known as ruhe18:30
*** NikitaKonovalov has joined #openstack-infra18:33
*** vipul is now known as vipul-away18:35
*** krtaylor has quit IRC18:36
lifelessmkoderer: pong18:37
*** NikitaKonovalov has quit IRC18:37
*** NikitaKonovalov has joined #openstack-infra18:38
*** krtaylor has joined #openstack-infra18:39
*** hogepodge has joined #openstack-infra18:40
*** pcrews_ has joined #openstack-infra18:40
*** hogepodge has quit IRC18:41
*** ryanpetrello has quit IRC18:41
*** krotscheck has quit IRC18:42
mkodererlifeless: hi18:42
mkodererlifeless: I already wrote an email18:42
*** gyee_ has joined #openstack-infra18:42
mkodererlifeless: I tried to define a decorator for the testscenarios18:43
*** hartsocks has joined #openstack-infra18:44
hartsockshi, I have an identity crisis18:44
mordredhartsocks: I recommend spending some time alone in the desert to sort that out18:44
fungihartsocks: double-account in gerrit?18:44
hartsocksI am both http://www.stackalytics.com/report/users/hartsocks and http://www.stackalytics.com/report/users/hartsocks18:44
hartsocksfungi: yeah18:44
hartsocksboth hartsock and hartsocks … not sure how I managed to screw that up18:45
fungii seem to recall you had both as separate accounts at one point, maybe a 6mos to a year ago18:45
fungibecause i think we got asked to combine them back then18:46
hartsocksfungi: yeah, so I wanted to fold everything into hartsocks IIRC.18:46
hartsocksfungi: but I think I left my launchpad id hartsock or something… oops.18:46
*** krotscheck has joined #openstack-infra18:47
fungiyeah, we don't control stackalytics and i'm not sure how they're doing account classification18:47
*** NikitaKonovalov has quit IRC18:47
hartsocksso, on launchpad… should I change my id to hartsocks? will that break things worse?18:48
fungiat first glance, the gerrit activity counted for hartsock and hartsocks seem to map back to the same e-mail address and gerrit account number18:49
*** hogepodge has joined #openstack-infra18:49
hartsocksbtw on review.openstack.org when you search "shawn" I show up twice too...18:49
hartsocksnot sure what's going on there.18:49
hartsocks(that's on the "add reviewer" dialog)18:50
*** jamielennox is now known as jamielennox|away18:50
fungihartsocks: that means there are still two accounts in gerrit. i'll make sure the e-mail address gets cleared from the old one18:50
fungiduplicate e-mail addresses tends to cause issues when selecting a reviewer for a change or group membership18:51
hartsocksfungi: sorry to be a pain. I should have just kept my hartsock@acm.org identity18:51
*** jcoufal-m has quit IRC18:51
fungihartsocks: part of this is that gerrit wasn't designed well to deal with openid providers, and lp does unexpected things with openid18:52
fungiso it's not really your fault18:52
*** NikitaKonovalov has joined #openstack-infra18:52
hartsocksfungi: I want to clean it up, and I also don't want to make it worse...18:53
lifelessfungi: hi18:53
fungilifeless: hi there18:53
lifelessfungi: so, new cloud :)18:53
*** vipul-away is now known as vipul18:53
fungilifeless: yeah18:53
lifelessfungi: I believe it to be all setup18:54
*** rnirmal has joined #openstack-infra18:55
*** rnirmal_ has joined #openstack-infra18:57
*** rnirmal_ has quit IRC18:57
fungihartsocks: should be all set now, hopefully. i nulled-out the inactive account's preferred e-mail address and reassigned the stray external id to your active account18:57
*** rnirmal_ has joined #openstack-infra18:57
*** NikitaKonovalov has quit IRC18:57
*** rnirmal_ has quit IRC18:57
fungilifeless: excellent. also, you helped us expose bug 126900118:57
uvirtbotLaunchpad bug 1269001 in nodepool "Nodepool stops building any new nodes when one provider is down" [High,Triaged] https://launchpad.net/bugs/126900118:57
*** rnirmal_ has joined #openstack-infra18:57
lifelessfungi: ahahahahaha, sorry.18:57
fungiyou can't be held accountable for our bugs18:58
hartsocksfungi: thanks. Should I edit my launchpad ID from https://launchpad.net/~hartsock to hartsocks or will that break something?18:58
mordredfungi: hold him accountable18:58
*** SergeyLukjanov has quit IRC18:58
*** NikitaKonovalov has joined #openstack-infra18:58
fungihartsocks: i would leave it as is. changing your username in lp has been seen to generate yet another new openid18:58
hartsocksfungi: thanks again.18:58
*** SergeyLukjanov_ has joined #openstack-infra18:59
fungihartsocks: don't mention it18:59
openstackgerritGuido Günther proposed a change to openstack-infra/jenkins-job-builder: maven-deploy: don't require id  https://review.openstack.org/6603318:59
*** atiwari has quit IRC18:59
lifelessfungi: so I'd like to get confirmation that you can use said cloud18:59
*** rnirmal has quit IRC18:59
*** rnirmal_ is now known as rnirmal18:59
fungilifeless: me too. hopefully after the tc meeting i can give it a whirl19:00
fungiinfra meeting time now19:00
*** pblaho has joined #openstack-infra19:00
*** nicedice has quit IRC19:02
*** SergeyLukjanov_ has quit IRC19:03
*** nicedice has joined #openstack-infra19:03
*** NikitaKonovalov has quit IRC19:03
*** SergeyLukjanov has joined #openstack-infra19:03
*** krtaylor has quit IRC19:05
openstackgerritA change was merged to openstack-dev/hacking: Remove unnecessary headers  https://review.openstack.org/6213719:05
openstackgerritA change was merged to openstack-dev/hacking: checking for metaclass to be Python 3.x compatible  https://review.openstack.org/5689019:06
*** MarkAtwood has quit IRC19:07
*** NikitaKonovalov has joined #openstack-infra19:10
*** gyee_ has quit IRC19:10
*** elasticio has joined #openstack-infra19:11
*** ryanpetrello has joined #openstack-infra19:12
*** NikitaKonovalov has quit IRC19:14
*** dizquierdo has quit IRC19:15
*** markmcclain has quit IRC19:15
*** NikitaKonovalov has joined #openstack-infra19:19
*** yolanda has quit IRC19:20
*** dcramer_ has joined #openstack-infra19:21
*** herndon has quit IRC19:24
*** dcramer_ has quit IRC19:27
*** vkozhukalov has quit IRC19:29
*** ruhe is now known as ruhe_away19:30
*** ruhe_away is now known as ruhe19:33
*** NikitaKonovalov has quit IRC19:34
*** beagles is now known as beagles_brb19:35
*** NikitaKonovalov has joined #openstack-infra19:38
*** krtaylor has joined #openstack-infra19:45
*** NikitaKonovalov has quit IRC19:46
*** rfolco has quit IRC19:50
*** NikitaKonovalov has joined #openstack-infra19:51
*** vipul is now known as vipul-away19:52
*** vipul-away is now known as vipul19:52
*** dkliban is now known as dkliban_afk19:54
*** pballand has quit IRC19:55
*** dcramer_ has joined #openstack-infra19:56
*** CaptTofu has joined #openstack-infra19:56
*** ianw has joined #openstack-infra19:59
*** denis_makogon_ has joined #openstack-infra19:59
*** markmcclain has joined #openstack-infra19:59
*** jcoufal has quit IRC20:00
*** sdake has quit IRC20:01
*** pblaho has quit IRC20:01
*** sdake has joined #openstack-infra20:01
*** sdake has quit IRC20:01
*** sdake has joined #openstack-infra20:01
*** markmcclain1 has joined #openstack-infra20:01
*** markmcclain1 has quit IRC20:01
*** markmcclain1 has joined #openstack-infra20:01
*** madmike has joined #openstack-infra20:02
*** jcoufal has joined #openstack-infra20:02
*** yolanda has joined #openstack-infra20:03
*** mfink has quit IRC20:03
*** markmcclain has quit IRC20:04
*** ArxCruz has quit IRC20:05
*** beagles_brb is now known as beagles20:05
*** dkranz has quit IRC20:05
*** dkranz has joined #openstack-infra20:05
*** denis_makogon_ is now known as denis_makogon20:06
*** sparkycollier has joined #openstack-infra20:06
*** NikitaKonovalov has quit IRC20:06
*** vipul is now known as vipul-away20:08
*** esker has quit IRC20:08
*** thuc has quit IRC20:10
*** resker has joined #openstack-infra20:10
*** thuc has joined #openstack-infra20:10
*** senk has quit IRC20:11
*** CaptTofu has quit IRC20:12
*** vipul-away is now known as vipul20:12
*** NikitaKonovalov has joined #openstack-infra20:13
*** thuc has quit IRC20:15
*** Ajaeger1 has quit IRC20:18
*** gothicmindfood_ has left #openstack-infra20:20
*** gothicmindfood has joined #openstack-infra20:21
*** sarob has quit IRC20:23
*** HenryG has joined #openstack-infra20:24
*** NikitaKonovalov has quit IRC20:25
*** niska has quit IRC20:27
*** vipul is now known as vipul-away20:27
*** yolanda has quit IRC20:29
*** dkliban_afk is now known as dkliban20:32
*** NikitaKonovalov has joined #openstack-infra20:33
*** herndon has joined #openstack-infra20:34
*** NikitaKonovalov has quit IRC20:35
*** gyee has quit IRC20:38
*** gyee has joined #openstack-infra20:38
*** gothicmindfood has left #openstack-infra20:46
*** miqui has quit IRC20:46
*** gothicmindfood has joined #openstack-infra20:47
*** sarob has joined #openstack-infra20:48
*** sarob has quit IRC20:50
*** sarob has joined #openstack-infra20:51
*** senk has joined #openstack-infra20:53
*** niska has joined #openstack-infra20:53
*** senk1 has joined #openstack-infra20:55
*** senk has quit IRC20:57
*** dcramer_ has quit IRC20:58
*** DinaBelova has quit IRC20:59
*** markmcclain has joined #openstack-infra21:02
fungimordred: i've added you back to the administrators group in gerrit, so you should be able to add yourself to the other groups you need (i couldn't find where jeblair kept a record of what groups he removed you from)21:03
*** AaronGr is now known as AaronGr_afk21:03
fungimordred: i'll add you back in lp groups and github orgs shortly21:03
*** markmcclain2 has joined #openstack-infra21:03
fungiassuming you've changed passwords anywhere you feel you need to do so21:03
mordredfungi: awesome. thanks21:04
*** markmcclain1 has quit IRC21:04
*** ^d has joined #openstack-infra21:05
*** ^d has joined #openstack-infra21:05
*** dprince has quit IRC21:06
*** julim has quit IRC21:06
*** markmcclain has quit IRC21:07
*** ^demon has quit IRC21:08
mikaljog0: can I get a new revision of https://review.openstack.org/#/c/63478/ from you please?21:13
*** mrda has joined #openstack-infra21:14
*** praneshp has quit IRC21:15
*** mfink has joined #openstack-infra21:16
*** madmike has quit IRC21:17
*** Hefeweizen has quit IRC21:17
fungimordred: added you back to openstack-ci-admins on lp for now. jeblair will have to add you back to openstack-admins later (he's the only admin of that group now)21:18
*** senk1 has quit IRC21:18
mordredfungi: k. thanks!21:18
mordredif anybody is bored and looking for things to do - https://review.openstack.org/#/c/56760/21:19
mordredthat ^^ doesn't actually work, but it would be nice and magical if a fairy flew in and fixed it21:19
* fungi is fresh out of pixie dust21:21
*** gothicmindfood has quit IRC21:21
*** melwitt has joined #openstack-infra21:21
fungimordred: also remember that this revert happened... https://review.openstack.org/6004221:23
fungiand https://review.openstack.org/56660 needed it21:24
mordredoh right21:24
* fungi has been trying to keep state of all that in his notes21:25
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Stop running manage-projects  https://review.openstack.org/6669221:25
mordredfungi: well, luckily - wheels are enabled by default in pip 1.5 .. so the matrix of things we have to screw with is smaller to totally land everything21:25
mordredfungi: ^^ the patch above should block manage-projects from running until we get it fixed21:25
mordredfungi: I think that for now we can batch up and run by hand, and then figure out why it works by hand but not in puppet21:26
mordredfungi: yeah?21:26
*** vipul-away is now known as vipul21:26
fungimordred: why does gerrit::remotes need to be absent?21:27
*** _david_ has joined #openstack-infra21:27
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations  https://review.openstack.org/6641421:28
mordredfungi: to remote the cron job that runs manage-projects21:28
fungii keep forgetting it's there, and then i see it in the syslog when i go to debug, and then i forget again21:28
*** thuc has joined #openstack-infra21:28
*** thuc has quit IRC21:29
*** thuc has joined #openstack-infra21:29
dstufftfungi: mordred clarkb whoever, we're down to basically 1 issue before we start the 1.5.1/1.11/1 RC process for real, and that is basically just waiting on a guy to release a new version of distlib so we can vendor it21:30
dstufftjust as an ETA kind of thing21:30
mordreddstufft: thank you!21:31
*** senk has joined #openstack-infra21:31
*** ildikov_ has quit IRC21:32
dstufftmordred: I know openstack deals with helping people new to python get involved some too, pip 1.5.1 comes with a much much much simplified bootstrap instruction too21:32
mordreddstufft: oh yeah? what is?21:32
dstufftget-pip.py basically handles everything for you, setuptools, pip, the whole 9 yards21:32
mordreddstufft: WOOT!21:33
mordredthat's excellent21:33
dstufftit won't install virtualenv by default, but it'll get the package manager stuff setup for you21:33
fungidstufft: stellar. sounds like our existing get-pip machinations will get even simpler21:33
mordredAaronGr_afk: ^^ the puppet module we have that hacks up the puppet pip provider should be able to be smaller now21:33
fungiyes, that21:33
dstufftthis basically happened because pip itself no longer depends on setuptools to install from a Wheel21:34
dstufft(in 1.5.1 that is)21:34
fungidie setuptools die?21:34
dstufftit still uses setuptools for sdists21:34
dstufftbut it's able to install setuptools itself now21:34
dstufftinstead of needing to have it installed before hand21:34
mordredfungi: https://review.openstack.org/#/dashboard/9660 both of these are nice cleanups you may want to look at - AaronGr_afk is starting to walk through cleanup refactoring stuff21:34
fungiexcellent. one more chicken-and-egg hatched before its laid21:34
fungimordred: AaronGr_afk: i already had the first one starred. other added now. last week was not conducive to getting much reviewed21:35
mordredfungi: :)21:36
mordredfungi: no, I know it - and I could not be happier that you did, you know, real things21:36
fungii think the main thing i accomplished was to not get around to approving more changes which could break everything ;)21:36
*** rnirmal has quit IRC21:40
fungimordred: added you back to all the owner teams in github again21:40
mordredfungi: tahnks!21:44
*** senk has quit IRC21:45
*** senk has joined #openstack-infra21:46
mordredfungi: also - AaronGr_afk and I are working on some package building stuff - what's teh status of the stuff from pabelanger?21:46
mordredI'd ove to not overlap21:46
*** thomasem has quit IRC21:47
fungimordred: i think we added a packaging project parallel to one of our infra projects... zuul maybe?21:47
mordredfungi: and he has shown me https://github.com/rcrowley/freight#readme21:47
mordredfungi: which is apparently a modern and easy way to manage the package repos21:47
fungimordred: http://git.openstack.org/cgit/openstack-infra/zuul-packaging/21:47
fungifreight, huh?21:48
pabelangerI have some reviews up, but not much exposure on it: https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul-packaging,n,z21:48
mordredAaronGr_afk: we shoudl sync with pabelanger21:48
mordredbecause the machinery is something that we should want to be useful to infra's repos too21:48
*** jhesketh_ has joined #openstack-infra21:48
fungimordred: so the high-level idea was a packaging repo per project being packaged, and each distro+release gets its own branch21:49
*** markmc has quit IRC21:49
mordredfungi: hrm. why repo per project?21:49
mordred"packaging repo"21:49
mordredwith you21:49
fungiyes, sorry, git repo21:50
mordredno - you said it properly - I can't read21:50
fungisince, for example, packaging for debian-jessie would end up being potentially different from debian-wheezy and definitely different than fedora 2021:50
fungibut they could still benefit from diverging from some common branches (branch debian-jessie off from debian-sid for example)21:50
fungiso it makes keeping the commit history a little easier21:51
fungithe thought was that pabelanger and others bang on the zuul-packaging git repo until the workflow emerges on its own, then codify that and start propagating more packaging git repos21:52
*** AaronGr_afk is now known as AaronGr21:53
mordredk. well, fyi - I have an internal deadline to have tools for it soonish21:53
fungimordred: that's awesome21:53
pabelangerYa, I haven't had much time to play with it lately.  However, that will be changing shortly21:53
mordredso hopefully we can take a stab at workflow and stuff21:54
*** dkliban has quit IRC21:56
pabelangerone item on the workflow is how to handle chroots for building packages.  I've used pbuilder in the past for debian / ubuntu, but others use cowbuilder.  Honestly I can go either way, however building / maintain the chroot takes some time and energy.  So, I think we'd need to get a specific jenkins-slave snapshot that runs up to date chroots nightly to avoid delays in builds21:57
*** markmcclain2 has quit IRC21:58
*** gothicmindfood has joined #openstack-infra21:58
fungipabelanger: i'd lean toward a nodepool image which gets an updated set of chroots built nightly, then we can launch single-use slaves from it as needed21:59
openstackgerritA change was merged to openstack-infra/config: Added new SSH key for Monty  https://review.openstack.org/6634322:00
fungimordred: ^22:00
pabelangerfungi, Ya, I think that might be best22:01
mordredfungi: ++ on single-use-slaves22:01
fungilong-running slaves have proven to be a liability, so the fewer of them we have doing generic tasks, the safer we'll be22:02
*** jasondotstar has quit IRC22:02
*** sparkycollier has quit IRC22:04
*** smarcet has left #openstack-infra22:04
EmilienMfungi: Hi, could you please tell me who else is devstack-gate core ? I would need https://review.openstack.org/#/c/65822/ and https://review.openstack.org/#/c/66142/ merged when possible.22:04
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations  https://review.openstack.org/6641422:04
*** resker has quit IRC22:05
fungiEmilienM: me, mordred, clarkb and jeblair at the moment22:08
EmilienMfungi: ok, I need it because tomorrow we start the neutron/tempest sprint, and we will need those plugins.22:09
EmilienMmordred: if you have 2 minutes for that, I would appreciate :-)22:09
*** praneshp has joined #openstack-infra22:10
*** ^d has quit IRC22:11
fungiEmilienM: if you could get some buy-off from neutron and qa on those patches, i'd be inclined to just approve them22:11
*** praneshp_ has joined #openstack-infra22:11
*** ^d has joined #openstack-infra22:12
fungiEmilienM: i wasn't involved in the previous disabling of those services in tempest, so don't have a lot of history/perspective on whether the situation has resolved itself22:12
fungiEmilienM: also, as i noted in review, those changes are going to conflict with one another, so will need to rebase one of them when the other merges (or rebase one on the other in review if you prefer)22:13
EmilienMfungi: yeah when one will be merged, I'll rebase22:14
EmilienMfungi: some buy-off?22:14
fungiEmilienM: confirmation from qa that this isn't going to cause the same problems which led to it being disabled the first time22:15
*** praneshp has quit IRC22:15
*** praneshp_ is now known as praneshp22:15
EmilienMfungi: they are not disabled, they just are not enabled by default in devstack :-)22:15
*** sarob has quit IRC22:15
EmilienMand in neutron too.22:15
EmilienMit's just plugins to add in neutron22:16
*** sarob has joined #openstack-infra22:16
fungiEmilienM: right. i gather we were testing them and had to stop because it was disruptive, but i don't recall the details. i think clarkb and sdague were involved in that22:16
EmilienMfungi: both plugins are working. I just don't know what to do more. We have to perform tempest tests, but if the plugins are disabled, i don't know what to do  more :-)22:17
fungimainly just wanted confirmation from neutron and qa devs that it was okay to reenable again22:17
EmilienMfungi: you have a +1 from enikanorov_ who is neutron-core22:18
fungipreferably who would confirm with a comment in the review that whatever cause us to remove it last time (whatever that was) has been addressed22:19
*** lcestari has quit IRC22:20
*** jamielennox|away is now known as jamielennox22:20
fungii guess i'll see if i can git pickaxe to where it was removed before and see if i can find a bug or commit message to explain the circumstances22:20
*** ruhe is now known as _ruhe22:21
EmilienMfungi: tbh, i'm not sure the plugins have already been enabled in the past. metering & fwaas are quite new plugins in neutron22:22
*** ^demon has joined #openstack-infra22:23
*** ^d has quit IRC22:24
*** Hefeweizen has joined #openstack-infra22:25
*** _david_ has left #openstack-infra22:25
*** julim has joined #openstack-infra22:29
fungiEmilienM: agreed, hunting in git history of config, devstack-gate, devstack and tempest i'm not finding any record of q-fwaas having been previously disabled in some way. bug search is also turning up nothing, so i must have imagined it22:31
*** mfer has quit IRC22:31
*** dkliban has joined #openstack-infra22:31
openstackgerritSergey Lukjanov proposed a change to openstack/requirements: Bump python-savannaclient version to >=0.4.1  https://review.openstack.org/6672322:32
*** ArxCruz has joined #openstack-infra22:33
SergeyLukjanovfungi, mordred, could you please take a look on ^^22:33
*** dcramer_ has joined #openstack-infra22:34
*** ArxCruz has quit IRC22:35
openstackgerritJeremy Stanley proposed a change to openstack-infra/pypi-mirror: Whitelist external packages  https://review.openstack.org/6636422:35
*** SergeyLukjanov has quit IRC22:36
fungiSergeyLukjanov: it looks like a more general requirements review, not anything infra-specific (there's a fairly large requirements-core group to review those). also, it's not going to make it through the requirements integration job in the gate until pypi-mirror fixes are in place22:36
EmilienMfungi: once rebased, I'll ping you to approve the metering one22:37
fungiEmilienM: okay22:37
fungimordred: latest iteration of 66364 has the --process-dependency-links option on22:38
*** sandywalsh has quit IRC22:38
fungimordred: i wonder now, though, whether that's going to cause old pip to fail to upgrade itself to new pip22:38
fungiguess i'll find out22:39
* fungi needs to take a break to cook dinner... bbiab22:39
*** dcramer_ has quit IRC22:41
*** dims has quit IRC22:42
*** jcoufal has quit IRC22:43
openstackgerritA change was merged to openstack-infra/config: Set up openstackid module  https://review.openstack.org/6331622:46
openstackgerritA change was merged to openstack-infra/storyboard: Add SQLalchemy database model  https://review.openstack.org/6223922:48
*** mfink has quit IRC22:48
*** CaptTofu has joined #openstack-infra22:49
*** oubiwann has quit IRC22:50
*** herndon has quit IRC22:51
*** pcrews_ has quit IRC22:52
*** julim has quit IRC22:52
*** oubiwann has joined #openstack-infra22:52
*** hogepodge has quit IRC22:53
*** sandywalsh has joined #openstack-infra22:54
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added .gitignore for IntelliJ specific configuration files  https://review.openstack.org/6295622:54
*** hogepodge has joined #openstack-infra22:55
*** jhesketh__ has joined #openstack-infra22:56
*** herndon has joined #openstack-infra22:57
*** dims_ has joined #openstack-infra22:57
*** markmcclain has joined #openstack-infra23:00
*** CaptTofu has quit IRC23:04
*** bknudson has quit IRC23:06
*** pelix has joined #openstack-infra23:07
*** rcleere has quit IRC23:08
*** prad has quit IRC23:09
*** CaptTofu has joined #openstack-infra23:10
*** burt has quit IRC23:11
*** kraman has quit IRC23:12
pelixzaro: Re: https://review.openstack.org/#/c/48783/ removing the leading '/' results in the specified file being left out of the API documentation. I suspect it's because sphinx assumes paths starting with '/' are relative to the top level build dir, so if documentation is being built at two different levels (html/ and html/api/) you can't use purely relative paths.23:13
*** denis_makogon has quit IRC23:16
*** dkranz has quit IRC23:17
*** CaptTofu has quit IRC23:18
*** CaptTofu has joined #openstack-infra23:19
fungijhesketh__: long time no see!23:19
*** CaptTofu has quit IRC23:19
jhesketh__fungi: indeed.. I've been slack^W on leave and then at LCA23:19
*** elasticio has quit IRC23:20
fungijhesketh__: no worries. mikal broke all your stuff while you were gone... enjoy ;)23:20
jhesketh__heh, yeah, I saw23:20
jhesketh__on the bright side, he's nearly done improving it23:21
fungi(not really--he did an admirable job)23:21
jhesketh__indeed, I kinda turned on voting and ran away23:21
fungii've pointed lots of random people at turbo-hipster, so maybe you'll get some outside contributions23:22
jhesketh__yes, thanks for that :-)23:22
fungimostly people who are running jenkins and question all the bloat when they use basically none of its extended features23:23
fungialso, it's written in a language which isn't java23:24
*** sarob has quit IRC23:32
*** mrda has quit IRC23:33
*** eharney has quit IRC23:33
*** mfink has joined #openstack-infra23:34
*** CaptTofu has joined #openstack-infra23:36
*** ^d has joined #openstack-infra23:36
*** mrda has joined #openstack-infra23:36
*** ^demon has quit IRC23:38
*** nati_ueno has joined #openstack-infra23:38
*** dkliban has quit IRC23:42
*** thedodd has quit IRC23:42
zaropelix: wow, thanks.  i didn't notice that.  curently most of the jjb plugins that are using literal include do not have the leading '/' .  probably should refactor that.23:45
zaropelix: would you like to review 52080?  then maybe i'll get another core to see if it passes muster.23:48
*** herndon has quit IRC23:51
*** changbl has quit IRC23:54
*** markmcclain has quit IRC23:54
*** mrodden has quit IRC23:55
*** thuc has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!