Tuesday, 2015-09-01

*** achanda has quit IRC00:01
fungikfox1111: ubuntu 14.04 lts00:02
*** MarkAtwood has quit IRC00:02
*** achanda_ has quit IRC00:02
clarkbfungi: you see request to rename the UX/personas mailing list00:02
*** dims has quit IRC00:03
clarkbI am trying to remember if that is a thing we have done efore00:03
fungiclarkb: i did, just haven't gotten around to replying00:03
fungiwe renamed the board ml00:03
clarkbit would require a coordinated change to puppet too to make sure puppet doesn't make a new one with the old name00:03
fungii can probably find my notes00:03
fungiand yeah, we'd need to delete it from puppet i think00:03
yushiroarmax: Thank you for your review.00:04
clarkbkrotscheck: ^ I distinctly remember asking about personas :P00:04
fungiin the case of the board ml we renamed one existing list to a new name and then another existing list to the first name to replace it, but neither of them were in puppet afair00:04
*** hdd has joined #openstack-infra00:04
*** mtanino has quit IRC00:05
*** puranamr has quit IRC00:05
*** puranamr has joined #openstack-infra00:06
*** baoli has joined #openstack-infra00:07
*** wenlock has joined #openstack-infra00:07
*** sdake_ has joined #openstack-infra00:08
fungissh slaves plugin has been upgraded and jenkins service cleaned up and restarted on 06 and 07. now i have 03 and 05 in prepare for shutdown00:09
*** mtanino has joined #openstack-infra00:09
fungilooks like zuul finally caught up with the backlog while i was at dinner00:10
*** tjones has joined #openstack-infra00:10
*** sdake has quit IRC00:11
*** wenlock has quit IRC00:11
clarkbfungi: 04 continues to look great fwiw00:12
*** Swami has joined #openstack-infra00:12
openstackgerritMerged openstack-infra/project-config: Add python-jobs into networking-fujitsu  https://review.openstack.org/21806700:13
*** alexsyip has quit IRC00:15
*** shashank_hegde has quit IRC00:15
*** Piet has quit IRC00:16
yushiroarmax, clarkb: Thank you for your review.00:17
*** tjones has left #openstack-infra00:17
*** sdake has joined #openstack-infra00:18
*** teran has joined #openstack-infra00:18
*** teran_ has joined #openstack-infra00:21
*** teran_ has quit IRC00:21
*** sdake_ has quit IRC00:21
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements  https://review.openstack.org/21782400:22
clarkbmriedem: still around? why does sqlalchemy migrate depend on keystonemiddleware?00:22
clarkbmriedem: I am trying to figure out how those things are related00:22
*** gyee has quit IRC00:23
*** teran has quit IRC00:23
clarkbnibalizer: https://review.openstack.org/#/c/219031/ is extra special sauce00:25
*** sharikapongubala has quit IRC00:25
clarkbnibalizer: comments welcome since I have no better ideas00:25
*** yushiro has quit IRC00:26
nibalizerclarkb: that looks legit00:27
nibalizerobviously a hack, but simple and I'm down00:27
*** vivekd has joined #openstack-infra00:28
*** zeih has joined #openstack-infra00:29
*** mtanino has quit IRC00:29
*** hdd has quit IRC00:30
*** ybathia has quit IRC00:31
*** mtanino has joined #openstack-infra00:32
*** hdd has joined #openstack-infra00:33
*** zeih has quit IRC00:34
*** puranamr has quit IRC00:36
*** Piet has joined #openstack-infra00:37
*** Swami has quit IRC00:40
*** pvaneck has quit IRC00:41
*** thorst has joined #openstack-infra00:45
*** geoffarnold has quit IRC00:46
*** aysyd has quit IRC00:47
*** Sukhdev has joined #openstack-infra00:47
*** sdake_ has joined #openstack-infra00:47
*** tiswanso has joined #openstack-infra00:48
*** geoffarnold has joined #openstack-infra00:49
*** sdake__ has joined #openstack-infra00:49
openstackgerritMerged openstack/requirements: Block sqlalchemy-migrate 0.9.8, cap at 0.10.0  https://review.openstack.org/21902700:49
*** sdake has quit IRC00:50
*** tiswanso has quit IRC00:51
*** abdelwas has quit IRC00:51
*** tiswanso has joined #openstack-infra00:51
*** sdake_ has quit IRC00:53
*** luqas has quit IRC00:53
*** luqas has joined #openstack-infra00:53
anteayaarmax: can you remove your -1 please? https://review.openstack.org/#/c/217446/00:54
*** hdd has quit IRC00:54
*** sdake__ is now known as sdake00:54
armaxanteaya: woot?00:55
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting some trusty beaker jobs  https://review.openstack.org/21907800:55
armaxmost certainly00:55
anteayaarmax: very much woot00:55
anteayaarmax: thanks00:55
anteayaarmax: who is around to send it in?00:55
armaxa recheck should do it00:55
* armax reckons00:55
anteayaI don't think that will send it into the gate00:56
anteayamtreinish: are you about?00:56
armaxbut can it go straight to the gate, unless someone with powerful powers can?00:56
anteayait needs a +1 from jenkins00:56
*** madhu_ak has quit IRC00:56
EmilienManteaya: I don't want to abuse your time at this time of the day but I have an outstanding request: https://review.openstack.org/21907800:56
anteayaso yes do recheckk00:56
armaxanteaya: fired away00:57
cp16netanyone with Canonical here could answer a question on cloud-images.ubuntu.com?00:57
anteayaarmax: thanks, let's watch00:57
armaxanteaya: with trepidation00:57
anteayaEmilienM: not sure you could abuse my time00:57
EmilienMcp16net: zul00:57
anteayaarmax: even my cat is watching!00:57
armaxoh boy, good thing cats have 7 lifes00:57
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting some trusty beaker jobs  https://review.openstack.org/21907800:58
anteayaEmilienM: you write great commit messages00:58
anteayaEmilienM: thank you00:58
anteayaI get context from them and everything00:58
EmilienManteaya: good night and have a good trip to MTL :)00:59
*** glauco_ has joined #openstack-infra00:59
EmilienMI'm off00:59
anteayaEmilienM: see you wednesday00:59
anteayaI travel tomorrow00:59
cp16netzul: we've seen a few times where the /current dir of https://cloud-images.ubuntu.com/trusty/current go missing. Our trove gate is dependent on this location to build an image for our integration tests. Do know why this would happen? Have any suggestions to mitgate this?00:59
anteayaEmilienM: come back01:00
*** bdemers has quit IRC01:00
openstackgerritMerged openstack-infra/project-config: Add python34 to list of jobs of EC2-API project  https://review.openstack.org/21855401:00
anteayaEmilienM: you have one glance too many01:00
*** sigmavirus24_awa is now known as sigmavirus2401:00
anteayaand you forgot -upgrade in there01:00
vipulcp16net: hmm.. we're seeing similar but different errors from cloud-images url http://logs.openstack.org/24/215824/19/check/gate-cue-integration-dsvm-rabbitmq/6016ee1/logs/devstacklog.txt.gz#_2015-09-01_00_16_27_60801:00
EmilienMno upgrade are non voting already01:01
anteayaEmilienM: then change the commit message01:01
*** bdemers has joined #openstack-infra01:01
cp16netvipul: thats exactly the same error we see because the current dir is missing01:01
cp16netvipul: if you browse through there you will notice its missing01:02
vipulcp16net: oh yea :P01:02
EmilienManteaya: I should not be here :P01:02
cp16netcausing the forbidden01:02
anteayavipul: let's have polite language please01:02
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting some trusty beaker jobs  https://review.openstack.org/21907801:02
EmilienManteaya: done ^ thanks a lot01:02
anteayaEmilienM: :)01:02
*** chlong has joined #openstack-infra01:02
*** bookwar has quit IRC01:02
anteayaEmilienM: that looks better01:03
*** calebb has quit IRC01:03
vipulanteaya: whoops.. apologies01:03
*** skylerberg has quit IRC01:03
anteayavipul: thank you01:03
*** vivekd_ has joined #openstack-infra01:03
*** glauco_ has quit IRC01:03
mmmporkSpamapS: quick question, I saw you did a "recheck" comment and it looks like some jenkins things failed that passed before01:03
cp16netvipul: i thought we had images we used for cases like this but i guess not?01:03
*** bodepd has quit IRC01:03
mmmporki'm looking at the console output and it's not obvious to me what broke01:03
*** bodepd has joined #openstack-infra01:03
*** alivigni has joined #openstack-infra01:04
*** vivekd has quit IRC01:04
*** vivekd_ is now known as vivekd01:04
mriedemclarkb: it doesn't01:04
*** bookwar has joined #openstack-infra01:04
clarkbmriedem how are tgey related tgen01:04
mriedemclarkb: but keystone depends on sqla-migrate and keystonemiddleware01:05
mmmporki just see "Build failed for element: apt-sources, test-element: test-sources" which i didn't touch01:05
vipulcp16net: Trove may.. but we always generate one one the fly01:05
anteayaclarkb: you are holding a baby01:05
mriedemthe conflict is in deps on pr01:05
*** calebb has joined #openstack-infra01:05
clarkbanteaya yes01:05
mriedemclarkb: keystonemiddleware and sqla-migrate were requiring conflicting versions of pbr in that gate bug01:05
clarkbmriedem but middleware depends on the old version01:05
anteayaclarkb: I can tell your typing changes when your arms are full of baby01:05
clarkbso it needs updating?01:05
mriedemclarkb: keystone?01:05
*** rossella_s has quit IRC01:05
clarkbya keystonemiddleware says pbr < 1.0 according to the job log01:06
*** rossella_s has joined #openstack-infra01:06
mriedemclarkb: yeah, and migrate 0.9.8 requires pbr>=1.301:06
mriedemso kaboom01:06
mriedemanyway, https://review.openstack.org/#/c/213877/01:06
*** asettle is now known as asettle-afk01:07
clarkboh I thought >1.0 was required all over01:07
clarkbso we forcefully install latest but thatay be a bad assumption01:07
*** sharikapongubala has joined #openstack-infra01:07
armaxanteaya: actually on second thoughts, me wonders if the https://review.openstack.org/#/c/219027/ needs to merge in neutron first?01:08
*** ramashri has quit IRC01:08
*** ramashri_ has joined #openstack-infra01:08
mriedemclarkb: it was just stable/kilo that i screwed up, but because of grenade it also wedges master01:08
mriedemso basically i ruined everyone's life today01:08
*** alivigni has quit IRC01:08
*** baoli has quit IRC01:08
anteayaarmax: well it is merged01:10
armaxright, but that wasn’t pulled into stable/neutron01:10
anteayaarmax: is there a patch01:10
armaxnot that I can find01:10
anteayaI think proposal bot should be offering one up01:10
*** fitoduarte has quit IRC01:10
anteayaclarkb: should it not?01:11
armaxanteaya: it should01:11
armaxbut a new PS hasn’t popped up yet01:11
anteayafor stable/kilo?01:11
anteayaokay I will take your word for it, perhaps it needs a moment to collect itself01:12
armaxI’ll watch it with the help of your cats01:12
clarkbir should but runs in post iirc01:12
clarkbso may be slow01:12
armaxclarkb: yeah…I am not worried01:13
*** DericHorn-HP has joined #openstack-infra01:13
anteayaclarkb: so something should be happening here, yes?01:13
*** spzala has joined #openstack-infra01:13
armaxbtw anteaya  I have no merge rights on stable01:13
anteayaarmax: thanks :)01:13
armaxso you’ll have to reach out to other elves01:13
anteayaarmax: ah okay, good to know01:13
anteayaarmax: when we have something we will start the trace to find them01:13
mmmporkhrm i was wondering if anyone could help me understand why this passed before but just suddenly stopped passing - https://review.openstack.org/#/c/218965/01:13
anteayaarmax: first we need a patch to point them at01:13
mmmporkaccording to the error console, it's failing on something i didn't touch as part of the patch, so that's why i'm a bit confused01:14
armaxwe’’l find out soon enough01:14
*** bdemers has quit IRC01:14
clarkboh ya thats special node so maybe not slow01:14
clarkbcheck build history of that slave via jenkins ui01:14
*** bdemers has joined #openstack-infra01:14
mmmporkah it looks like SpamapS triggered a recheck to rerun the tests01:15
mmmporki just saw the "recheck" and the failed tests01:15
anteayammmpork: http://logs.openstack.org/65/218965/4/check/gate-dib-dsvm-functests-devstack-trusty/ad72368/console.html#_2015-09-01_00_29_08_93001:15
mriedemarmax: i'm going through the stable/kilo g-r syncs right now01:15
mriedemand fast approving everything01:15
armaxmriedem: great01:15
mmmporkanteaya: thanks i just glossed over that, just saw the failed on the things i didn't touch ;D01:15
anteayamriedem: thank you01:15
armaxmriedem: can’t see anything popping up for neutron yet though01:15
fungiarmax: clarkb: anteaya: the proposal job for that failed http://logs.openstack.org/4c/4c08ed336fdb79a7e4aaa09c1f9b2a5cc3e4f940/post/propose-requirements-updates/e25835d/01:16
mriedemarmax: does neutron use sqla-migrate? i thought neutron was only alembic?01:16
mmmporki just got an email that the triple NO tests failed (which is normal) so i realized it's rerunning right now :)01:16
anteayammmpork: many times the job fails for strange reasons, the host can have issues, new dependecies get released, image issues, on and on01:16
mmmporkright, typical jenkins stuff ;D01:16
armaxmriedem: indeed it does use alembic01:16
anteayafungi: thanks fungi01:17
mmmporki just am not familiar enough with our logs yet ;D01:17
armaxmriedem: but we’re still affected because keystonemiddleware01:17
*** ramashri_ has quit IRC01:17
anteayammmpork: well actually dismissing it is unhelpful and a poor dev habit which we work hard to educate folks not to fall into01:17
fungibut yeah, looks like neutron didn't need an update anyway01:17
anteayaas it results in devs expecting others to fix it for them01:17
mriedemarmax: keystonemiddleware doesnw't matter01:17
anteayaso yes learning how to read the logs is very helpful01:17
mriedemkeystonemiddleware doesn't require sqla-migrate01:17
mriedemkeystone does01:17
fungiseems it failed attempting to clone stackforge/congress for some reason01:18
anteayaand starting by finding or asking for help to find the failure is agreat start01:18
anteayaso thank you for doing that01:18
* mmmpork has ridden this rodeo before ;)01:18
*** puranamr has joined #openstack-infra01:18
fungiand some others. but the job at least seems to skip projects which are able to be updated01:18
anteayammmpork: anytime I'm wasting your time let me know, I will move onto other things01:19
*** btully has joined #openstack-infra01:19
*** iraw_ has joined #openstack-infra01:19
fungifor example, it did propose an update to ceilo https://review.openstack.org/#/c/218022/2/requirements.txt01:19
*** dims has joined #openstack-infra01:20
fungiso looks like it proposed the things it was going to propose01:20
anteayaso are we agreed that neutron does not need a stable kilo patch for the -migrate cap?01:20
*** jyuso1 has joined #openstack-infra01:20
fungii don't see it in requirements or test-requirements at http://git.openstack.org/cgit/openstack/neutron/tree/?h=stable/kilo01:21
fungiso seems like stable/kilo of neutron shouldn't have gotten an update proposed from that merge anyway01:21
anteayafungi: thank you01:21
anteayaarmax: what else stands between us and mtreinish's patch being merged?01:22
anteayaclarkb: you said look on the jenkins ui for the proposal job, where do I look? https://jenkins.openstack.org/01:22
clarkbanteaya on the page for the slave on the left is a build history button01:23
anteayaarmax: just 217446?01:23
anteayaclarkb: thanks, /me looks01:23
anteayafungi: is this the same failed build you were looking at? https://jenkins.openstack.org/job/propose-requirements-updates/lastBuild/console01:25
fungianteaya: probably, but i identified it from the git commit id01:25
zulcp16net: passing the info along01:25
anteayafungi: thanks, I see the same error failure to clone congress01:26
fungianteaya: yeah, i think there are a handful in there which changed from stackforge to openstack and the projects.txt in openstack/requirements was never updated to match01:27
*** FallenPegasus has joined #openstack-infra01:27
cp16netzul: thanks! :) i have a script pinging it every min to alert me when it is available again. and trying to see if there is another image endpoint we can use in the mean time.01:27
anteayafungi: ah01:28
anteayaI can do that01:28
anteayaunless you already have a script for that01:28
armaxanteaya: yes01:28
fungianteaya: it would be a big help! just search for any clone failures like that one in the console log, check to see if that repo has the wrong namespace on it and then update them all in a batch01:29
armax217446 is the only one01:29
armaxwhich btw just passed01:29
anteayaarmax: grand01:29
anteayaarmax: yay01:29
armaxfor reasons that escape me01:29
* anteaya goes to +A mtreinish's patch01:29
*** baoli has joined #openstack-infra01:29
anteayaarmax: I'm not asking questions right now01:29
*** mtanino has quit IRC01:29
armaxI won’t dare01:29
*** Piet has quit IRC01:30
anteayaokay waiting on the gate01:30
*** zeih has joined #openstack-infra01:30
anteayaguard cat is still with me01:30
*** puranamr has quit IRC01:30
anteayafungi: okey dokie01:31
jyuso1anteaya: hi01:31
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Pin python-netaddr to 0.7.15  https://review.openstack.org/21909001:32
anteayajyuso1: hey there, how are you01:32
anteayajyuso1: you had a runaway system today01:32
fungiokay, jenkins03 and 05 have the ssh slaves plugin upgraded, and have been cleaned up and restarted. putting 01 and 02 into prepare for shutdown now01:32
anteayajyuso1: have you found the error?01:32
anteayajyuso1: we had to shut you down, sorry about that01:33
anteayafungi: ack01:33
jyuso1anteaya: hi,yes,that's my problem.something wrong with one of my environment01:33
jyuso1anteaya: I've found the reason01:33
*** baoli has quit IRC01:34
anteayajyuso1: figured you would01:34
*** baoli has joined #openstack-infra01:34
anteayajyuso1: what happened?01:34
jyuso1anteaya: we are try auto-retrigger function in our CI.01:34
*** DericHorn-HP has quit IRC01:34
anteayamorgan: what did python-keystoneclient-federation become?01:34
anteayajyuso1: ah, let's not do that01:34
morgananteaya: uhmm... keystoneauth-saml2? i think01:35
anteayajyuso1: other ci operators have gotten into problems with taht before01:35
jyuso1anteaya: and some step are broken,that's a exception i've never thought about.01:35
*** zeih has quit IRC01:35
anteayamorgan: I'll go with that and hope you can review the patch once it is up01:35
anteayajyuso1: okay glad you are thinking about it now01:35
morgananteaya: if you give me a moment I can chase it down01:35
anteayamorgan: all the time you need01:36
anteayaI'll tell jyuso1 a story01:36
morgananteaya: https://github.com/openstack/keystoneauth-saml201:36
jyuso1anteaya: you mean we'd better not use auto-retrigger?01:36
anteayajyuso1: one time about a year and a half ago01:36
anteayajyuso1: yes don't use auto-trigger01:36
anteayamorgan: thank you01:36
anteayajyuso1: about a year and a half ago someone used an auto retrigger01:37
jyuso1anteaya: I'm listening:)01:37
anteayaand retriggered about 600 patches at once01:37
anteayaour system was smaller then01:37
anteayabut it was enough of a load, plus we were busy anyway01:37
openstackgerritMerged openstack-infra/project-config: copy_puppet_logs: copy dstat logs if exist  https://review.openstack.org/21858601:37
anteayaand it took out our system01:37
anteayawe had to restart our gerrit01:37
anteayaso we aren't in favour of folks using auto-retriggers01:38
anteayaas they rarely have the consequences the author intends01:38
anteayaand usually end up doing weird things that are very upsetting for devs01:38
anteayaso if you can avoid any kind of automated commenting or retriggering from your system01:38
anteayathat works out best for us01:39
anteayajyuso1: make sense?01:39
jyuso1anteaya: ok01:40
jyuso1anteaya: we just want to avoid the wrong return information.some error happened caused by networking:)01:40
anteayajyuso1: I understand01:41
anteayafungi: I don't know why it is trying to clone python-keystoneclient-federation: http://logs.openstack.org/4c/4c08ed336fdb79a7e4aaa09c1f9b2a5cc3e4f940/post/propose-requirements-updates/e25835d/console.html#_2015-09-01_00_55_01_04101:41
anteayafungi: do you see it in projects? http://git.openstack.org/cgit/openstack/requirements/tree/projects.txt01:41
*** larainema has joined #openstack-infra01:41
*** vivekd has quit IRC01:42
fungianteaya: oh, that job is on the stable/kilo branch specifically01:42
fungianteaya: so looking at that log, the cleanup would be for stable/kilo branch requirements projects.txt file01:43
anteayaI was in the wrong branch01:43
anteayaright sorry01:43
anteayahow to do I clone the branch again01:43
anteayaI can look it up01:43
*** vivekd has joined #openstack-infra01:44
openstackgerritMerged openstack-infra/project-config: Tripleo CI reshuffle  https://review.openstack.org/20547901:44
*** mriedem is now known as mriedem_away01:44
jyuso1anteaya: another thing.It looks like our account is forbidden,could you help to resume it?01:44
fungiyou can do something like `git checkout -B requirements-renames --no-track origin/stable/kilo` to create a local non-tracking topic branch named "requirements-updates" from the current state of teh stable/kilo branch on the origin remote01:44
anteayajyuso1: well first reply to the thread in third-party-announce and give me a full report of what went wrong01:45
anteayajyuso1: as well as the steps you will take to avoid the problem in the future01:45
anteayajyuso1: and I'll read what you have01:46
anteayajyuso1: thanks01:46
anteayafungi: thanks01:46
*** shashank_hegde has joined #openstack-infra01:46
*** geoffarnold is now known as geoffarnoldX01:46
*** Zhongjun has quit IRC01:47
*** larainema has quit IRC01:47
openstackgerritJerry Zhao proposed openstack-infra/project-config: Add networking-fortinet project to neutron  https://review.openstack.org/21905501:47
*** larainema has joined #openstack-infra01:47
anteayafungi: that worked, thank you01:47
openstackgerritMerged openstack-infra/project-config: Add docs and publish jobs for kolla  https://review.openstack.org/21813701:48
*** wenlock has joined #openstack-infra01:49
*** stevemar has joined #openstack-infra01:50
*** Sukhdev has quit IRC01:51
openstackgerritMerged openstack-infra/project-config: copy_puppet_logs: copy tempest.conf in txt file  https://review.openstack.org/21824501:54
*** stevemar has quit IRC01:55
*** bdemers has quit IRC01:55
*** bdemers has joined #openstack-infra01:55
openstackgerritAnita Kuno proposed openstack/requirements: Updates moved project paths  https://review.openstack.org/21909101:56
*** sridhar_ram has quit IRC01:56
openstackgerritMerged openstack-infra/project-config: Kolla: Add build gate for ubuntu source  https://review.openstack.org/21353101:57
anteayamorgan: if you want in on the review: https://review.openstack.org/#/c/219091/101:57
*** madong_ has joined #openstack-infra01:57
anteayaarmax: yay it passed01:58
anteayaarmax: third in the gate, waiting on the two before it01:58
anteayamorgan: thank you01:59
* anteaya goes to make tea01:59
*** larainema has quit IRC02:01
*** sthillma has quit IRC02:04
*** sabeen has quit IRC02:04
*** larainema has joined #openstack-infra02:04
*** madong_ has quit IRC02:04
*** jyuso has joined #openstack-infra02:06
*** asettle-afk has quit IRC02:07
*** zz_dimtruck is now known as dimtruck02:08
*** jyuso1 has quit IRC02:09
*** spzala has quit IRC02:10
*** Piet has joined #openstack-infra02:12
anteayajyuso: I have to go to bed soon, once you post your reply to the third-party-announce mailing list thread it will need to be approved by me before you can see it at lists.openstack.org02:14
openstackgerritMotohiro/Yuanying Otsuka proposed openstack-infra/project-config: Enable Barbican in Magnum job definition  https://review.openstack.org/21693302:14
*** iraw_ has quit IRC02:14
anteayajyuso: I'll be back for the third party meeting at 0800 utc so we can talk more then02:14
jyusoanteaya: OK,good night.I'll send the mail later.see you.02:16
anteayathanks see you later02:17
*** woodster_ has quit IRC02:19
*** tlian2 has quit IRC02:19
*** geoffarnoldX is now known as geoffarnold02:20
anteayamtreinish: 212864 is +A'd finally and I'm going to bed02:20
*** rkukura has joined #openstack-infra02:24
*** bknudson has quit IRC02:24
*** thorst has quit IRC02:29
*** baoli has quit IRC02:30
*** zeih has joined #openstack-infra02:31
openstackgerritRyu Ishimoto proposed openstack-infra/project-config: Set DEVSTACK_GATE_SETTINGS for midonet gate job  https://review.openstack.org/20773102:33
*** zeih has quit IRC02:35
*** baoli has joined #openstack-infra02:37
*** FallenPegasus has quit IRC02:41
*** BharatK has joined #openstack-infra02:45
*** fitoduarte has joined #openstack-infra02:46
fungiand finally, jenkins01 and 02 have the ssh slave plugin upgraded, then cleaned up and restarted. that makes all the nodepool jenkins masters. i'll get the others (jenkins.o.o and -dev) dealt with tomorrow02:47
*** iraw_ has joined #openstack-infra02:48
*** adduarte has joined #openstack-infra02:48
*** FallenPegasus has joined #openstack-infra02:49
*** yamahata has quit IRC02:50
*** fitoduarte has quit IRC02:50
*** baoli has quit IRC02:52
*** larainema has quit IRC02:55
*** baoli has joined #openstack-infra02:55
*** shashank_hegde has quit IRC02:57
*** asettle has joined #openstack-infra02:58
*** glauco has joined #openstack-infra03:00
*** shashank_hegde has joined #openstack-infra03:02
*** glauco has quit IRC03:04
*** achanda has joined #openstack-infra03:07
*** larainema has joined #openstack-infra03:07
*** baoli has quit IRC03:08
*** Zhongjun has joined #openstack-infra03:10
*** baoli has joined #openstack-infra03:10
*** chlong has quit IRC03:12
*** xyang1 has quit IRC03:13
*** baoli has quit IRC03:14
*** shashank_hegde has quit IRC03:20
*** Apsu is now known as JorgeSonata03:20
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for infra 403 during devstack setup bug 1490806  https://review.openstack.org/21910903:21
*** larainema has quit IRC03:21
openstackbug 1490806 in OpenStack-Gate "devstack fails in gate with "Server returned an unexpected response code. [403]"" [Undecided,New] https://launchpad.net/bugs/149080603:21
*** larainema has joined #openstack-infra03:21
*** JorgeSonata is now known as Apsu03:22
*** bdemers has quit IRC03:26
*** bdemers has joined #openstack-infra03:26
*** ddieterly has quit IRC03:26
*** mriedem_away has quit IRC03:27
*** baoli has joined #openstack-infra03:28
*** darrenc is now known as darrenc_afk03:28
*** gildub has joined #openstack-infra03:29
*** unicell has quit IRC03:31
*** zeih has joined #openstack-infra03:31
*** shashank_hegde has joined #openstack-infra03:33
openstackgerritMerged openstack-infra/elastic-recheck: Add query for infra 403 during devstack setup bug 1490806  https://review.openstack.org/21910903:34
openstackbug 1490806 in OpenStack-Gate "devstack fails in gate with "Server returned an unexpected response code. [403]"" [Undecided,New] https://launchpad.net/bugs/149080603:34
*** zeih has quit IRC03:36
*** chlong has joined #openstack-infra03:38
*** tiswanso has quit IRC03:40
*** larainema has quit IRC03:42
*** rlandy has joined #openstack-infra03:42
*** larainema has joined #openstack-infra03:43
*** vivekd has quit IRC03:44
*** FallenPegasus has quit IRC03:46
*** links has joined #openstack-infra03:46
*** FallenPegasus has joined #openstack-infra03:47
*** bdemers has quit IRC03:48
*** sigmavirus24 has quit IRC03:48
*** bdemers has joined #openstack-infra03:48
*** sigmavirus24 has joined #openstack-infra03:50
*** achanda has quit IRC03:51
*** stevemar has joined #openstack-infra03:56
*** bhunter71 has quit IRC03:57
*** iraw_ has quit IRC03:58
*** sigmavirus24 is now known as sigmavirus24_awa03:59
*** jeadre has quit IRC03:59
*** bdemers has quit IRC04:00
*** markvoelker has joined #openstack-infra04:00
*** HeOS_ has quit IRC04:00
*** HeOS_ has joined #openstack-infra04:00
openstackgerritMotohiro/Yuanying Otsuka proposed openstack-infra/project-config: Enable Barbican in Magnum job definition  https://review.openstack.org/21693304:00
*** bdemers has joined #openstack-infra04:00
*** markvoelker_ has joined #openstack-infra04:02
*** chlong_ has joined #openstack-infra04:04
*** chlong has quit IRC04:04
*** markvoelker has quit IRC04:04
*** ildikov has joined #openstack-infra04:04
*** btully has quit IRC04:05
*** geoffarnold is now known as geoffarnoldX04:07
*** Sree has joined #openstack-infra04:08
*** Sree has quit IRC04:08
*** Sree has joined #openstack-infra04:09
openstackgerritIan Wienand proposed openstack-infra/project-config: Make gate-ceilometer-dsvm-functional-postgresql voting  https://review.openstack.org/21548504:10
*** jerryz has quit IRC04:11
*** yfried has joined #openstack-infra04:13
*** jyuso1 has joined #openstack-infra04:13
*** bdemers has quit IRC04:14
*** jyuso has quit IRC04:16
*** amotoki has joined #openstack-infra04:17
openstackgerritMerged openstack/requirements: Changing the bounds for Django  https://review.openstack.org/21646904:19
openstackgerritMerged openstack/requirements: Changing the bounds for Django  https://review.openstack.org/21646904:19
*** mrmartin has joined #openstack-infra04:21
*** bdemers has joined #openstack-infra04:21
*** robcresswell has quit IRC04:21
*** Sree_ has joined #openstack-infra04:22
*** Sree_ is now known as Guest4120604:22
*** robcresswell has joined #openstack-infra04:22
*** bdemers has quit IRC04:23
*** geoffarnoldX is now known as geoffarnold04:23
*** bdemers has joined #openstack-infra04:23
*** Sree has quit IRC04:24
nigelbfungi: I ended up banning the use of + on our mailing lists. Got tired of the whack-a-mole with spammers.04:24
*** FallenPegasus has quit IRC04:24
*** ddieterly has joined #openstack-infra04:27
*** ayoung has quit IRC04:29
*** dims has quit IRC04:30
*** ddieterly has quit IRC04:32
*** zeih has joined #openstack-infra04:32
*** bdemers has quit IRC04:34
*** baoli has quit IRC04:34
*** puranamr has joined #openstack-infra04:35
*** yfried has quit IRC04:36
*** shashank_hegde has quit IRC04:36
*** rlandy has quit IRC04:36
*** bdemers has joined #openstack-infra04:37
*** zeih has quit IRC04:37
*** watanabe_isao has joined #openstack-infra04:39
*** yonglihe has joined #openstack-infra04:41
*** jyuso has joined #openstack-infra04:44
*** achanda has joined #openstack-infra04:45
*** jyuso1 has quit IRC04:47
*** jerryz has joined #openstack-infra04:49
ianwdoes anyone know any names of people that have to do with the ubuntu cloud mirror?04:52
*** darrenc_afk is now known as darrenc04:54
*** sharikapongubala has quit IRC04:57
*** sharikapongubala has joined #openstack-infra04:57
StevenKianw: About /trusty/current not existing? I've bought it up in #ubuntu-devel, but no answer yet04:57
StevenKianw: But https://launchpad.net/~ubuntu-cdimage is what you're looking for04:58
*** puranamr has quit IRC04:58
ianwStevenK: ok cool, yeah i pinged in ubuntu-mirrors but also nobody around04:58
*** Sukhdev has joined #openstack-infra04:58
openstackgerritMerged openstack-infra/project-config: Make gate-ceilometer-dsvm-functional-postgresql voting  https://review.openstack.org/21548504:59
*** sharikapongubala has quit IRC05:01
*** unicell has joined #openstack-infra05:02
*** puranamr has joined #openstack-infra05:04
*** vivekd has joined #openstack-infra05:05
*** Adri2000 has joined #openstack-infra05:09
*** sthillma has joined #openstack-infra05:10
*** geoffarnold has quit IRC05:11
*** geoffarnold has joined #openstack-infra05:11
*** ildikov has quit IRC05:12
*** FallenPegasus has joined #openstack-infra05:16
*** sthillma_ has joined #openstack-infra05:16
*** puranamr has quit IRC05:19
*** sthillma has quit IRC05:20
*** sthillma_ is now known as sthillma05:20
*** stevemar has quit IRC05:20
*** yfried has joined #openstack-infra05:21
*** jyuso1 has joined #openstack-infra05:23
*** yfried_ has joined #openstack-infra05:25
*** jyuso has quit IRC05:26
*** yfried has quit IRC05:26
*** ddieterly has joined #openstack-infra05:28
*** jerryz has quit IRC05:33
*** stevemar has joined #openstack-infra05:33
*** zeih has joined #openstack-infra05:33
*** ddieterly has quit IRC05:33
*** shashank_hegde has joined #openstack-infra05:34
*** larainema has quit IRC05:35
*** larainema has joined #openstack-infra05:36
*** zeih has quit IRC05:37
*** stevemar has quit IRC05:38
*** ildikov has joined #openstack-infra05:38
*** larainema has quit IRC05:41
*** larainema has joined #openstack-infra05:41
*** maishsk has joined #openstack-infra05:46
*** dimtruck is now known as zz_dimtruck05:46
*** mrmartin has quit IRC05:47
*** jyuso1 has quit IRC05:48
*** jyuso1 has joined #openstack-infra05:50
*** geoffarnold is now known as geoffarnoldX05:51
*** markvoelker_ has quit IRC05:53
*** yfried_ has quit IRC05:54
*** kushal has joined #openstack-infra05:55
*** mrmartin has joined #openstack-infra05:58
*** yfried has joined #openstack-infra05:58
*** flepied has quit IRC05:58
*** hdd has joined #openstack-infra05:58
*** dims has joined #openstack-infra05:59
*** FallenPegasus has quit IRC06:02
*** dims has quit IRC06:04
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/21914806:04
*** sdake_ has joined #openstack-infra06:05
*** kushal has quit IRC06:05
*** jtomasek has joined #openstack-infra06:08
*** sdake has quit IRC06:08
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/21873706:09
*** FallenPegasus has joined #openstack-infra06:11
*** sdake has joined #openstack-infra06:11
*** sdake has quit IRC06:11
*** sdake has joined #openstack-infra06:13
*** jerryz has joined #openstack-infra06:13
*** yfried is now known as yfried|afk06:13
*** mrunge has joined #openstack-infra06:13
*** unicell has quit IRC06:15
*** sdake_ has quit IRC06:15
*** unicell has joined #openstack-infra06:15
*** yfried|afk is now known as yfried06:15
*** abregman has joined #openstack-infra06:16
*** btully has joined #openstack-infra06:17
*** Zhongjun has quit IRC06:17
*** ParsectiX has joined #openstack-infra06:22
*** asettle has quit IRC06:26
*** BharatK has quit IRC06:27
yamamotowhere can i find results of periodic jobs?06:28
*** ddieterly has joined #openstack-infra06:29
*** flepied has joined #openstack-infra06:29
*** liam__ has joined #openstack-infra06:29
*** zeih has joined #openstack-infra06:34
*** deepakcs has joined #openstack-infra06:34
*** ddieterly has quit IRC06:34
*** kushal has joined #openstack-infra06:35
*** sharikapongubala has joined #openstack-infra06:36
*** afazekas__ has joined #openstack-infra06:36
*** armax has quit IRC06:37
*** zeih has quit IRC06:38
*** flepied1 has joined #openstack-infra06:39
*** sharikapongubala has quit IRC06:40
*** liam__ has quit IRC06:42
*** kiran-r has joined #openstack-infra06:43
*** flepied has quit IRC06:43
*** sshnaidm has joined #openstack-infra06:44
*** hdd has quit IRC06:44
*** kdas_ has joined #openstack-infra06:45
*** b10n1k_ has quit IRC06:46
*** Sukhdev has quit IRC06:46
*** kushal has quit IRC06:46
*** jyuso has joined #openstack-infra06:49
*** jyuso1 has quit IRC06:52
*** kiran-r has quit IRC06:53
*** k4n0 has joined #openstack-infra06:55
*** dtantsur|afk is now known as dtantsur06:57
*** peristeri has joined #openstack-infra06:58
*** glauco has joined #openstack-infra07:01
*** FallenPegasus has quit IRC07:03
*** markus_z has joined #openstack-infra07:03
*** glauco has quit IRC07:06
*** slagle has quit IRC07:06
*** fhubik has joined #openstack-infra07:08
*** achanda has quit IRC07:11
*** sirushti has quit IRC07:15
*** liam__ has joined #openstack-infra07:16
*** achanda has joined #openstack-infra07:17
*** deepakcs has quit IRC07:22
*** chlong_ has quit IRC07:24
*** achanda has quit IRC07:25
*** achanda has joined #openstack-infra07:26
*** liam__ has quit IRC07:28
*** aukhan has joined #openstack-infra07:29
*** sharikapongubala has joined #openstack-infra07:29
*** zeih has joined #openstack-infra07:29
*** achanda has quit IRC07:30
*** ddieterly has joined #openstack-infra07:30
*** liam__ has joined #openstack-infra07:30
*** liam__ has quit IRC07:30
*** Zhongjun_ has joined #openstack-infra07:33
*** Ala has joined #openstack-infra07:34
*** ifarkas has joined #openstack-infra07:34
*** ddieterly has quit IRC07:35
*** shardy has joined #openstack-infra07:35
*** Zhongjun_ is now known as Zhongjun07:38
*** kiran-r has joined #openstack-infra07:38
*** bharathm has quit IRC07:41
*** dizquierdo has joined #openstack-infra07:41
*** bexelbie_ has joined #openstack-infra07:43
*** sharikapongubala has quit IRC07:44
*** vivekd has quit IRC07:46
*** bexelbie_ is now known as bexelbie07:47
*** yfried is now known as yfried|afk07:47
*** kdas_ is now known as kushal07:48
*** kushal has joined #openstack-infra07:49
*** HeOS has quit IRC07:52
*** sharikapongubala has joined #openstack-infra07:52
*** jamespage has joined #openstack-infra07:53
*** jlanoux has joined #openstack-infra07:53
openstackgerritEvgeny Antyshev proposed openstack-infra/devstack-gate: Third-party rhel-based oses in devstack-gate  https://review.openstack.org/21502907:53
*** yfried|afk is now known as yfried07:57
*** luqas has quit IRC07:58
*** arxcruz has joined #openstack-infra08:02
*** ihrachys has joined #openstack-infra08:03
*** sthillma has quit IRC08:04
*** gildub has quit IRC08:07
*** jistr has joined #openstack-infra08:08
*** lennyb has joined #openstack-infra08:11
*** snikitin has joined #openstack-infra08:11
*** fhubik has quit IRC08:13
*** fhubik has joined #openstack-infra08:13
*** e0ne has joined #openstack-infra08:13
snikitinAll nova patches are failing because of job gate-tempest-dsvm-ironic-pxe_ssh08:14
ianwsnikitin: maybe try a rerun, it should be ok now.  see https://bugs.launchpad.net/openstack-gate/+bug/149080608:16
openstackLaunchpad bug 1490806 in OpenStack-Gate "https://cloud-images.ubuntu.com/trusty/current/ returning 403 ; various tests failing" [Undecided,New]08:16
*** cdent has joined #openstack-infra08:17
snikitinianw: thanks, I'll try08:18
openstackgerritAntoine Musso proposed openstack/diskimage-builder: Disable DIB_DEBUG_TRACE by default  https://review.openstack.org/21917608:23
*** yamahata has joined #openstack-infra08:26
*** fedora has quit IRC08:26
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Create job for infra ansible  https://review.openstack.org/21918108:30
*** ddieterly has joined #openstack-infra08:31
*** maishsk has quit IRC08:31
*** sirushti has joined #openstack-infra08:32
*** HeOS has joined #openstack-infra08:33
*** jcoufal has joined #openstack-infra08:34
*** ddieterly has quit IRC08:35
*** e0ne has quit IRC08:36
*** bexelbie_ has joined #openstack-infra08:38
*** bexelbie is now known as Guest237808:38
*** bexelbie_ is now known as bexelbie08:38
*** jerryz has quit IRC08:40
*** bharathm has joined #openstack-infra08:41
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Gate infra-ansible with gate-infra-ansible job  https://review.openstack.org/21918508:42
*** dims has joined #openstack-infra08:42
*** hichihara has quit IRC08:45
*** bharathm has quit IRC08:46
*** katyafervent is now known as katyafervent_awa08:47
*** e0ne has joined #openstack-infra08:48
*** dims has quit IRC08:48
openstackgerrityolanda.robla proposed openstack-infra/system-config: Move manage_projects logic to puppet-jeepyb  https://review.openstack.org/17156508:49
*** derekh has joined #openstack-infra08:49
openstackgerrityolanda.robla proposed openstack-infra/system-config: Remove graphite wrapper class  https://review.openstack.org/21669308:50
anteayafungi: once you are up and about, can we get Intel PCI CI re-enabled please? gerrit account id 1575108:50
anteayaor anyone else who cares to do so08:50
*** katyafervent_awa is now known as katyafervent08:51
*** ociuhandu has quit IRC08:54
*** dingyichen has quit IRC08:57
*** kiran-r has quit IRC08:57
openstackgerritAdam Coldrick proposed openstack-infra/infra-specs: storyBoard: Worklists and Boards  https://review.openstack.org/20298909:00
*** boris-42 has quit IRC09:00
*** jaosorior has joined #openstack-infra09:00
*** maishsk has joined #openstack-infra09:01
*** marzif has joined #openstack-infra09:01
*** yamahata has quit IRC09:01
*** kiran-r has joined #openstack-infra09:04
*** marzif has quit IRC09:04
*** marzif has joined #openstack-infra09:05
*** pelix has joined #openstack-infra09:05
*** sharikapongubala has quit IRC09:05
*** shashank_hegde has quit IRC09:06
*** sshnaidm has quit IRC09:07
*** sirushti has quit IRC09:08
*** yfried is now known as yfried|afk09:09
*** hashar has joined #openstack-infra09:10
*** sshnaidm has joined #openstack-infra09:10
*** sirushti has joined #openstack-infra09:11
*** ildikov has quit IRC09:13
*** ociuhandu has joined #openstack-infra09:16
*** abregman is now known as abregman|lunch09:19
*** fhubik has quit IRC09:19
openstackgerritLucas Alvares Gomes proposed openstack/diskimage-builder: Map ubuntu versions to match the SHA256SUMs  https://review.openstack.org/21919909:20
openstackgerritAdam Coldrick proposed openstack-infra/infra-specs: StoryBoard: Worklists and Boards  https://review.openstack.org/20298909:24
openstackgerrityolanda.robla proposed openstack-infra/puppet-jenkins: Don't configure memory swap bytes on cgroups for Trusty  https://review.openstack.org/21822709:26
*** fhubik has joined #openstack-infra09:28
*** yfried|afk is now known as yfried09:29
*** ddieterly has joined #openstack-infra09:31
*** ddieterly has quit IRC09:36
*** john-davidge has joined #openstack-infra09:36
*** dims has joined #openstack-infra09:36
*** mrmartin has quit IRC09:36
snikitinianw: Now http://cloud-images.ubuntu.com doesn't return 403, but job gate-tempest-dsvm-ironic-pxe_ssh is still failing with09:38
snikitinsha256sum: standard input: no properly formatted SHA256 checksum lines found09:38
*** yfried is now known as yfried|afk09:39
tsufievhey, folks! Where in openstack-infra/config can I find the details about the Devstack configuration being used for Horizon integration tests?09:39
*** john-davidge_ has joined #openstack-infra09:40
rcarrillocruztsufiev: you want to look at openstack-infra/project-config09:40
rcarrillocruzthat's where the project tests are kept09:40
*** john-davidge has quit IRC09:40
*** john-davidge_ is now known as john-davidge09:40
*** dims has quit IRC09:41
*** dtantsur is now known as dtantsur|brb09:41
tsufievrcarrillocruz, thanks, that was indeed the wrong repo )09:41
*** yfried|afk is now known as yfried09:45
*** asettle has joined #openstack-infra09:48
*** ildikov has joined #openstack-infra09:48
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements  https://review.openstack.org/21782409:48
*** claudiub has joined #openstack-infra09:56
*** maishsk has quit IRC09:56
*** coolsvap has quit IRC10:01
*** coolsvap has joined #openstack-infra10:02
*** Guest41206 has quit IRC10:03
*** rkukura_ has joined #openstack-infra10:03
*** yfried is now known as yfried|afk10:04
*** Sree has joined #openstack-infra10:04
*** maishsk has joined #openstack-infra10:05
*** maishsk has quit IRC10:05
*** sharikapongubala has joined #openstack-infra10:06
*** rkukura has quit IRC10:06
*** rkukura_ is now known as rkukura10:06
*** Zhongjun has quit IRC10:07
*** Sree has quit IRC10:08
*** maishsk has joined #openstack-infra10:08
*** yfried|afk is now known as yfried10:10
*** sharikapongubala has quit IRC10:10
*** mrmartin has joined #openstack-infra10:11
*** skolekonov has joined #openstack-infra10:13
*** jeadre has joined #openstack-infra10:17
*** dizquierdo has quit IRC10:18
*** claudiub has quit IRC10:18
openstackgerritMerged openstack-infra/groups: Remove local Leaflet library and use version 0.7.3  https://review.openstack.org/21860310:20
*** abregman|lunch is now known as abregman10:22
*** dtantsur|brb is now known as dtantsur10:23
*** marzif has quit IRC10:24
*** yfried is now known as yfried|afk10:25
openstackgerritMerged openstack-infra/groups: Use '#!/usr/bin/env bash' as shebang in script files  https://review.openstack.org/21860210:26
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip  https://review.openstack.org/21101610:29
*** btully has quit IRC10:29
*** yfried|afk is now known as yfried10:29
*** hashar has quit IRC10:29
openstackgerritMerged openstack-infra/groups: Cleanup and convert the README file to RST  https://review.openstack.org/21860110:30
*** Sree has joined #openstack-infra10:30
*** fhubik is now known as fhubik_brb10:30
*** dims has joined #openstack-infra10:30
*** Sree has quit IRC10:30
*** bharathm has joined #openstack-infra10:31
*** Sree has joined #openstack-infra10:31
openstackgerritEkaterina Chernova proposed openstack/requirements: Add django-formtools to global requirements  https://review.openstack.org/21922210:31
*** yamamoto has quit IRC10:32
*** ddieterly has joined #openstack-infra10:32
*** pc_m has joined #openstack-infra10:32
*** bharathm has quit IRC10:35
*** dims has quit IRC10:36
*** ddieterly has quit IRC10:37
*** maishsk has quit IRC10:39
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip  https://review.openstack.org/21101610:39
*** fhubik_brb has quit IRC10:40
*** bexelbie has quit IRC10:42
*** bexelbie has joined #openstack-infra10:43
*** marzif has joined #openstack-infra10:44
*** ociuhandu_ has joined #openstack-infra10:44
*** ociuhandu has quit IRC10:45
*** ociuhandu_ is now known as ociuhandu10:45
*** ZZelle has joined #openstack-infra10:48
*** haypo has joined #openstack-infra10:53
*** ParsectiX has quit IRC10:53
haypohi. my change https://review.openstack.org/#/c/213817/ was approved but sdague changed the commit message to add "(Pulled from gate, cinder still needs unit test fix before this can pass)"10:54
haypoi'm not aware of cinder issues. what is the problem?10:54
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Clean any yum cached data  https://review.openstack.org/21922710:54
sdaguehaypo: https://github.com/openstack/cinder/commit/5b45dcf3a097efa307e31ab7c8c50719d4d7696910:56
sdagueit's fixed now10:56
sdagueso you can delete that part and get tests run on it10:57
sdaguehaypo: you will notice after I did that, pep8 and python27 failed your patch, right?10:57
hayposdague: a cinder changed introduced a requirement to OpenSSL and it broke the gates?10:58
sdagueglanceclient used to require openssl10:58
*** Sree has quit IRC10:58
sdaguethe new release does not10:58
*** Sree has joined #openstack-infra10:59
sdaguecinder introduced code that used openssl and didn't add the requirement, because it worked when glanceclient installed it for it10:59
sdaguewhich then broke after the new glanceclient release10:59
hayposdague: ah no, i didn't notice the pep8 and py27 failures. i will see if it's now fixed ;)10:59
*** bexelbie is now known as bexelbie|lunch10:59
*** hrubi has quit IRC10:59
openstackgerritMarton Kiss proposed openstack-infra/groups: Remove duplicate declaration in groups ambassadors ask form.  https://review.openstack.org/21922911:00
sdagueyeh, based on what's happening in check, I think it is11:00
*** spzala has joined #openstack-infra11:01
*** marzif has quit IRC11:02
*** FallenPegasus has joined #openstack-infra11:02
*** aukhan has quit IRC11:03
*** Sree has quit IRC11:03
*** yfried is now known as yfried|afk11:04
*** vinay_yadhav has joined #openstack-infra11:06
*** FallenPegasus has quit IRC11:07
*** sharikapongubala has joined #openstack-infra11:07
*** btully has joined #openstack-infra11:07
openstackgerritMerged openstack-infra/groups: Remove duplicate declaration in groups ambassadors ask form.  https://review.openstack.org/21922911:08
hayposdague: oh. pep8 failed locally, "Error importing module cinder.volume.drivers.emc.emc_vmax_common: No module named OpenSSL"11:09
sdaguehaypo: rebase on master11:10
sdagueand rebuild your tox envs11:10
*** vinay_yadhav has quit IRC11:10
hayposdague: oh ok :) it's that simple, cool. thanks11:11
*** sharikapongubala has quit IRC11:12
*** slagle has joined #openstack-infra11:12
*** btully has quit IRC11:12
*** iraw_ has joined #openstack-infra11:13
openstackgerritJohn Trowbridge proposed openstack/diskimage-builder: [ironic-agent] Use svc-map for enabling agent  https://review.openstack.org/21887911:13
*** claudiub has joined #openstack-infra11:14
*** jaypipes has joined #openstack-infra11:14
*** ParsectiX has joined #openstack-infra11:17
dtantsurhey folks! could anyone check my suspection that gate-tempest-dsvm-ironic-pxe_ssh-dib runs on DIB from PyPI instead of a DIB patch in question?11:17
*** maishsk has joined #openstack-infra11:17
dtantsurI'm suspecting it because of https://review.openstack.org/#/c/219199 I don't see the results of the patch in the devstack logs11:17
sdaguedtantsur: yes, that's intentional11:18
dtantsuroh? so it's not possible to fix DIB gate?11:18
dtantsurwhat was the reasoning behind it?11:19
*** wenlock has quit IRC11:19
sdaguedtantsur: the reasoning was to decouple all of these things11:19
*** asettle has quit IRC11:20
sdaguebut apparently no one responsible for this all understands the gate architecture11:20
*** wenlock has joined #openstack-infra11:20
dtantsursdague, so, do you have any hints how to unbreak DIB gate, if it does not pick the change that fixes it?11:20
*** zeih__ has joined #openstack-infra11:21
*** lucasagomes has joined #openstack-infra11:21
dtantsurlucasagomes, failure on https://review.openstack.org/#/c/219199/ looks like we're not testing DIB from source on DIB itself (unless I'm confusing things)11:22
*** zeih has quit IRC11:22
lucasagomesdtantsur, lol seriously?11:22
dtantsurthat's what I suspect, I'm asking people to confirm :)11:23
lucasagomesouch hmm right11:23
* lucasagomes look at the logs11:23
*** hodos has joined #openstack-infra11:23
*** yfried|afk is now known as yfried11:25
*** dims has joined #openstack-infra11:25
openstackgerritSean Dague proposed openstack-infra/project-config: make gate-tempest-dsvm-ironic-pxe_ssh non voting  https://review.openstack.org/21924011:25
*** dizquierdo has joined #openstack-infra11:26
dtantsursdague, "no longer pulling from the open internet" is something that all ironic gates do, and probably it's not trivial to fix..11:27
sdaguedtantsur: I'm sure it's not trivial to fix, but if you want jobs to cross vote on other projects, you have to11:27
sdaguewe've gone through extradinary effort in building caching layers into nodepool for this reason11:28
dtantsursdague, fair enough, not sure why DIB does not support it (or it does?)11:28
sdagueno idea, someone else needs to figure out a sane architecture here, I just know what is currently done is not sane.11:29
*** dims has quit IRC11:29
*** hashar has joined #openstack-infra11:29
sdaguesorry for being grumpy, but this is the 5th or 6th "100% fail breaks everything" change I've looked at in 3 days11:29
sdagueso my patience is low11:29
dtantsursdague, no problem, I hear you11:30
dtantsurI just don't know how to make DIB work with infra mirrors (?)11:30
*** vinay_yadhav has joined #openstack-infra11:31
sdaguedtantsur: is there a reason you can't just build a static image perodically and use that?11:31
sdaguedoes ironic actually have to build the image every time?11:31
dtantsursdague, we'll probably do it with move to IPA (i.e. pretty soon). the problem is DIB itself: would be nice to have a job verifying that we can still build an ironic ramdisk11:32
dtantsur* IPA = ironic-python-agent11:32
lucasagomessdague, can we change the base OS for that job to something else?11:32
lucasagomesuntil we fix stuff on ubuntu?11:32
sdaguedtantsur: right, so that's a functional test job on dib11:32
*** thorst has joined #openstack-infra11:33
sdaguenot integration tests with tempest and nvoa11:33
*** wenlock has quit IRC11:33
*** fhubik has joined #openstack-infra11:33
*** ddieterly has joined #openstack-infra11:33
dtantsursdague, probably tempest is an overkill, but we still have to make sure that image we just built works11:33
sdaguedtantsur: then build just the test that does that, and run that in isolation. The current test architecture here is a big ball of mud.11:34
*** yfried is now known as yfried|afk11:35
dtantsuryeah, something like that...11:35
dtantsurlucasagomes, we need to rethink our testing with move to IPA11:35
dtantsurgood topic for the summit, also11:35
*** gordc has joined #openstack-infra11:36
sdaguelucasagomes: changing out the base os is probably no really helpful, plus, all the projects where that is changable are blocked by this11:36
sdaguewhere things went really wrong here is the -dib job is supposed to be using dib from git11:36
sdaguehowever no one implemented that11:36
lucasagomessdague, yeah I hear you. Let's make it non-vote then11:37
lucasagomesI will update the patch I put for DIB to map the ubuntu names to versions11:37
sdaguelucasagomes: yeh, that's https://review.openstack.org/21924011:37
sdaguefungi: you up for fast approve on https://review.openstack.org/219240  ?11:38
lucasagomessdague, thanks11:38
*** ddieterly has quit IRC11:38
sdaguelucasagomes: also, if you post up changes to devstack so that DIB supports libs from git, we can get the -dib job to do the right thing11:38
*** yfried|afk is now known as yfried11:38
sdaguebut the support isn't even in devstack at the moment11:38
sdagueso there was a massive disconnect somewhere when I told folks about spliting up these jobs11:39
*** bexelbie|lunch is now known as bexelbie11:39
sdaguebecause, the theory would be that the -dib job would be a version of the pxe_ssh job that used dib from git (which would be sensible). And in the rest of the jobs it's from the last release11:40
sdagueso dib can test new changes and make sure they won't break folks11:40
sdaguethat's how we do most of the client libs now11:40
sdagueexcept, apparently the message was lost intranslation, and only the job got renamed, without doing the other work11:40
*** ParsectiX has quit IRC11:41
lucasagomessdague, yeah seems a ball of wax, but I will take a look at it11:41
lucasagomes1) fix devstack to git clone and install dib 2) update the job to do it11:42
lucasagomesand ofc fix ubuntu11:42
fungianteaya: jyuso: i've reactivated the intel pci ci account now11:42
*** nadya has joined #openstack-infra11:42
*** glauco has joined #openstack-infra11:45
openstackgerritJoseph Lanoux proposed openstack-infra/project-config: Enable SSL in devstack for experimental checks.  https://review.openstack.org/18096811:45
*** e0ne has quit IRC11:46
sdaguelucasagomes: the devstack patches aren't huge, look at how they were done for oslo libs or novaclient11:46
sdaguelucasagomes: thanks for looking at it11:47
lucasagomessdague, will do! thanks11:47
mescanefhi! folks, regarding stackforge migration and its wiki page: https://wiki.openstack.org/wiki/Stackforge_Namespace_Retirement - what in case of fuel-plugins? lots of them are already there, and there was a idea, that for keeping "consistency" these fuel-plugin-* projects would *stay* in stackforge namespace. is that possible?11:49
*** jyuso has quit IRC11:49
*** jyuso1 has joined #openstack-infra11:49
mescaneffungi: mordred ^^11:49
*** nadya has quit IRC11:50
fungimescanef: you probably misunderstood the consistency argument. we said it was okay to create new fuel plugins in the stackforge namespace to keep them together for now, and then we'll move them all at the same time together into the openstack namespace in a couple months11:50
*** nadya has joined #openstack-infra11:50
*** pradk has joined #openstack-infra11:50
fungirather than creating new ones in the openstack namespace now while the old ones are still in the stackforge namespace for the next couple months11:51
openstackgerritMerged openstack-infra/project-config: make gate-tempest-dsvm-ironic-pxe_ssh non voting  https://review.openstack.org/21924011:51
anteayafungi: thank you11:53
*** sergmelikyan has joined #openstack-infra11:54
*** jyuso has joined #openstack-infra11:54
mescaneffungi: so every fuel-plugin-* we shall add to the wiki page, yes?11:54
*** rfolco has joined #openstack-infra11:55
sdaguelucasagomes: you have a bug up for the dib / sha256 mismatch?11:55
lucasagomessdague, yeah 1 sec, dtantsur opened one.11:55
lucasagomessdague, https://bugs.launchpad.net/diskimage-builder/+bug/149086411:56
openstackLaunchpad bug 1490864 in diskimage-builder "Ubuntu mirror file renaming broken "ubuntu" element" [Undecided,In progress] - Assigned to Lucas Alvares Gomes (lucasagomes)11:56
*** mase_x200 has joined #openstack-infra11:56
sdaguelucasagomes: great, thanks, I wanted to get that signature into our system11:56
*** jyuso1 has quit IRC11:58
*** jyuso has quit IRC11:58
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add query for sha256 dib / ironic issue  https://review.openstack.org/21924611:58
*** iraw_ has quit IRC12:01
*** nadya has quit IRC12:01
*** lucasagomes is now known as lucas-hungry12:01
* lucas-hungry brb's quick getting a sandwich 12:02
*** iraw_ has joined #openstack-infra12:02
*** ParsectiX has joined #openstack-infra12:03
*** yfried is now known as yfried|afk12:03
*** iraw_ has quit IRC12:06
*** yfried|afk is now known as yfried12:07
*** mwagner_lap has quit IRC12:08
*** iraw_ has joined #openstack-infra12:08
*** chlong_ has joined #openstack-infra12:09
*** annegentle has joined #openstack-infra12:11
openstackgerritJoseph Lanoux proposed openstack-infra/project-config: Enable SSL in devstack for experimental checks.  https://review.openstack.org/18096812:13
*** salv-orlando has quit IRC12:14
*** nicodemos has joined #openstack-infra12:15
EmilienMgood morning infra, I have an outstanding request for puppet CI: https://review.openstack.org/#/c/219078/ - thanks a lot :-)12:16
*** jcoufal has quit IRC12:17
*** vinay_yadhav has quit IRC12:17
*** trown|outttypeww is now known as trown12:18
*** e0ne has joined #openstack-infra12:18
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Add support for listing folder indexes  https://review.openstack.org/21420712:18
*** vinay_yadhav has joined #openstack-infra12:18
*** bharathm has joined #openstack-infra12:20
*** bswartz has joined #openstack-infra12:21
fungimescanef: yes, if your repository is active (you don't want it retired) and it's in the stackforge namespace now, it should go on the list of repos to migrate in that wiki article12:21
*** baoli has joined #openstack-infra12:21
*** salv-orlando has joined #openstack-infra12:22
openstackgerritMerged openstack-infra/elastic-recheck: add query for sha256 dib / ironic issue  https://review.openstack.org/21924612:23
*** salv-orl_ has joined #openstack-infra12:23
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul  https://review.openstack.org/21773212:23
*** bharathm has quit IRC12:24
*** dizquierdo has quit IRC12:25
*** dkranz has joined #openstack-infra12:26
*** yfried has quit IRC12:26
*** salv-orlando has quit IRC12:26
sdaguefungi: any idea if the ironic non voting change has propogated yet?12:27
*** abregman is now known as abregman|afk12:29
fungisdague: i'll check zuul's logs12:29
mescaneffungi: ok tnx12:29
*** aysyd has joined #openstack-infra12:30
EmilienMfungi: thx for +a12:30
*** burgerk has joined #openstack-infra12:30
*** dprince has joined #openstack-infra12:31
fungisdague: yep, it showed up nonvoting on a zuul config reload about 26 minutes ago12:31
*** cburroughs has joined #openstack-infra12:31
EmilienMcan we read zuul logs somewhere ?12:32
*** dims has joined #openstack-infra12:33
*** salv-orlando has joined #openstack-infra12:33
*** ddieterly has joined #openstack-infra12:34
openstackgerritsvasheka proposed openstack-infra/project-config: Change screenshots source destination for  gate-horizon-dsvm-integration job  https://review.openstack.org/21925412:34
sdaguefungi: thanks12:34
openstackgerritMerged openstack-infra/project-config: zuul/layout: make non-voting some trusty beaker jobs  https://review.openstack.org/21907812:34
*** woodster_ has joined #openstack-infra12:36
fungiEmilienM: unfortunately no, i have to ssh into the server and pull up the logs manually. we have a general distrust of logs for applications which might leak credentials12:36
*** salv-orl_ has quit IRC12:37
EmilienMfungi: ok no problem :-) - I was just curious to know12:37
*** ddieterly has quit IRC12:38
*** mase_x200 has quit IRC12:38
*** fhubik has quit IRC12:39
*** hrubi has joined #openstack-infra12:39
*** dkranz has quit IRC12:40
*** jcoufal has joined #openstack-infra12:40
*** lucas-hungry is now known as lucasagomes12:40
*** dkranz has joined #openstack-infra12:42
openstackgerritPaul Michali proposed openstack-infra/project-config: Create experimental jobs for VPN DevStack plugin  https://review.openstack.org/20988712:43
*** Sree has joined #openstack-infra12:44
*** puranamr has joined #openstack-infra12:44
*** hashar_ has joined #openstack-infra12:45
*** fedora has joined #openstack-infra12:45
*** amuller has joined #openstack-infra12:46
*** hashar has quit IRC12:46
*** dprince has quit IRC12:46
*** dprince has joined #openstack-infra12:48
*** nadya has joined #openstack-infra12:48
*** mase_x200 has joined #openstack-infra12:49
*** ddieterly has joined #openstack-infra12:49
*** hashar has joined #openstack-infra12:49
*** thiagop has joined #openstack-infra12:50
*** FallenPegasus has joined #openstack-infra12:50
*** sigmavirus24_awa is now known as sigmavirus2412:51
*** puranamr has quit IRC12:51
*** rlandy has joined #openstack-infra12:52
*** derekh has quit IRC12:52
*** hashar_ has quit IRC12:52
*** yamamoto has joined #openstack-infra12:54
*** FallenPegasus has quit IRC12:55
*** puranamr has joined #openstack-infra12:56
*** amotoki has quit IRC12:57
*** ddieterly has quit IRC12:58
*** iraw_ has quit IRC12:58
*** mase_x200 has quit IRC12:58
*** oomichi has quit IRC12:58
*** iraw_ has joined #openstack-infra12:58
*** fhubik has joined #openstack-infra12:58
*** tlian has joined #openstack-infra12:59
*** mase_x200 has joined #openstack-infra12:59
*** abregman|afk is now known as abregman13:00
*** bexelbie_ has joined #openstack-infra13:01
*** bexelbie has quit IRC13:02
*** oomichi has joined #openstack-infra13:03
*** mase_x200 has quit IRC13:03
*** salv-orlando has quit IRC13:04
*** samuelBartel has joined #openstack-infra13:04
*** k4n0 has quit IRC13:05
*** rossella_s has quit IRC13:05
*** rossella_s has joined #openstack-infra13:06
*** vinay_yadhav has quit IRC13:07
*** sergmelikyan has quit IRC13:07
*** mjturek1 has joined #openstack-infra13:07
*** vinay_yadhav has joined #openstack-infra13:07
*** miqui has joined #openstack-infra13:08
*** gordc has quit IRC13:09
*** sergmelikyan has joined #openstack-infra13:12
*** watanabe_isao has quit IRC13:13
*** nadya has quit IRC13:13
*** esker has quit IRC13:14
*** amitgandhinz has joined #openstack-infra13:16
*** sdake has quit IRC13:17
*** burgerk has quit IRC13:17
*** puranamr has quit IRC13:20
*** iraw__ has joined #openstack-infra13:20
*** iraw__ has quit IRC13:21
*** sdake has joined #openstack-infra13:21
*** julim has joined #openstack-infra13:22
*** iraw_ has quit IRC13:22
*** puranamr has joined #openstack-infra13:23
*** sigmavirus24 is now known as sigmavirus24_awa13:24
*** mriedem has joined #openstack-infra13:24
*** dkranz has quit IRC13:25
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Script to release cycle-with-intermediary  https://review.openstack.org/21927413:26
*** tiswanso has joined #openstack-infra13:26
*** changbl has quit IRC13:28
*** dkranz has joined #openstack-infra13:28
*** puranamr has quit IRC13:28
*** geoffarnoldX is now known as geoffarnold13:29
*** bhunter71 has joined #openstack-infra13:29
*** mtanino has joined #openstack-infra13:31
*** dizquierdo has joined #openstack-infra13:31
mordredfungi: I'm going to release shade - I do not think it should affect us negatively, but should allow us to unpin puppet on nodepool13:31
fungimordred: please do13:32
*** bdemers has quit IRC13:33
ttxclarkb, fungi: we'll need to enable the requirements cross-check jobs this week, ideally today, so progress needed on https://review.openstack.org/#/c/21730013:33
*** alivigni has joined #openstack-infra13:33
fungittx: yep, was mostly waiting to see if clarkb had replies to your comments on it13:34
mordredfungi: tag pushed13:34
*** bexelbie_ is now known as bexelbie13:35
*** bdemers has joined #openstack-infra13:36
*** fhubik is now known as fhubik_brb13:38
*** mwagner_lap has joined #openstack-infra13:38
*** Piet has quit IRC13:38
*** ddieterly has joined #openstack-infra13:38
*** ayoung has joined #openstack-infra13:39
*** sigmavirus24_awa is now known as sigmavirus2413:39
openstackgerritEvgeny Antyshev proposed openstack-infra/devstack-gate: Third-party rhel-based oses in devstack-gate  https://review.openstack.org/21502913:39
*** kiran-r has quit IRC13:40
*** kiran-r has joined #openstack-infra13:41
*** kiran-r has quit IRC13:41
*** fhubik_brb is now known as fhubik13:42
*** eharney has joined #openstack-infra13:44
*** jsavak has joined #openstack-infra13:45
*** edmondsw has joined #openstack-infra13:47
clarkbI didnt realize there were comments as its +1 I can address this morning13:48
*** sergmelikyan has quit IRC13:48
clarkbI didnt add to master because for other cross branch thing we typically only do check13:49
*** annegentle has quit IRC13:49
clarkbdevstack gate and tempest for example13:49
*** gordc has joined #openstack-infra13:52
ttxfungi, clarkb: could anyone add maishsk@gmail.com to openstack-planet-core at https://review.openstack.org/#/admin/groups/93,members ? Been very helpful reviewing those lately13:53
mordredttx: done13:54
fungittx: done13:54
ttxmaishsk: ^13:54
ttxwelcome to the group and thanks again for helping there13:54
*** sergmelikyan has joined #openstack-infra13:54
maishskttx: fungi: Thanks.13:55
maishskHappy to help out.13:55
fungi(seems adding someone to a gerrit group multiple times is a quietly successful no-op)13:55
ttxgerrit is reentrant13:55
*** zul has quit IRC13:56
*** zul has joined #openstack-infra13:56
*** glauco has quit IRC14:01
*** mriedem is now known as mriedem_meetings14:01
*** glauco has joined #openstack-infra14:01
openstackgerritPaul Michali proposed openstack-infra/project-config: Create experimental jobs for VPN DevStack plugin  https://review.openstack.org/20988714:02
*** Kennan has joined #openstack-infra14:03
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Use pxe_ipa job in Nova  https://review.openstack.org/21928514:03
*** Kennan2 has quit IRC14:03
*** ihrachys has quit IRC14:06
*** burgerk has joined #openstack-infra14:06
*** bharathm has joined #openstack-infra14:08
*** sharikapongubala has joined #openstack-infra14:09
openstackgerritMerged openstack-infra/release-tools: clarify duties in library release process  https://review.openstack.org/21901714:10
*** peristeri has quit IRC14:13
*** zeih__ has quit IRC14:13
*** bharathm has quit IRC14:13
*** ParsectiX has quit IRC14:13
*** sharikapongubala has quit IRC14:13
*** geoffarnold has quit IRC14:14
*** angdraug has joined #openstack-infra14:15
fungittx: we still have a summit.openstack.org server at 2001:4800:7810:512:3bc3:d7f6:ff04:94fc even though that name in dns is now a cname to static.openstack.org14:18
*** ihrachys has joined #openstack-infra14:18
fungittx: was there still anything on the old summit.o.o server you needed, or can i delete it now?14:19
fungittx: this is _not_ the design-summit-prep.openstack.org server, but rather its predecessor14:20
openstackgerritMonty Taylor proposed openstack-infra/shade: Pass timeout to session, not constructors  https://review.openstack.org/21929014:20
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different  https://review.openstack.org/21929114:20
mordredfungi, clarkb: ^^ not urgent - just two cleanups that came out of yesterday's real patch14:20
fungimordred: do we need another release with those ^ before we can unleash puppet agent on nodepool again?14:20
*** amotoki has joined #openstack-infra14:20
fungiahh, sounds like no14:21
mordredthey're just to clean up behind myself while I'm thinking about it still14:21
fungigreat idea14:21
mordredfungi: while I have your attention ... https://review.openstack.org/#/c/212177/14:21
*** wenlock has joined #openstack-infra14:22
*** links has quit IRC14:23
*** annegentle has joined #openstack-infra14:25
ttxfungi: I'm only using design-summit-prep now, so you can scrap the old summit.o.o server14:25
ttx(if those are different ;)14:25
fungittx: thanks! yeah, we're getting tight on quota there so every little bit counts14:25
ttxI thought it was dead already14:26
fungittx: looks like the last time you logged into it was april 2314:28
rcarrillocruzfungi: heya, can you advise why https://review.openstack.org/#/c/211925/ wasn't merged by Jenkins?14:28
rcarrillocruzchange it depends on already merged, dunno what happened14:28
*** dkranz has quit IRC14:29
*** regXboi has joined #openstack-infra14:29
fungircarrillocruz: the change it depends on doesn't share a change queue with it, and the depending change was approved before the dependency merged14:29
*** vinay_yadhav has quit IRC14:29
openstackgerritMerged openstack-infra/glean: Create systemwide lock for glean  https://review.openstack.org/21217714:30
fungi211923 didn't merge until 08-27 16:34 but 211925 was approved before that, at 08-27 16:1014:30
*** shardy_ has joined #openstack-infra14:30
fungircarrillocruz: so you need ttx to remove and re-add his approval on 21192514:30
*** shardy has quit IRC14:32
rcarrillocruzttx: ^14:32
*** sharikapongubala has joined #openstack-infra14:32
rcarrillocruzcould you please have a look?14:32
*** maishsk has quit IRC14:33
fungittx: and yeah, this old summit.o.o server does seem to have been the same, consistent with my recollection. it has a /var/www/summitsched with a wsgi app in there14:33
fungiso deleting it now14:33
*** shardy_ has quit IRC14:35
openstackgerritMonty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface  https://review.openstack.org/21424314:36
*** shardy has joined #openstack-infra14:36
*** zz_dimtruck is now known as dimtruck14:37
*** vinay_yadhav has joined #openstack-infra14:37
*** arxcruz has quit IRC14:41
*** afazekas__ has quit IRC14:41
*** boris-42 has joined #openstack-infra14:42
*** tiswanso has quit IRC14:44
*** vinay_yadhav has quit IRC14:44
fungiclarkb: https://jenkins.openstack.org/pluginManager/ warns that the ssh slaves plugin 0.21 installed there uses different configuration than 1.10. did you need to make any adjustments when upgrading previously from really old versions of it? should we just leave the ancient version there?14:44
*** tiswanso has joined #openstack-infra14:44
*** vinayyadhav has joined #openstack-infra14:45
*** shardy_ has joined #openstack-infra14:47
*** hdd has joined #openstack-infra14:47
*** shardy has quit IRC14:48
*** adduarte has quit IRC14:50
clarkbfungi: I didn't that seems like a really old version though14:51
fungiyep, like really, really, really old14:51
fungisort of thinking i won't actually upgrade it until things quiet down, maybe on a weekend14:51
clarkbsounds good14:51
clarkbespecially since it doesn't have the problem with many adds and removes14:52
*** shardy_ has quit IRC14:52
*** jlanoux has quit IRC14:52
*** sharikapongubala has quit IRC14:53
*** geoffarnold has joined #openstack-infra14:53
*** glauco has quit IRC14:53
*** shardy has joined #openstack-infra14:53
*** armax has joined #openstack-infra14:53
*** sdake_ has joined #openstack-infra14:53
clarkbttx: fungi I realize now I probably want to restrict those new stable/liberty jobs to the master branch in zuul14:53
fungithat sounds like a good idea, agreed14:54
clarkbso making that change and the commit message updates and will push new patch in a minute14:54
*** tonytan4ever has joined #openstack-infra14:56
*** kgiusti has joined #openstack-infra14:56
openstackgerritClark Boylan proposed openstack-infra/project-config: Add stable/liberty crosscheck jobs for reqs  https://review.openstack.org/21730014:57
clarkbttx: fungi ^14:57
*** links has joined #openstack-infra14:57
*** sdake has quit IRC14:58
*** jistr is now known as jistr|call14:58
*** angdraug_ has joined #openstack-infra14:58
*** erlon has quit IRC14:58
clarkbStevenK: if you are still awake please let me know which bits of the scripting I can help with14:58
clarkbStevenK: either running scripts or debugging or writing new ones etc14:58
clarkbStevenK: the big ones I am curious about are the populate transifex from zanata script(s) and the daily update projects from zanata scripts14:59
*** bknudson has joined #openstack-infra14:59
*** angdraug has quit IRC15:00
*** daemontool_ has quit IRC15:01
*** fhubik is now known as fhubik_brb15:01
*** flepied has joined #openstack-infra15:05
openstackgerritMarton Kiss proposed openstack-infra/groups: Enable message email notification  https://review.openstack.org/21930715:05
*** sdake_ is now known as sdake15:06
*** flepied1 has quit IRC15:06
*** burgerk has quit IRC15:09
*** vivekd has joined #openstack-infra15:12
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: don't assume we will get an existing tag  https://review.openstack.org/21930815:12
*** salv-orlando has joined #openstack-infra15:13
*** pblaho has quit IRC15:13
*** zeih has joined #openstack-infra15:13
*** yuanying has quit IRC15:13
*** cbader has joined #openstack-infra15:14
*** bexelbie has quit IRC15:14
*** gmann has quit IRC15:14
openstackgerritSHIGEMATSU Mitsuhiro proposed openstack-infra/project-config: Gate infra-ansible with gate-infra-ansible job  https://review.openstack.org/21918515:15
*** yamahata has joined #openstack-infra15:16
*** links has quit IRC15:16
*** dims has quit IRC15:17
*** zeih has quit IRC15:18
*** salv-orlando has quit IRC15:18
*** vinayyadhav has quit IRC15:18
*** vinayyadhav has joined #openstack-infra15:19
*** salv-orlando has joined #openstack-infra15:19
*** mjturek1 has quit IRC15:22
*** krickwix has joined #openstack-infra15:23
*** annegentle has quit IRC15:23
*** annegentle has joined #openstack-infra15:23
*** fhubik_brb is now known as fhubik15:24
*** gmann has joined #openstack-infra15:24
*** sputnik13 has joined #openstack-infra15:25
openstackgerritKurt Taylor proposed openstack-infra/infra-specs: Host an initial CI monitoring dashboard service  https://review.openstack.org/19443715:25
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Use Ironic pxe_ipa job in Nova  https://review.openstack.org/21928515:25
*** salv-orlando has quit IRC15:26
jrollsdague: ^ I think that's what I want, I'm still a bit unsure about project-config things15:26
*** jlanoux has joined #openstack-infra15:27
sdaguejroll: ok, will look in a second15:27
openstackgerritMichael Jeanson proposed openstack-infra/jenkins-job-builder: Add 'do-not-fingerprint' to 'copyartifact' builder  https://review.openstack.org/21931215:27
jrollyeah, no rush. thanks :)15:28
*** flepied has quit IRC15:29
*** yuanying has joined #openstack-infra15:30
*** stevemar has joined #openstack-infra15:30
*** flepied has joined #openstack-infra15:31
sdaguejroll: yeh, that looks fine15:33
*** vinayyadhav has quit IRC15:34
*** bharathm has joined #openstack-infra15:35
*** ihrachys has quit IRC15:36
openstackgerritMonty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions  https://review.openstack.org/18317615:36
*** puranamr has joined #openstack-infra15:36
*** flepied has quit IRC15:36
clarkbfungi: are all but jenkins.o.o done now?15:37
*** tiswanso has quit IRC15:37
fungiclarkb: yep15:38
clarkbI am checking thread counts now, 04 still looks great only ~10 forremoting threads15:38
*** puranamr has quit IRC15:38
*** salv-orlando has joined #openstack-infra15:38
clarkblifeless sdague: from this point forwrad we don't expect to see the duplicate node usage problem assuming the jenkins ssh slaves plugin upgrades fixed things15:39
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder: ironic-agent: ensure dmidecode and ipmitool are installed  https://review.openstack.org/21152115:40
sdagueclarkb: \o/15:40
clarkblifeless: sdague so let us know if you see that fail case again15:40
*** dkranz has joined #openstack-infra15:40
clarkbsdague: and you saw that the pbr thing was related to sqlalchemy migrate and keystonemiddleware and only affected jobs doing stuff with kilo?15:40
sdagueclarkb: yep15:41
sdagueI gave that to mriedem last night, he fixed it15:41
*** tiswanso has joined #openstack-infra15:41
sdagueit was also his fault, so all's good now15:41
*** burgerk has joined #openstack-infra15:41
sdagueI got to say - http://status.openstack.org//elastic-recheck/ is just one meteor strike after another recently15:41
*** glauco has joined #openstack-infra15:42
*** sridhar_ram has joined #openstack-infra15:42
clarkbthats what happens when the entire project is trying to push code through quickly in order to get into the rlease ...15:42
clarkbat least it seems like 3rd milestone timeframe is pretty typically crazyness15:42
mtreinishsdague: but there's no dust cloud :)15:42
sdagueclarkb: yeh15:43
clarkbfungi: just checked thread counts across all but jenkins.o.o andthey look good15:43
sdaguealso the fact that these things exist in a fragile state for a long time, and no one fixes them during the good times15:43
sridhar_raminfra team - need your help - https://review.openstack.org/#/c/211405/ has been Approved yesterday but it hasn't merged and I couldn't find an entry on zuul as well15:44
sridhar_ramplease help15:44
clarkbat least in the case of jenkins weirdness upstream ignored my bug for forever and appears someone may have fixed it independently after finding a different bug injira15:44
clarkbso we were tracking it there was just fail to communicate with upstream15:44
*** dangers_away is now known as dangers15:45
clarkbsridhar_ram: the depends on was merged after you approved that change, if they do not share a gate queue then it will not be automatically gated15:46
fungisridhar_ram: that change and its dependency don't share a change queue, and you approved it before the one it depends on merged15:46
*** vikram has joined #openstack-infra15:46
vikramHi there15:46
fungisridhar_ram: someone needs to remove and reapply that approval vote15:46
sridhar_ramfungi: clarkb: thanks ..15:46
vikramCan someone please help me to understand the release process of a big tent project15:47
sridhar_ramwill ask the reviewer to -A and +A15:47
clarkbvikram: I think you want to talk to dhellmann lifeless and or ttx15:47
vikrammy project belongs to neutron big tent..15:47
vikramclarkb: thanks for your guidance15:47
sridhar_ramfungi: for next time, what is the correct procedure ? I thought merging the dependency first is the correct order15:49
*** samuelBartel has quit IRC15:49
*** j_king has left #openstack-infra15:50
*** bknudson has quit IRC15:50
*** abregman has quit IRC15:51
clarkbsridhar_ram: it is, but you have to approve the child after the parent merges if they do not share a gate queue15:51
*** puranamr has joined #openstack-infra15:51
clarkbsridhar_ram: if you approve it before the dependency merges then zuul nops because the dependency has not merged yet15:51
*** mriedem_meetings is now known as mriedem15:51
SwansonHad been looking at Ramy's CI stuff.  Saw anteaya mention puppet-openstackci as a common CI solution.  Is there documentation for this?  I see some pages marked as inaccurate...15:52
sridhar_ramclarkb: got it, zuul doesn't retry ?15:52
clarkbsridhar_ram: no15:52
clarkbits event based and that event happened and was processed15:52
asselin_Swanson, updating docs is wip15:52
*** spzala has quit IRC15:53
sridhar_ramclarkb: I see. All is well, lesson learnt. thanks team!15:53
*** fhubik has quit IRC15:53
*** sharikapongubala has joined #openstack-infra15:53
fungivikram: you should read http://docs.openstack.org/project-team-guide/ and also talk to the neutron team to see where you fit in15:53
vikramfungi: ok15:54
*** Swami has joined #openstack-infra15:54
fungivikram: since it sounds like your project is related to neutron, and perhaps in similar ways to lots of others15:54
*** yfried has joined #openstack-infra15:55
*** ivar-lazzaro has joined #openstack-infra15:55
vikramfungi: Since this will my first release.. so checking out things :-)15:55
*** sshnaidm has quit IRC15:55
*** ivar-lazzaro has quit IRC15:55
*** ivar-lazzaro has joined #openstack-infra15:56
*** alivigni has quit IRC15:57
*** flepied has joined #openstack-infra15:57
*** juggler has joined #openstack-infra15:57
*** sharikapongubala has quit IRC15:57
*** ociuhandu_ has joined #openstack-infra15:58
*** esker has joined #openstack-infra15:59
*** esker has quit IRC16:00
*** zhenguo has quit IRC16:00
*** skylerberg has joined #openstack-infra16:00
*** eharney has quit IRC16:01
*** esker has joined #openstack-infra16:01
openstackgerritAntoine Musso proposed openstack-infra/nodepool: doc: FLUSH PRIVILEGES; is unneeded  https://review.openstack.org/21932416:01
*** puranamr has quit IRC16:02
*** sridhar_ram has quit IRC16:03
*** ajmiller has joined #openstack-infra16:04
*** sridhar_ram has joined #openstack-infra16:04
*** baoli has quit IRC16:04
*** hashar has quit IRC16:04
*** baoli has joined #openstack-infra16:05
*** hashar has joined #openstack-infra16:05
*** dims has joined #openstack-infra16:05
*** sdake_ has joined #openstack-infra16:05
*** e0ne has quit IRC16:05
*** jtomasek has quit IRC16:06
*** dtantsur is now known as dtantsur|afk16:06
*** bknudson has joined #openstack-infra16:06
*** ociuhandu_ has quit IRC16:07
*** sdake has quit IRC16:09
clarkbany objections to me deleting the old git08 now? the centos7 machine has been doing well16:09
clarkbI would also like to replace git0716:09
*** changbl has joined #openstack-infra16:10
*** baoli has quit IRC16:10
clarkbthat will give me a chance to test my latest iptables module update on rax images16:10
clarkbneed to go test it on hpcloud centos too16:10
*** fhubik has joined #openstack-infra16:11
*** Sukhdev has joined #openstack-infra16:11
*** fhubik has quit IRC16:11
*** burgerk has quit IRC16:11
*** Piet has joined #openstack-infra16:12
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Switch ironic full tempest to use IPA  https://review.openstack.org/21933016:13
*** bcourt has quit IRC16:13
jrollmtreinish: ^^ gets rid of the dib stuff there16:13
mtreinishjroll: cool, +116:13
greghaynesjroll: dib method is broken?16:13
JayFjroll: is that 'devstack-virtual-ironic' section new?16:13
JayFjroll: it looks 10x more slicker way to configure that than before16:13
jrollgreghaynes: yes, upstream ubuntu broke the heck out of it16:14
jrollJayF: I don't remember how new16:14
greghaynesjroll: oh, you mean the 403 from cloud-images.ubuntu?16:14
jrollgreghaynes: it's not dib, it's dib + ubuntu I think? lucasagomes is digging16:14
jrollmaybe, idk16:14
lucasagomesgreghaynes, couple of errors there...16:15
lucasagomesgreghaynes, the ubuntu element is broken due: https://review.openstack.org/#/c/219199/16:15
greghaynesI am +1 on removing the dib test btw, just surprised it was broken16:15
lucasagomesgreghaynes, and the pxe_ssh-dib job wasn't testing DIB from source16:15
*** jaypipes has quit IRC16:16
lucasagomesso it actually never tested building disk from source to see if changes in dib broke ironic16:16
*** eharney has joined #openstack-infra16:16
lucasagomesgreghaynes, later on I found that "pip install -e <diskimage-builder repo>" is also broken, that leaves the DIB commands partially installed16:16
*** bcourt has joined #openstack-infra16:16
greghaynesyes, -e (documentedly) not usable16:17
openstackLaunchpad bug 1491035 in diskimage-builder "Can't install diskimage-builder via "pip install -e"" [Undecided,New]16:17
lucasagomesgreghaynes, right, that's what the libs in devstack does when installing things from source16:17
*** jlanoux has quit IRC16:17
lucasagomesso to fix devstack we also need to fix that pip install -e :-(16:17
lucasagomesso it's a ball of wax16:18
lucasagomesI will put a workaround on devstack first and leave that pip install -e for later16:18
greghaynesIts not that hard to fix, I just didnt because im not really a -e fan anyhow16:18
lucasagomesgreghaynes, right, well that's needed :-/16:18
greghaynesyea, I might do that if someone doesnt beat me to it then16:18
*** enikanorov__ has joined #openstack-infra16:18
lucasagomesgreghaynes, heh please do16:19
*** shashank_hegde has joined #openstack-infra16:19
*** ifarkas has quit IRC16:20
lucasagomesgreghaynes, btw where it's documented that -e doesn't work for dib?16:20
*** burgerk has joined #openstack-infra16:20
*** esker has quit IRC16:20
greghayneslucasagomes: trying to find that now... I just know we had a big deal about this and that was the resolution...16:20
*** tjones has joined #openstack-infra16:20
*** dolphm has joined #openstack-infra16:20
clarkbit is really hard to boot servers in hpcloud ...16:21
*** enikanorov has quit IRC16:21
*** dizquierdo has quit IRC16:21
*** sdake_ is now known as sdake16:21
fungiclarkb: i am wholly in favor of allowing centos 7 to take over more of the cluster16:21
clarkbyou have to first use neutron client to get a network lits16:22
clarkbthen go back over to nova client to boot a node passing in the network16:22
clarkband when thats done use neutron client again to make a floating ip then associate it16:22
clarkboh and nova boot says HTTP 500 unknown error :/16:23
greghayneslucasagomes: hrm, having a hard time locating, so sounds like we failed at documenting well ;)16:25
lucasagomesgreghaynes, heh well I opened that bug16:25
lucasagomessomehow it's documented that it doesn't work now16:25
clarkbsorry you hav eto use nova client for the floating IPs because neutron client thinks creat has too few args16:25
*** jaypipes has joined #openstack-infra16:27
*** jistr|call has quit IRC16:28
*** tonytan4ever has quit IRC16:28
*** dannywilson has joined #openstack-infra16:28
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin  https://review.openstack.org/17393216:29
clarkbnibalizer: as soon as centos7 is done updating in hpcloud I will be testing my modifications to the iptables module16:30
*** dannywilson has quit IRC16:30
greghaynescrinkle: Is your launchpad ssh key correct?16:31
greghaynescrinkle: I am about to have two boxes for you16:31
*** jeadre has quit IRC16:32
*** haypo has left #openstack-infra16:32
crinklegreghaynes: i don't have an ssh key in launchpad so whatever you're looking at is probably not correct16:33
*** jeadre has joined #openstack-infra16:33
greghaynescrinkle: heh, im not looking at anything yet, just asking before I go hunting16:33
greghaynes(you should totally add one to lp)16:34
crinklegreghaynes: okay https://launchpad.net/~krinkle/+sshkeys16:35
greghaynescrinkle: awesome16:35
docaedoyolanda: thanks for your feedback on https://review.openstack.org/#/c/21889816:35
docaedoyolanda: have a question for you on that topic (and for rest of infra team to)16:35
docaedoWe are debating whether the script that creates a JSON file full of the assets should live in infra repo, or app-catalog repo (ala https://review.openstack.org/#/c/218670)16:36
clarkbnibalizer: notify implies before right?16:36
clarkbdocaedo: if its an app catalog utility regardless of config management system I would put it there16:37
docaedoThis is not super long term, but until we have a more complete backend for app-catalog (i.e. something more than a flat yaml file), we want to have reasonable way to retrieve the assets from the horizon plugin16:37
clarkbdocaedo: if it is related to some puppet bug or behavior it should go in the puppet module16:37
*** trown is now known as trown|lunch16:37
docaedoclarkb: cool, that makes it a very easy decision, thank you!16:38
*** ajmiller has quit IRC16:38
*** pblaho has joined #openstack-infra16:39
nibalizerclarkb: i think notify implies require16:40
clarkbnibalizer: I think we mean the same thing, it sets up an ordering16:41
*** thedodd has joined #openstack-infra16:41
clarkbsuch that the notifier happens before the notified resource16:41
openstackgerritKevin Fox proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889816:41
*** MarkAtwood has joined #openstack-infra16:45
*** ashleighfarnham has joined #openstack-infra16:45
openstackgerritKevin Fox proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889816:45
openstackgerritKaitlin Farr proposed openstack/requirements: Add Castellan to global requirements  https://review.openstack.org/18487416:45
*** ashleighfarnham has quit IRC16:46
*** ashleighfarnham has joined #openstack-infra16:46
*** skylerberg has quit IRC16:46
*** sergmelikyan has quit IRC16:48
*** david-ly_ has joined #openstack-infra16:49
*** david-lyle has quit IRC16:49
*** amuller is now known as amuller_afk16:49
*** sergmelikyan has joined #openstack-infra16:50
*** tonytan4ever has joined #openstack-infra16:50
*** geoffarnold has quit IRC16:50
*** amotoki has quit IRC16:52
*** ajmiller has joined #openstack-infra16:52
akraminskyclarkb, can we add in a block_migrate_cinder_iscsi compute feature in the tempest config?16:54
*** tjones has quit IRC16:54
*** geoffarnold has joined #openstack-infra16:54
*** ajmiller has quit IRC16:54
clarkbakraminsky: maybe? mtreinish is a better person to ask, I am not sure what tempests rules for feature flags are16:55
mtreinishclarkb: well that flag exists, I think it's a devstack question for setting it to true16:56
mtreinishbut that would only be on the multi node job16:56
clarkbpabelanger: nibalizer ok I have more infos, the problem is systemd doesn't appear to be loading the iptables service config into itself (doing a systemctl daemon-reload) and that is why it fails16:56
clarkbpabelanger: I am not sure how to debug that furhter? maybe a package bug?16:56
akraminskymtreinish, i don't think there is a flag for it yet16:56
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source  https://review.openstack.org/21935116:57
lucasagomessdague, ^ and https://review.openstack.org/#/c/219349/16:57
lucasagomessdague, not sure the project-config patch is correct tho. I'm not familiar with the yaml files there16:57
mtreinishakraminsky: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/config.py#n36116:57
clarkbpabelanger: but maybe also a bug in puppet, perhaps its systemd service provider should do a daemon-reload prior to each service state change? that is very shotgun approach though16:58
akraminskymtreinish, oh I thought the flags get set in devstack/lib/tempest with the iniset call16:58
sdaguelucasagomes: https://review.openstack.org/#/c/219349 looks right16:58
lucasagomessdague, yeah that I tested locally. But the project config which depends on this one I'm not 100% sure16:59
mtreinishakraminsky: that's how they get set if the configuration differs from the default value of the option. Devstack is the configuration engine more or less16:59
mtreinishakraminsky: but the options are all defined in tempest17:00
*** julim has quit IRC17:00
*** markus_z has quit IRC17:00
*** Sukhdev has quit IRC17:00
sdaguelucasagomes: yeh, I think project-config is good as well17:01
lucasagomeso wow17:01
lucasagomessdague, ok thanks very much!17:01
*** skylerberg has joined #openstack-infra17:01
sdagueclarkb: you want to take a look?17:01
*** hashar has quit IRC17:01
lucasagomesI will see if I can fix the pip install -e tomorrow and the ubuntu element for DIB17:01
lucasagomesthat should fix it all17:02
*** ZZelle_ has joined #openstack-infra17:02
akraminskymtreinish, yeh I think that's what I'd like to do is configure it, just so upstream's default isn't true..17:02
*** glauco has quit IRC17:02
*** puranamr has joined #openstack-infra17:02
*** julim has joined #openstack-infra17:03
*** glauco__ has joined #openstack-infra17:03
*** unicell has quit IRC17:03
*** w_ has joined #openstack-infra17:03
*** ajmiller has joined #openstack-infra17:03
*** rossella_s has quit IRC17:06
*** sdake_ has joined #openstack-infra17:06
*** rossella_s has joined #openstack-infra17:06
clarkblucasagomes: I think your -z condition is wrong17:06
*** sdake has quit IRC17:06
clarkblucasagomes: should be -n ya?17:06
lucasagomesclarkb, my bash scripting is quite bad17:07
*** sdake has joined #openstack-infra17:07
*** olaph has quit IRC17:07
lucasagomeslemme do some tests locally17:07
* lucasagomes thought -z was for non set env vars17:07
clarkbI think it should be -n17:07
clarkblucasagomes: it is17:07
*** david-ly_ has quit IRC17:07
*** alivigni has joined #openstack-infra17:07
clarkblucasagomes: but the code in that block assumes it is set17:07
clarkbso you need to invert and use -n17:07
lucasagomesclarkb, ohh good catch17:07
lucasagomesclarkb, end of the day sorry... I will update the patch17:08
lucasagomesclarkb, that's correct yes! updating...17:09
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin  https://review.openstack.org/17393217:09
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source  https://review.openstack.org/21935117:10
*** vivekd_ has joined #openstack-infra17:10
*** sdake_ has quit IRC17:10
*** vivekd has quit IRC17:11
*** vivekd_ is now known as vivekd17:11
*** puranamr has quit IRC17:12
*** sergmelikyan has quit IRC17:12
*** shashank_hegde has quit IRC17:12
*** krickwix has quit IRC17:13
*** lucasagomes is now known as lucas-dinner17:14
*** sergmelikyan has joined #openstack-infra17:14
glauco__I have a question about gerrit: one gerrit-initial-init shouldn't it be executing the reindex only if the secondary index type was set?17:14
glauco__Because, reading gerrit's source code, it says it will fail if the index type is SQL17:15
*** zeih has joined #openstack-infra17:15
*** w_ is now known as olaph17:15
glauco__please, take a look on line 9517:15
mmmporkhey folks i'm seeing more errors in the jenkins logs and i'm wondering if someone can help me to troubleshoot17:15
mmmporkfor my particular patch17:15
mmmpork"sha256sum: standard input: no properly formatted SHA256 checksum lines found"17:15
*** jerryz has joined #openstack-infra17:16
anteayaclarkb: fungi would like some comments here, did you want to respin then I will approve? https://review.openstack.org/#/c/217300/2/zuul/layout.yaml17:16
mmmporkit's a minor patch, these have passed before, but since last night there was some issue connecting to the ubuntu cloud17:16
mmmporki've looked at the logs of all the things erroring out and they all have this same checksum error17:17
*** ihrachys has joined #openstack-infra17:17
*** tjones has joined #openstack-infra17:17
greghaynesmmmpork: looks like ubuntu broke us17:18
*** tonytan4ever has quit IRC17:18
*** sergmelikyan has quit IRC17:19
mmmporkdid something change on the ubuntu boxes? like a system update or something?17:19
mmmpork"boxes" whatever they are ;)17:19
greghaynesmmmpork: I think the format of their sha256sums file changed17:19
mmmporkah ok17:19
fungianteaya: clarkb: i was mostly asking whether the comments would help, or if we were going to remember when to rip that back out17:20
*** zeih has quit IRC17:20
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Switch ironic full tempest to use IPA  https://review.openstack.org/21933017:20
openstackgerritSean Dague proposed openstack-infra/project-config: enable nova v20 compat testing jobs  https://review.openstack.org/21937017:20
anteayafungi: depends on the definition of we17:20
anteayayou and clarkb probably will know when it should come out but not remember17:20
jrollsdague: 219330 there is a blocker to working on this tempest full thing17:20
*** ybathia has joined #openstack-infra17:20
anteayaAJeager will come the file and have to ask17:21
mmmporkgreghaynes: i'm happy to look into it but i'm not quite sure where to start :) if you could point me in the right direction i'm happy to see if i can figure it out17:21
*** bharathm has quit IRC17:21
anteayafungi: clarkb I can live with or without comments, just wanted clarkb to offer an opinion before I close the door17:21
greghaynesmmmpork: https://review.openstack.org/#/c/219199/ looks like a fix from lucas-dinner17:21
anteayafungi: AJeager will comb the file17:22
anteayaI can't type17:22
sdaguejroll: +217:22
anteayait is like I have my arms full of baby17:22
mmmporkoh excellent17:22
openstackgerritClint Adams proposed openstack-infra/infra-specs: WIP - tracking of priority efforts in yaml  https://review.openstack.org/21937217:22
mmmporkhrm although looks like his build failed, i'll take a look17:22
*** b10n1k_ has joined #openstack-infra17:23
greghaynesmmmpork: yea, that is a second ironic fail that I think he was looking at17:23
mmmporkoh man17:23
greghayneswell, second dib + ironic fail17:23
*** tjones has left #openstack-infra17:24
mmmporkyeah.. looks like devstack couldn't launch so i was seeing if i could figure out anything more :)17:24
mmmporki "love" how it says "the setup script failed" err ok17:24
mmmporki still see the checksum fail in the devstack log17:25
*** angdraug_ has quit IRC17:25
*** achanda has joined #openstack-infra17:25
mmmporkoh wait but it's getting a new thingy ahh ok17:25
mmmporki can read!17:25
clarkbfungi anteaya I can propose revert today and wip it17:26
fungiclarkb: i like that solution even better than adding comments17:26
mmmporkhrm.. still seeing that error in the devstack log17:26
anteayaclarkb: hey that works17:26
* anteaya +As17:27
*** tonytan4ever has joined #openstack-infra17:27
greghaynesjroll: left review on https://review.openstack.org/#/c/219330/17:27
pc_manteaya: hi17:28
clarkbanteaya I have two babies currently. one rocked by a foot and the other being fed17:28
clarkb3 things at once17:28
anteayaclarkb: you multitasking father you17:28
anteayaclarkb: and I'm the one who can't type17:28
anteayapc_m: how are you today?17:29
greghaynesjroll: er, nvm, DIB doesnt do -full17:29
anteaya<-- train wifi for those who care17:29
pc_manteaya: nice. And you?17:29
jrollgreghaynes: heh yeah17:29
anteayapc_m: good thanks, love the train17:29
pc_manteaya: train?17:29
anteayapc_m: to montreal, puppet sprint tomorrow17:29
*** sergmelikyan has joined #openstack-infra17:30
*** sergmelikyan has quit IRC17:30
*** sharikapongubala has joined #openstack-infra17:30
pc_manteaya: cool!17:30
pc_manteaya: Had a question for ya...17:30
anteayapc_m: thought so17:30
* anteaya listens17:30
pc_manteaya: Is there an IRC channel for neutronclient development questions?17:30
*** unicell has joined #openstack-infra17:30
anteayanot that I'm aware17:30
anteayajust -neutron17:30
anteayayou seem to be low on reasources there17:31
sdaguegreghaynes: so I think you folks should figure out how to break up the test matrix more, because it seems like dib should have some narrow test of "I built a thing that can boot".17:31
pc_manteaya: OK. Will see if anyone responds there (I posted already). Thanks.17:31
greghaynessdague: That review exists, the problem is were hitting lots of issues because of network fails because we only have upstream mirrors to use17:32
anteayapc_m: sure, approvers for neutronclient are neutron core: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/python-neutronclient.config17:32
greghaynessdague: so our ability to increase test coverage has kind of stalled17:33
*** adduarte has joined #openstack-infra17:33
pc_manteaya: Seeing odd operation, so looking for someone with experience in that area.17:33
*** maishsk has joined #openstack-infra17:33
*** zz_ja has quit IRC17:34
*** maurosr has quit IRC17:34
sdaguegreghaynes: why can't you stash the base image in infra somewhere?17:34
greghaynesIts not just base images - package repo's seem to fail more17:35
*** sharikapongubala has quit IRC17:35
*** puranamr has joined #openstack-infra17:35
clarkbalso our images are 16GB huge need to stash less not more on them17:35
anteayapc_m: yup  I support that but I have no idea whom that would be17:35
greghaynesoh yea, we definitely wouldnt put them in the test images17:36
*** baoli has joined #openstack-infra17:36
zaroianw: ping17:36
mordredgreghaynes, sdague: yeah - and local package mirrors are a TDL item, but are several things down on the current list17:36
sdaguemordred: well, it being several items down the list means it wrecked a bunch of unrelated projects for 24 hours17:37
sdagueso I would ask that it gets brought up the list17:37
*** abitha has joined #openstack-infra17:37
greghaynesWas there a bug relating to dib being unable to boot?17:37
sdagueno, it was a bug in dib not being able to build images because of changes in ubuntu's repos17:38
greghaynesyea, so that we test and caught17:38
anteayaso I have this error twice in this patch, failed two runs: http://logs.openstack.org/64/212864/3/gate/gate-project-config-layout/af79132/console.html#_2015-09-01_02_22_22_38417:38
anteayalogstash only has 5 incidents of it in the last 12 hours, so fairly isolated17:39
anteayawhat would cause zuul to fail to traverse parents of a commit?17:39
sdaguegreghaynes: if by test and caught you mean blocked all nova patches from having valid test results for 24 hours, I think we have different definitions of test and caught :P17:39
anteayathe most recent failure hasn't posted yet, one job is still running17:39
greghaynessdague: Ubuntu does not gate on us, so catching it does not mean preventing the change form going in17:40
*** baoli has quit IRC17:40
*** sergmelikyan has joined #openstack-infra17:40
sdaguegreghaynes: but not depending on the internet in tests would have17:40
*** abregman has joined #openstack-infra17:40
greghaynessdague: Until our mirrors updated...17:40
mordredsdague: it's important - there are just a few other things it's blocking on. however, I'll bring it up ta infra meeting today17:41
sdagueno, then you gate mirror updates on it passing17:41
*** baoli has joined #openstack-infra17:41
greghaynesor new images went out/etc17:41
sdaguebasically like the whole constraints updates model lifeless is building for pip17:41
greghaynesoh my17:41
*** Sukhdev has joined #openstack-infra17:41
mordredI do not think we're planning on gated mirror updates17:41
mordredthe only thing we were planning on for mirror updates was making sure we always had a working mirror copy before publishing it17:41
mordrednot gating the contents on openstack working17:42
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements  https://review.openstack.org/21782417:42
*** stevemar has quit IRC17:42
mordredand honestly - just getting a mirror at all is probably a good first step in either case17:42
*** pvaneck has joined #openstack-infra17:42
sdaguemordred: that's probably fine, however we should test isolation things there17:43
sdagueso only dib should have to pass tests that use that mirror17:43
mordredclarkb, docaedo: https://pbs.twimg.com/media/CN1BDvPUkAE8eTY.jpg:large <-- btw, this happened17:43
sdaguenothing else should17:43
greghaynesBeing broken by upstream distro packages is a completely unrelated issue to anything dib - the tooling to prevent that hasnt even been theorized and that problem exists for all our projects17:43
mordredsdague: it's possible you and I are talking about different things, for which I'm sorry if that's the case17:43
sdaguemordred: it's possible17:44
mordredsdague: what I'm talking about is that infra has been discussing general local package mirrors for resilience of access to package repositories - not any sort of further curated thing17:44
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add launchpad-login command  https://review.openstack.org/21937717:44
mordredwith the one exception being17:44
sdaguemordred: sure, here is the issue17:44
freerunnerHi folks! I would like to ask you for quick review. https://review.openstack.org/#/c/219222/ . Murano gate is broken because of django..17:44
sdagueubuntu changed a thing17:45
mordredthat we want to solve for atomic updates, ecuase the normal package mirroring approaches have those holes where the index is out of sync with the contents17:45
sdaguewhich impacts dib not working17:45
mordredsdague: right. which is then tied into a whole bunch of other things - isn't the solution there to untangle dib from the other things for that?17:45
sdagueit 100% blocked nova, ironic, tempest and possibly other projects from getting any valid test results17:45
*** pelix has quit IRC17:45
sdaguemordred: yes17:45
mordred(like, I feel like jroll sent a message to the list outlining a solution to this)17:45
*** trown|lunch is now known as trown17:46
*** sharikapongubala has joined #openstack-infra17:46
jrollwhat'd I do17:46
greghaynesI dont understand how that issue is any different than our other distro package deps. Say libvirt updating and breaking nova17:46
*** sharikapongubala has quit IRC17:46
sdaguehe did, and that's a big chunk of this, it just seemed like I was hearing there being no belief that there was a real issue17:46
*** afazekas__ has joined #openstack-infra17:46
mordredgreghaynes: it's different because it's an interface change in the repo itself17:46
*** sergmelikyan has quit IRC17:47
mordredgreghaynes: the sha sums are a thing dib specifically looks at17:47
mordrednormal usage of apt repos is not affected by that17:47
greghaynesYes, so its not a package problem, but the issue is that were a downstream of that repo17:47
mordredfor the other thing17:47
mordredif libvirt breaks the world - we _do_ want to know about that17:47
mordredeven if it's painful17:47
*** sharikapongubala has joined #openstack-infra17:47
mordredbecause it means the libvirt will be breaking people for real in the world17:47
*** mmedvede has joined #openstack-infra17:47
clarkbisnt that true of dib here too?17:48
mordreddib would be broken for people in the world17:48
clarkbdib stopped working17:48
mordredbut nova need not be broken by dib being broken17:48
jrollnothing depends on dib at runtime17:48
greghaynesyes, so I agree with the point that nova shouldnt care that dib is broken17:48
*** sergmelikyan has joined #openstack-infra17:48
mordredso yeah - we want dib to notice that ubuntu broke them17:48
mordredso that they can fix it17:48
anteayado we identify which zuul merger is involved in a given job in the log?17:48
greghaynestotally unrelated to the issue of us needing to solve pinning for distro packaging17:48
sdagueright, the point jroll made, dib is not a runtime dep17:48
anteayazm02 looks odd compared to the others17:48
clarkbanteaya yes its the zuul url parameter17:49
*** bharathm has joined #openstack-infra17:49
anteayagreat thanks17:49
greghaynesbecause I do still want to know that dib is broken17:49
*** Sukhdev has quit IRC17:49
jrollright, I don't think anyone disagrees with that17:49
jrollbut dib being broken shouldn't break the world17:49
greghaynesnow, does the IPA change actually fix that issue? Isnt that just moving us on to another upstream?17:50
jrollthe upstream is now tarballs.o.o17:50
jrollwhich is published to by a post job on IPA17:50
greghaynesoh, so coreos-i-b posts artifacts to there?17:50
*** FallenPegasus has joined #openstack-infra17:50
jrollso while it could still be a problem, it's far more controllable17:51
greghaynessweet :)17:51
*** afazekas__ has quit IRC17:51
mordrednow, in parallel to that - we do want better local mirrors - which would not have solved this, but which would/might solve other things17:52
anteayaclarkb: I see zuul-0.log and zuul-1.log does that mean that zuul merger 1 and 2 were involved?17:52
anteayaI might not be looking in the right spot17:52
docaedomordred: yes that did, that meetup (moderated by clarkb!) is going to be awesome :)17:52
*** vivekd_ has joined #openstack-infra17:52
* mordred is SO SAD he can't be there for clarkb moderating that panel17:52
greghaynesIt is going to be fun17:52
*** puranamr has quit IRC17:52
*** vivekd has quit IRC17:53
*** vivekd_ is now known as vivekd17:53
*** amuller_afk is now known as amuller17:53
*** david-lyle has joined #openstack-infra17:54
anteayaclarkb: found it: http://logs.openstack.org/64/212864/3/gate/gate-project-config-layout/af79132/console.html#_2015-09-01_02_22_14_92317:54
*** sigmavirus24 is now known as sigmavirus24_awa17:54
anteayaclarkb: I was in the wrong spot17:54
*** afazekas__ has joined #openstack-infra17:54
sdaguemordred: right, better mirrors will help with other failures17:55
clarkbanteaya ok17:55
*** shashank_hegde has joined #openstack-infra17:56
kfox1111docaedo: hows the cors patch looking?17:56
sdaguealso, in theory, the only artifacts posting to tarballs.o.o are ones we know work, right jroll ?17:56
odyssey4mewith regards to the mirrors, would it not make sense to make use of cloud-specific mirror configurations instead of the generalised ubuntu archive and such?17:56
jrollsdague: yes, they're published on every merge to master, and the gate jobs verify those17:57
*** sputnik13 has quit IRC17:57
sdaguejroll: great17:57
jrollsdague: in theory it's a different build, so MAYBE we could race a mirror update17:57
*** sharikapongubala has quit IRC17:57
*** madhu_ak has joined #openstack-infra17:57
odyssey4megor ubuntu, as an example, hpcloud has http://az[1-3].clouds.archive.ubuntu.com as mirrors in their standard image17:57
jrollsdague: gate does build+tempest, post does build+upload17:57
kfox1111I don't quite understand yolanda's ordering issue. Do you?17:57
mordredodyssey4me: well17:58
*** sharikapongubala has joined #openstack-infra17:58
mordredodyssey4me: you'd think17:58
*** sputnik13 has joined #openstack-infra17:58
anteayaclarkb: logstash tells you the build_node, I'm going with we don't have it set up to tell you the zm though17:58
mordredodyssey4me: except that we've found that doing so actually _increases_ our failure rate17:58
sdaguejroll: yeh, honestly, that's pretty good validation17:58
mordredodyssey4me: due to operational concerns on those mirrors themselves17:58
odyssey4memordred ugh17:58
* odyssey4me shakes his fist at the sky17:59
mordredodyssey4me: this is why we're looking at running per-cloud-region mirrors ourselves17:59
jrollsdague: that's by intention :)17:59
*** gyee has joined #openstack-infra17:59
mordredodyssey4me: and doing it in a way that we won't publish them when they're bad/broken17:59
sdaguejroll: right, so that bits is good. Now just to use it :)17:59
jrollyep, just waiting on a +2+W17:59
*** esmute_ has joined #openstack-infra18:00
*** david-lyle has quit IRC18:01
*** mpopow has joined #openstack-infra18:01
mordredjroll: patch?18:02
*** tiswanso has quit IRC18:02
jrollmordred: https://review.openstack.org/219285 https://review.openstack.org/21933018:02
jrollthe former being more important18:02
*** tjones has joined #openstack-infra18:03
*** tiswanso has joined #openstack-infra18:03
docaedokfox1111: CORS patch looks good to me except need to clarify what yolanda was saying about mod_http and needing to call notify http18:03
anteayaall 5 failures in the last 12 hours are from zuul merger 218:03
anteayamessage:"fatal: Failed to traverse parents of commit"18:03
kfox1111k. thx.18:04
mordredjroll: +318:04
*** zeus- has joined #openstack-infra18:04
*** wznoinsk has quit IRC18:04
anteayaas well as this one: https://jenkins07.openstack.org/job/gate-project-config-layout/1637/console18:04
jrollmordred: <318:04
anteayawhich I'm sure isn't indexed yet18:05
anteayaso zuul merger 2, are we able to take it out of rotation safely?18:05
*** puranamr has joined #openstack-infra18:05
clarkbanteaya: we should index the zuul url too18:06
anteayaclarkb: we should18:06
anteayaclarkb: can I help to do that?18:06
anteayaclarkb: as in I have no idea how to do that and would like to learn18:07
clarkbanteaya: I think it may just be a simple change to the jenkins zmq client and gearman client18:07
anteayaI will see what I can grep there18:07
*** HeOS has quit IRC18:07
anteayaand toss up a place holder which you can tell me how to fix18:07
clarkbanteaya: add it to https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-client.py#n114'18:07
anteayaI will do that18:07
* anteaya does that18:07
*** afazekas__ has quit IRC18:07
*** wznoinsk has joined #openstack-infra18:07
*** alex_xu has quit IRC18:08
*** adduarte has quit IRC18:09
*** fitoduarte has joined #openstack-infra18:09
*** alex_xu has joined #openstack-infra18:11
sdagueso how long do we expect the merge of the top of the queue to take after it's passed all the tests?18:11
*** Sukhdev has joined #openstack-infra18:12
sdaguebecause 215658,1 is going on 5 minutes now18:13
openstackgerritAnita Kuno proposed openstack-infra/puppet-log_processor: Add build_zm parameter  https://review.openstack.org/21938818:14
*** Hal has joined #openstack-infra18:14
clarkbsdague: however long it takes the job result event to get processed18:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface  https://review.openstack.org/21424318:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Pass timeout to session, not constructors  https://review.openstack.org/21929018:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different  https://review.openstack.org/21929118:15
clarkblooks like there are a few hundred in the queue18:15
*** sharikapongubala has quit IRC18:15
*** sharikapongubala has joined #openstack-infra18:15
anteayaclarkb: so I created a new variable (I think) ZUUL_MERGER18:15
anteayaclarkb: where do I need to set or get the value for that, to send it to the logs18:16
*** maishsk_ has joined #openstack-infra18:16
*** puranamr has quit IRC18:17
clarkbanteaya: just copy that line I linked and update the names basically18:18
clarkbanteaya: what that does is copy it from the jenkins paylod and convert it into the gearman payload that will get indexed18:18
anteayaclarkb: this ran on zm02 and failed for the same bug: https://review.openstack.org/#/c/217300/18:18
*** puranamr has joined #openstack-infra18:19
*** david-lyle has joined #openstack-infra18:19
*** maishsk has quit IRC18:19
*** maishsk_ is now known as maishsk18:19
*** w_ has joined #openstack-infra18:19
openstackgerritMerged openstack/requirements: Updated from generate-constraints  https://review.openstack.org/21873718:20
anteayaclarkb: I copied and updated18:20
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-gerrit: Trigger reindex only when secondary index is set.  https://review.openstack.org/21939118:20
anteayawhat would it take to get zm02 out of rotation?18:20
clarkbanteaya: I think the variable is ZUUL_URL18:20
anteayaclarkb: I will fix18:20
clarkbanteaya: you can confirm by looking at a parameters list on a jenkins job in jenkins18:20
clarkbanteaya: go to a job page, login in via lp and on the left it should have a parameters link18:21
*** olaph has quit IRC18:22
openstackgerritMerged openstack-infra/project-config: Use Ironic pxe_ipa job in Nova  https://review.openstack.org/21928518:22
*** ajmiller has quit IRC18:23
*** puranamr has quit IRC18:23
*** thorst has quit IRC18:23
*** thorst has joined #openstack-infra18:23
sdagueclarkb: ok, I guess I was surprised that it took about 15 minutes to merge the 17 passing jobs at top of gate, and that the sliding window didn't adjust until they all were merged18:23
openstackgerritAnita Kuno proposed openstack-infra/puppet-log_processor: Add build_zuul_url parameter  https://review.openstack.org/21938818:24
anteayaclarkb: ah okay, thanks18:24
clarkbI have delted te old git0818:24
clarkbgoing to make a new git07 now18:25
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515118:27
*** abregman is now known as abregman|afk18:28
*** e0ne has joined #openstack-infra18:29
*** vikram has quit IRC18:30
*** tjones has quit IRC18:30
*** jtomasek has joined #openstack-infra18:32
*** aysyd has quit IRC18:32
*** aysyd has joined #openstack-infra18:33
*** w_ is now known as olaph18:34
*** esker has joined #openstack-infra18:35
*** esker has quit IRC18:35
*** MarkAtwood has quit IRC18:35
*** baoli has quit IRC18:35
anteayaclarkb: it is ZUUL_URL and nets us: http://zm06.openstack.org/p18:35
clarkbanteaya: thats it18:35
pabelangernice, finally got nodepool running under centos718:36
anteayaclarkb: wonderful, thank you18:36
glauco__nibalizer: thanks for you review. I didn't understood what you meant by having an puppet if.18:36
clarkbpabelanger: did you see my notes about why iptables is broken on hpcloud centos7?18:36
yolandahi docaedo, kfox1111, so my review is about the usage of httpd_mod , currently is buggy18:36
pabelangerclarkb: no, missed them sorry. Look in scroll back now18:37
nibalizerglauco__: if $secondary_index { exec... }18:37
yolandathe way your change was done, won't work, because it will not reboot httpd service after modules are installed18:37
pabelangerclarkb: I know ianw was looking into is also18:37
*** baoli has joined #openstack-infra18:37
yolandaand i was pointing you to some samples to fix it18:37
glauco__nibalizer: would we be able to notify that exec when $secondary_index is false?18:37
yolandawe need to do a proper fix on httpd_mod, but it's still a TODO18:37
anteayanibalizer: thank you18:38
pabelangerclarkb: okay, caught up. That sucks18:38
glauco__nibalizer: that exec will be notified every time gerrit gets up and down, no matter if it has a secondary type or not. Don't it?18:38
*** e0ne has quit IRC18:39
nibalizerglauco__: the exec can subscribe to the other execs18:39
*** puranamr has joined #openstack-infra18:40
pabelangerclarkb: Ya, I'm not sure how to proceed either. But, does sound like a packaging bug18:40
*** dims has quit IRC18:40
glauco__nibalizer: at first we tried using puppet if, but we noticed that the exec will not be defined when the flag is false, and subscribe will not find the resource18:40
clarkbok new git07 is built, I am gonna take old git07 out of the haproxy rotation now, update dns, force a sync from gerrit then push a change to update the haproxy config18:40
*** dims has joined #openstack-infra18:40
glauco__nibalizer: do you mean to inverse the dependency order and have the exec subscribe instead of being notified?18:41
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Clean any yum cached data  https://review.openstack.org/21922718:41
anteayaclarkb: ack18:41
*** puranamr has quit IRC18:41
bdemersto request a new release via the _new_ process, all i need to do is create a yaml file for a given project name, similar to what was done here: https://review.openstack.org/#/c/217766/ ?18:42
nibalizerglauco__: subscribe and notify are inverses18:42
nibalizerso subscribe in one direction is equivalent to notify in the other direction18:42
bdemersand are the releases cut automaticly once it gets merged ?18:43
nibalizerso you should be able to express the dependency you want without using onlyif= >18:43
anteayabdemers: release questions are best answered by the release managers18:43
anteayabdemers: who tend to be available in #openstack-relmgr-office18:44
pabelangerLooks like ubuntu trusty dibs are broken. Looks like a patch however: https://review.openstack.org/#/c/219199/18:44
openstackgerritMerged openstack-infra/tripleo-ci: Clean any yum cached data  https://review.openstack.org/21922718:44
anteayathat channel is often used for meetings though so when you join wait a bit in case a meeting is in progress18:44
glauco__nibalizer: got it. I will fix that.18:45
*** ivar-lazzaro has quit IRC18:45
bdemersanteaya: i18:45
*** dims has quit IRC18:45
glauco__nibalizer: thank you18:45
pabelangerhttp://paste.openstack.org/show/438357/ for ubuntu trusty error18:45
bdemersanteaya: I ask more about the process / infra around the releas vs. an actual release itself18:46
*** puranamr has joined #openstack-infra18:46
*** sharikapongubala has quit IRC18:47
anteayathe release managers know the current status best18:47
anteayaand my have questions for you about your individual project18:47
*** sharikapongubala has joined #openstack-infra18:48
*** puranamr has quit IRC18:48
clarkbok haproxy and dns are done, have to wait for dns to resolve properly on review.o.o then I will trigger replication18:48
*** puranamr has joined #openstack-infra18:48
*** sharikapongubala has quit IRC18:49
*** sharikapongubala has joined #openstack-infra18:49
greghaynesSpamapS: ok, just deployed one node for crinkle using the baked in bridge/vlan network config + dhcp and it seems to work fine18:50
*** sridhar_ram has quit IRC18:50
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-gerrit: Trigger reindex only when secondary index is set.  https://review.openstack.org/21939118:50
greghaynesSpamapS: deploying second node now...18:50
*** sridhar_ram has joined #openstack-infra18:50
glauco__nibalizer: ^^^18:50
*** sharikap_ has joined #openstack-infra18:50
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-gerrit: Setup preconditions for acceptance tests.  https://review.openstack.org/21940018:52
*** FallenPe_ has joined #openstack-infra18:53
*** sergmelikyan has quit IRC18:53
*** eharney has quit IRC18:53
*** FallenPegasus has quit IRC18:53
mordredgreghaynes, SpamapS: I wanna cut a new glean - the changes look like bugfixes to me - but I could be convinced that the lock file is a new feature and should make it 1.5.0 - thoughts?18:53
greghaynes1.5 due to the lock18:54
*** sharikapongubala has quit IRC18:54
*** sergmelikyan has joined #openstack-infra18:54
openstackgerritMerged openstack-infra/infra-specs: Host Stackalytics Service  https://review.openstack.org/18771518:54
greghaynesdoesnt hurt to err on that side18:54
clarkblooking at review.o.o error logs we appear to have broken the welcome message script18:54
*** sharikapongubala has joined #openstack-infra18:54
greghaynesI think18:54
*** sridhar_ram has quit IRC18:54
greghayneswell, actually, I really dont care either way, the change isnt that behaviroal18:54
greghayneser, behavioral18:54
clarkbI think stevenks fix logging change may have done that? we are passing --verbose but it should be -v18:54
greghaynesmordred: flip a coin ;)18:54
*** e0ne has joined #openstack-infra18:55
*** sergmelikyan has quit IRC18:55
mordredjeblair: ^^ thoughts?18:55
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul  https://review.openstack.org/21773218:55
mordredjeblair: new glean release introduces a lockfile to serialize creation of common files - is that a 1.4.2 or a 1.5.0 to you?18:55
*** zeus- is now known as zeus18:56
*** zeus has quit IRC18:56
*** zeus has joined #openstack-infra18:56
jeblairmordred: 1.4.218:57
*** salv-orlando has quit IRC18:57
mordredjeblair: thanks18:57
jeblairmordred: np. i also supply losing lottery numbers.18:58
mordredrcarrillocruz, jeblair, greghaynes, SpamapS: new glean cut18:58
*** sharikap_ has quit IRC18:58
openstackgerritClark Boylan proposed openstack-infra/jeepyb: Add --verbose flag as valid option for logging  https://review.openstack.org/21940318:59
clarkbthere is the verbose flag fix18:59
*** HeOS has joined #openstack-infra18:59
*** jhesketh has quit IRC18:59
*** john-davidge has quit IRC18:59
*** annegent_ has joined #openstack-infra19:00
jeblairmeeting time!19:00
*** angela-s has joined #openstack-infra19:00
*** changbl has quit IRC19:02
anteayaclarkb: I filed a story for it: https://storyboard.openstack.org/#!/story/200034119:03
*** nicodemos has quit IRC19:03
*** marcusvrn_ has joined #openstack-infra19:03
*** annegentle has quit IRC19:03
*** jhesketh has joined #openstack-infra19:03
clarkbanteaya: thanks I think it may be related to the performance trouble we saw yesterday19:03
clarkbpossible noisy neighbor was one theory I heard19:03
*** Piet has quit IRC19:04
*** sigmavirus24_awa is now known as sigmavirus2419:04
*** afazekas has quit IRC19:04
anteayaclarkb: fair enough, I need it fixed though as it keeps my approved patheces from merging19:05
*** salv-orlando has joined #openstack-infra19:06
SpamapSmordred: ACK19:06
*** esker has joined #openstack-infra19:07
*** stevemar has joined #openstack-infra19:07
SpamapSgreghaynes: wanna have another call to get the ship moving again on the terrible cloud? I feel like we lost the thread a bit.19:08
*** MarkAtwood has joined #openstack-infra19:08
*** annegent_ has quit IRC19:08
*** esmute_ has quit IRC19:08
*** dims has joined #openstack-infra19:09
*** watanabe_isao has joined #openstack-infra19:09
*** hashar has joined #openstack-infra19:09
greghaynescrinkle: ok, you now have .132 available19:09
crinklegreghaynes: \o/19:09
greghaynesSpamapS: Next big thing for me is to get https://review.openstack.org/#/c/219026/ and the two dhcp servers codified19:10
sdagueclarkb / mordred - how you folks feeling about - https://review.openstack.org/#/c/218966 - (though I need to recheck it because of the migrate bug yesterday) - as I'm watching a bunch of the large-ops jobs fail under timeout preasure again, and that would get us tracking that19:10
*** puranamr has quit IRC19:10
rcarrillocruzmordred: \o/19:10
greghaynesSpamapS: basically I have a lot of one-off work on the bastion I need to formulate into reviews, the one off work is what is letting us do the separate conductor and deploy networks19:10
clarkbsdague: seems fine19:10
openstackgerritClint Adams proposed openstack-infra/infra-specs: WIP - tracking of priority efforts in yaml  https://review.openstack.org/21937219:10
clarkbsdague: I really do want to trim the list of PROJECTS too19:10
sdagueclarkb: sure19:11
clarkbwill have to take that on once transifex to zanata migration is complete19:11
sdaguehonestly, it would be really nice if we could generate the project list per job19:11
*** annegentle has joined #openstack-infra19:11
*** krtaylor has quit IRC19:11
clarkbsdague: yes thats the idea19:11
greghaynesSpamapS: then from there I think were going to cycle on a node to get the bifrost vlan config-drive stuff working, then the upstart script to move interface to a bridge19:12
sdagueclarkb: neat19:12
clarkbsdague: with a base set in d-g for simplicity19:12
sdagueclarkb: sure19:12
clarkbsdague: I think base set in d-g will reflect default devstack list19:12
clarkbthen everything else will be job specific19:12
SpamapSgreghaynes: k.19:12
*** sdake_ has joined #openstack-infra19:12
lifelessclarkb: cool thanks19:13
*** sdake has quit IRC19:13
*** dims has quit IRC19:13
*** watanabe_isao has quit IRC19:13
SpamapSgreghaynes: right, thats what I was going to do. Just wanted to make sure we're all on the same page.19:13
*** sdake has joined #openstack-infra19:13
*** julim has quit IRC19:14
greghaynesSpamapS: Sweet, I'll try to be careful and not break the bastion host while I move the chagnes there upstream19:14
greghaynesSpamapS: if you want to poke at that19:14
greghaynes(I want to evacuate the changes from my head ASAP before I forget them)19:15
SpamapSgreghaynes: best way to do that is just to deploy a new baremetal (lets call them baremetals when we deploy them with automation. ;)19:15
greghaynesoh, works for me19:15
*** jaosorior has quit IRC19:15
SpamapSgreghaynes: it becomes bastion when we screw with it for 2 weeks though. ;)19:15
clarkbsdague: maybe a function wrapper for other functions to do timing is something we should add in if we are going to time lots of things?19:15
greghaynesSpamapS: new host or take the plunge and redeploy the original bastion?19:16
*** zeih has joined #openstack-infra19:16
sdagueclarkb: we could do that down the road if we do more of this19:16
sdague3 seemed still in the copy/paste realm19:16
*** sdake_ has quit IRC19:18
sdaguemordred / jeblair - https://review.openstack.org/#/c/202494 commented back about the ansible config, assuming that working directory is stable during d-g run is not a valid assumption I don't think19:18
*** dims has joined #openstack-infra19:18
greghaynesclarkb: you were messing with git stuffs earlier right? https://review.openstack.org/#/c/219330/ exploded on a clone, curious if I should try and recheck now19:19
*** esmute_ has joined #openstack-infra19:19
*** cdent has quit IRC19:20
*** sabeen1 has joined #openstack-infra19:20
clarkbgreghaynes: I did but that should be unrelated unfortunately there is almost no logging there19:20
clarkbgreghaynes: I disabled an haproxy backend which should keep existing connections open iirc19:21
*** krtaylor has joined #openstack-infra19:21
greghaynesclarkb: hrm, didnt you also reboot that backend?19:21
*** zeih has quit IRC19:21
clarkbgreghaynes: no19:21
greghaynesIt does look like a socket that got axed19:21
*** sharikapongubala has quit IRC19:22
clarkbgreghaynes: I can check nodepool logs for IP addrs and do a lookup in the haproxy log after the meteing19:22
greghaynesok, ty19:22
*** shashank_hegde has quit IRC19:23
*** esmute_ has quit IRC19:24
*** dims has quit IRC19:25
*** sharikapongubala has joined #openstack-infra19:25
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515119:25
*** armax has quit IRC19:26
*** geoffarnold has quit IRC19:27
*** pvaneck has quit IRC19:29
mriedemi want to applaud the people that wrote this doc http://docs.openstack.org/infra/zuul/gating.html19:30
*** yamamoto has quit IRC19:30
*** sharikapongubala has quit IRC19:31
*** pcrews has quit IRC19:31
openstackgerritDoug Hellmann proposed openstack-infra/project-config: publish docs built from releases repository  https://review.openstack.org/21942019:31
*** sharikapongubala has joined #openstack-infra19:31
*** pcrews has joined #openstack-infra19:31
dhellmannanteaya: I would appreciate a thorough review of ^^, since I'm cargo-culting those changes19:32
openstackgerritIan Wienand proposed openstack/diskimage-builder: Make README.rst a bit more generic  https://review.openstack.org/21804319:32
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable  https://review.openstack.org/20832119:32
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable  https://review.openstack.org/21143419:32
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions  https://review.openstack.org/21810019:32
*** ivar-lazzaro has joined #openstack-infra19:32
*** pvaneck has joined #openstack-infra19:33
anteayadhellmann: you will get the best I can offer19:33
*** krtaylor has quit IRC19:34
*** HeOS has quit IRC19:34
*** aysyd has quit IRC19:34
*** thorst has quit IRC19:34
*** bharathm has quit IRC19:34
*** claudiub has quit IRC19:34
*** ZZelle has quit IRC19:34
*** mrunge has quit IRC19:34
*** lifeless has quit IRC19:34
*** matbu has quit IRC19:34
*** mtreinish has quit IRC19:34
*** Mmike has quit IRC19:34
*** mrodden has quit IRC19:34
*** raginbajin has quit IRC19:34
*** harlowja has quit IRC19:34
*** goodygum has quit IRC19:34
*** nithyag__ has quit IRC19:34
*** sobersabre has quit IRC19:34
*** dtantsur|afk has quit IRC19:34
*** kfox1111 has quit IRC19:34
*** pabelanger has quit IRC19:34
*** Sam-I-Am has quit IRC19:34
*** cschwede has quit IRC19:34
*** wenlock has quit IRC19:34
*** cloudnull has quit IRC19:34
*** sc68cal has quit IRC19:34
*** Gareth has quit IRC19:34
*** sdague has quit IRC19:34
*** niska has quit IRC19:34
*** jhesketh has quit IRC19:34
*** Piet has joined #openstack-infra19:34
*** baoli has quit IRC19:34
*** mriedem is now known as mattykins19:34
*** wenlock has joined #openstack-infra19:34
*** cloudnull has joined #openstack-infra19:34
*** sc68cal has joined #openstack-infra19:34
*** Gareth has joined #openstack-infra19:34
*** sdague has joined #openstack-infra19:34
*** krtaylor has joined #openstack-infra19:35
*** HeOS has joined #openstack-infra19:35
*** aysyd has joined #openstack-infra19:35
*** thorst has joined #openstack-infra19:35
*** bharathm has joined #openstack-infra19:35
*** claudiub has joined #openstack-infra19:35
*** ZZelle has joined #openstack-infra19:35
*** mrunge has joined #openstack-infra19:35
*** lifeless has joined #openstack-infra19:35
*** matbu has joined #openstack-infra19:35
*** mtreinish has joined #openstack-infra19:35
*** kfox1111 has joined #openstack-infra19:35
*** Mmike has joined #openstack-infra19:35
*** mrodden has joined #openstack-infra19:35
*** raginbajin has joined #openstack-infra19:35
*** goodygum has joined #openstack-infra19:35
*** nithyag__ has joined #openstack-infra19:35
*** sobersabre has joined #openstack-infra19:35
*** dtantsur|afk has joined #openstack-infra19:35
*** pabelanger has joined #openstack-infra19:35
*** Sam-I-Am has joined #openstack-infra19:35
*** cschwede has joined #openstack-infra19:35
*** harlowja has joined #openstack-infra19:35
*** harlowja has quit IRC19:35
*** harlowja has joined #openstack-infra19:35
*** jhesketh has joined #openstack-infra19:35
*** dannywilson has joined #openstack-infra19:35
*** w_ has joined #openstack-infra19:35
*** baoli has joined #openstack-infra19:35
*** dims has joined #openstack-infra19:35
*** eharney has joined #openstack-infra19:35
clarkbgreghaynes: I see 3 connections that had a CD state which I think is closed disconnected19:35
greghaynesclarkb: hrm19:35
clarkbnow I have to go look up docs to hve it tell me which side did the closing19:36
*** tiswanso has quit IRC19:36
*** niska has joined #openstack-infra19:36
clarkbgreghaynes: but only one of them was to the backend I was touching and the timeframes don't line up for when I disabled that particular backend19:36
clarkbgreghaynes: if I had to guess right now, I would pin this on hpcloud networking not the git farm19:36
greghaynesclarkb: ok. so might just be network blip19:36
greghaynesrecheck it is19:36
*** abitha has quit IRC19:37
*** tiswanso has joined #openstack-infra19:38
*** unicell has quit IRC19:38
*** olaph has quit IRC19:38
openstackgerritClint Adams proposed openstack-infra/infra-specs: WIP - tracking of priority efforts in yaml  https://review.openstack.org/21937219:39
*** esmute_ has joined #openstack-infra19:39
*** w_ has quit IRC19:40
*** erlon has joined #openstack-infra19:44
*** sdake_ has joined #openstack-infra19:45
greghayneslucas-dinner: jroll So, trying to get up to speed on what the state of the DIB ironic pxe_ssh job is.19:46
clarkbgit07 looks good from my local clone testing, will have a change up to put it in the ortation as soon as git remote update works on that repo19:46
greghayneslucas-dinner: jroll Is there any path forward for fixing it that doesnt involve fixing dib pip install -e as well as devstack? that seems a bit overly complex to get done right now19:46
fungimattykins: looks like hashar and jeblair are primarily to blame for that excellent document19:47
*** mattykins is now known as mriedem19:47
jrollgreghaynes: I'm not really up to speed, lucas-dinner left me these to look at https://review.openstack.org/#/c/219351/ https://review.openstack.org/#/c/219349/19:47
jrollgreghaynes: not sure if that's all19:47
anteayamattykins: I just wanted to tab complete this name for you19:47
greghaynesah, that fixes from source19:47
*** harlowja has quit IRC19:47
*** krtaylor has quit IRC19:47
*** HeOS has quit IRC19:47
*** aysyd has quit IRC19:47
*** thorst has quit IRC19:47
*** bharathm has quit IRC19:47
*** claudiub has quit IRC19:47
*** ZZelle has quit IRC19:47
*** mrunge has quit IRC19:47
*** lifeless has quit IRC19:47
*** matbu has quit IRC19:47
*** mtreinish has quit IRC19:47
*** Mmike has quit IRC19:47
*** mrodden has quit IRC19:47
*** raginbajin has quit IRC19:47
*** goodygum has quit IRC19:47
*** nithyag__ has quit IRC19:47
*** sobersabre has quit IRC19:47
*** dtantsur|afk has quit IRC19:47
*** kfox1111 has quit IRC19:47
*** pabelanger has quit IRC19:47
*** Sam-I-Am has quit IRC19:47
*** cschwede has quit IRC19:47
greghaynesso then were still broke by pip install -e...19:48
jrollgreghaynes: stepping away, I'll bbiab, ping me if I can help other than reviewing those19:48
anteayamriedem: while I could19:48
greghaynesjroll: ok, ty19:48
mriedemanteaya: and it's not even friday19:48
openstackgerritClark Boylan proposed openstack-infra/system-config: Swap out git07 to new centos7 node  https://review.openstack.org/21942719:48
clarkband there it is, ready whenever you are19:48
fungimriedem: you could declare it pre-friday19:49
*** esmute_ has quit IRC19:49
*** sdake has quit IRC19:49
anteayamriedem: I knkow, stepping out19:49
*** sdake has joined #openstack-infra19:49
fungiall non-fridays are hereby designated pre-fridays19:49
*** sputnik13 has quit IRC19:49
fungilet the early fridaying commence19:49
*** ashleighfarnham has quit IRC19:49
anteayaoh I feel so much better now19:49
*** ayoung has quit IRC19:50
nibalizerclarkb: i am poking the archive module with a stick, hopefully will have something for you soon19:50
clarkbnibalizer: cool and thanks19:50
*** davideagnello has quit IRC19:50
fungimordred: can i safely reenable puppet agent on nodepool.o.o now (or did you do it yourself after the shade release)?19:51
*** sharikapongubala has quit IRC19:51
*** ashleighfarnham has joined #openstack-infra19:52
*** harlowja has joined #openstack-infra19:52
*** krtaylor has joined #openstack-infra19:52
*** HeOS has joined #openstack-infra19:52
*** aysyd has joined #openstack-infra19:52
*** thorst has joined #openstack-infra19:52
*** bharathm has joined #openstack-infra19:52
*** claudiub has joined #openstack-infra19:52
*** ZZelle has joined #openstack-infra19:52
*** mrunge has joined #openstack-infra19:52
*** lifeless has joined #openstack-infra19:52
*** matbu has joined #openstack-infra19:52
*** mtreinish has joined #openstack-infra19:52
*** kfox1111 has joined #openstack-infra19:52
*** Mmike has joined #openstack-infra19:52
*** mrodden has joined #openstack-infra19:52
*** raginbajin has joined #openstack-infra19:52
*** goodygum has joined #openstack-infra19:52
*** nithyag__ has joined #openstack-infra19:52
*** sobersabre has joined #openstack-infra19:52
*** dtantsur|afk has joined #openstack-infra19:52
*** pabelanger has joined #openstack-infra19:52
*** Sam-I-Am has joined #openstack-infra19:52
*** cschwede has joined #openstack-infra19:52
*** geoffarnold has joined #openstack-infra19:52
*** bgmccollum has joined #openstack-infra19:52
*** sharikapongubala has joined #openstack-infra19:52
mordredfungi: I have not done it - but it should be safe19:52
fungimordred: thanks, done19:53
*** sdake_ has quit IRC19:53
*** MarkAtwood has quit IRC19:53
*** 64MADQRLY has joined #openstack-infra19:53
*** davideagnello has joined #openstack-infra19:53
*** olaph has joined #openstack-infra19:53
*** 64MADQRLY has quit IRC19:53
*** sputnik13 has joined #openstack-infra19:53
*** mrmartin has quit IRC19:53
*** e0ne has quit IRC19:54
*** zz_ja has joined #openstack-infra19:55
*** e0ne has joined #openstack-infra19:55
*** ociuhandu has quit IRC19:55
*** maurosr has joined #openstack-infra19:55
hasharfungi: o/19:56
hasharmriedem: ah thank you very much!19:56
*** ayoung has joined #openstack-infra19:56
hasharmriedem: took me a while to figure out while asking question in this channel  and thought it would save others a bunch of time :}    There is more that needs to be documented though :/19:57
*** glauco__ has quit IRC19:58
*** julim has joined #openstack-infra19:58
*** glauco has joined #openstack-infra19:58
*** ayoung has quit IRC19:58
clarkbfungi: jenkins04 still looking good19:58
*** jtomasek has quit IRC19:59
fungitwo days in a row with crazy load/io spikes on zm02, huh?19:59
fungibad enough we stopped being able to poll snmp on it for 20-30 minutes at a stretch20:00
*** ayoung has joined #openstack-infra20:00
anteayafungi: https://storyboard.openstack.org/#!/story/200034120:00
*** boris-42 has quit IRC20:00
anteayacan we pull it out of rotation?20:00
*** MarkAtwood has joined #openstack-infra20:00
anteayamtreinish wants this patch in and it keeps hitting zm02: https://review.openstack.org/#/c/212864/20:01
*** krtaylor has quit IRC20:01
fungijeblair: ^ is it safe to just stop the zuul-merger service on that worker? i'll open a trouble ticket with rackspace about poor i/o performance20:01
*** mrunge has quit IRC20:01
mtreinishanteaya: I want many patches in :)20:02
anteayamtreinish: :)20:02
fungii want all patches in, except the ones i don't20:02
*** krtaylor has joined #openstack-infra20:02
jeblairfungi: yes20:02
anteayayay \o/20:02
anteayathank you20:02
lucas-dinnergreghaynes, hi there, yes the pip install -e is being worked around on the devstack patch20:03
lucas-dinnerwe can fix it later20:03
clarkbya should be safe just dont stop apache20:03
mtreinishmordred: is the ansible run taking a ~40 min a known thing: http://logs.openstack.org/79/219379/1/check/gate-tempest-dsvm-large-ops/6f83839/console.html#_2015-09-01_18_07_50_88420:03
*** sergmelikyan has joined #openstack-infra20:03
hasharfungi: maybe the merger git repos have too many references and needs some cleanup ?20:03
hasharfungi: there is no garbage collection of .git refs and with time that really slow down fetches operations20:04
*** glauco__ has joined #openstack-infra20:04
*** sridhar_ram has joined #openstack-infra20:04
fungihashar: we checked yesterday and it didn't look like that was the issue (and we have 7 other merger servers not exhibiting this behavior)20:04
hasharfungi: just striking a potential slowness option :}20:05
pabelangerfungi: clarkb: any issues adding the previous maintainers of trystack.org into the trystack-site-core group?20:05
*** ayoung has quit IRC20:05
*** armax has joined #openstack-infra20:05
*** unicell has joined #openstack-infra20:05
pabelangerhashar: ohai! I'm checking out your systemd stuff for nodepool20:05
pabelangertrying to get it working locally too20:06
hasharpabelanger: happy you found it :}20:06
hasharpabelanger: I have a debian package for nodepool as well though I have yet to publish it anywhere :/20:06
*** glauco has quit IRC20:06
pabelangerhashar: having some issues with dib right now, and sudo requiring tty. Trying to fix that now20:07
pabelangerhashar: you should import it here!20:07
*** ayoung has joined #openstack-infra20:07
mordredmtreinish: not to me - that seems excessive20:07
mtreinishmordred: well it trigger the jenkins timeout and killed it20:07
mtreinishclarkb: were you looking at that before, it seems familiar for some reason20:07
hashar!createrepo openstack-infra/zuul-packaging from https://gerrit.wikimedia.org/r/operations/debs/nodepool20:08
openstackhashar: Error: "createrepo" is not a valid command.20:08
hasharpabelanger: so there is a pending patch at https://gerrit.wikimedia.org/r/#/c/224390/1/debian/nodepool.systemd,unified20:09
*** ociuhandu has joined #openstack-infra20:09
clarkbmtreinish is it running workspace setup?20:09
clarkbanisble is less important and the command being run ia what is ibteresting20:09
hasharpabelanger: and another in puppet https://github.com/wikimedia/operations-puppet/blob/production/modules/nodepool/templates/initscripts/nodepool.systemd.erb20:09
greghayneslucas-dinner: oh, sweet20:10
lucas-dinnergreghaynes, yeah, one thing I need to do yet is to extend that patch mapping the ubuntu versions20:10
hasharpabelanger: the idea is to let systemd handle the daemon / pid file by itself. So nodepool is launched with -d (our puppet source)20:10
lucas-dinnerto map the other versions20:10
*** jcoufal has quit IRC20:11
lucas-dinnergreghaynes, that was just a quick test to see if it would fix the gate, but then we found out that dib wasn't being tested from source there :-(20:11
*** shashank_hegde has joined #openstack-infra20:11
*** ihrachys has quit IRC20:11
mtreinishclarkb: yeah it looks like setup workspace is the slow thing: http://logs.openstack.org/79/219379/1/check/gate-tempest-dsvm-large-ops/6f83839/logs/devstack-gate-setup-workspace-new.txt.gz20:11
clarkbI think thats zm02 hitting us which fungi is addressing20:11
* lucas-dinner will do it tomorrow since it's quite late over here today (but feel free to extend it if you have some free time)20:11
hasharpabelanger: and systemd does not wait after a graceful stop, so you need to ship a script that polls for the process while it stops : /20:11
clarkbyou can confirm by checking the zuul url20:11
mtreinishclarkb: yep: http://logs.openstack.org/79/219379/1/check/gate-tempest-dsvm-large-ops/6f83839/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-09-01_18_10_40_29620:12
*** Sukhdev has quit IRC20:12
greghayneslucas-dinner: got it. Trying to figure out how we can unravel the two breaks we have in DIB right now, so sounds like we need to wait for that devstack change to merge then the dib fix20:12
lucas-dinnergreghaynes, yeah20:12
*** vivekd has quit IRC20:13
lucas-dinnergreghaynes, we can merge 1) devstack 2) project config to use dib from source non-voting 3) fix dib ubuntu element 4) project-config to set it to vote on gate again20:13
*** kdas_ has joined #openstack-infra20:14
lucas-dinner2) and 3) can be done in parallel, don't need to be in order20:14
greghaynesoh, we need to make it non-voting? I was thinking just merge devstack, project config to pull from source, then ubuntu element fix20:14
*** abregman|afk is now known as abregman20:15
lucas-dinnergreghaynes, it's already non-voting20:15
*** eharney has quit IRC20:15
lucas-dinnerall the pxe_ssh jobs were set to non-voting cause it was blocking not only ironic but also nova, dib etc...20:15
*** jeadre has quit IRC20:15
*** kushal has quit IRC20:15
greghaynesoh, so it is :)20:15
greghaynesok, im going to merge the ubuntu element fix then20:16
*** rlandy has quit IRC20:16
lucas-dinnergreghaynes, ack, thanks! But please add more versions to that map20:16
*** jeadre has joined #openstack-infra20:16
lucas-dinnersince I only put trusty there because that was the one broken20:16
lucas-dinnerI think all the others still using their names for version20:16
*** rlandy has joined #openstack-infra20:17
lucas-dinnerprecise = precise in the mapping (needs to check the SHA256SUMS file in the ubuntu website)20:17
*** alivigni has quit IRC20:17
lucas-dinnergreghaynes, and thanks for the helP!20:17
greghayneslucas-dinner: np, thanks for the fixes :)20:17
pabelangerhashar: okay, checking out the reviews now.  Just caching centos6 right now20:17
lucas-dinnerit's all good20:18
hasharI am upstreaming the nodepool graceful stop script20:18
fungiclarkb: we have at least some logs where we fail to retrieve some refs (also zm02) not just slowness20:18
*** shashank_hegde has quit IRC20:18
clarkbfungi: gotcha so some are fails and others are slow20:19
clarkband sometimes the slow ones are also fails due to timeouts20:19
fungimight all be timeouts, just at different layers20:19
*** kgiusti has left #openstack-infra20:19
fungifor example, the error in the previous round of your liberty cross-check going into the gate (i've rechecked since)20:20
*** esker has quit IRC20:20
*** salv-orl_ has joined #openstack-infra20:20
fungi"fatal: remote did not send all necessary objects" http://logs.openstack.org/00/217300/2/gate/gate-project-config-layout/219cba8/console.html#_2015-09-01_17_28_30_60120:21
*** pblaho has quit IRC20:21
*** eharney has joined #openstack-infra20:22
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source  https://review.openstack.org/21935120:23
*** esmute_ has joined #openstack-infra20:24
openstackgerritAntoine Musso proposed openstack-infra/nodepool: tools: shell script to gracefully stop and wait  https://review.openstack.org/21944220:24
hasharpabelanger: ^^^20:24
pabelangerhashar: ack20:24
*** salv-orlando has quit IRC20:24
anteayafungi: is the story I filed lacking in detail? https://storyboard.openstack.org/#!/story/200034120:25
hasharpabelanger: the version in our puppet is the one deployed ( https://github.com/wikimedia/operations-puppet/blob/production/modules/nodepool/templates/initscripts/nodepool.systemd.erb )20:25
hasharpabelanger: it misses a few bits though such as the   After=network.target  and [Install] WantedBy=multi-user.target20:26
*** jcoufal has joined #openstack-infra20:26
hasharpabelanger: I am rebuilding our nodepool package this week so will have something more usable soonish20:26
*** jeadre has quit IRC20:26
pabelangerWhen an infra-core has time, if they could add wfoster@redhat.com / kambiz@redhat.com to trystack-site-core, that would be great.20:27
clarkbhashar: do you run as daemon or foreground and let systemd daemonize?20:27
anteayayay Dorval!20:27
clarkbI am curious what the more systemdy way of doing that is20:27
*** jeadre has joined #openstack-infra20:27
kambizpabelanger: thanks!20:27
*** kdas_ has quit IRC20:28
*** olaph has quit IRC20:28
*** julim has quit IRC20:29
*** mrmartin has joined #openstack-infra20:29
*** vivekd has joined #openstack-infra20:30
hasharclarkb: foreground and systemd handles everything20:30
*** mrmartin has quit IRC20:30
hasharclarkb: so we don't need to write a .pid file and could even remove the python-daemon dependency20:30
clarkbhashar only if we assume systemd is the only way20:31
hasharclarkb: from discussions we had with Wikimedia ops, they tend to prefer letting systemd handle the demonization / pid tracking20:31
clarkbwhich is bad assumption for now20:31
clarkbya and thats supported20:31
fungianteaya: nah, it's sufficient, it's just not likely a bug we're able to fix (unless you count opening a ticket with the service provider and waiting for them to act on it a "bug fix")20:31
anteayafungi: ah okay, thanks20:32
hasharclarkb: yeah nobody is forced to use systemd and python-daemon should stick in Nodepool.  That was more a rhetorical question :  should dev bother reinventing the wheel of pid tracking / demonization ? :}20:32
fungianteaya: pretty much "zm02 is performing terribly" is about all the detail we really need in that storyboard story (if we even need a storyboard story)20:32
clarkbhashar I wrote my first unit files last week which is why I am curious20:33
anteayafungi: ah okay thanks, wasn't sure20:33
anteayawanted to set a good example20:33
hasharclarkb: there is a bunch of glitches.  Took  me a while to figure out even the most basic but eventually ended up with that file.  There is probably a lot more achievable with systemd though20:33
*** baoli has quit IRC20:34
*** baoli has joined #openstack-infra20:34
*** baoli has quit IRC20:36
clarkbya I wrote them due to bugs in the centos7 unit files20:36
clarkblearned a lot in the process20:37
*** abregman is now known as abregman|afk20:37
* anteaya packs up to prepare to disembark the train20:37
*** baoli has joined #openstack-infra20:37
*** dprince has quit IRC20:38
*** afranc202 has joined #openstack-infra20:39
*** julim has joined #openstack-infra20:42
*** olaph has joined #openstack-infra20:43
*** baoli has quit IRC20:43
*** bswartz has quit IRC20:44
openstackgerritMerged openstack-infra/project-config: Remove the local copy of subunit2html  https://review.openstack.org/21286420:45
*** olaph has quit IRC20:47
*** jeadre has quit IRC20:48
*** jeadre has joined #openstack-infra20:49
*** yamahata has quit IRC20:49
*** sdake has quit IRC20:53
*** thedodd has quit IRC20:53
*** thedodd has joined #openstack-infra20:54
*** achanda has quit IRC20:54
*** earlephilhower has quit IRC20:55
*** olaph has joined #openstack-infra20:55
openstackgerritMonty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions  https://review.openstack.org/18317620:56
mordredfungi: ^^ more code deleted20:56
nibalizerclarkb: okay I argued on the wildfly PR20:56
fungimordred: i also saw a neutron one fly by earlier?20:56
mordredfungi: yeah. although I'm not sure that one works yet20:57
mordredjamielennox, morgan: ^^ there's the latest patch for using keystoneauth20:57
* morgan looks.20:57
morganmordred: that looks pretty straight forward20:58
openstackgerritAntoine Musso proposed openstack/diskimage-builder: dib-run-parts: make cp to target root more robust  https://review.openstack.org/21945320:58
*** thedodd has quit IRC20:59
*** flepied has quit IRC20:59
*** trown is now known as trown|outttypeww20:59
mordredmorgan: it's not terrible20:59
mordredmorgan: I, in fact, removed some tests which would have wound up being tests of "does this library work"21:00
*** olaph has quit IRC21:00
*** shashank_hegde has joined #openstack-infra21:00
mordredmorgan: if that passes tests, I think releasing ksa will have my blessing21:01
openstackgerritAdam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config  https://review.openstack.org/21733021:01
jamielennoxmordred: cool, what i want to be sure of there is that none of the clients are doing anything stupid that will be different from ksc -> ksa21:01
morganthere are a few more lined up patches from jamielennox that would be good to get too21:01
rm_workianw: fixed https://review.openstack.org/#/c/217330/ to depend on the project CR, and updated that CR to only ADD the new method, not delete the old one at the same time -- should be good now21:01
rm_workianw: let me know if that looks OK when you have a moment? :) thanks for the feedback! this really is a safer way to do it, not sure why i didn't in the first place21:02
*** changbl has joined #openstack-infra21:02
*** aysyd has quit IRC21:04
*** bgmccollum has left #openstack-infra21:04
*** rfolco has quit IRC21:05
*** rossella_s has quit IRC21:05
*** rossella_s has joined #openstack-infra21:06
jrolllooks like I'm having sporadic git issues here https://review.openstack.org/#/c/219330/21:07
*** sridhar_ram has quit IRC21:07
jroll2015-09-01 19:54:41.794 | fatal: Failed to traverse parents of commit dc28559b05ce38f1780b85359f14aeaf9211544521:07
jroll2015-09-01 19:54:41.795 | fatal: remote did not send all necessary objects21:07
jrolltwice now :(21:07
*** achanda has joined #openstack-infra21:07
fungijroll: is there a url for zm02 in the log? if so, it was struggling so i've taken it out of the cluster and opened a ticket with the service provider21:08
jrolllifeless: do I need to add things to upper-constraints manually? readme says it's generated but CI says not? https://review.openstack.org/#/c/219011/21:08
jrollfungi: no, zm05 it seems21:08
lifelessjroll: its both, and sometimes. See the README and if tht doesn't clear it up we'll patch it until it does :)21:08
* jroll reads again21:09
*** changbl has quit IRC21:09
fungijroll: hrm, that's unfortunate. i wonder if more than just zm02 are misbehaving now21:10
jrollfungi: yeah :(21:11
jrollI'll recheck for now21:11
*** pradk has quit IRC21:11
jrolllifeless: so adding a new library will require me to at a minimum add that library to upper-constraints, and I should also check the diff21:11
*** sridhar_ram has joined #openstack-infra21:11
*** pradk has joined #openstack-infra21:12
*** changbl has joined #openstack-infra21:13
fungiclarkb: jeblair: yikes, i'm finding this on several more zuul-merger servers now21:14
*** abregman|afk has quit IRC21:14
jeblairfungi: are we sure that's because of the merger?21:15
fungiacross workers in multiple providers, and impacting more than one repo21:15
lifelessjroll: yes21:15
*** e0ne has quit IRC21:15
*** cburroughs has quit IRC21:15
fungijeblair: i'm not certain yet21:15
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-jenkins: Fix puppet-lint errors  https://review.openstack.org/21945421:15
*** e0ne has joined #openstack-infra21:16
fungijeblair: looks like in those examples we see it affecting a clone of jenkins-job-builder and a clone of zuul21:16
jeblairfungi, clarkb: the git mirror changed recently, right?21:16
*** dkranz has quit IRC21:16
*** esmute_ has quit IRC21:16
*** ldnunes has quit IRC21:16
fungiyep, i bet this is zuul-cloner updating from git.o.o21:16
*** geoffarnold has quit IRC21:17
fungii'm going to take 07 out of the haproxy rotation right now21:17
clarkbits already out?21:18
clarkbI didnt see that my change merged21:18
*** zeih has joined #openstack-infra21:18
fungihrm, yeah it's not merged yet21:18
fungijust 0821:19
fungitrying to correlate to logs21:19
clarkbfungi https://review.openstack.org/#/c/219427/ is not merged and I took 07 out when the dns change was made21:19
clarkbto avoid it going stale21:19
fungiyep, makes sense21:19
*** tiswanso has quit IRC21:19
clarkbso 07 shouldnt affect anything right now unless loss of 1/8 capacity broke us21:20
lifelessfungi: when the crisis is over, I'd like to get you and clarkb to +2A https://review.openstack.org/#/c/207262/21:20
*** maishsk has quit IRC21:20
jeblairclarkb: what's the status of 08?21:20
clarkbjeblair its in use and was happy last I checked21:21
clarkbI did delete old 08 earlier today21:21
clarkbI was running top on it this mroning to keep an eye on general memory and cpu use and it seemed fine21:22
*** zeih has quit IRC21:22
*** esmute_ has joined #openstack-infra21:23
clarkbhas been in ise since early saturday utc21:23
fungianalyzing http://logs.openstack.org/00/217300/2/gate/gate-project-config-layout/219cba8/console.html the 4693760 node was and at 17:28:30 haproxy directed it to balance_git_https/git08.openstack.org21:24
*** ddieterly has quit IRC21:24
*** glauco__ has quit IRC21:24
*** maishsk has joined #openstack-infra21:24
fungithough it was balanced to a couple other backends around that timeframe too, so it's not a slam dunk21:24
clarkbgreghaynes had a job fail like that too hit a bunch of backends21:24
*** julim has quit IRC21:25
clarkbcould it be hp networking? any rax fails?21:25
clarkband haproxy put state in CD for three of the connectiona in that case and each was a different backend21:25
fungiwe have similar failures with the same error from rax workers too21:26
*** geoffarnold has joined #openstack-infra21:26
fungifor example http://logs.openstack.org/00/217300/2/check/gate-project-config-layout/83add3c/console.html21:26
openstackgerritPaul Van Eck proposed openstack-infra/puppet-refstack: Add param to specify app protocol  https://review.openstack.org/21946021:27
openstackgerritPaul Van Eck proposed openstack-infra/puppet-refstack: Update refstack.conf template  https://review.openstack.org/21946221:27
*** thorst has quit IRC21:29
*** hashar has quit IRC21:30
*** achanda has quit IRC21:30
*** tlian2 has joined #openstack-infra21:31
fungiclarkb: i've correlated the ipv6 address in that one to a request which got directed to git08 as well21:32
jeblairfungi: which protocol was it using?21:32
clarkbgit:// should syslog and http/https will got to apache log21:32
fungiSep  1 20:51:41 localhost haproxy[34107]: 2001:4801:7828:101:be76:4eff:fe11:99bf:53916 [01/Sep/2015:20:51:41.011] balance_git_https balance_git_https/git08.openstack.org 1/0/449 211385 -- 61/45/45/6/0 0/021:33
jeblairfungi: ah, thanks; and i see you included that in your earlier statement too, sorry.21:33
fungijeblair: well, this is a different one, but same protocol and backend21:34
clarkbfungi that doesnt look like an errored connection from haproxies perspective so likely failed above tcp21:34
*** bpoulos has joined #openstack-infra21:34
*** pc_m has quit IRC21:34
clarkbthe two --s are where the error flags go21:34
*** tlian has quit IRC21:34
jeblairfungi: okay, which repo was that last request for?21:35
fungijeblair: zuul21:35
fungii don't see any time-correlated errors in git08's /var/log/httpd/git-error.log21:37
clarkbneither do I21:37
fungi104.239.158.55 - - [01/Sep/2015:20:51:41 +0000] "POST /openstack-infra/zuul/git-upload-pack HTTP/1.1" 200 202928 "-" "git/1.9.1"21:38
clarkbmy guess is that the cgi script successfully returned failure to the cleitn and didn't treat that as exceptional?21:38
fungithat's the only request i can find which correlates time-wise and was for the correct repo21:38
*** puranamr has joined #openstack-infra21:38
clarkbI can clone from git08 and git show 7ff604053aa220bd8466b326350c4bc419e038f7 succesffuly now21:39
fungii can locally clone from the copy of the repo it's serving and checkout the commit it wanted21:39
clarkband am using trusty git 1.9.1 locally21:39
fungiyep, that too21:40
clarkbmost of the errors in the httpd logs appear related to repos that have been renamed so not even seeing any errors there that could be related at different times21:40
clarkbthere are no OOMkillers in dmesg21:41
clarkbso not seeing that behavior again21:41
*** miqui has quit IRC21:42
*** sergmelikyan has quit IRC21:42
clarkband messages doesn't show any system problems during that time21:43
clarkbwhat if git pushes from gerrit are not atomic ?21:43
clarkbcould the child commit be in the repo before the parent and cause that error?21:44
fungihow recent are those objects?21:44
clarkbhttps://review.openstack.org/#/c/187268/ couple weeks old21:45
clarkbso likely not it21:45
fungithe ones i see in the current example are 2-week-old objects21:45
*** mriedem is now known as mriedem_away21:46
clarkbthe other thing it could be is selinux maybe? let me look at the audit log21:47
clarkbthats mostly just successful git-daemon starts from systemd's socket activation inetd replacement21:48
*** annegentle has quit IRC21:48
openstackgerritDoug Hellmann proposed openstack-infra/project-config: publish docs built from releases repository  https://review.openstack.org/21942021:49
fungiyeah, i'm not finding any obvious signs as to what might be going on there21:50
jeblairfungi, clarkb; oh these are shallow clones21:50
clarkboh hrm :/21:50
clarkblet me test that then21:50
fungithis might explain why we're just seeing it on the layout job21:50
clarkbwe did test it but maybe corner case21:50
fungior at least that's the only place i've spotted it thus far21:50
jeblairyeah, both of the jjb and zuul clones in the layout job are shallow21:51
greghaynesMy fail was also in the layout job21:51
*** burgerk has quit IRC21:51
jeblairthey use https21:51
jeblairi am unable to repro locally though21:51
clarkbgit clone http://git08.openstack.org:8080/openstack-infra/zuul --depth 1 zuul-shallow wfm21:52
clarkblet me try https21:52
fungii'm a little less concerned about this being an openstack-wide emergency now if it's only impacting jobs that do shallow clones21:52
*** yamamoto has joined #openstack-infra21:53
clarkbagreed my client crashes when I set GIT_SSL_NO_VERIFY=1 I think I have it all set up proprely and when I tested I did test http, https and git protocl21:53
jeblairi had to set up an /etc/hosts entry21:54
clarkboh maybe thats it let me try21:54
clarkbI was using ip addrs when I tested21:54
jeblairbut after doing so, it wfm; i also did the same on a bare-trusty-hpcloud-b2 node and it also works21:54
clarkbyup IP addr works and is fine21:54
*** PinkFreud has quit IRC21:55
jeblairoh that's easier then :)21:55
clarkbany opposition to set -x'ing that script?21:56
clarkbwould've made catching its shallow cloning slightly easier21:56
jeblairclarkb: that doesn't seem necessary :(21:56
clarkbdefinitely won't help debug the rest of this, this is so weird that it works locally21:57
*** asettle has joined #openstack-infra21:57
jeblairit may have been a transient thing that only happens under certain conditions21:57
*** changbl has quit IRC21:57
*** yamamoto has quit IRC21:58
clarkb/tmp is part of / and has many gigs available so don't thin kwe filled a scratch space21:59
*** Hal has quit IRC21:59
*** stevemar has quit IRC21:59
clarkboh wait21:59
clarkbjeblair: fungi the error messages says it couldn't traverse parents of commit foo which makes sense since its a depth of one so only foo would exist22:00
clarkbjeblair: fungi I wonder if the server is fine but its something client side22:00
*** bpoulos has left #openstack-infra22:00
*** Piet has quit IRC22:01
*** shardy has quit IRC22:01
lifelesspbr and the new release notes stuff are somewhat incompatible with thin clones22:01
lifelessin that they look at history22:01
lifeless(perhaps irrelevant to the issue at hand - sorry if so)22:01
clarkblifeless: good point22:01
jeblairclarkb, lifeless: it appears to be the git clone itself that's failing22:01
clarkbjeblair: hard to say without -x but I think you may be right22:01
*** amitgandhinz has quit IRC22:02
jeblairclarkb: we see "Cloning into 'zuul'" but not "Cloning into 'jenkins-job-builder'" in http://logs.openstack.org/00/217300/2/gate/gate-project-config-layout/219cba8/console.html22:02
*** stevemar has joined #openstack-infra22:02
jeblairclarkb: so we know that it's failing on the first of two clone commands that are adjacent22:02
*** hdd has quit IRC22:02
*** stevemar has quit IRC22:02
jeblair(see http://logs.openstack.org/30/219330/2/gate/gate-project-config-layout/0b6afac/console.html for a successful zuul clone followed by an unsuccessful jjb clone)22:02
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515122:06
*** rlandy has quit IRC22:08
*** puranamr has quit IRC22:08
*** jsavak has quit IRC22:08
*** jcoufal has quit IRC22:09
*** e0ne has quit IRC22:10
*** glauco_ has joined #openstack-infra22:10
*** mquin_ has joined #openstack-infra22:12
clarkbthe error there is one I would expect if we attempted to look at the parent commit since the parent commit does not exist22:12
*** mquin has quit IRC22:13
clarkbwhat is confusing is why anything would want to traverse to the parent during a shallow clone22:13
*** e0ne has joined #openstack-infra22:14
*** Rockyg has joined #openstack-infra22:14
*** glauco_ has quit IRC22:15
openstackgerritAdam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config  https://review.openstack.org/21733022:16
clarkbcould it be something silly and git clone cares about the the order of args, maybe local side sees it as not shallow but remote sees it as shallow? (I am thinking out loud here don't really expect thta to be the issue)22:18
openstackgerritAdam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config  https://review.openstack.org/21733022:18
*** sigmavirus24 is now known as sigmavirus24_awa22:18
*** zeih has joined #openstack-infra22:19
rm_workhey, any chance anyone can look at this CR soon? we're trying to get this stuff in before the freeze -- I've corrected the job updates to be non-breaking (per ianw's suggestion), so hopefully not a big deal? https://review.openstack.org/21733022:20
rm_workI know it's super busy around here pretty much permanently -- if we don't make the freeze, it's not the end of the world22:20
*** regXboi has quit IRC22:21
clarkb`while git clone --depth 1 ; do rm -rf zuul ; done` is now running22:21
*** zeih has quit IRC22:23
fungijhesketh: around yet? was there a proposed implementation for creating parent tree indices for swift-hosted job logs?22:23
clarkbfungi: there is a first draft iirc22:24
clarkbmy while loop has yet to fail22:24
fungioh, cool. rbradfor is hacking on solutions to do coverage job analysis and since those are all uploaded to swift now we don't have any way to guess the urls (because of the embedded zuul uuid)22:24
rm_workdougwig: ^^22:25
clarkbfungi: we made coverage go to the log server again because of that22:25
rm_workdougwig: err, https://review.openstack.org/217330 -- should be right now I think22:25
clarkbfungi: so it should be working now22:25
*** mwagner_lap has quit IRC22:26
fungiclarkb: when did we do that? it's still going to swift right now (as in looking at samples from the last few minutes)22:26
*** gordc has quit IRC22:27
clarkbfungi: https://review.openstack.org/#/c/210114/22:27
*** edmondsw has quit IRC22:28
clarkboh wait wrong one22:28
clarkbgah I think I had the thing inverted22:28
clarkbso I fixed one issue with that change but it created the issue you are talking about22:29
openstackgerritAlan Pevec proposed openstack/requirements: Allow updated pyScss and django-pyscss on stable/kilo  https://review.openstack.org/21947422:29
clarkbwhile loop still looping22:29
*** thiagop has quit IRC22:30
openstackgerritJim Rollenhagen proposed openstack/requirements: Add ironic-lib at initial release  https://review.openstack.org/21901122:31
jrolllifeless: I should have trusted my instincts :P22:31
lifelessjroll: ?22:31
jrollthat requirements patch right there22:31
jrolllifeless: only needed the new library in upper-constraints, because that code is copied from ironic, duh :)22:32
*** earlephilhower has joined #openstack-infra22:32
lifelessping me when its passed CI22:32
jrollI guess I learned how to do the thing, at least22:32
lifelessjroll: also22:32
lifelessjroll:  you *will* want requirements syncin for it22:32
lifelessjroll: so you need check-requirements configured in zuul, and you need to add it to projects.txt too22:33
*** TheJulia has quit IRC22:33
jrolllifeless: it's already syncing22:33
jrolljust ready to be used now22:33
*** e0ne has quit IRC22:34
clarkbjeblair: fungi any other debugging ideas? should we take 08 out of the rotation and put old 07 back?22:35
jrollfungi: hitting zm04 now :/ http://logs.openstack.org/30/219330/2/gate/gate-project-config-layout/cf054de/console.html22:35
clarkbmy while loop is still running so if it is intermittent I don't see it locally22:35
rm_workianw: thanks for the feedback and the +2, this is a lot cleaner than it was originally22:39
rm_workianw: not sure why doing it this way wasn't obvious to me before :P22:39
rbradforfungi, I can now determine the directory via status.json so I believe I can build an index of directories dynamically. A challenge for tomorrow22:39
*** ddieterly has joined #openstack-infra22:39
*** jaypipes has quit IRC22:40
*** annegentle has joined #openstack-infra22:40
*** esmute_ has quit IRC22:41
jeblairjroll: it's not a zm problem, that's a git server problem (1 out of our 8 git servers is running centos7 and having problems with shallow clones)22:44
jrolljeblair: ah, fungi asked about zm earlier22:44
jeblairjroll: yeah, we, er, adjusted our hypothesis :)22:44
jrolljeblair: should I just keep rechecking for now?22:44
jrollit keeps flapping between verified and unverified22:45
*** puranamr has joined #openstack-infra22:45
jeblairjroll: yes, rechecking may produce a different outcome depending on load balancer rotation22:45
jrollk, thank you22:45
*** asselin_ has quit IRC22:46
*** esmute_ has joined #openstack-infra22:46
*** baoli has joined #openstack-infra22:48
*** gildub has joined #openstack-infra22:49
jeblairclarkb: neat i just got a local repo22:49
clarkbjeblair I really thibk it has to be a client issue22:49
clarkboh how? just repeated or something different?22:50
jeblairclarkb: going through the load balancer22:50
jeblairtook a couple of times...22:50
jeblairclarkb: but i'm wondering if the issue is due to git version mismatch across the cluster22:51
clarkbare we not using source hash balancing anymore?22:51
jeblairclarkb: we did switch to leastconn22:51
clarkbah ok22:51
jeblairso maybe the solution is to rip the bandaid off.22:51
jeblair(or, if we want to go more slowly, temp switch back to source hash)22:52
clarkbmaybe do thst so we arent rushed, but still pull bandaid quickly22:52
*** Rockyg has quit IRC22:53
jeblairit may cause other problems (like load-related timeouts), though we have a few more servers now, so maybe not as severe.22:53
openstackgerritClint Adams proposed openstack-infra/infra-specs: WIP - tracking of priority efforts in YAML  https://review.openstack.org/21937222:53
jeblairfungi: fyi ^ possible root cause22:54
*** achanda has joined #openstack-infra22:54
*** Sukhdev has joined #openstack-infra22:55
clarkbspinning up new backends is easy now, all patches landed. the only two special steps have been stopping the backend in haprpxy when dns flips and copyibg the rsa host key over so gerrit doesnt see a mismatch22:55
*** angela-s has quit IRC22:55
*** wenlock has quit IRC22:55
*** watanabe_isao has joined #openstack-infra22:55
clarkbI can roll through them quickly if we need to22:56
*** ddieterly has quit IRC22:57
*** boris-42 has joined #openstack-infra22:57
openstackgerritMerged openstack-infra/project-config: Switch ironic full tempest to use IPA  https://review.openstack.org/21933022:58
clarkb(if you don't copy the host key we would need to restart gerrit in order to get replication working)22:58
*** achanda has quit IRC22:59
*** jogo has quit IRC22:59
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support default string values for variables  https://review.openstack.org/18011723:00
*** vivekd has quit IRC23:00
*** bharathm has quit IRC23:01
*** vivekd has joined #openstack-infra23:01
clarkbStevenK: awake yet? you have details for item 3 https://etherpad.openstack.org/p/zanata-deploy ? would be good to get those into the etherpad23:02
*** dims has quit IRC23:03
*** jogo has joined #openstack-infra23:05
*** jogo has quit IRC23:05
*** jogo has joined #openstack-infra23:05
*** sridhar_ram has quit IRC23:05
*** asselin has joined #openstack-infra23:06
*** ivar-laz_ has joined #openstack-infra23:07
*** Piet has joined #openstack-infra23:08
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889823:10
fungijroll: clarkb: yeah, the clone failures and the performance issues on zm02 seem to be unrelated, but zm02 has still been having other issues (for which i opened a ticket with rax)23:10
*** ivar-lazzaro has quit IRC23:10
jrollfungi: ew tickets23:11
*** legoktm is now known as egoktm23:11
*** jswarren has quit IRC23:11
fungiclarkb: yeah, other potential challenge with spinning up the rest of them is we're very close to quota23:11
clarkbI think switching balance type will also confirm if thats the bug23:13
*** yamamoto has joined #openstack-infra23:13
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889823:13
*** asselin has quit IRC23:14
*** amuller has quit IRC23:14
*** egoktm is now known as legoktm23:14
*** asselin has joined #openstack-infra23:14
*** yamamoto has quit IRC23:16
fungieasy enough to toggle23:16
*** eantyshev has quit IRC23:17
*** eantyshev has joined #openstack-infra23:17
*** chlong_ is now known as chlong23:18
*** olaph has joined #openstack-infra23:18
openstackgerritMerged openstack/requirements: require SQLAlchemy >=0.9.9  https://review.openstack.org/21636023:18
*** puranamr has quit IRC23:18
*** puranamr has joined #openstack-infra23:19
*** zeih has joined #openstack-infra23:19
*** watanabe_isao has quit IRC23:20
*** shashank_hegde has quit IRC23:21
*** mdbooth has quit IRC23:21
docaedoyolanda: took me all day to circle back to it, but I think I made 218898 do what you suggested - let us know how it looks now, thanks!23:21
*** tonytan4ever has quit IRC23:22
*** ZZelle_ has quit IRC23:23
*** puranamr has quit IRC23:23
*** dims__ has joined #openstack-infra23:24
*** zeih has quit IRC23:25
*** MarkAtwood has quit IRC23:25
*** mdbooth has joined #openstack-infra23:26
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889823:29
*** dannywil_ has joined #openstack-infra23:30
*** ivar-laz_ has quit IRC23:33
*** wenlock has joined #openstack-infra23:33
*** dannywilson has quit IRC23:33
*** FallenPe_ has quit IRC23:36
*** hodos has quit IRC23:38
*** puranamr has joined #openstack-infra23:39
jgriffithfungi: clarkb either of you know how to fix a patch that refuses to get picked up by the verify stage?23:40
jgriffithfungi: clarkb mtanino and I have tried a number of things, "reverify" just add another +2/A etc... even took it back through check23:40
jgriffithfungi: clarkb but it seems to be *stuck*23:41
*** bharathm has joined #openstack-infra23:41
*** geoffarnold has quit IRC23:43
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support default string values for variables  https://review.openstack.org/18011723:43
*** salv-orlando has joined #openstack-infra23:43
*** salv-orl_ has quit IRC23:47
*** puranamr has quit IRC23:48
jgriffithmtanino: well I tried anyway... need to head out, now23:52
jgriffithmtanino: I'll check in later tonight if I can, or in the morning23:52
*** eantyshev has quit IRC23:52
mtaninojgriffith: yeah, I hope infra member will check that message.23:52
mtaninojgriffith: Thanks :)23:52
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues  https://review.openstack.org/21951423:53
*** puranamr has joined #openstack-infra23:53
*** esker has joined #openstack-infra23:53
*** jhesketh has quit IRC23:54
*** jhesketh has joined #openstack-infra23:55
fungijgriffith: looking23:55
openstackgerritDarragh Bailey proposed stackforge/python-jenkins: Migration to using requests  https://review.openstack.org/20563723:55
*** dingyichen has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!