Thursday, 2015-12-17

mikalWhat does "not registered" mean as a test result?00:00
zaroclarkb: fungi is seeing timeouts with some of the diffs00:00
fungijeblair: we'll likely want to deploy a replacement server, but it should be trivial00:00
*** alexsyip has joined #openstack-infra00:00
Sam-I-Amanteaya: how are you this week?00:00
jeblairof course i'm uploading a patch at utc midnight00:01
fungihello database backup time!00:01
*** tjones has quit IRC00:01
zaroactually diff cache memory is defaulted to 10M, maybe we should just double it?00:01
docaedojeblair: huh .. I get it every time, triggers immediately for me - I'll try to recreate somewhere else before I pester any more :)00:02
jeblairzaro, clarkb: perhaps "rate" or "count" of timeouts could be measurable00:02
asselin_fungi, thanks for answering my question. didn't even have to ask.00:02
*** baoli has quit IRC00:02
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command
fungizaro: yes, i have no trouble believing we could be totally blowing out a 10mb diff cache and getting a lot more cache misses than intended00:03
*** sabeen3 has joined #openstack-infra00:03
openstackgerritJames E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release
jeblairclarkb, fungi, greghaynes, mordred: is a companion to but merits more thought/discussion00:04
anteayaSam-I-Am: I'm good thanks00:04
anteayaSam-I-Am: thanks for asking00:04
zarojeblair: do you an idea how to measure, besides greps logs i guess?00:05
Sam-I-Amanteaya: cold up there?00:05
docaedojeblair: oh so sorry .. that's the new SNI warning you just fixed!00:05
fungii still find it odd that it would take 5 seconds to do a diff, but i'm sure gerrit is plenty busy and maybe jgit is not well optimized?00:05
anteayaSam-I-Am: no, I'm disappointed00:05
anteayaSam-I-Am: september weather00:05
anteayajust a skift of snow today00:05
Sam-I-Amanteaya: had 8" of snow here... went snow-biking for lunch.00:05
anteayano frost in the ground00:05
jeblairneat, the ps2 i uploaded at 00:00 disappeared00:05
anteayaSam-I-Am: nice work00:05
*** sabeen has quit IRC00:06
*** thorst has quit IRC00:06
Sam-I-Amrode for an hour at full power... average speed: 300:06
anteayajeblair: :\00:06
jeblairlet me make a trivial change to it00:06
jlvillalSo I seem to be unable to push a rebased patch. It keeps saying, "! [remote rejected] HEAD -> refs/publish/master (no new changes)"00:06
*** spzala has joined #openstack-infra00:06
anteayaSam-I-Am: well done00:06
jeblairjlvillal: did you start trying to do that 6 minutes ago?00:06
anteayajlvillal: which patch?00:06
jlvillalThe patch is the same, but I did rebase it so it is based off tip of tree.00:06
jlvillaljeblair, About then00:06
jeblairjlvillal: yeah, same thing happened to me; the patch is in gerrit's git tree, but not in the database because db backups happened then and i think the query timed out00:07
jlvillaljeblair, First time I pushed it gave a different error00:07
jlvillaljeblair, " ! [remote rejected] HEAD -> refs/publish/master (internal error)00:07
jlvillalWas first error00:08
*** Qiming has quit IRC00:08
jlvillaljeblair, Do I just wait?00:08
anteayajlvillal: well if we are losing patches during a db backup that is new and isn't great00:08
jlvillalanteaya, I agree!00:09
openstackgerritJames E. Blair proposed openstack-infra/system-config: Switch to using twine from packages
openstackgerritJames E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release
anteayajlvillal: I'm guessing jeblair is checking something00:09
*** harlowja has joined #openstack-infra00:09
jlvillalOkay. I'll be patient. Thanks00:09
jeblairjlvillal: no, i doubt they will ever show up.  i made insubstantial changes to my commit messages for gerrit to accept them as new patchsets.00:09
nibalizerclarkb: hey you about? my nodepool isn't cooperating00:09
*** amuller has quit IRC00:09
nibalizeri'm getting Authentication (publickey) failed.00:09
jeblairjlvillal: so if you look at you'll see PS1 and then PS300:09
jlvillaljeblair, Okay. I will do something similar00:09
nibalizerthis is from snapshot, not dib00:09
clarkbnibalizer: ya just sat back in front of laptop00:09
nibalizerbut when i use the provided credentials i can ssh in just fine00:09
clarkbnibalizer: nodepool will use the pubkey for hte user it runs as by default00:10
clarkbnibalizer: er key00:10
clarkbso you have to put that pubkey on the hosts00:10
nibalizerit is00:10
nibalizerunder the jenkins user00:10
jlvillaljeblair, I did that and I jumped for Patch set 4 to 6.
nibalizerso i ssh jenkins@template-host just fine from the nodepool user00:11
nibalizerbut when image-update tries it fails00:11
clarkbnibalizer: and when you say you do it by hand are you logging in as jenkins using that same key or ar eyou using cloudinit/glean stuff?00:11
jeblairjlvillal: nice! you jumped twice as far as i did!00:11
clarkbnibalizer: image-update doesn't use that key, it makes a temp key iirc00:11
nibalizerclarkb: using jenkins00:11
Sam-I-Amanteaya: do you often spy in #openstack ?00:11
nibalizerclarkb: oh00:11
anteayaSam-I-Am: I click around00:11
nibalizerclarkb: then why does it have a key?00:11
anteayaSam-I-Am: I see you working openstack a lot and leave you to it00:11
clarkbnibalizer: the key on the image is the key to use once image is created00:12
Sam-I-Amanteaya: front line support!00:12
anteayamost questions in there I can't see how anything I have to offer can help (and make me happy for doing so)00:12
Sam-I-Ambut yields lots of happy openstack users00:12
clarkbnibalizer: but in order to bootstrap image creation it keygens and passes that along via metadata/config-drive00:12
nibalizeromfra-cloud doesn't have key support00:12
anteayaSam-I-Am: complete with safety glasses00:12
nibalizerer metadata/config-drive support00:12
anteayaSam-I-Am: indeed00:12
Sam-I-Amplus i think its a really good feedback mechanism for docs, among other things00:12
clarkbnibalizer: it needs that00:12
clarkbnibalizer: config-drive is probably preferable00:12
*** sabeen3 has quit IRC00:12
anteayaSam-I-Am: you contribute a lot to the happy users bottom line00:12
Sam-I-Amalso makes me realize i could work on docs 24/7 for years and not make a dent :/00:12
clarkbmetadata server is less reliable00:13
anteayaSam-I-Am: true that00:13
* anteaya waits00:13
jeblairwe don't need that for omfra00:13
jeblairnibalizer, clarkb: ^00:13
greghaynesJust need dhcp00:13
clarkbjeblair: the way nodepool works we do00:14
clarkbwe can change how nodepool works if we want00:14
jeblairclarkb: we can just upload images00:14
anteayaokay I'm going to evening for a few hours00:14
clarkboh you are saying no snapshots00:14
clarkbya we can do that instead00:14
jeblairclarkb: yes00:14
Sam-I-Amanteaya: evening before it becomes nighting00:14
anteayasee y'all later or tomorrow00:14
clarkbgreghaynes: nodepool needs one or the other when building snapshot images to set up keys00:14
anteayaSam-I-Am: exactly00:14
jeblairi've been following the no-snapshots policy for new clouds00:14
*** Daisy has quit IRC00:14
zaroanteaya: have a good evening, thanks for helping00:14
clarkbjeblair: roger00:14
*** tjones1 has quit IRC00:14
anteayaSam-I-Am: thanks you too00:14
clarkbnibalizer: so then ignore snapshots and just do dib00:14
greghaynesclarkb: ya, just bake in keys00:14
anteayazaro: thanks, awesome work today00:14
anteayazaro: talk to you tomorrow00:14
clarkbthta said, config-drive being an option is probably reasonable00:15
jeblairso if we want to add it as a learning experience or for some other thing, cool, just saying we don't need to block on it for nodepool00:15
crinkleour infra-cloud doc says "The cloud will have config drive enabled because we believe it to be more robust  than the EC2-style metadata service.", we'll still need that i think?00:15
clarkbthen you can opt into it00:15
jeblairclarkb: right00:15
zaroyep, may see a 'ha ha ha' patch from me00:15
*** zz_dimtruck is now known as dimtruck00:15
*** Daisy has joined #openstack-infra00:15
clarkboh I know what we need it for00:15
clarkbso its actually required00:15
clarkbgreghaynes: why doesn't that work on clouds that dhcp then?00:16
greghaynesGood question00:16
clarkbis it because cloud-init/glean overwrite?00:16
crinklethe way it is now the nodes take a while to boot because they're waiting to time out finding the metadata service, so i'd like to fix that00:16
greghaynesIt shouldn't, but it might00:16
zaroclarkb: am i reading that right? it says cache "changes" is disabled by default?00:16
greghaynesclarkb: it could also be dib setting a fail hostname that doesn't get overwritten00:17
clarkbzaro: ya its disabled by default00:17
clarkbgreghaynes: or neutron/nova don't provide hostname via dhcp00:17
greghaynescrinkle: use non cloud-init nodes00:17
openstackgerritBen Kero proposed openstack/diskimage-builder: Replace sfdisk partitioning with parted
zaroclarkb: ohh, i think that should be enabled.00:18
greghaynesclarkb: possible, we should look into that00:18
greghaynesSounds like a useful thing to do00:18
clarkbjeblair: +2 on system twine00:20
*** masber has joined #openstack-infra00:20
clarkbjeblair: for the wheel change I can look at wheel's changelog and see if there is anything important there we would miss00:21
greghaynesThe wheel thing doesn't have implications for constraints, correct?00:21
fungias soon as i get settled i'll approve the twine change and deploy a replacement release.s.o.o, then retrigger the upload jobs for dims00:21
clarkbgreghaynes: not for the release slave since it doesn't run install code00:21
clarkbgreghaynes: its straight get file upload file00:21
*** Daisy has quit IRC00:22
greghaynesAh, awesome00:22
pabelangerpuppetdb.o.o seems quiet.  111nodes unreported in the last 1.5 hours00:22
clarkbtrusty has 0.2400:22
pabelangersomething going on with ansible?00:22
clarkb0.26 looks like latest00:22
dimsfungi : thanks for reference and did not make it to pypi00:22
clarkbpabelanger: we turned it off for gerrit upgrade00:22
*** apuimedo_ has quit IRC00:22
pabelangerclarkb: Ah, when do you suspect it will come back online?00:23
clarkbpabelanger: I am not sure I had to pop out to visit the eye doctor and am not up to date on upgrade state00:23
*** oomichi has joined #openstack-infra00:23
*** Daisy has joined #openstack-infra00:23
clarkbjeblair: based on that I htink if we don't use it with python3 we will be ok00:23
pabelangerfungi: when you break away from the pizza ^00:23
jeblairdo we use wheel at all there?00:24
clarkbjeblair: the only thing that we may want is the 'arch' tag support00:24
clarkbjeblair: I htink it may be used to grok the wheels we upload00:24
jeblairah, i'll grep around00:24
clarkbjeblair: but at that point the wheels are already generated00:24
fungioh, we haven't reenabled puppet. any objections to turning it back on now?00:24
fungipabelanger: thanks for the reminder00:24
clarkbjeblair: so its use would be minimal00:24
fungiour maintenance plan had a disable puppet step, no reenable step :/00:24
fungii should have caught that00:24
jeblairclarkb: yes in jenkins/scripts/pypi-extract-name.py00:25
jeblairthat's all we use it for00:25
jeblairso cool00:25
jeblairi'll amend the commit message for posterity00:25
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command
*** thorst has joined #openstack-infra00:26
jeblairnot sure why no gerritbot00:26
*** dimtruck is now known as zz_dimtruck00:27
*** Daisy has quit IRC00:27
mattoliveraufyi, the new gerrit's reply (review) doesn't mention workflow -1 is WIP, so this could get confusing for new users.00:27
*** apuimedo_ has joined #openstack-infra00:28
jeblairmattoliverau: unless you click on it, you mean?00:28
*** zz_dimtruck is now known as dimtruck00:29
*** dims has quit IRC00:29
mattoliveraudon't know just saw it, so thought I'd mention it00:29
*** thorst has quit IRC00:29
fungii haven't seen any objections to reenabling puppet, so doing that now00:29
jeblairoh, gertty failed to update the commit message00:30
*** thorst has joined #openstack-infra00:30
jeblair2015-12-16 16:30:59,102 Received: Not found: message00:30
openstackgerritApoorva Deshpande proposed openstack-infra/ciwatch: Separate votes from Jenkins queues
jeblairit totally sent "message"00:30
jeblairthe endpoint changed?00:31
jeblairPUT /changes/{change-id}/edit:message00:32
jeblairthat used to just be "/changes/{change-id}/message" before00:32
fungiapi break00:32
* fungi shakes fist ineffectually00:32
jeblairgreat... can we query the version?00:32
fungizaro: ^ is the gerrit api versioned?00:33
fungi(rest api)00:33
fungipabelanger: if puppetboard is still quiet half an hour from now, let me know00:34
sdaguezaro: ok, so I've got 3 days left in the "office" this year. I'd love to be part of the design conversation for that plugin from the UX side as I hacked a lot of the current interface. But I won't be able to really do anything about that until 201600:36
*** apuimedo has quit IRC00:36
*** apuimedo_ has quit IRC00:37
*** apuimedo has joined #openstack-infra00:38
jeblairi just checked headers... nothing there.00:38
docaedojeblair: I think my issue has to do with ubuntu and openssl (the ssl v3 handshake thing), just for my sanity, what OS did you try under?00:38
*** thorst has quit IRC00:38
cloudnullNew gerrit looks nice! well done everybody.00:39
jeblairdocaedo: Ubuntu 14.04.3 LTS00:39
* cloudnull goes to push buttons00:39
clarkbjeblair I think you can check gerrit version00:39
clarkbbut not api version00:39
jeblairclarkb: that would work well enough; any clues as to where?00:39
docaedojeblair: unpossible! drat! but thanks, I'll try in a clean VM and see what happens :)00:40
clarkbgerrit version is ssh command00:40
jeblairclarkb: that doesn't work so well for gertty :(00:40
clarkbwell it may be hhtp too00:40
clarkbI just dont know what that is00:41
jeblairright, i'm looking and not finding00:41
jeblairfound it!00:41
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Skip dsvm jobs for ironic-python-agent if code doesn't change
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Add reno jobs for ironic-python-agent
*** apuimedo_ has joined #openstack-infra00:42
*** otter768 has joined #openstack-infra00:42
openstackgerritJames E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release
jeblairi'll fix gertty tomorrow00:43
fungii take it that works00:43
*** dimtruck is now known as zz_dimtruck00:43
jeblairnah, i updated that in the web ui00:43
fungioh. poo00:43
fungibut good that there's a path forward00:44
jeblairfungi: yep.  (though i don't have a 2.8 server anymore!)00:44
jeblairediting in the ui is a little weird...00:44
jeblairyou open the file, then click "edit"... that's normal enough00:44
jeblairwhen you're done, you click save, then go back to the change screen, then click "done editing", then click "publish edit"00:45
openstackgerritTianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support
jeblairi was not quite expecting so many "yes i'm done" steps.00:45
fungithat does seem more cumbersome than necessary00:45
fungianyway, once those two merge i'll launch a new release.s.o.o00:45
jeblaircurl  ->  2.11.4-11-ga14450f00:45
jeblairso yeah, looks like a path forward00:46
funginot what i'd call a "versioned api" but it's better than "no clue" i guess00:46
*** apuimedo_ has quit IRC00:46
fungii suppose you'll also want to know which gerrit version changed that method00:47
*** gildub has quit IRC00:47
fungioh, and of course you have no way to confirm that the version lookup even works the same way on 2.800:47
*** otter768 has quit IRC00:47
jeblairyeah, i was going to go with "try GET; except: assume 2.8" :)00:47
jeblairwe can always refine later00:48
*** mgrosz has joined #openstack-infra00:48
jeblair(i should probably check the version everytime gertty switches back into online mode)00:49
jeblair(since, after all, like today, it might change)00:49
*** mgrosz has quit IRC00:50
*** rcernin has quit IRC00:50
fungiindeed it might00:51
*** ayoung has quit IRC00:51
zarook. i've met my quota of excitement for the day.  going to decompress00:51
*** apuimedo_ has joined #openstack-infra00:52
fungiit was a very good day zaro--thanks for working on the upgrade!00:52
pleia2yes, thanks zaro! have a nice evening00:53
*** masber has quit IRC00:53
*** pvaneck has quit IRC00:53
olaphnice work, zaro!00:53
*** Swami_ has quit IRC00:54
*** angdraug has quit IRC00:55
*** xarses has quit IRC00:55
*** Qiming has joined #openstack-infra00:56
*** MarkAtwood has quit IRC00:56
*** ajmiller has quit IRC00:57
*** chenli has joined #openstack-infra00:58
chenlihello, the page show for is different today ??00:59
chenlianyone know how to change it back ???01:00
*** eil397 has quit IRC01:00
clarkbchenli: for the change screen? the version we upgraded to today no longer has the old screen01:00
pleia2chenli: yes, it upgraded it to version 2.11 today, the old dashboard is no longer available01:01
chenli... I can't find the line I get comment...01:01
*** apuimedo_ has quit IRC01:01
*** apuimedo has quit IRC01:01
pleia2chenli: I recommend trying it out for a little while, it gets easier :)01:01
openstackgerritSachi King proposed openstack-dev/pbr: Handle markers to support sdist on pip < 6
clarkbchenli: can you link to an example change where you can't see the comment?01:02
*** thorst has joined #openstack-infra01:02
chenlipleia2: oo....  it is a very big change01:02
*** dguitarbite has joined #openstack-infra01:02
*** apuimedo has joined #openstack-infra01:02
chenliclarkb: I just find it.... enter into files, the same as the old one...01:02
pleia2there you go01:03
openstackgerritMerged openstack-infra/system-config: Switch to using twine from packages
*** bharathm has quit IRC01:03
*** bpokorny has quit IRC01:04
openstackgerritTianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support
fungithere's one01:04
chenlipleia2: clarkb : thanks !01:04
*** masber has joined #openstack-infra01:05
*** bpokorny has joined #openstack-infra01:05
*** apuimedo_ has joined #openstack-infra01:06
*** chlong has joined #openstack-infra01:07
*** apuimedo_ has quit IRC01:11
*** gyee has quit IRC01:11
openstackgerritSean Dague proposed openstack-infra/system-config: narrow the 3 columns progressively with screen width
*** thorst has quit IRC01:12
dguitarbiteHello, I need some help with regards to releasing for training-labs01:13
*** thorst has joined #openstack-infra01:13
dguitarbitefungi: Can you help me with releasing kilo for training-labs?01:13
fungidguitarbite: probably not at this time of my night. my dance card is still pretty full. might help though to describe what you mean by "releasing kilo for training-labs"01:14
*** alexsyip has quit IRC01:16
*** apuimedo_ has joined #openstack-infra01:16
dguitarbitefungi: I just want to add tags for Juno and Kilo for training-labs. Which I assume acts as releases.01:16
dguitarbitefungi: I remember doing the same for Icehouse (pinging you for it). Is there a better way?01:17
*** ParsectiX has joined #openstack-infra01:17
*** ashleighfarnham has quit IRC01:17
fungidguitarbite: i honestly don't remember what we did for the icehouse training-labs release, but that sounds possible01:17
*** tjones has joined #openstack-infra01:17
dguitarbitefungi: Ok, how do I release? By that I want to add git tags.01:18
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Use separated ACL group for OSprofiler
ianwgreghaynes / anyone : so is configdrive the default on RAX?  or do you have to be special to get it?01:18
clarkbianw: its a nova boot flag01:18
jrollianw: it is not default for virt01:19
jroll--config-drive ''01:19
fungidguitarbite: is our documented process. no idea if the training-labs repo is typical of most projects though01:19
ianwjroll: ok ... it is not exposed via the web interface at all?01:19
openstackgerritYan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job
jrollianw: not that I know of - I think if it allows passing "user data" that will trigger configdrive01:19
jrollbut not sure01:20
ianwok, i have never started a rax node using !web ... let me see here01:20
ianwi know horizon has a tick box for it01:20
jrollnova boot --image bla --flavor bla name01:20
* jroll has to run; good luck!01:20
*** thorst has quit IRC01:21
*** mudassirlatif has quit IRC01:22
dguitarbitefungi: thanks, let me take a look.01:22
*** skath_ has quit IRC01:23
*** skath_ has joined #openstack-infra01:24
*** etoews has quit IRC01:24
*** bgmccollum has joined #openstack-infra01:25
*** zhiyan has quit IRC01:25
*** mgkwill has quit IRC01:25
*** etoews has joined #openstack-infra01:25
*** zxiiro has quit IRC01:25
fungistill waiting on two devstack-centos7 nodes for 258726 to get through check01:26
*** maximov has quit IRC01:26
*** mgkwill has joined #openstack-infra01:28
*** maximov has joined #openstack-infra01:28
*** zhiyan has joined #openstack-infra01:28
*** ningy has joined #openstack-infra01:28
*** zxiiro has joined #openstack-infra01:28
fungirackspace is taking upwards of an hour to boot instances again01:28
fungi| 6625225 | rax-ord                | None | devstack-centos7       | jenkins06 | devstack-centos7-rax-ord-6625225            | devstack-centos7-rax-ord-6625225            | 9b605aba-17be-4512-85cc-2342dae94f69 | None            | building | 00:00:59:15 |01:28
ningy@here could anyone help me taking a look at this patch to make sure I'm doing the right thing?
*** apuimedo_ has quit IRC01:30
ningyI'm trying to only enable certain services for devstack install and disable all other services that i don't need to have. any help is much appreciated!01:31
*** apuimedo_ has joined #openstack-infra01:35
*** zhenguo has joined #openstack-infra01:38
openstackgerritTianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support
fungiit looks like failure to boot in rackspace is starving out most of our devstack-centos7 capacity. we've got a few in used state in hpcloud, and a bunch building forever in rackspace01:40
*** apuimedo_ has quit IRC01:40
*** mgrosz has joined #openstack-infra01:42
*** mgrosz has quit IRC01:43
openstackgerritMathieu Gagné proposed openstack-infra/system-config: Fix top element in hideci.js for new change screen
mgagnezaro this fixes hideci.js which isn't hiding the correct element anymore ^01:44
*** apuimedo_ has joined #openstack-infra01:45
mgagnejeblair about hideci.js ^01:45
*** rvasilets___ has quit IRC01:45
mgagneoh it's 8:45pm EST already :-/ will go home then01:46
*** annegentle has joined #openstack-infra01:48
*** mriedem has joined #openstack-infra01:48
*** hdd has quit IRC01:48
*** signed8bit has quit IRC01:49
*** apuimedo_ has quit IRC01:50
*** apuimedo_ has joined #openstack-infra01:55
openstackgerritYan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job
*** tjones has quit IRC01:57
*** masber has quit IRC01:58
*** mtanino has joined #openstack-infra01:59
*** apuimedo_ has quit IRC02:00
*** aeng has quit IRC02:02
mriedemanyone noticed that the multinode jobs are failing as NOT_REGISTERED?02:03
mriedemsomething up with test nodes?02:03
*** amotoki has joined #openstack-infra02:03
*** apuimedo_ has joined #openstack-infra02:04
*** amotoki_ has joined #openstack-infra02:05
greghaynesmriedem: there were/are issues with rax going on02:06
*** thorst has joined #openstack-infra02:08
*** amotoki has quit IRC02:09
*** thorst has quit IRC02:09
*** thorst has joined #openstack-infra02:09
*** chlong has quit IRC02:10
*** triggerwarning has joined #openstack-infra02:11
*** bpokorny_ has joined #openstack-infra02:12
openstackgerritJoey D. proposed openstack/diskimage-builder: Closes bug: 1486689
openstackbug 1486689 in diskimage-builder "Ramdisk should consider the size unit when inspecting the amount of RAM" [Medium,In progress] - Assigned to Brad Lodgen (lodgenbd)02:12
*** yamamoto has joined #openstack-infra02:14
*** bpokorny has quit IRC02:16
*** apuimedo_ has quit IRC02:16
*** bpokorny_ has quit IRC02:16
mriedemgreghaynes: ok, thanks, rechecked02:18
*** thorst has quit IRC02:18
*** sridhar_ram has quit IRC02:19
*** yamahata has quit IRC02:19
*** baoli has joined #openstack-infra02:20
*** baoli has quit IRC02:22
*** apuimedo_ has joined #openstack-infra02:22
*** chlong has joined #openstack-infra02:22
*** baoli has joined #openstack-infra02:22
*** [1]Thelo has joined #openstack-infra02:23
fungi17 devstack-centos7 nodes building in rackspace many for close to an hour. none ready or used at this point anywhere02:25
*** Thelo has quit IRC02:25
*** [1]Thelo is now known as Thelo02:25
* fungi sighs02:25
fungiso still waiting for 258726 to pass check02:26
*** apuimedo_ has quit IRC02:26
fungiit looks like it may just be a problem with the centos7 images, so likely we've merged something that's broken them in rackspace but not in hpcloud02:29
fungiianw: ^ any ideas?02:29
fungii'm going to roll back the devstack-centos7 images in rax regions to see whether that works around it for now02:30
ianwfungi: not immediately ...02:30
ianwmy monitoring shows the nodepool builds all seem ok...02:31
*** sputnik13 has quit IRC02:32
openstackgerritgreghaynes proposed openstack/diskimage-builder: Make check for image-should-fail quiet
*** apuimedo_ has joined #openstack-infra02:32
greghaynesbkero: ^02:32
fungiianw: i mean they build fine, but aren't able to boot in rackspace02:33
*** yamamoto has quit IRC02:33
ianwyeah, so ... well, no idea, any logs will help :)02:33
fungiat this point i just want that pending system-config change to finish getting tested and merge so i can replace the release worker and finish fixing the stuck releases02:34
openstackgerritIan Wienand proposed openstack-infra/glean: install systemd service as 0644
greghaynesianw: monitoring?02:35
*** baoli has quit IRC02:35
ianwgreghaynes: -> "monitoring" ... at least lets me see big failiures02:35
*** baoli has joined #openstack-infra02:36
greghaynesianw: so, I have a dream of something like that up to an irc bot02:36
ianwin build, but doesn't help with boot02:36
ianwgreghaynes: well, i can easily put you on the cc list if you want it sent every day02:36
*** mgrosz has joined #openstack-infra02:36
fungiright, we're mostly blind to problems (like this) with failing to boot an instance from a successfully built image02:36
greghaynesIt wouldn't be hard to do a similar thing to the dib CI where we qemu boot an image02:37
greghaynesand just ensure we ping it02:37
fungiwell, it seems to be provider-specific02:37
*** piet has joined #openstack-infra02:37
fungiso in this case we upload one image to two providers, it's booting in one and not the other02:38
greghaynesIt is stuck in building?02:38
fungioh, though these are probably snapshot images not dib02:38
*** mgrosz has quit IRC02:38
greghaynesor they just dont get networking?02:38
greghaynesah, I cant help you there ;)02:38
fungigreghaynes: nodepool shows devstack-centos7 nodes "building" in rackspace for about an hour and then switches them to delete02:39
fungiand since it counts building nodes as satisfying demand, we basically just wind up with no capacity and accumulate nodes stuck in a building state in the broken provider02:39
*** dguitarbite has quit IRC02:40
* greghaynes tries to remember when nodepool considers something done building02:40
*** dguitarbite has joined #openstack-infra02:41
fungiaccording to "nova show" they're in status BUILD the entire time02:42
fungiso nova finally gives up waiting for them to go active02:42
fungiwhich in cases like this usually means some sort of boot-time problem where the instance never fully boots up02:42
greghaynesIf they are dib I have seen things like that happen if you dont set hvm on the images02:43
greghaynesalso saw similar things in hpcloud when we didnt set qcow2 compat02:43
fungii don't think devstack-centos7 is dib in rax yet, but i'll confirm02:43
*** otter768 has joined #openstack-infra02:43
*** gongysh_ has joined #openstack-infra02:44
greghaynesYa, I think it isnt02:44
fungiyeah, not dib02:44
fungisnapshots still02:44
* greghaynes points fingers at rax02:44
*** apuimedo_ has quit IRC02:44
fungimeh, we could still easily be setting something on them that's not working as expected02:44
fungifor example, the mem= kernel command line change we made in the last 24 hours02:45
greghaynesright, that02:45
*** otter768 has quit IRC02:48
pabelangerianw: greghaynes: I want to get grafana tracking dib builds too. I should see what needs to be added into nodepool to publish into to statsd02:50
pabelangerwe have some info, but I don't think enough02:50
*** apuimedo_ has joined #openstack-infra02:50
greghaynesI just want something to tell me when they failed, and I am bad at email02:50
pabelangerianw: script does that today :)02:51
pabelangershow up in my inbox at noon every day02:51
*** piet has quit IRC02:52
*** jyuso1 has joined #openstack-infra02:52
*** gongysh_ has quit IRC02:53
zaro                  move forward [];02:53
pabelangerSo, how is OVH holding up to nodepool?02:53
fungiokay, the last job for 258726 finally got a worker in the gate02:54
*** apuimedo_ has quit IRC02:54
zaro              move right [];02:55
ianwfungi: so obviously centos is priority, not sure what i can do but willing to help.  on f23 ... i'm quite out of ideas about why it's not working.  manual testing of glean on rax (with configdrive) seems to show it configuring interfaces ok02:56
fungiput your hands on your hips, then pull your knees in tight?02:56
greghaynesPhone derp02:57
fungiianw: yeah, i'm not really sure about my centos7 theory either. even rolling back to yesterday's images the only ones i see coming online still are booting in hpcloud not rax02:57
greghaynesianw: did you get a change in nodepool to use glean from git?02:57
*** amotoki_ has quit IRC02:57
zaro                o02:57
openstackgerritMerged openstack-infra/system-config: Switch to using wheel from distro on release
*** harlowja has quit IRC02:58
ianwgreghaynes: yes, the f23 images are installing glean from git02:58
greghaynesianw: oh, and they are still broken?02:58
ianwgreghaynes: seemingly, yes.  NOT_REGISTERED when i try a job.  could be specific ... might even be related to centos issues?02:59
greghaynesYa, I wouldn't be surprised02:59
ianwgreghaynes: best i can do is bring up a f22 node in rax, uninstall cloud-init, remove the interface stuff and reboot with glean02:59
ianwthat *seems* to work02:59
*** baoli has quit IRC03:00
greghaynesI can resurrect my set up to upload vhds to rax if it turns out to not be that03:00
*** apuimedo_ has joined #openstack-infra03:00
ianwgreghaynes: do you need special permissions to do that?03:00
ianwbecause it would be super duper handy to be able to test the image without nodepool reaping it03:00
greghaynesI don't think so, but you need vhd-util03:00
*** masber has joined #openstack-infra03:01
greghaynesAnd you need to go through the motions of uploading to swift, importing l, setting hvm03:01
*** elo has quit IRC03:01
*** apoorvad has quit IRC03:02
*** piet has joined #openstack-infra03:02
openstackgerritYan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job
*** apuimedo_ has quit IRC03:05
ianwfungi: i wonder, can you "share" the f23 image?
fungiianw: i could stick a copy of it in the web root, i bet03:10
fungiianw: hrm, nope. we're tight on space in that filesystem03:10
*** apuimedo_ has joined #openstack-infra03:10
fungiwould require some tweaking to expose another location, and i'm still trying to get the release.slave replacement online03:11
*** EricGonczer_ has joined #openstack-infra03:11
fungiand desperately want to finish fixing the missing releases so i can call it a night03:11
openstackgerritYan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job
*** mriedem has quit IRC03:13
*** yonglihe has joined #openstack-infra03:14
*** yamamoto has joined #openstack-infra03:15
*** thorst has joined #openstack-infra03:15
ianwgreghaynes: sooo... the fedora vhd-util doesn't seem to "convert"03:15
*** hdd has joined #openstack-infra03:17
*** xarses has joined #openstack-infra03:18
*** ashtokolov has quit IRC03:18
*** apuimedo_ has quit IRC03:20
openstackgerritYan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job
*** ashtokolov has joined #openstack-infra03:20
fungianybody know why we're trying to install etcd on trusty?03:23
*** thorst has quit IRC03:23
fungiE: Unable to locate package etcd03:23
fungiso sayeth apt-get03:23
fungiwhen trying to build this replacement server03:23
*** unicell1 has quit IRC03:24
fungiError: /Stage[main]/Openstack_project::Thick_slave/Package[etcd]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install etcd' returned 100: Reading package lists...03:24
*** apuimedo_ has joined #openstack-infra03:25
fungii sort of feel like i don't need yet another thing to dig into tonight, so i'm just going to hope it's benign03:26
ianwhmm, smells like dlm03:26
*** woodster_ has quit IRC03:26
*** jamielennox is now known as jamielennox|away03:26
ianwgreghaynes: "If using the VHD output format you need to have a patched version of vhd-util installed for the image" so yeah ... you just gotta read :)03:26
fungiyeah, seems unnecessary on our release slave03:26
*** links has joined #openstack-infra03:28
fungiugh, someone approved addition of etcd package on our slaves without confirming it actually existed in those distros?!?03:29
fungii couldn't help but look03:30
*** mgrosz has joined #openstack-infra03:31
lifelessfungi: in that vein you'll *adore* the consul driver setup script for tooz tests03:31
*** tzn has joined #openstack-infra03:31
*** jamielennox|away is now known as jamielennox03:31
*** apuimedo_ has quit IRC03:32
*** mgrosz has quit IRC03:32
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Install etcd on nodes for tooz unit tests"
fungiyeah, i'm sure03:32
funginot really in the mood i'm afraid03:33
lifelessfungi: can I help?03:33
fungilifeless: nah, it's just been a longish process trying to get to the point i can retrigger the broken release jobs on a rebuilt slave that's no longer installing twine and requests from pypi03:34
lifelesspypi twine is bust? Sorry, I don't have context.03:34
fungididn't want to go to sleep before it's settled and i can not worry about people pushing more tags that end up requiring more manual intervention03:35
fungilifeless: twine is suffering from a bug in requests 2.9.0 at the momnt03:35
fungiso we're switching to distro packages for it03:35
*** tzn has quit IRC03:35
*** docaedo has quit IRC03:36
fungidon't really need bleeding-edge python lib releases injected into the middle of our release automation03:36
fungiwhen there are distro packages we can use03:36
*** zhurong has joined #openstack-infra03:36
lifelesswe could of course pin it, but distro should be approximately good enough03:37
*** apuimedo_ has joined #openstack-infra03:37
fungiright, and pinning pypi package versions in puppet can be a bit fiddly03:39
fungisince the problem lib is being pulled in as a dependency of the tool we had puppet installing03:40
*** apuimedo_ has quit IRC03:42
*** _denisra has joined #openstack-infra03:45
*** njirap has quit IRC03:47
fungiokay, replacement slave is built now and checks out. once i give dns a few minutes to settle i'll get it registered in jenkins properly to replace the old one03:47
*** docaedo has joined #openstack-infra03:47
*** apuimedo_ has joined #openstack-infra03:47
*** amotoki has joined #openstack-infra03:47
greghaynesfungi: I am betting etcd is for tooz03:48
greghaynesianw: yep, you need a patched vhd-util which you get the patch off some random ml03:49
fungigreghaynes: yes, the change for which i proposed the revert (258779) said as much03:49
greghaynesianw: mordred made a ppa03:49
*** EricGonczer_ has quit IRC03:49
ianwgreghaynes: yeah, doesn't help when everything i'm building on is fedora :)03:49
fungigreghaynes: i'm going to guess they only wanted to install it on fedora, or maybe just assumed it also existed in current ubuntu lts without actually checking03:50
*** amotoki has quit IRC03:50
greghaynesfungi: they also can install it on their own rather than by default03:51
fungiwhich is what we prefer at this point03:52
*** ddieterly has quit IRC03:53
*** ddieterly has joined #openstack-infra03:54
*** apuimedo_ has quit IRC03:54
*** amotoki has joined #openstack-infra03:55
*** ddieterly has quit IRC03:58
*** apuimedo_ has joined #openstack-infra04:00
*** amotoki_ has joined #openstack-infra04:02
*** hdd has quit IRC04:02
*** apuimedo_ has quit IRC04:04
funginow i'm mildly concerned that the oslo.db tag ref i've reenqueued is just sitting there in the release pipeline getting no workers assigned to any of its jobs04:04
fungi6 minutes and counting04:05
*** amotoki has quit IRC04:05
bkeroany idle workers?04:08
*** dguitarbite has quit IRC04:08
ianwgreghaynes: that patch for vhd-convert is ... interesting04:09
*** apuimedo_ has joined #openstack-infra04:09
greghaynesianw: yep. It is extremely annoying that we need to use it.04:09
fungiyay, it worked!04:10
fungijust took a bit to pick up04:10
fungijeblair: ^ distro packaged twine in use04:10
fungii've got the django_openstack_auth release reenqueued now as well04:11
fungii already took care of swauth when it was pointed out earlier today and manually completed its upload job, but from it looks like we're also short uploads for releases of oslo.utils, gnocchi and tripleo-image-elements04:13
fungii'll take care of those three before i call it a night04:15
greghayneswhoa, t-i-e released04:15
*** apuimedo_ has quit IRC04:16
greghaynesfirst time in 6mo. and they hit this ;)04:16
*** EricGonczer_ has joined #openstack-infra04:16
*** EricGonczer_ has quit IRC04:17
fungiyeah, just their luck04:17
*** maishsk has quit IRC04:20
*** thorst has joined #openstack-infra04:20
*** abregman has joined #openstack-infra04:20
*** apuimedo_ has joined #openstack-infra04:21
fungias proven by the fact that retriggering the job also failed04:21
*** abregman has quit IRC04:22
fungioh, their wheel builds are noncomforming04:22
*** mahatic has joined #openstack-infra04:22
*** maishsk has joined #openstack-infra04:23
*** chenli has quit IRC04:23
fungiand the pypi upload job is looking for tripleo_image_elements-0.9.7-py2.py3-none-any.whl04:23
fungipresumably they've been broken for a while04:23
*** mgrosz has joined #openstack-infra04:25
ianwgreghaynes: that *should* work for fedora ... let's see ...
*** apuimedo_ has quit IRC04:25
*** piet has quit IRC04:26
fungignocchi's tarball job takes a while to complete04:26
*** mgrosz has quit IRC04:26
*** thorst has quit IRC04:28
*** apuimedo_ has joined #openstack-infra04:30
*** mriedem has joined #openstack-infra04:32
fungiokay, pypi uploads have now completed for all releases broken by the requests release yesterday04:32
fungii'm going to hit the hay--night all!04:33
*** EricGonczer_ has joined #openstack-infra04:35
*** apuimedo_ has quit IRC04:35
olaph'night fungi!04:35
*** Ramanjaneya has joined #openstack-infra04:36
*** chenli has joined #openstack-infra04:37
*** spzala has quit IRC04:37
*** apuimedo_ has joined #openstack-infra04:39
*** tsg has joined #openstack-infra04:41
*** EricGonczer_ has quit IRC04:43
*** tlian has quit IRC04:44
*** otter768 has joined #openstack-infra04:44
*** apuimedo_ has quit IRC04:44
*** hichihara has joined #openstack-infra04:44
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Publish per-branch astara-appliance images
*** rguillebert has quit IRC04:47
*** gongysh_ has joined #openstack-infra04:48
*** zhurong has quit IRC04:48
*** otter768 has quit IRC04:49
*** apuimedo_ has joined #openstack-infra04:49
*** Nakato has quit IRC04:51
*** Nakato has joined #openstack-infra04:51
*** gongysh_ has quit IRC04:52
*** gsagie_ has joined #openstack-infra04:53
*** boris-42_ has quit IRC04:53
*** apuimedo_ has quit IRC04:53
gsagie_Where in the new gerrit look, can i see the line that fetch me the change?04:57
gsagie_ohh i saw04:58
*** apuimedo_ has joined #openstack-infra04:58
*** ramineni1 has joined #openstack-infra04:59
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Disable tempest plugin for stable branches
*** gongysh_ has joined #openstack-infra05:01
*** ramineni has joined #openstack-infra05:02
*** ramineni1 has quit IRC05:03
*** ramineni1 has joined #openstack-infra05:04
*** gus has quit IRC05:04
*** zhurong has joined #openstack-infra05:05
*** ramineni_ has joined #openstack-infra05:06
*** ramineni has quit IRC05:07
*** mtreinish has quit IRC05:08
*** bpokorny has joined #openstack-infra05:08
*** ramineni1 has quit IRC05:08
*** apuimedo_ has quit IRC05:10
adam_ggsagie, where did you find it? im lookin for the copy-able line for a quick local cherry-pick, with no luck05:10
adam_goh nvm. there it is, in that drop-down :p05:11
*** ningy has quit IRC05:14
*** apuimedo_ has joined #openstack-infra05:15
*** masber has quit IRC05:15
*** mtreinish has joined #openstack-infra05:17
*** hdd has joined #openstack-infra05:18
*** apuimedo_ has quit IRC05:20
*** signed8bit has joined #openstack-infra05:20
*** maishsk has quit IRC05:20
*** gsagie_ has quit IRC05:22
*** apuimedo_ has joined #openstack-infra05:25
*** thorst has joined #openstack-infra05:25
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: test
*** tsg has quit IRC05:27
*** apuimedo_ has quit IRC05:29
*** mtanino has quit IRC05:30
*** yonglihe has quit IRC05:31
*** thorst has quit IRC05:33
*** apuimedo_ has joined #openstack-infra05:35
*** MarkAtwood has joined #openstack-infra05:36
*** apuimedo has quit IRC05:36
*** maishsk has joined #openstack-infra05:37
*** apuimedo has joined #openstack-infra05:37
*** mgrosz has joined #openstack-infra05:38
*** maishsk has quit IRC05:38
*** apuimedo_ has quit IRC05:39
*** abregman has joined #openstack-infra05:42
*** gongysh_ has quit IRC05:43
mgroszGood Morning05:44
*** apuimedo has quit IRC05:44
*** apuimedo has joined #openstack-infra05:44
*** apuimedo_ has joined #openstack-infra05:45
*** apuimedo has quit IRC05:48
*** bpokorny has quit IRC05:49
*** gus has joined #openstack-infra05:49
*** piet has joined #openstack-infra05:50
*** apuimedo_ has quit IRC05:50
*** apuimedo has joined #openstack-infra05:51
*** jerryz has quit IRC05:52
*** apuimedo_ has joined #openstack-infra05:54
*** apuimedo has quit IRC05:55
*** apuimedo has joined #openstack-infra05:56
*** camunoz has quit IRC05:57
*** apuimedo_ has quit IRC05:58
*** aeng has joined #openstack-infra06:00
*** harlowja_at_home has joined #openstack-infra06:01
*** apuimedo_ has joined #openstack-infra06:03
*** apuimedo has quit IRC06:03
*** apuimedo has joined #openstack-infra06:04
*** piet has quit IRC06:04
*** harlowja_still_a has joined #openstack-infra06:05
*** apuimedo_ has quit IRC06:08
*** aeng has quit IRC06:08
*** harlowja_at_home has quit IRC06:08
*** apuimedo has quit IRC06:09
*** camunoz has joined #openstack-infra06:09
*** jswarren_ has joined #openstack-infra06:09
*** piet has joined #openstack-infra06:09
*** apuimedo has joined #openstack-infra06:09
*** asselin_ has quit IRC06:10
*** ryanpetrello has quit IRC06:10
*** asselin_ has joined #openstack-infra06:11
*** davidlenwell has quit IRC06:12
*** jswarren has quit IRC06:12
*** xarses has quit IRC06:13
*** xarses has joined #openstack-infra06:14
*** davidlenwell has joined #openstack-infra06:14
*** apuimedo_ has joined #openstack-infra06:14
*** xarses has quit IRC06:16
*** calebb has quit IRC06:16
*** signed8bit is now known as signed8bit_ZZZzz06:16
*** xarses has joined #openstack-infra06:16
*** calebb has joined #openstack-infra06:16
*** zhurong has quit IRC06:16
*** apuimedo has quit IRC06:16
*** xarses has quit IRC06:17
*** apuimedo has joined #openstack-infra06:17
*** xarses has joined #openstack-infra06:18
*** zhurong has joined #openstack-infra06:18
*** mudassirlatif has joined #openstack-infra06:18
*** xarses has quit IRC06:18
*** xarses has joined #openstack-infra06:19
*** apuimedo_ has quit IRC06:19
*** xarses has quit IRC06:19
*** ryanpetrello has joined #openstack-infra06:20
*** xarses has joined #openstack-infra06:20
*** gsagie_ has joined #openstack-infra06:20
*** xarses has quit IRC06:20
*** xarses has joined #openstack-infra06:20
gsagie_Hello, i am trying to use docker client (python binding) in the gate , but the connection is aborted:  (this works locally) is there anything inn the gate blocking this? and is there any solution?06:21
*** piet has quit IRC06:22
*** apuimedo_ has joined #openstack-infra06:24
*** xarses has quit IRC06:24
*** xarses has joined #openstack-infra06:24
*** camunoz has quit IRC06:25
*** maishsk has joined #openstack-infra06:27
*** apuimedo_ has quit IRC06:28
openstackgerritGal Sagie proposed openstack-infra/project-config: Add Dragonflow fullstack job
*** calebb has quit IRC06:30
*** signed8bit_ZZZzz has quit IRC06:30
*** thorst has joined #openstack-infra06:31
mgroszHey... I got lost with the new Gerrit UI... I have pushed a new patch set to, number 18. For some reason I got Code-Review -1 from Iury Gregory Melo Ferreira - but there is no comment about that. I am waiting for more than 50 minutes for Jenkins to start.... Anyone?06:32
mgroszI believe this guy is even sleeping at the moment (Brazil time)06:33
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** apuimedo_ has joined #openstack-infra06:34
*** xarses has quit IRC06:34
*** xarses has joined #openstack-infra06:35
*** xarses has quit IRC06:36
*** xarses has joined #openstack-infra06:36
*** ramineni_ has quit IRC06:36
*** xarses has quit IRC06:37
*** jffische has quit IRC06:38
*** thorst has quit IRC06:38
*** hdd has quit IRC06:38
*** camunoz has joined #openstack-infra06:38
*** xarses has joined #openstack-infra06:38
*** maishsk has quit IRC06:40
*** maishsk_ has joined #openstack-infra06:40
*** calebb has joined #openstack-infra06:42
*** apuimedo_ has quit IRC06:43
*** otter768 has joined #openstack-infra06:45
*** apuimedo_ has joined #openstack-infra06:48
*** mgrosz has quit IRC06:49
*** camunoz has quit IRC06:49
*** otter768 has quit IRC06:50
*** apuimedo has quit IRC06:51
*** apuimedo has joined #openstack-infra06:51
*** aopincar has joined #openstack-infra06:52
*** apuimedo_ has quit IRC06:53
*** jerryz has joined #openstack-infra06:54
*** AJaeger has joined #openstack-infra06:57
*** maishsk_ has quit IRC06:57
*** apuimedo_ has joined #openstack-infra06:58
*** mgrosz has joined #openstack-infra06:58
*** isaacb has joined #openstack-infra07:00
*** apuimedo_ has quit IRC07:02
*** HeOS has quit IRC07:05
*** apuimedo has quit IRC07:05
*** apuimedo has joined #openstack-infra07:05
*** isaacb has quit IRC07:07
*** apuimedo_ has joined #openstack-infra07:08
*** apuimedo has quit IRC07:10
*** apuimedo has joined #openstack-infra07:11
*** apuimedo_ has quit IRC07:12
*** gsagie_ has quit IRC07:13
lifelessbah, sorry07:13
*** triggerwarning has quit IRC07:14
*** apuimedo has quit IRC07:15
*** apuimedo has joined #openstack-infra07:17
*** apuimedo_ has joined #openstack-infra07:18
*** apuimedo has quit IRC07:21
*** apuimedo_ has quit IRC07:23
AJaegerjhesketh: please check - the commit message job fails since the script cannot be found.07:23
*** apuimedo has joined #openstack-infra07:24
*** irenab has quit IRC07:24
mgroszAJaeger: Good Morning.Hey... I got lost with the new Gerrit UI... I have pushed a new patch set to, number 18. For some reason I got Code-Review -1 from Iury Gregory Melo Ferreira - but there is no comment about that. I am waiting for more than 2 hours ago for Jenkins to start....07:26
mgroszMaybe a bug?07:27
*** apuimedo_ has joined #openstack-infra07:28
*** apuimedo has quit IRC07:28
mgroszYou can see in the history of the change that this guy Gregory last -1was a week ago. Since then he already gave +1...07:29
*** apuimedo has joined #openstack-infra07:30
* mkoderer is also very confused about the new Gerrit UI07:30
*** Guest6859 has quit IRC07:32
*** apuimedo_ has quit IRC07:33
AJaegermgrosz: was your upload just a rebase or something like that? Then -1s stay...07:34
*** thorst has joined #openstack-infra07:35
AJaegermgrosz: this looks strange indeed - but I can't help.07:35
mgroszJust did what I use to... git commit amend and then git review. I am sure this guy is sleeping now (Brazil time)07:35
AJaegerzaro: are you still around and can have a look at 255431?07:36
mgroszAJaeger: then how helps here?07:36
*** chlong has quit IRC07:36
mgroszAJaeger: do you advice to send an email infra mailing list?07:36
AJaegermgrosz: Our CI systems are quite busy, I see on that your job is still in the queue...07:37
AJaegermgrosz: either ask on openstack-infra mailing list or come back here in the US morning - or wait and in most cases infra-root reads the backscroll for 25543107:37
jheskethAJaeger: yep, that's because the nodes need to be rebuilt07:38
jheskeththat'll happen overnight sometime07:38
*** apuimedo_ has joined #openstack-infra07:38
AJaegermgrosz: you can approve a change even if there's a -1 review comment...07:38
jhesketh*node images07:38
AJaegerjhesketh: ah, that problem again. Thanks07:38
jheskethAJaeger, mgrosz: so no jobs for 255431 is the problem?07:39
mgroszAJaeger: I am saying that the -1 was there at first place... You don't see any entry in the history list nest to my latest "Uploaded patch set 18." entry.07:40
jheskethmgrosz: looks like jenkins just reported on it07:41
jheskethbut yes, the -1 is odd07:41
openstackgerritDavid Pursehouse proposed openstack-infra/lodgeit: Remove unused image file
openstackgerritDavid Pursehouse proposed openstack-infra/lodgeit: Update i18n files to remove unused translations
jheskethmgrosz: diffing patchset 18 to 13 there looks like nothing is changed07:41
mgroszAJaeger: No... there should be several of jobs (I had 17 patches so far so I know that...).OK... saw the email from Jenkins coming... 2 hour and 10 minutes after submitting.. I think things are not working good....07:42
*** apuimedo has quit IRC07:42
jheskethmgrosz: so that vote has carried over because Gregorgy hasn't updated it along the line07:42
jheskethactually I lie, I can see his newer vote07:43
*** thorst has quit IRC07:43
*** apuimedo has joined #openstack-infra07:44
AJaegerjhesketh: do you see a vote on patch set 17?07:46
jheskethAJaeger: from who?07:46
AJaegerthe -1 from Iury Gregory...07:47
jheskethAJaeger: I see a +1 from lury07:47
*** apuimedo_ has quit IRC07:47
AJaegerbut the summary on the top shows a -1 from him for me07:48
* AJaeger is confused07:48
jheskethAJaeger: right, yes, I see that in the summary.. I see a +1 in the comment stream07:48
*** apuimedo has quit IRC07:48
AJaegerjhesketh: I see that on patch 17 but not on 18 which is the most recent07:48
jheskethyeah I don't see anything on 1807:49
*** apuimedo has joined #openstack-infra07:50
jheskethAJaeger: so ps18 is identical to ps13 (although 17 and others were different)07:50
jheskethso what's happened is that the -1 from ps13 has been applied07:50
jheskethsince the patches match07:50
AJaegerjhesketh: AH! Good spotting, that explains it...07:50
jheskethlury will need to put a new vote in on ps18 now07:50
jheskethand gerrit expands his -1 from ps13 in the comment section07:50
*** apuimedo_ has joined #openstack-infra07:52
AJaegermgrosz: see the explanation above ^07:52
*** mgrosz has quit IRC07:54
*** mgrosz has joined #openstack-infra07:55
*** apuimedo_ has quit IRC07:57
*** k4n0 has joined #openstack-infra07:58
*** k4n0 has quit IRC07:58
*** apuimedo has quit IRC07:59
*** sflanigan has quit IRC07:59
*** apuimedo has joined #openstack-infra08:01
*** jerryz has quit IRC08:01
mgroszAJaeger: OK, now I understand. this guy, lury, has reverted his code he did here My patch 13 was not compliance with this changes, and 14 was. Now, he reverted his code here , So now patch 13 should be good... I will need to wait for him when he wakes up...08:01
*** apuimedo_ has joined #openstack-infra08:02
mgroszAJaeger: And also patch 18 should be good08:02
*** harlowja_still_a has quit IRC08:03
*** jcoufal has joined #openstack-infra08:04
*** rvasilets___ has joined #openstack-infra08:05
*** zeih has joined #openstack-infra08:06
*** jtomasek has joined #openstack-infra08:06
*** scheuran has joined #openstack-infra08:08
*** apuimedo_ has quit IRC08:09
*** zeih has quit IRC08:10
*** Liuqing has joined #openstack-infra08:10
*** apuimedo has quit IRC08:10
*** apuimedo has joined #openstack-infra08:12
*** irenab has joined #openstack-infra08:12
*** unicell has joined #openstack-infra08:13
*** hashar has joined #openstack-infra08:14
*** apuimedo_ has joined #openstack-infra08:14
*** thorst has joined #openstack-infra08:16
*** unicell1 has joined #openstack-infra08:17
*** zeih has joined #openstack-infra08:18
*** unicell has quit IRC08:18
*** apuimedo_ has quit IRC08:18
*** apuimedo has quit IRC08:19
openstackgerritDenis Laxalde proposed openstack-infra/jenkins-job-builder: Fix docstring of downstream-ext publisher
*** apuimedo has joined #openstack-infra08:19
*** shardy has joined #openstack-infra08:22
*** thorst has quit IRC08:23
*** zeih has quit IRC08:23
*** apuimedo_ has joined #openstack-infra08:23
*** yaume has joined #openstack-infra08:23
*** e0ne has joined #openstack-infra08:23
*** egonzalez has joined #openstack-infra08:23
*** apuimedo has quit IRC08:24
*** apuimedo has joined #openstack-infra08:25
*** mudassirlatif has quit IRC08:26
*** rcernin has joined #openstack-infra08:26
*** Hal has joined #openstack-infra08:27
*** Hal is now known as Guest8470308:27
*** markus_z has joined #openstack-infra08:27
*** ifarkas has joined #openstack-infra08:29
*** apuimedo_ has quit IRC08:32
*** apuimedo_ has joined #openstack-infra08:37
*** apuimedo has quit IRC08:37
*** MarkAtwood has quit IRC08:38
*** apuimedo has joined #openstack-infra08:38
*** e0ne has quit IRC08:38
*** e0ne has joined #openstack-infra08:40
*** rcernin has quit IRC08:40
*** rossella_s has joined #openstack-infra08:40
*** rcernin has joined #openstack-infra08:41
*** apuimedo_ has quit IRC08:42
*** _nadya_ has joined #openstack-infra08:43
*** otter768 has joined #openstack-infra08:46
*** apuimedo_ has joined #openstack-infra08:46
*** zeih has joined #openstack-infra08:48
*** FallenPegasus has joined #openstack-infra08:48
*** otter768 has quit IRC08:50
*** apuimedo has quit IRC08:52
*** zeih has quit IRC08:52
*** ashtokolov has quit IRC08:53
*** e0ne has quit IRC08:53
*** apuimedo_ has quit IRC08:53
*** apuimedo has joined #openstack-infra08:54
*** patrickeast has quit IRC08:56
*** ashtokolov has joined #openstack-infra08:57
*** apuimedo_ has joined #openstack-infra08:58
*** dingyichen has quit IRC08:59
*** larainema has quit IRC08:59
*** sshnaidm has joined #openstack-infra09:00
*** matrohon has joined #openstack-infra09:01
*** apuimedo_ has quit IRC09:03
openstackgerritMerged openstack-infra/project-config: Publish per-branch astara-appliance images
*** larainema has joined #openstack-infra09:03
*** apuimedo has quit IRC09:04
*** briancurtin has quit IRC09:04
*** ashtokolov has quit IRC09:04
*** apuimedo has joined #openstack-infra09:05
*** patrickeast has joined #openstack-infra09:06
*** fhubik has joined #openstack-infra09:06
*** amotoki_ has quit IRC09:06
*** apuimedo_ has joined #openstack-infra09:07
*** fhubik is now known as fhubik_brb09:08
*** fhubik_brb is now known as fhubik09:08
*** derekh has joined #openstack-infra09:09
*** mgrosz has quit IRC09:09
*** jistr has joined #openstack-infra09:09
*** HeOS has joined #openstack-infra09:09
*** fhubik has quit IRC09:09
*** zhurong has quit IRC09:10
*** ParsectiX has quit IRC09:11
*** ashtokolov has joined #openstack-infra09:11
*** ParsectiX has joined #openstack-infra09:12
*** briancurtin has joined #openstack-infra09:12
openstackgerritMerged openstack-infra/project-config: Use separated ACL group for OSprofiler
*** Guest84703 has quit IRC09:14
*** apuimedo_ has quit IRC09:14
*** patrickeast has quit IRC09:15
openstackgerritMerged openstack-infra/project-config: Make gate-tempest-dsvm-neutron-dvr job voting in the check queue
openstackgerritMerged openstack-infra/project-config: Skip dsvm jobs for ironic-python-agent if code doesn't change
openstackgerritMerged openstack-infra/project-config: Make python34 job voting for Congress
*** placko_ has joined #openstack-infra09:17
*** briancurtin has quit IRC09:17
*** betherly has quit IRC09:19
*** apuimedo_ has joined #openstack-infra09:19
*** mgrosz has joined #openstack-infra09:20
*** thorst has joined #openstack-infra09:20
*** ashtokolov has quit IRC09:21
*** rossella_s has quit IRC09:21
*** rossella_s has joined #openstack-infra09:23
*** apuimedo_ has quit IRC09:24
*** jlanoux has joined #openstack-infra09:25
*** zigo has quit IRC09:25
*** zigo_ has joined #openstack-infra09:26
*** FallenPegasus has quit IRC09:26
*** zeih has joined #openstack-infra09:26
*** ParsectiX has quit IRC09:27
*** thorst has quit IRC09:27
*** salv-orlando has joined #openstack-infra09:28
*** larainema has quit IRC09:28
openstackgerritJulia Kreger proposed openstack-infra/project-config: Correct typo in diskimage-builder job
*** hichihara has quit IRC09:29
*** apuimedo_ has joined #openstack-infra09:29
*** Hal has joined #openstack-infra09:30
openstackgerritMerged openstack-infra/project-config: Creation of the Coda Repo
*** Hal is now known as Guest9828209:31
*** chenli has quit IRC09:31
*** amotoki has joined #openstack-infra09:33
*** apuimedo has quit IRC09:33
*** placko_ has quit IRC09:33
*** apuimedo_ has quit IRC09:34
*** apuimedo has joined #openstack-infra09:34
*** ashtokolov has joined #openstack-infra09:37
*** betherly has joined #openstack-infra09:38
*** apuimedo_ has joined #openstack-infra09:39
*** patrickeast has joined #openstack-infra09:39
openstackgerritMerged openstack-infra/project-config: [sahara] job for sahara-dashboard integration tests
*** larainema has joined #openstack-infra09:40
*** apuimedo_ has quit IRC09:43
*** amotoki has quit IRC09:44
openstackgerritMerged openstack-infra/project-config: Remove check-requirements for Solar.
*** briancurtin has joined #openstack-infra09:44
openstackgerritMerged openstack-infra/project-config: Fix for sahara grenade job
*** maishsk has joined #openstack-infra09:46
*** apuimedo has quit IRC09:46
openstackgerritJulia Kreger proposed openstack-infra/project-config: Build a Debian simple-init guest image when DIB changes merge
*** apuimedo has joined #openstack-infra09:47
*** openstackgerrit has quit IRC09:47
*** openstackgerrit has joined #openstack-infra09:47
*** rguillebert has joined #openstack-infra09:49
*** apuimedo_ has joined #openstack-infra09:49
*** placko_ has joined #openstack-infra09:51
*** apuimedo has quit IRC09:52
*** apuimedo has joined #openstack-infra09:54
*** apuimedo_ has quit IRC09:54
*** e0ne has joined #openstack-infra09:55
*** sfinucan has joined #openstack-infra09:57
*** dizquierdo has joined #openstack-infra09:58
*** apuimedo_ has joined #openstack-infra09:59
*** mgrosz has quit IRC10:02
*** placko_ has quit IRC10:02
*** ihrachys has joined #openstack-infra10:03
gsagieHello, i am trying to use docker client (python binding) in the gate , but the connection is aborted:  (this works locally) is there anything inn the gate blocking this? and is there any solution?10:03
*** apuimedo has quit IRC10:03
gsagierequests.exceptions.ConnectionError: ('Connection aborted.', error(13, 'Permission denied'))10:04
*** apuimedo_ has quit IRC10:04
*** apuimedo has joined #openstack-infra10:05
*** e0ne has quit IRC10:05
*** apuimedo_ has joined #openstack-infra10:08
*** bexelbie has joined #openstack-infra10:10
*** apuimedo has quit IRC10:12
*** Liuqing has quit IRC10:12
*** apuimedo has joined #openstack-infra10:12
*** electrofelix has joined #openstack-infra10:13
*** Liuqing has joined #openstack-infra10:13
*** apuimedo_ has quit IRC10:13
*** pigmej has left #openstack-infra10:15
*** apuimedo_ has joined #openstack-infra10:18
*** apuimedo has quit IRC10:19
*** armax has quit IRC10:20
*** apuimedo has joined #openstack-infra10:20
*** Qiming has quit IRC10:20
*** unicell1 has quit IRC10:20
*** unicell has joined #openstack-infra10:20
*** ihrachys has quit IRC10:21
*** apuimedo_ has quit IRC10:22
*** ihrachys has joined #openstack-infra10:22
*** ihrachys has quit IRC10:23
*** armax has joined #openstack-infra10:23
*** dizquierdo has quit IRC10:24
*** ihrachys has joined #openstack-infra10:24
*** Liuqing has quit IRC10:25
*** thorst has joined #openstack-infra10:26
*** apuimedo_ has joined #openstack-infra10:27
*** apuimedo has quit IRC10:29
*** mpavone has joined #openstack-infra10:30
*** apuimedo has joined #openstack-infra10:31
*** mpavone has quit IRC10:31
*** apuimedo_ has quit IRC10:32
*** thorst has quit IRC10:32
*** zhurong has joined #openstack-infra10:35
*** apuimedo has quit IRC10:35
*** apuimedo has joined #openstack-infra10:37
*** apuimedo_ has joined #openstack-infra10:37
*** apuimedo has quit IRC10:41
*** unicell has quit IRC10:42
*** apuimedo_ has quit IRC10:42
*** unicell has joined #openstack-infra10:42
*** apuimedo has joined #openstack-infra10:43
*** masber has joined #openstack-infra10:43
*** e0ne has joined #openstack-infra10:43
*** yolanda has quit IRC10:46
*** otter768 has joined #openstack-infra10:47
*** apuimedo_ has joined #openstack-infra10:47
*** apuimedo has quit IRC10:48
*** rfolco_ has quit IRC10:48
*** yamahata has joined #openstack-infra10:48
*** apuimedo has joined #openstack-infra10:49
*** rfolco has joined #openstack-infra10:50
*** Liuqing has joined #openstack-infra10:50
*** otter768 has quit IRC10:51
*** yolanda has joined #openstack-infra10:51
*** zhurong has quit IRC10:52
*** zhurong has joined #openstack-infra10:53
*** apuimedo has quit IRC10:54
*** apuimedo_ has quit IRC10:54
*** apuimedo has joined #openstack-infra10:55
*** hashar has quit IRC10:56
*** hashar has joined #openstack-infra10:57
*** apuimedo_ has joined #openstack-infra10:59
*** apuimedo_ has quit IRC11:04
*** apuimedo has quit IRC11:05
*** apuimedo has joined #openstack-infra11:05
*** aysyd has joined #openstack-infra11:07
*** dims has joined #openstack-infra11:07
*** lucas-dinner is now known as lucasagomes11:07
*** sambetts-afk is now known as sambetts11:08
*** Qiming has joined #openstack-infra11:08
*** salv-orlando has quit IRC11:08
openstackgerritMerged openstack-infra/storyboard-webclient: Add UI for complex permissions on boards and worklists
*** apuimedo_ has joined #openstack-infra11:09
*** salv-orlando has joined #openstack-infra11:11
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: DO NOT MERGE changes needed for delorean promotion
*** salv-orlando has quit IRC11:12
*** salv-orlando has joined #openstack-infra11:12
*** apuimedo_ has quit IRC11:14
*** Liuqing has quit IRC11:14
*** salv-orl_ has joined #openstack-infra11:15
*** yamahata has quit IRC11:18
*** _nadya_ has quit IRC11:18
*** salv-orlando has quit IRC11:18
*** apuimedo_ has joined #openstack-infra11:19
*** mgrosz has joined #openstack-infra11:20
*** _nadya_ has joined #openstack-infra11:23
*** christiana_ has quit IRC11:23
*** apuimedo_ has quit IRC11:24
*** apuimedo has quit IRC11:26
*** apuimedo has joined #openstack-infra11:27
*** sfinucan has quit IRC11:28
*** sfinucan has joined #openstack-infra11:28
*** apuimedo_ has joined #openstack-infra11:29
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes
*** thorst has joined #openstack-infra11:31
*** apuimedo has quit IRC11:31
*** apuimedo has joined #openstack-infra11:32
freerunnerAJaeger: Hey! Are you here? kzaitsev_ws and me have a little question about caps in requirements. Is this possible to cap our muranoclient in stable/liberty requirements branch? Or we should to cap it directly in murano requirements?11:33
*** christiana_ has joined #openstack-infra11:36
*** zeih has quit IRC11:37
*** apuimedo has quit IRC11:37
*** kzaitsev_mb has joined #openstack-infra11:37
*** jlanoux has quit IRC11:37
*** apuimedo_ has quit IRC11:38
*** apuimedo has joined #openstack-infra11:38
*** thorst has quit IRC11:38
kzaitsev_mbfreerunner: probably gotta ping the guys later today11:39
kzaitsev_mbwonder what sahara folks did in the same situation11:39
*** betherly has quit IRC11:39
openstackgerritMerged openstack-infra/storyboard-webclient: Make kanban cards be the correct width
*** claudiub has joined #openstack-infra11:41
*** mika has quit IRC11:41
*** larainema has quit IRC11:42
*** patrickeast has quit IRC11:42
*** yamamoto has quit IRC11:42
*** apuimedo_ has joined #openstack-infra11:42
*** ashtokolov has quit IRC11:43
*** betherly has joined #openstack-infra11:44
*** apuimedo has quit IRC11:45
*** apuimedo has joined #openstack-infra11:45
*** mika has joined #openstack-infra11:46
*** mika has joined #openstack-infra11:46
*** mgrosz has quit IRC11:48
*** masber has quit IRC11:51
*** apuimedo_ has quit IRC11:52
*** mgrosz has joined #openstack-infra11:52
*** apuimedo has quit IRC11:52
*** jswarren_ is now known as jswarren11:53
*** briancurtin has quit IRC11:53
*** betherly has quit IRC11:53
*** apuimedo has joined #openstack-infra11:53
*** irenab has quit IRC11:56
*** ashtokolov has joined #openstack-infra11:56
lennyb_Hi, is someone using lastcomment-scoreboard ( by jogo )? It stopped working after gerrit upgrade. I did not debug it yet.11:57
*** apuimedo_ has joined #openstack-infra11:57
*** abregman is now known as abregman|afk12:00
*** rcernin has quit IRC12:00
*** apuimedo_ has quit IRC12:02
*** zeih has joined #openstack-infra12:03
*** ashtokolov has quit IRC12:04
*** tzn has joined #openstack-infra12:05
*** julim has joined #openstack-infra12:05
*** patrickeast has joined #openstack-infra12:05
*** _nadya_ has quit IRC12:06
*** apuimedo_ has joined #openstack-infra12:08
*** zeih has quit IRC12:08
AJaegerfreerunner: now I'm here...12:11
AJaegerfreerunner: for liberty you can cap directly in requirements.12:11
*** apuimedo_ has quit IRC12:12
AJaegergsagie: does it need root access?12:12
openstackgerrityolanda.robla proposed openstack-infra/puppet-diskimage_builder: Add lvm2 package to diskimage builder
*** rcernin has joined #openstack-infra12:15
*** patrickeast has quit IRC12:15
*** u_glide1 has joined #openstack-infra12:16
*** apuimedo has quit IRC12:17
*** apuimedo has joined #openstack-infra12:17
u_glide1Hello folks!12:18
u_glide1Could anyone help me to investigate why gate-manila-buildimage-generic  job is NOT_REGISTERED ?12:18
*** apuimedo_ has joined #openstack-infra12:18
*** tsg has joined #openstack-infra12:21
*** weshay_xchat has joined #openstack-infra12:21
*** dprince has joined #openstack-infra12:22
*** bexelbie has quit IRC12:22
*** [1]Thelo has joined #openstack-infra12:23
*** yamamoto has joined #openstack-infra12:23
*** maishsk has quit IRC12:24
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Bump the lower boundary of pysnmp to 4.2.3
*** Thelo has quit IRC12:25
*** [1]Thelo is now known as Thelo12:25
*** rvasilets___ has quit IRC12:25
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Bump the lower boundary of pysnmp to 4.2.3
*** jaypipes has joined #openstack-infra12:26
*** apuimedo has quit IRC12:26
*** patrickeast has joined #openstack-infra12:26
*** irenab has joined #openstack-infra12:28
*** briancurtin has joined #openstack-infra12:29
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Tidy up project groups states
*** _nadya_ has joined #openstack-infra12:29
*** amuller has joined #openstack-infra12:30
ekarlsowould it be an idea to have the zuul status thing push SSE events to the browser instead of pulling ?12:30
openstackgerritfumihiko kakuma proposed openstack/requirements: Add pylint to upper-constraints.txt
*** apuimedo has joined #openstack-infra12:32
*** larainema has joined #openstack-infra12:38
*** apuimedo has quit IRC12:39
*** ashtokolov has joined #openstack-infra12:40
*** betherly has joined #openstack-infra12:41
*** smarcet has joined #openstack-infra12:43
*** amotoki has joined #openstack-infra12:43
*** Daisy has joined #openstack-infra12:44
*** apuimedo has joined #openstack-infra12:44
*** tsg has quit IRC12:44
*** amotoki has quit IRC12:46
*** tsg has joined #openstack-infra12:47
*** otter768 has joined #openstack-infra12:47
*** abregman|afk is now known as abregman12:48
*** apuimedo has quit IRC12:48
*** apuimedo_ has quit IRC12:49
*** smarcet has quit IRC12:50
*** apuimedo has joined #openstack-infra12:50
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Tidy up project groups states
*** thorst has joined #openstack-infra12:52
*** otter768 has quit IRC12:52
*** apuimedo_ has joined #openstack-infra12:53
*** thorst_ has joined #openstack-infra12:54
*** baoli_ has joined #openstack-infra12:54
*** smarcet has joined #openstack-infra12:55
*** apuimedo has quit IRC12:55
*** apuimedo has joined #openstack-infra12:55
openstackgerritClaudiu Belu proposed openstack/requirements: Adds networking-hyperv to projects.txt
openstackgerritMerged openstack-infra/tripleo-ci: Make flavor-creation failure non-fatal
*** thorst has quit IRC12:56
*** links has quit IRC12:58
*** smarcet has quit IRC12:59
*** smarcet has joined #openstack-infra12:59
*** apuimedo_ has quit IRC13:00
*** bdemers has quit IRC13:01
*** jyuso1 has quit IRC13:01
*** apuimedo has quit IRC13:02
*** Daisy has quit IRC13:02
*** apuimedo has joined #openstack-infra13:03
*** Daisy has joined #openstack-infra13:03
*** salv-orl_ has quit IRC13:04
*** apuimedo_ has joined #openstack-infra13:06
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds check-requirements job in networking-hyperv
*** yamamoto has quit IRC13:08
*** salv-orlando has joined #openstack-infra13:09
smarcetfungi:clarkb: morning, i am gonna to merge
smarcetOIDC implementation13:10
*** zeih has joined #openstack-infra13:11
*** apuimedo has quit IRC13:12
*** odlD2 has quit IRC13:12
*** apuimedo_ has quit IRC13:13
*** apuimedo has joined #openstack-infra13:13
*** tsg has quit IRC13:14
*** rhallisey has joined #openstack-infra13:15
*** odlD2 has joined #openstack-infra13:16
*** fhubik has joined #openstack-infra13:17
*** briancurtin has quit IRC13:17
*** briancurtin has joined #openstack-infra13:17
*** apuimedo_ has joined #openstack-infra13:17
*** gordc has joined #openstack-infra13:19
*** fhubik is now known as fhubik_brb13:19
lucasagomesthe font size of the new gerrit interface is very small, there's nothing in settings to change it either13:20
*** yamamoto has joined #openstack-infra13:20
* lucasagomes is using firefox13:21
*** rcernin has quit IRC13:21
*** ujuc_ has joined #openstack-infra13:21
*** flepied1 has quit IRC13:21
*** kgiusti has joined #openstack-infra13:22
*** vgridnev has joined #openstack-infra13:22
*** apuimedo has quit IRC13:23
*** briancurtin_ has joined #openstack-infra13:23
*** rossella_s has quit IRC13:23
*** alivigni has joined #openstack-infra13:23
*** apuimedo has joined #openstack-infra13:23
*** rossella_s has joined #openstack-infra13:23
*** rlandy has joined #openstack-infra13:26
*** apuimedo has quit IRC13:28
lucasagomesand the ctrl+f javascript regex thing is so bad!13:28
*** ashtokolov has quit IRC13:28
*** ashtokolov has joined #openstack-infra13:29
*** betherly has quit IRC13:29
*** betherly has joined #openstack-infra13:29
*** apuimedo_ has quit IRC13:29
*** apuimedo has joined #openstack-infra13:30
*** patrickeast has quit IRC13:30
*** patrickeast has joined #openstack-infra13:30
*** pradk has joined #openstack-infra13:31
*** ashtokolov_ has joined #openstack-infra13:32
*** betherly_ has joined #openstack-infra13:32
*** ayoung has joined #openstack-infra13:32
*** ashtokolov has quit IRC13:33
*** ashtokolov_ is now known as ashtokolov13:33
*** betherly has quit IRC13:33
*** betherly_ is now known as betherly13:33
*** links has joined #openstack-infra13:33
*** apuimedo_ has joined #openstack-infra13:34
*** mgrosz has quit IRC13:35
openstackgerritAndreas Jaeger proposed openstack/requirements: Adds networking-hyperv to projects.txt
*** flepied has joined #openstack-infra13:35
*** flepied1 has joined #openstack-infra13:36
*** trown|outttypeww is now known as trown13:37
*** apuimedo_ has quit IRC13:38
*** apuimedo has quit IRC13:39
*** sigmavirus24_awa is now known as sigmavirus2413:39
*** flepied has quit IRC13:40
*** mgrosz has joined #openstack-infra13:40
*** apuimedo has joined #openstack-infra13:40
*** ashtokolov_ has joined #openstack-infra13:40
*** betherly_ has joined #openstack-infra13:41
*** Ramanjaneya has quit IRC13:42
*** AJaeger has quit IRC13:42
*** hdd has joined #openstack-infra13:44
*** apuimedo_ has joined #openstack-infra13:44
*** patrickeast_ has joined #openstack-infra13:45
*** fhubik_brb is now known as fhubik13:45
ttxfungi: when up, could you investigate why the bandit 0.17.0 tag did apparently not result in a tarball build or Pypi upload ?13:45
ttxafaict the post job didn't run13:46
*** yamamoto has quit IRC13:46
*** Daisy_ has joined #openstack-infra13:47
ttxhmm, maybe some jobs were purged, the zuul graph looks funny around that hour13:47
*** Daisy has quit IRC13:48
yolandainfra-root, what is best way to remove a jenkins target from nodepool? for dropping a provider, there is that min-ready=-1 trick, but for dropping a jenkins master?13:50
yolanda(i need to do that downstream)13:50
*** yamamoto has joined #openstack-infra13:50
*** aopincar has quit IRC13:53
*** tlian has joined #openstack-infra13:53
*** placko has joined #openstack-infra14:00
*** apuimedo_ has quit IRC14:00
*** lykinsbd has joined #openstack-infra14:01
openstackgerritEvgeniy L proposed openstack-infra/project-config: Create openstack/bareon-specs repository
*** nmagnezi has joined #openstack-infra14:04
ujuc_hey #openstack-meeting bot is problem..... how to...14:04
*** apuimedo_ has joined #openstack-infra14:05
*** mattt has joined #openstack-infra14:06
ujuc_ah, relived. :)14:08
*** xyang1 has joined #openstack-infra14:09
*** hdd has quit IRC14:09
*** apuimedo_ has quit IRC14:10
*** apuimedo has quit IRC14:11
*** Daisy_ has quit IRC14:11
*** automagically has joined #openstack-infra14:12
*** apuimedo has joined #openstack-infra14:12
Sam-I-Amanteaya: in #openstack its as if people sense when i wake up14:12
*** jcoufal_ has joined #openstack-infra14:13
openstackgerritJim Rollenhagen proposed openstack-infra/devstack-gate: Bump ironic test timeout to 900
*** fhubik is now known as fhubik_brb14:16
*** apuimedo_ has joined #openstack-infra14:16
*** jcoufal has quit IRC14:16
*** spzala has joined #openstack-infra14:16
*** lucasagomes is now known as lucas-hungry14:17
*** lykinsbd has quit IRC14:17
automagicallyFYI - looks like post Gerrit upgrade, the fails with a 500 error “Cannot store contact information"14:17
automagicallyUnclear if this a good place to report that or if another location is preferable14:18
*** jlanoux has joined #openstack-infra14:19
*** markvoelker has quit IRC14:21
openstackgerritSean Dague proposed openstack-infra/system-config: narrow the 3 columns progressively with screen width
*** apuimedo has quit IRC14:24
bgmccollumanyone seen this before? -- ERROR: Publisher be.certipost.hudson.plugin.SCPRepositoryPublisher aborted due to exception14:24
*** dkranz has joined #openstack-infra14:24
*** apuimedo has joined #openstack-infra14:25
*** bhunter71 has quit IRC14:26
*** apuimedo_ has quit IRC14:27
sdaguethe inline fix is pretty nice for gerrit14:28
sdagueespecially for simple stuff14:28
*** markvoelker has joined #openstack-infra14:29
*** vkramskikh has joined #openstack-infra14:29
*** peristeri has joined #openstack-infra14:30
*** ilyashakhat has joined #openstack-infra14:30
*** EricGonczer_ has joined #openstack-infra14:31
*** ddieterly has joined #openstack-infra14:31
*** apuimedo_ has joined #openstack-infra14:32
*** bdemers has joined #openstack-infra14:32
*** zhurong has quit IRC14:34
*** thiagop has joined #openstack-infra14:34
*** piet has joined #openstack-infra14:34
*** placko has quit IRC14:35
*** ociuhandu has joined #openstack-infra14:35
*** fhubik_brb is now known as fhubik14:35
*** zhurong has joined #openstack-infra14:37
dansmithdoes anyone have a pointer to something like a doc that explains what query string differences there may be in the new version of gerrit?14:40
dansmiththe query I've been using for a couple years is suddenly giving me a 40014:40
sdaguedansmith: what is your query string14:40
*** jlanoux has quit IRC14:40
dansmith((owner:danms OR is:starred) OR (project:openstack/nova)) AND status:open14:40
*** pradk has quit IRC14:41
sdagueis there an encoding issue14:41
*** pradk has joined #openstack-infra14:41
dansmithwell, that's the pre-encoded version14:41
*** apuimedo_ has quit IRC14:41
*** apuimedo has quit IRC14:41
*** ChuckC has joined #openstack-infra14:42
sdaguehmmm,,, that seems to work for me, though you need to be signed in14:42
dansmithhmm, well, I cut part of it off:14:43
*** apuimedo has joined #openstack-infra14:43
dansmithbut the link above does not work for me in a browser14:43
dansmithwhere I am signed in14:43
*** dizquierdo has joined #openstack-infra14:43
sdague&o=DETAILED_ACCOUNTS ?14:43
dansmither, wait14:43
*** hdd has joined #openstack-infra14:44
dansmithI dunno, I had to add that some long time ago for some reason I guess14:44
dansmithokay without that it does work for me in a browser if I'm signed in, but I'm doing http auth with the http password in dash14:44
dansmithmaybe my password changed with the upgrade?14:45
*** redixin has left #openstack-infra14:45
dansmithnope, didn't change14:45
*** mgrosz has quit IRC14:45
jrolldansmith: this works for me
*** yamamoto has quit IRC14:46
*** yamamoto has joined #openstack-infra14:46
jrollis the result of pasting your query in14:46
dansmithjroll: yeah, me too, but not with the http auth and json output for some reason14:46
jrolldansmith: ohhhhh.14:46
jrollright, because # is a browser thing14:47
*** apuimedo_ has joined #openstack-infra14:47
jrollthis seems to make my browser give me json
jrolltry that with http auth?14:48
*** jlanoux has joined #openstack-infra14:48
*** eharney has joined #openstack-infra14:48
dansmiththat's what I'm running except %20 for spaces instead of +14:48
*** otter768 has joined #openstack-infra14:48
gsagieHello, i am trying to use docker client (python binding) in the gate , but the connection is aborted:  (this works locally) is there anything inn the gate blocking this? and is there any solution?14:49
gsagieam i doing something wrong?14:49
jrolldansmith: well, does it work with + ? :P14:50
dansmithjroll: well, I'm using urllib.quote, but I guess I'll try14:50
* jroll suddenly wonders if gertty has a dashboards feature14:51
dansmithso, if I try to load your line with curl and my http auth creds, I get "400 not signed in"14:51
dansmithso something seems weird with the http auth14:52
*** asselin has quit IRC14:52
*** otter768 has quit IRC14:53
*** ashtokolov_ has quit IRC14:54
*** betherly_ has quit IRC14:54
*** patrickeast_ has quit IRC14:54
*** bogdando has quit IRC14:55
sdagueummm... what happened to all our test nodes?14:55
sdaguefungi / yolanda ?14:55
sdaguenodepool seems empty14:56
yolandasdague, don't know... let me see14:56
gsagiethe gate might be blocking docker client somehow?14:56
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ansible-role-functional-jobs-(centos|trusty)-nv
*** apuimedo_ has quit IRC14:56
*** fhubik is now known as fhubik_brb14:56
sdagueyolanda: seems like zuul is just wedged as well14:57
sdaguewe've got a completely completed gate queue just sitting there14:57
yolandachecking logs14:58
yolandalooks more a zuul problem than nodepool, nodepool shows no queue14:58
*** tjones has joined #openstack-infra14:58
yolandasdague... 2015-12-17 14:59:30,305 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.                    Can not get account information.14:59
pabelangerquestion about signal, if you kill -1 a process that doesn't support SIGHUP, what is the expected outcome?  I would assume, the process stops since you killed it15:00
yolandainfra-root, you know about this error?15:00
*** odlD2 has quit IRC15:00
*** asselin has joined #openstack-infra15:00
*** briancurtin_ has quit IRC15:01
*** nmagnezi has quit IRC15:01
*** apuimedo_ has joined #openstack-infra15:01
*** dtardivel has joined #openstack-infra15:01
*** dslev has joined #openstack-infra15:01
SotKis there a way to see more than the 10 most recent reports in puppetboard?15:01
pabelangerSotK: you need to drill down to the specific node15:02
SotKpabelanger: I'm looking at , but it still only shows the most recent 10 reports15:03
pabelangerSotK: Oh, sorry. I missed the more then part15:03
pabelangerya, it only displays 1015:03
pabelangernot sure why15:03
yolandasdague, how long has it been happening, you know?15:04
SotKheh, thanks anyway :)15:04
sdagueyolanda: no idea, I just see that it's all jammed up15:04
*** HeOS has quit IRC15:05
*** rvasilets___ has joined #openstack-infra15:05
*** piet has quit IRC15:05
sdaguemy guess is there are new events from gerrit after the upgrade, and zuul doesn't understand some of them15:06
*** automagically_ has joined #openstack-infra15:06
*** automagically has quit IRC15:06
*** automagically_ is now known as automagically15:06
yolandai see several errors in zuul, i don't know if they are critical right now15:06
yolandaseveral errors about replication15:06
*** Mmike has quit IRC15:06
sdagueok, well nothing is progressing in the gate15:07
yolandayes, saw that :(15:07
*** Mmike has joined #openstack-infra15:08
SotKfungi: is there any chance you could provide me with some storyboard logs?15:08
*** Mmike has quit IRC15:09
*** Mmike has joined #openstack-infra15:09
pabelangeryolanda: sdague: Here are the 2 I am seeing:15:09
pabelanger2015-12-17 08:10:21,703 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.15:09
*** yaume_ has joined #openstack-infra15:09
pabelanger2015-12-17 08:10:49,191 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.                    Can not get account information.15:09
yolandandepool queue is not receiving any demand15:09
yolandazuul seems to be receiving comment added, patchset events.. but translating that to demand15:10
*** apuimedo_ has quit IRC15:10
*** vgridnev has quit IRC15:10
*** apuimedo has quit IRC15:11
*** apuimedo has joined #openstack-infra15:12
*** yaume has quit IRC15:12
*** odlD2 has joined #openstack-infra15:13
*** lucas-hungry is now known as lucasagomes15:13
*** piet has joined #openstack-infra15:13
yolandasdague, i hope that this can be solved without downgrading gerrit, but it looks so bad now15:14
*** apuimedo_ has joined #openstack-infra15:16
*** ayoung has quit IRC15:17
*** apuimedo has quit IRC15:19
*** apuimedo has joined #openstack-infra15:20
*** xarses_ has joined #openstack-infra15:20
*** xarses_ has quit IRC15:20
*** xarses_ has joined #openstack-infra15:21
*** apuimedo_ has quit IRC15:21
*** xarses has quit IRC15:21
*** jlanoux has quit IRC15:21
*** hashar is now known as hasharAway15:22
*** kzaitsev_mb has quit IRC15:23
fungittx: i'll have a look. at the bandit release issue once i'm done looking into nodepool. we had some release issues related to pypi uploads but i thought i had that solved before i went to bed (it was exercised by rather a number of rerun release jobs)15:24
*** jlanoux has joined #openstack-infra15:24
sdagueyolanda: who do we need to look into the issue deeper?15:24
yolandahi fungi, i restarted nodepool and now i don't see activity in its logs15:24
fungiyolanda: sdague: nodepool doesn't know about gerrit, so in theory this is unrelated. usually it's a sign that nodepool has lost its connection to zuul's gearman server15:24
yolandafungi, i thought that and i rebooted, but it doesn't come up normally15:24
yolandai was thiking in running my command line...15:25
ttxfungi: the release was yesterday so maybe that was the same issue15:25
yolandadoh, not rebooted, but restarted... bad words15:25
fungittx: oh, well it wasn't in the list of pypi upload jobs that got triggered yesterday, so probably it died earlier, like during tarball/wheel generation15:25
yolandafungi, are you looking at it?15:26
yolandai don't want to overlap15:26
fungiyolanda: ahh, reading more of scrollback yes it's possible this is a zuul bug with it receiving additional gerrit events it doesn't recognize15:26
fungiyolanda: starting to look, yes15:26
*** apuimedo_ has joined #openstack-infra15:27
yolandafungi, i'm afraid of some nodepool problem, that unactivity in logs is bad15:27
*** daemontool has joined #openstack-infra15:27
fungiyolanda: well, nodepoold isn't running. did you stop it?15:27
yolandawas going to ask you the same15:28
fungiheh, then perhaps nodepool crashed?15:28
*** BobBall has quit IRC15:28
yolandafungi yep, but i restarted a pair of times15:28
yolandaand after creating jenkins masters, there is no activity in logs15:28
u_glide1Hello folks!  Could anyone help me to investigate why gate-manila-buildimage-generic  job is NOT_REGISTERED ? #link
fungidmesg has rather a lot of errors about buffer issues on the loopback15:29
fungiahh, that looks like dib-related errors in dmesg maybe15:29
yolandachecked at disk , 96% of disk space usage, but we are not full15:29
fungiwe seem to get those daily, so probably unrelated15:30
funginodepoold is running again. did you start it?15:30
yolandabut look at debug.log15:30
yolandano activity15:30
*** bdemers has quit IRC15:30
yolandaprocess is there, but at least... doesn't log15:30
yolandawe can strace15:30
*** angdraug has joined #openstack-infra15:31
fungiperhaps there have been changes merged to nodepool since the last time we restarted. i'll see about rolling back the install a bit15:31
yolandafungi, changes in nodepool are not normally applied after they land? i mean, restart nodepool after a change that can affects it lands15:32
*** jaosorior has joined #openstack-infra15:34
fungiyolanda: no, we manually restart nodepoold when convenient15:34
fungiso i'm not sure what version of nodepool was running yesterday15:34
fungipresumably something crashed it and now it's trying to run a newer broken version (at least that's one theory i have)15:35
fungii'm going to try rolling back to 39b306ac3b4cdd0e7982527ec2d216ef5855039c from a couple weeks ago and see what happens15:35
yolandastrace looks bad..15:35
yolandastrace -p 241415:35
yolandaProcess 2414 attached15:35
*** bdemers has joined #openstack-infra15:35
*** mtanino has joined #openstack-infra15:36
*** bhunter71 has joined #openstack-infra15:36
fungiokay, i've restarted it at that commit installed15:37
*** sridhar_ram has joined #openstack-infra15:37
yolandafungi... i saw some activity in zuu logs, with jobs merged15:37
yolandastill no activity in debug.log, however15:37
fungilooks like it's possibly still not logging anything after the jenkins masters15:37
yolandait should log stuff, even just the regular events15:38
*** zeih has quit IRC15:39
funginext theory is that the requests 2.9.0 release from a couple days ago is to blame. i'm going to put nodepool back to origin/master but downgrade requests to 2.8.115:39
yolandabut zuul log shouws better now15:39
bgmccollumso, anecdotal evidence...but i have a change in the gate check that failed...but hasn't popped off, nor updated gerrit for almost 3 hours15:39
fungiwell, it looks like nodepool is actually building nodes based on the recent ages of some in its node table15:39
sdaguebgmccollum: yes, that's what fungi and yolanda are looking at15:39
fungibut no idea why it's not logging anything15:40
bgmccollumsdague: cool...15:40
sdaguefungi: so, is this actually a nodepool issue or a zuul issue?15:40
fungiwell, i still haven't stepped back and looked at the big picture either... when you say the change has failed, did it run all jobs?15:40
sdaguebecause the nodepool graph actually just looks like it's not being fed any work15:40
fungii just started looking at nodepool because that's where yolanda ended up15:40
sdagueok, sorry, I started with that15:41
sdaguesometime around 3 hours ago, we stopped processing work15:41
bgmccollumfungi: yes...all have run15:41
fungithat sounds like a zuul problem15:41
fungii'm going to put nodepool back like i found it, step back and start troubleshooting this from the top15:41
openstackgerritEvgeniy L proposed openstack-infra/project-config: Create openstack/bareon-specs repository
fungigoing to move to a different computer where i have a better view of everything. brb15:43
*** placko has joined #openstack-infra15:43
*** zhurong has quit IRC15:45
yolandafungi, unrelated ... but i saw logs in nodepool saying jenkins02 is offline. And i cannot reach it. Is that known?15:45
*** AJaeger has joined #openstack-infra15:45
*** fhubik_brb is now known as fhubik15:45
*** ningy has joined #openstack-infra15:46
fungiyolanda: not known to me, jenkins02 likely crashed and needs to be cleaned up/restarted15:47
fungiif you can't reach it via ssh? then maybe kernel panic or something15:47
yolandaby ui15:47
bgmccollumone other bit...not sure if related, jenkins scp publisher exception --
yolandai can reach by ssh15:47
dimsfungi : thanks for the re-run of those 2 pypi uploads last night!15:47
fungiyolanda: yeah, then you'll probably find evidence of the oom killer's work in dmesg15:47
fungidims: sure thing. apparently oslo.utils release was bitten by it too and you were unaware, but i retriggered that as well (along with gnocchi)15:48
dimsfungi : ah. thanks!15:48
*** links has quit IRC15:50
*** mriedem has quit IRC15:50
*** cfarquhar has quit IRC15:51
fungiokay, so zuul load in the nodepool graphs looks like it started to trail off around 14:00 utc15:51
fungiimage updates start at 14:14 but that could simply be coincidence15:51
*** xarses_ is now known as xarses15:51
*** ddieterly has quit IRC15:51
fungialso the fact that nodepoold isn't writing to its debug log could simply be an unrelated nuisance blinding us to the nodepool end of the problem15:52
evgenylAJaeger: hi, thanks for the review , I have a small question, do I need to additionally define "bareon group" somewhere? Or it's enough to write "group: - bareon" for the project?15:52
*** apuimedo_ has quit IRC15:52
fungihowever changes sitting in pipelines unserviced by zuul even though their jobs are complete makes me think the issue is zuul side15:52
fungimaking image updates timing and possibly lack of nodepool debug logging seem much more like a coincidence15:53
yolandafungi, last thing i saw in nodepool logs, is a queue 0 for all node types15:53
yolandai did a telnet to gear server, and status showed jobs there15:53
openstackgerritJulien Danjou proposed openstack-infra/project-config: Set delay-release for gnocchiclient
fungii'm going to start with a zuul thread dump, though it's a little like reading tea leaves for me15:54
*** esker has joined #openstack-infra15:54
fungisort of wondering if someone merged a zuul config change around 14:00 and that was maybe the first time it attempted a config reload since the restart yesterday15:55
clarkbthe fact that not even noop jobs are making it through zuul right now indicates a zuul issue15:55
*** zeih has joined #openstack-infra15:55
*** david-lyle_ has joined #openstack-infra15:55
*** esker has quit IRC15:56
funginot seeing any candidates for that theory either15:56
*** esker has joined #openstack-infra15:56
fungi merging at 09:44 seems to be the most recent thing to hit zuul configuration, unless i'm missing something15:57
*** david-lyle_ has quit IRC15:57
*** esker has quit IRC15:57
*** apuimedo has quit IRC15:57
fungiclarkb: agreed15:57
fungiLast reconfigured: Thu Dec 17 2015 10:06:51 GMT+0000 (UTC)15:58
*** apuimedo has joined #openstack-infra15:58
clarkb2015-12-17 14:01:59,919 DEBUG zuul.Scheduler: Project None not found there is a lot of that in the log15:58
fungiand the nodes graph shows evidence nodepoold was still getting demand info for some hours15:58
fungiclarkb: yep, is that the first incidence?15:58
*** rbrndt has joined #openstack-infra15:58
clarkblet me dig back15:58
*** david-lyle has quit IRC15:59
*** apuimedo_ has joined #openstack-infra15:59
fungilooks like we've had that further back15:59
*** dizquierdo has quit IRC15:59
*** ChuckC has quit IRC15:59
*** luqas is now known as lezbar15:59
fungiso probably unrelated15:59
fungifirst hit is 20:49:42 yesterday16:00
*** hasharAway is now known as hashar16:00
clarkbwhich is when we turned gerrit on16:00
fungilikely something introduced in zuul before yesterday's restart but after the prior restart16:00
clarkbso probably something in an event it doesn't understand and is getting ignored16:00
AJaegerfungi, yolanda approved a few changes this morning (European morning), zuul should have been reconfigurd there16:00
fungioh, or that16:00
*** zeih has quit IRC16:00
fungiyeah could be we're seeing events with an empty project?16:01
yolandai approved changes for project-config yes16:01
*** rhedlind has left #openstack-infra16:01
AJaegerfungi: and status../zuul says "Last reconfigured: Thu Dec 17 2015 11:06:51 GMT+0100 (CET)"16:01
sdaguefungi: well earlier yolanda linked the following message16:01
sdague<yolanda> sdague... 2015-12-17 14:59:30,305 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.                    Can not get account information.16:01
AJaegerthat's 10:06 UTC16:01
yolandabut those don't look critical16:01
yolandajust some event ignored16:01
*** kmartin has quit IRC16:02
fungiright, we have those all the way back to when zuul restarted for new gerrit16:02
*** kmartin has joined #openstack-infra16:02
fungiwhile the problem seems to have started around 14:00 utc16:02
AJaegeris it time to send out a notice that we have a stuck zuul before everybody asks?16:02
*** tjones has left #openstack-infra16:04
sdagueAJaeger: yeh, that's probably a good idea16:04
anteayaSam-I-Am: they know your schedule16:04
*** david-lyle has joined #openstack-infra16:05
yolandanodepool started logging now16:06
yolandabut queue is still to zero16:06
*** egonzalez has quit IRC16:06
yolandaso logging problem looks as a red herring16:06
pabelangeryolanda: possible somebody tried to SIGINT nodepool? Cause, it won't shutdown properly just using that.16:06
clarkbyolanda: I think that nodepool is trying to connect to jenkins02 and timing out16:07
clarkbthat causes the delay in logging16:07
fungiyeah, that explains the lengthy pause in the logs16:07
pabelangerI can reproduce an issue with SIGING and nodepool hanging16:07
yolandaclarkb, so that's one of the things i fixed then. I restarted jenkins0216:07
yolandaand it started logging again16:07
jeblairthis is the first restart since jhesketh's branch merged16:07
yolandabut still no demand received16:07
yolandaso lack of debug was caused by jenkins02 beng down.. problem solved. But we have the no queue problem now16:08
*** sridhar_ram has quit IRC16:09
e0nehi. is any news about CI status? looks like we don't recieve response to gerrit16:09
AJaegere0ne: we're debugging..16:09
e0neI checkt few patches to cinder in zuul: all tests passed but results not publiished16:09
fungiokay, there are a pair of zuul stack dumps one minute apart along with the thread stats info in ~fungi/stack.dump on zuul16:10
*** vgridnev has joined #openstack-infra16:10
fungii'm seeing what i cam make of that now, but other infra-root feel free to open that file and see what jumps out at you16:10
jeblairthe second one makes me suspicious that line numbers have changed since the restart16:11
*** yamahata has joined #openstack-infra16:11
fungioh, like we've merged more zuul changes today?16:11
jeblairyeah, which, if so, makes them a little hard to read16:11
*** boris-42_ has joined #openstack-infra16:11
jeblairor at least merged more since...what, last week?16:12
jeblairno, i guess since yesterday16:12
*** ihrachys has quit IRC16:12
*** apuimedo has quit IRC16:12
jeblairbut the git log doesn't seem to back that up16:12
jeblair  File "/usr/local/lib/python2.7/dist-packages/zuul/connection/", line 285, in query16:12
jeblair    return data16:12
jeblairi just don't expect the traceback to go into 'pprint' after that.16:12
fungiyeah, that's more than strange16:13
*** cfarquhar has joined #openstack-infra16:14
*** cfarquhar has quit IRC16:14
*** cfarquhar has joined #openstack-infra16:14
*** Qiming has quit IRC16:14
fungiPrettyPrinter seems to have a lot of process time16:14
*** odlD2 has quit IRC16:15
jeblairboth times you took thread dumps the main thread was pprinting16:17
*** apuimedo has joined #openstack-infra16:17
fungibad timing on my part? i can try a couple more16:17
*** rebase has joined #openstack-infra16:18
clarkbcouod be inefficient handling of our data structure too?16:18
*** rebase has quit IRC16:18
jeblairfungi: or perhaps it's just sinking a lot of time into that16:18
fungiseems more likely, but i'm doing two more 90 seconds apart to see if we catch it at something else16:18
jeblairthe logger for connections is wrong16:19
jeblairso we aren't logging that16:19
jeblair(this at least explains why we aren't seeing the results of all this pretty printing)16:20
*** apuimedo_ has quit IRC16:20
*** apuimedo_ has joined #openstack-infra16:20
*** markus_z has quit IRC16:22
*** tonytan4ever has joined #openstack-infra16:23
fungiokay, second pair of stack dumps and accompanying stats are in ~fungi/stack.dump.116:24
*** apuimedo has quit IRC16:24
*** pigmej has joined #openstack-infra16:24
fungia lot of PrettyPrinter in the gerrit query function here too16:26
*** yamamoto has quit IRC16:26
*** matrohon has quit IRC16:26
*** bexelbie has joined #openstack-infra16:27
*** sshnaidm has quit IRC16:28
*** odlD2 has joined #openstack-infra16:28
jeblairi'm tempted to try altering the logging config and sighuping16:29
*** bpokorny has joined #openstack-infra16:29
fungiwe do have rather a lot of varied tracebacks in zuul's logs too, which look like bugs in need of fixing, though not necessarily corresponding to when this seems to have started falling apart on us16:29
*** Swami_ has joined #openstack-infra16:30
*** apuimedo has joined #openstack-infra16:30
jeblairfrom what i can tell, it is not stuck16:30
jeblairit is just moving very, very slowly16:31
fungiso slowly that it outwardly appears to have been standing almost still for something like 2.5 hours16:32
*** tmcpeak has joined #openstack-infra16:32
stevemarhow can i find out the keyboard shortcuts for the new gerrit ui?16:32
tmcpeakhey guys, I've got the automatic push to PyPI enabled on Bandit.  I pushed a new version of Bandit yesterday but it didn't show up in PyPI.  Any idea why and what I should do about it?16:33
sdaguestevemar: "?"16:33
stevemarit used to be that when i pressed "r" it would take me to the revie form16:33
*** asselin__ has joined #openstack-infra16:33
sdaguestevemar: it's 'a' now16:33
stevemarsdague: ugh, i bet it says it somewhere on the page huh16:33
stevemarPress '?' to view keyboard shortcuts16:33
yolandainfra-root, if that helps, the only queue that nodepool seems to be receiving, is nodes for rax-iad16:33
sdaguethe "?" for help thing is a standard google product interface, I don't know if it's documented though16:34
sdagueoh, yeh, it is there on the bottom (but it's pretty light font). So not surprised you didn't catch it.16:34
*** abregman is now known as abregman|nb16:34
jeblairit's taking about 15 seconds to process each event16:34
anteayastevemar: I think you made a typo on thank you16:34
stevemardidn't know that about google interfaces16:34
*** apuimedo_ has quit IRC16:34
stevemaranteaya: yes i did!16:34
stevemarsdague: thanks :)16:35
*** apuimedo has quit IRC16:35
*** asselin has quit IRC16:35
*** scheuran has quit IRC16:35
*** ningy has quit IRC16:35
sdaguejeblair: hmmm... any theory about why it downshifted 100 gears?16:35
stevemaranteaya: and thanks to you for keeping me in check16:35
anteayastevemar: :)16:36
fungitmcpeak: yeah, ttx mentioned it earlier. we're dealing with a bigger issue right now but i'll take a look once the ci system is moving at a typical pace again. most likely the tarball job log will have the answers16:36
*** apuimedo has joined #openstack-infra16:36
tmcpeakfungi: ok sounds good.  Thank you16:36
anteayastevemar: thank you just has such a lovely ring to it16:36
jeblairsdague: signs point to spending a lot of time in pprint16:36
sdaguejeblair: because the new event structures are more complicated?16:37
jeblairsdague: i don't know yet16:37
fungistatus Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating.16:39
*** aorourke has joined #openstack-infra16:40
*** apuimedo_ has joined #openstack-infra16:40
fungialso this graph makes it look like it didn't slow down all at once, but was a little more gradual16:40
jeblairthe bug in connections i pointed out to jhesketh the other day not only means that we're now potentially operating on outdated information, we've effectively moved quite a lot of gerrit query processing from a separate thread into the main thread16:40
fungi#status notice Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating.16:41
openstackstatusfungi: sending notice16:41
*** automagically has left #openstack-infra16:41
*** _nadya_ has quit IRC16:41
jeblairthat would explain things being slower, but i would not expect that to cause it to slow down gradually16:41
*** _nadya_ has joined #openstack-infra16:41
*** marcusvrn_ has joined #openstack-infra16:42
-openstackstatus- NOTICE: Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating.16:42
*** ujuc_ has quit IRC16:42
fungiit at least looks like nodepoold may have gotten some demand numbers after 13:30 but spaced further out16:42
jeblairfungi: is there a coincident nodepool problem?16:43
yolandai still saw some demand, but very sporadic16:43
openstackstatusfungi: finished sending notice16:43
fungijeblair: possible there's also something going on with nodepoold, but hard to say until zuul is back up to speed16:44
jeblairfungi: why?16:44
jeblairi mean, why is it hard to say?16:44
fungioh, actually i guess this is easily explainable by the current zuul situation16:45
yolandafungi, i don't think there is a nodepool problem now. The lack of logging was likely caused by jenkins02 not reacting16:45
fungithere are no demand numbers to be had16:45
jeblair2015-12-17 16:04:42,424 DEBUG nodepool.NodePool:   Demand from gearman: devstack-centos7: 116:45
fungiaccording to statsd, zuul doesn't think it has a job backlog at the moment16:45
fungiso there's little reason for nodepool to see any demand16:45
jeblairright, it's correctly seeing no demand every ~10 seconds though16:46
jeblairso i believe that means it's operating normally16:46
*** HeOS has joined #openstack-infra16:46
jeblair(almost no demand -- it wants centos7)16:46
*** tmcpeak has left #openstack-infra16:46
fungiwe had some separate confusion earlier with nodepoold. it seems to have died of its own accord before yolanda and i began looking into it, and when it was restarted jenkins02 was apparently in a crashed state so it was timing out trying to connect to it and leaving the debug log silend for a very long period at startup16:47
*** apuimedo_ has quit IRC16:47
jeblairi just wanted to make sure that we weren't looking at a gearman problem16:47
yolandafungi, yep, a red herring caused by jenkins0216:47
jeblairdon't want to ignore evidence.  at least by accidenty.16:47
fungiyolanda: so anyway, double-checking that, it looks like you went ahead and cleaned up any pending slaves in the jenkins02 config and then started it up around 16:00 utc?16:48
*** Guest98282 has quit IRC16:48
jeblairokay, back to zuul... i'm going to try to figure out one of the queries the main thread is doing and then pprint it16:49
yolandai restarted safely using the play16:49
jeblairif that doesn't tell us anything, i'm going to try to live-reconfigure logging16:49
fungii'll take a quick look and see if i can tell when/why nodepoold died16:49
fungijeblair: that sounds reasonable16:49
*** otter768 has joined #openstack-infra16:49
*** burgerk has joined #openstack-infra16:49
fungiif mildly dangerous, but then again we don't have much to lose (grab a snapshot of check and gate pipelines first, presumably?)16:50
*** davideag_ has joined #openstack-infra16:50
jeblair(which will be of little overall use considering the 3k unprocessed events)16:50
clarkbjeblair: sounds good16:50
*** davideagnello has quit IRC16:51
jeblairthese tracebacks don't make any sense.16:52
*** apuimedo has quit IRC16:52
fungithe ones in the zuul log? yeah, they're a little random and strange16:52
jeblairthat doesn't call any of the methods underneath it16:53
anteayais it safe to share any of them in a paste?16:53
*** apuimedo has joined #openstack-infra16:53
*** annegentle has quit IRC16:53
fungilike it's skipping around in the source when mapping up the lines16:53
fungisimilar to the line number oddities in the stack dump?16:53
*** apuimedo_ has joined #openstack-infra16:53
anteayathank you16:53
* anteaya goes to stare at that a while16:54
stevemaranteaya: release note jobs should not run on stable branches for clients right?16:54
fungioh, you meant in the stack dump16:54
*** otter768 has quit IRC16:55
jeblairthe *second* tracebacks don't make any sense16:55
jeblairthe first ones do16:55
jeblairclarkb: could there be something funny about the way our signal handler is doing that?16:55
fungiin both pairs of stack dumps?16:56
jeblairfungi: yep16:56
fungiif so, seems the most likely place for that to be happening16:56
jeblair2 is a pattern :)16:56
* jeblair ignores second tracebacks and gets back on track16:56
*** davideag_ has quit IRC16:56
clarkbjeblair: possibly? maybe if there is a context switch while dumping tracebacks it races the code that is running16:56
jeblairclarkb: or maybe the profiler is mucking things up?16:57
clarkbI am not sure if the GIL locks up around traceback writing16:57
fungior we're somehow reusing the first context in the second dump16:57
*** placko has quit IRC16:58
anteayastevemar: um check with ttx and dhellmann when doug is back from vacation but you sound like you believe the answer is no16:58
*** claudiub has quit IRC16:58
clarkbI would be surprised if the profiler could do things that confuse where stack pointers are in threads16:58
*** placko has joined #openstack-infra16:58
clarkbthe profiler can add things to the stack but the stack pointer is pretty fundamental to operation.16:58
clarkbthough maybe its just messing up the recording of such things16:59
fungiwe've got ip allocation over quota errors for bluebox-sjc1 (not tat it's important for now)16:59
*** ddieterly has joined #openstack-infra16:59
*** ihrachys has joined #openstack-infra16:59
fungialso we've got ports leaked in hpcloud preventing node launches there16:59
*** annegentle has joined #openstack-infra17:00
*** abregman|nb is now known as abregman17:00
*** rhallisey_ has joined #openstack-infra17:00
*** changbl has quit IRC17:00
fungiand possibly also floating ips leaked too since there are a lot of http 500 unknown error responses from nova there in addition to the port overlimit errors17:01
clarkbjeblair: for the trace you pasted I think data = change._data calls connection.gerrit query()17:01
clarkband htats trying to render?17:01
*** rhallisey_ has quit IRC17:02
clarkboh maybe I am wrong17:02
pleia2good morning17:02
jeblairclarkb: did someone make that a property?17:02
*** maishsk has joined #openstack-infra17:02
clarkbjeblair: trying to see if that is the case now, it is odd that the source updates change's _data17:02
*** maishsk has quit IRC17:03
*** fhubik has quit IRC17:03
clarkbisMerged() updates _data17:03
anteayamorning pleia217:04
*** lykinsbd has joined #openstack-infra17:04
anteayapleia2: we have a bit of an issue with zuul, it is moving slowly, we don't know why yet17:04
clarkband isMerged() runs query() but I don't see it getting called17:05
jeblairclarkb: _isMerged does not17:05
anteayapleia2: fungi has some thread dumps on his zuul user if you want to catch up by looking there17:05
jeblair(with the tail)17:05
clarkbjeblair: right17:05
*** eharney has quit IRC17:05
*** yaume_ has quit IRC17:05
pleia2anteaya: thanks, I saw the notice and have been catching up :)17:06
anteayapleia2: wonderful17:06
jeblairto start with, the info for 252250 is 4MB.17:07
jeblairi believe we're looking at a looping third party ci as the root cause17:08
*** eharney has joined #openstack-infra17:09
fungioh, is zuul getting flooded with events?17:09
sdaguex-io seems to be tight looping there17:10
sdagueevery 5 - 10 seconds17:10
openstackgerritAntoine Musso proposed openstack-infra/zuul: Update merge status after merge:merge is submitted
anteayaI think we have close connection on gerrit 2.11 yay17:10
anteayaif we can find out which ci it is17:10
sdagueanteaya: x-io17:10
reedYour voter key is invalid. You should have received a correct URL by email.17:11
jeblairit takes 5 seconds to query gerrit for 252250 and another 1.5 to parse and pretty-print it.  so that's about half of our time.17:11
reedknown issue? trying to vote for O17:11
*** hashar has quit IRC17:11
anteayareed: you need to find mordred17:11
clarkbprobably need to disable the account first to prevent it reconnecting17:11
reedanteaya, thanks17:11
anteayareed: welcome17:11
sdagueyeh, it's still posting17:11
anteayaclarkb: good point17:11
pleia2reed: as a data point some folks had problems the day it was sent out, but for the most part they sorted themselves out, so yeah, ask mordred :)17:12
*** mudassirlatif has joined #openstack-infra17:12
sdagueit started going crazy 12 hrs ago17:12
sdaguefrom what I can see in the logs17:12
clarkbI am setting that email addr inactive now17:12
anteayasdague: oh you told me before and I didn't understand what you were saying17:13
clarkbok thats done, now tofigure out the session17:13
anteayaas x-io means nothing to me17:13
fungiyeah, i'm getting the connections list now17:13
sdagueanteaya: oh, osrry17:13
*** ayoung has joined #openstack-infra17:13
anteayasdague: no worries, our brainwaves diverge17:13
fungiit takes a while, i think because it tries reverse dns on all the ip addresses17:13
fungithere we go17:13
jeblairfungi: there's an -n option17:14
fungii always forget about that17:14
*** automagically_ has joined #openstack-infra17:14
fungiwhat's the username associated with
anteayayeah I can't get them by email17:14
clarkbfungi: not sure may have to ask the db17:15
pleia2morning zaro17:15
*** apoorvad has joined #openstack-infra17:15
fungiyep, digging in the db now17:15
fungijust didn't know if someone already knew17:15
anteayazaro: can you help me here, this command used to work and is failing for me today:
anteayazaro: has the syntax changed?17:15
*** salv-orl_ has joined #openstack-infra17:15
anteayafungi: was trying to find it, my usual method is failing17:16
jeblairof course there is no cinder-ci group to make this convenient17:16
anteayafungi:  xio-ise-iscsi-ci17:16
anteayafungi: according to their wikipage17:16
fungiyep, they have two accounts... x-io-ise-ci and xio-ise-iscsi-ci17:17
zarocan you get your own account?17:17
zarocase sensitive?17:17
fungii unfortunately don't see either of those in the gerrit connection list17:17
clarkbfungi: maybe it went away as part of the inactivation?17:18
zaromine works ok, curl
*** ayoung has quit IRC17:18
fungishow-connections -n also doesn't lookup usernames17:18
*** automagically has joined #openstack-infra17:18
*** salv-orlando has quit IRC17:18
jeblairjust ids17:18
clarkbfungi: does it return account id because that is numeric?17:18
zaroahh case sensitvie, curl, does not work17:18
sdagueit looks like it stopped 4 minutes ago17:18
anteayazaro: thanks17:18
anteayasdague: someone is watching the channel mayhap?17:18
sdagueso the deactivation17:19
*** automagically_ has quit IRC17:19
jeblairclarkb: disabled it, so perhaps they were also reconnecting17:19
fungiwe also have an emc-xio-ci conencting a couple of times but presumably they're not involved17:19
jeblairfungi: what 2 account numbers did you find?17:19
sdagueanyway, that change is no longer getting updated17:19
*** jistr has quit IRC17:19
*** changbl has joined #openstack-infra17:19
fungi13142 and 14969 the latter of which has an open connection so i'm killing that now17:20
fungi75a0b1ad 20:39:00 00:00:04   a/14969
jeblairxio-ise-iscsi-ci a/1496917:20
xyang1fungi: what do you mean17:20
anteayaxyang1: hi17:20
*** hemna has joined #openstack-infra17:20
xyang1anteaya: hi17:20
zaronice to see new gerrit haters coming out of the woodwork.17:20
jeblairfungi: yeah that's from the sshd log, that was the one doing it17:20
xyang1what is the problem17:20
fungiclosing connection 75a0b1ad...17:20
hemnalots of us don't like the new UI17:20
fungiclosed connection 75a0b1ad17:20
*** unicell has quit IRC17:20
anteayaxyang1: is emc-xio-ci at all associated with
anteayaxyang1: thank you17:21
xyang1anteaya: that is a different company17:21
anteayafungi: the answer to your question about emc-xio-ci is no17:21
fungixio-ise-iscsi-ci was the one connecting17:21
anteayaxyang1: thank you17:21
zarohemna: yes, nice hearing from you.17:21
fungithanks xyang117:21
jeblairfungi, clarkb: sshd log is probably a good way to confirm these in the future.17:21
anteayaxyang1: we had a ci auto generating comments and it took out our system17:22
*** yamahata has quit IRC17:22
anteayaxyang1: you alread know not to auto generate comments, I'm sure17:22
*** rhedlind has joined #openstack-infra17:22
xyang1I think so17:22
anteayaxyang1: yes wanted to make sure your system wasn't related17:22
sdaguejeblair: so is the belief now that this just needs to burn down the event queue and we'll be back to normal?17:23
*** bradjones has quit IRC17:23
anteayaxyang1: can you check with your operators and make sure they know, no auto generated comments please17:23
xyang1anteaya: sure, thanks17:23
clarkbI have to step out for a bit have an appointment in just over half an hour I need to get ready for17:23
jeblairfungi, clarkb: if we weren't swamped with irrelevant log entries for the new gerrit events (i thought someone fixed that?), we might have noticed that zuul is actually querying the same change repeatedly.17:23
anteayaxyang1: thank you17:23
*** trown is now known as trown|lunch17:23
*** rossella_s has quit IRC17:23
xyang1anteaya: I'll pass that along to make sure17:23
anteayaclarkb: I hope it goes well17:23
*** rossella_s has joined #openstack-infra17:23
anteayaxyang1: thank you17:23
xyang1anteaya: thank you17:23
*** bradjones has joined #openstack-infra17:24
*** bradjones has quit IRC17:24
*** bradjones has joined #openstack-infra17:24
jeblairsdague: sort-of.  i suspect there are N events in the queue from that change, and it'll take 15 seconds for each.  we should probably edit the gerrit db, delete the comments from that system on that change, and then restart zuul, taking the data loss hit to get moving again.17:24
anteayaI will email the third party announce mailing list regarding xio-ise-iscsi-ci17:25
jeblairi could do the math, but my gut says hours or days to get through that.17:25
sdaguejeblair: coolio17:25
rhedlindanteaya: what's up with the X-IO CI?17:25
jeblairclarkb, fungi: i will edit the db now.17:25
sdagueyeh, at this point I think whatever gets work flowing again fastest is cool17:25
fungijeblair: thanks17:26
sdagueonce we get a zuul restart I'll go and see what's in approved state and stuck17:26
jeblair(also, i believe we can *vastly* improve this in zuul; i believe i'm going to hit some of this code in my v3 work, i'll try to keep this case in mind)17:26
anteayarhedlind: it took out our zuul with auto generated comments17:26
fungiand yeah, i'm playing with analyzing the gerrit ssh logs now for a good way to spot these17:27
anteayarhedlind: it is disabled and disconnected17:27
*** jlanoux has quit IRC17:27
rhedlindanteaya: huh? how could it do that?17:27
*** elo has joined #openstack-infra17:27
anteayarhedlind: auto generated comments17:27
sdaguerhedlind: it was reporting every 5 - 10 seconds17:27
sdagueon review 25225017:27
sdague8455 comments in the last 12 hours17:28
fungiso these are the 10 most active accounts commenting via ssh connection during the utc 16:00 hour
*** jcoufal_ has quit IRC17:28
jeblairactually, it's possible that removing the comment data might make zuul fast enough to process the backlog in a reasonable time17:29
jeblairso i will wait to asses performance before restarting zuul17:29
sdaguejeblair: that's worth trying17:29
fungilooks like it was averaging just shy of 10 a minute17:29
rhedlindsdague: anyone has a theory on how it could do that? My zuul server is eating up 7.17 gig of memory and jenkins crashed due to out of memory17:30
*** leeantho has joined #openstack-infra17:30
*** HeOS has quit IRC17:30
rhedlindwhy would it keep commenting on the same change?17:31
jeblairselect count(*) from change_messages where change_id=252250 and author_id=14969;17:31
jeblair 846317:31
jeblairturning that into a delete17:31
*** ddieterly has quit IRC17:31
*** eharney has quit IRC17:32
jeblairdelete from change_messages where change_id=252250 and author_id=14969;17:32
jeblairUPDATE 8463; 133 ms17:32
jeblairzuul's cycle is about 1s per event now17:32
jeblaira lot of those events are ignorable17:32
jeblairso we may just be able to let it run17:33
anteayarhedlind: it has an auto generated commenter which loops on something17:33
*** apuimedo_ has quit IRC17:33
anteayarhedlind: such as if it posts a message with recheck in the comment and triggers on the word recheck17:33
fungilooks like it started at 05:06:38, so i guess zuul wan't really crippled by that until load picked up elsewhere?17:33
*** automagically has left #openstack-infra17:34
sdagueyeh, the event queue seems to be dropping17:34
jeblairfungi: it would have had to leave enough comments to generate enough data to slow things down17:34
sdaguepretty quickly17:34
jeblairfungi: this matches the 'gradually got worse' profile you observed17:34
fungixio-ise-iscsi-ci posted 1634 comments during the 05:00 hour when it was first starting to get going17:34
anteayais today the first appearance of that ci system sshing into gerrit?17:35
sdagueso, I'm not sure if it's helpful or not, but given that gerrit 2.11 does merge conflict checking internally, those merge conflict checks could be disabled now in zuul I think17:35
fungiand yeah, it looks like its rate slowed over time (as the performance of our systems began to degrade?)17:35
sdagueyou can even query for mergable17:35
*** ayoung has joined #openstack-infra17:36
jeblairsdague: yeah, i think we'd want to have zuul consider gerrit mergability as a gate pipeline requirement first, but once that's in place, i think we can drop the merge-check pipeline17:36
rhedlindanteaya: our CI has been online for quite some time (since this spring) and I have not changed the zuul scripts recently.17:36
anteayathis is a ui style nit: I can't see what project the patch is for before I scroll down17:37
jeblairsdague: i'm not sure whether the needed bits for that are in zuul already and therefore it would just be a layout.yaml config change, or if we'd need to add something to zuul first to pick up mergeability17:37
anteayarhedlind: which ci is yours?17:37
electrofelixzaro waynr: wondering if either of you have looked at it's to remove the need to having to update setup.cfg for ever new addition to JJB. Only if it's a brand new module.17:37
fungirhedlind: oh! you're the one we just disabled17:37
anteayarhedlind: oh well time to take a look at its logs then17:37
rhedlindthe one causing all the trouble17:37
rhedlindand making me sick right now17:38
AJaegerstevemar: release notes job should run on stable branches - why do you ask?17:38
anteayarhedlind: well take a walk around17:38
AJaegerstevemar: but we publish from master the content including all branches you configured17:38
anteayarhedlind: we need to decompress ourselves and assure ourselves that our backlog can clear17:38
*** apuimedo_ has joined #openstack-infra17:39
anteayarhedlind: your system isn't doing any more harm at the moment17:39
stevemarAJaeger: i'm asking cause of this:
electrofelixwaynr: I like your JJB CLI changes proposed. I would however like if we could make a push to land a lot of the other work that is in flight at the moment, do a release and then merge your CLI rework.17:39
anteayarhedlind: after you have had a walk and a cup of tea, and we have assured ourselves our backlog will clear17:39
*** ddieterly has joined #openstack-infra17:39
stevemarAJaeger: i was under the impression that we don't include reno for liberty for libraries and clients17:39
anteayarhedlind: then you can go through some of your event logs, and we can look too if you wish to share17:40
anteayarhedlind: and we can figure out what happened17:40
anteayarhedlind: let us decompress though17:40
*** ifarkas has quit IRC17:40
stevemarAJaeger: the release note fail, not the python test fail :P17:40
sdaguethe event queue has drained to 0, but the integrated gate still seems stuck17:40
rhedlindanteaya:ok. I am really sorry for all this. as far as I knew it was running along fine.17:41
stevemarAJaeger: actually... that is really old, sorry17:41
anteayarhedlind: okay thanks, let's take a break for a bit and then come back to this and figure out a way forward17:41
anteayarhedlind: thanks for contacting us so quickly17:41
jeblairit looks like it's reporting changes in check...17:41
stevemarAJaeger: cause this one, same repo and branch doesn't include the releasenotes job:
jeblairsdague: oh17:42
jeblairsdague: it's only just now processing the queues for the first time since the dos started17:42
zaroelectrofelix: i have not but that's very cool.17:42
jeblairsdague: so it's got to go through all of the pipelines now17:43
sdaguejeblair: ok, is there a known order for that?17:43
*** shardy has quit IRC17:43
jeblairsdague: yeah, check then gate17:43
zaroelectrofelix: i've been busy with gerrit lately so haven't gotten around to reviewing much.  now i we to fix stuff after upgrading.17:44
sdagueI guess I am surprised the gate doesn't have priority on that17:44
anteayafungi: also when you get to email, while all the fun was going on, ovh posted to the infra list with a suggestion of how to help us more17:44
openstackgerritApoorva Deshpande proposed openstack-infra/ciwatch: Separate votes from Jenkins queues
jeblairsdague: i don't think it's come up before :)  but we could probably sort them in priority order17:44
anteayafungi: I replied saying we will discuss the suggestion as a group and get back to him and we appreciate all his help and his commitment to supporting us17:45
dmelladoHi here, is zuul still having issues? I'm seeing a gate queued like...forever17:45
dmelladoI guess so, but just wanted to double check ;)17:45
anteayadmellado: yes we just identified the issue and it is dealt with17:45
anteayadmellado: it will take some time for the backlog to clear17:45
dmelladothanks anteaya ;)17:45
anteayadmellado: thank you17:45
*** kushal has joined #openstack-infra17:46
*** kushal has quit IRC17:46
*** kushal has joined #openstack-infra17:46
*** bexelbie has quit IRC17:46
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Workaround 500 error when moving items in worklists
fungittx: i can't find any evidence that zuul ever saw the bandit 0.17.0 tag upload (according to its logs). tmcpeak doesn't seem to stick around in channel or i'd ask him, but my guess is that he pushed it the _moment_ gerrit came back online, a few minutes before we started zuul up after the upgrade maintenance17:47
jeblair(so, before we announced the maint window was over)17:47
fungii'll just reenqueue it here in a bit once things calm down17:47
fungijeblair: no idea but that's my guess17:47
fungii could check the gerrit logs actually17:48
*** ddieterly has quit IRC17:48
*** yamahata has joined #openstack-infra17:48
*** asselin_ has quit IRC17:48
*** eharney has joined #openstack-infra17:49
*** ociuhandu has quit IRC17:49
fungi[2015-12-16 20:42:16,138 +0000] ecb50f4c travis-mcpeak a/11861 git-receive-pack./openstack/bandit.git 0ms 436ms 017:49
fungiso... yep17:49
*** gyee has joined #openstack-infra17:49
sdaguegate queue finally moving again17:49
sdagueno longer staring at that tempest change at the top of it17:50
AJaegerstevemar: let's discuss on #openstack-keystone, I'll join you there to get noise out of here..17:50
*** apuimedo_ has quit IRC17:50
*** electrofelix has quit IRC17:51
*** tonytan4ever has quit IRC17:52
jaypipesfungi, clarkb, jeblair: sorry if stupid question, but in the latest gerrit, is there a way to go back to the old code review screen?17:52
*** e0ne has quit IRC17:53
olaphunfortunately, no17:53
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix various style bugs with cards
*** u_glide1 has quit IRC17:54
sdaguejaypipes: see -
fungijaypipes: the old review screen was finally dropped completely from gerrit upstream in 2.11. the "new" (not so new as it's been available as an option for years) review screen is now the only web interface. they're working on an even newer review screen to replace it in upcoming gerrit versions, but it's still in the design stages17:55
*** apuimedo_ has joined #openstack-infra17:55
*** abregman is now known as abregman|mtg17:55
pkoniszewskihi, is there an option to start (implement) a test in gate with, let's say, custom image? I need something that will prevent live migration from finishing...17:56
*** asselin has joined #openstack-infra17:56
anteayajaypipes: and jeblair readily hands out links to gertty to all who ask17:57
jaypipessdague, fungi: OK, thanks guys.17:57
anteayapkoniszewski: what do you mean when you say custom image17:58
anteayapkoniszewski: do you mean use your image instead of bare-trusty?17:58
*** dangers_away has quit IRC17:58
*** SumitNaiksatam has joined #openstack-infra17:59
*** apuimedo_ has quit IRC17:59
pkoniszewskianteaya: it can be bare-trusty if I can modify it a bit somehow, e.g., install and start additional software during test18:00
*** derekh has quit IRC18:00
openstackgerritMerged openstack-infra/openstackid: OIDC - OpenId Connect Implementation
*** ashleighfarnham has joined #openstack-infra18:00
fungipkoniszewski: the jobs have full root access on the machines where they run and can do pretty much anything. you don't need a custom image for what you're describing18:00
*** thiagop has quit IRC18:01
pkoniszewskifungi: so there is an option to start a VM with stress installed on so it will dirty memory and therefore prevent live migration from finishing?18:01
*** bgmccollum has left #openstack-infra18:01
pkoniszewskifungi: or just install a stress tool post-boot18:01
fungipkoniszewski: why does it need to be installed before booting?18:02
fungiyes, you can run any commands you want in a job18:02
fungiincluding installing additional packages18:02
SumitNaiksatamhi, the horizon team is trying to end the meeting on openstack-meeting-3 but the bot does not seem to be responding18:04
david-lyleI think it ended18:04
david-lylebut the channel status is stuck18:04
david-lyleother meetings have been started and stopped since18:04
sdaguewe still don't seem to be demanding nodes as fast as I'd have guessed18:05
*** esker has joined #openstack-infra18:05
*** sputnik13 has joined #openstack-infra18:05
*** apuimedo_ has joined #openstack-infra18:05
fungidavid-lyle: which channel?18:05
sdaguethat might be a pathological issue in the work that zuul does before it asks for any nodes18:05
SumitNaiksatamdavid-lyle: thats interesting18:06
david-lylefungi: openstack-meeting-318:06
SumitNaiksatamfungi: meeting-318:06
fungilooks like meeting is also in a similar state18:06
fungiprobably the meetbot got netsplit or restarted in the middle of those meetings18:06
*** kdas_ has joined #openstack-infra18:06
*** kdas_ has quit IRC18:06
*** dangers_away has joined #openstack-infra18:07
jeblairsdague: i don't think zuul has fully worked through everything yet18:07
sdaguejeblair: that could be18:07
*** kushal has quit IRC18:07
jeblairsdague: the 'queued' jobs are actually probably mostly still in merge-pending state18:08
SumitNaiksatamfungi: i just started a new meeting, the topic changed18:08
*** unicell has joined #openstack-infra18:08
SumitNaiksatamfungi: so i think we are good :-)18:08
*** _nadya_ has quit IRC18:08
jeblairsdague: which means they're waiting on results from mergers before the actual jobs get launched18:08
sdagueI guess I'm surprised that's true even for the gate18:09
*** _nadya_ has joined #openstack-infra18:09
fungiSumitNaiksatam: david-lyle: yeah, it's just the default channel topic which is getting set back to an incorrect string at the end of meetings now. i need to catch that channel between meetings to fix it safely (like i just did for #openstack-meeting)18:09
SumitNaiksatamfungi: okay thanks18:09
jeblairsdague: gate does get priority there,  but there's actually still a backlog of _results_ it's working through now18:10
*** davideagnello has joined #openstack-infra18:11
sdagueok, sure, there were just 12 patches in the gate above the current work set, I thought as soon as the sliding window was open we started running things, based on the same speculation criteria as before18:11
david-lylethanks fungi18:11
*** jerryz has joined #openstack-infra18:11
sdaguebut I guess that was my misunderstanding18:11
*** zeih has joined #openstack-infra18:11
pkoniszewskifungi: just one more thing, by default devstack uploads only a cirros image to glance during installation, do I need to use image builder to build trusty image or is there a store that contains other images than cirros?18:11
*** eharney has quit IRC18:11
jeblairsdague: this is "as soon as" when zuul hasn't caught up with 5 hours of backlog :)18:11
*** esker has quit IRC18:12
sdagueok, I guess I don't have the model of what gets processed fifo before we get there :)18:12
*** ihrachys has quit IRC18:12
*** _nadya__ has joined #openstack-infra18:13
*** ihrachys has joined #openstack-infra18:13
jeblairi think it just launched a bunch of jobs because nodepool is now hyperactive18:13
*** _nadya_ has quit IRC18:13
fungipkoniszewski: do you expect to actually boot the images? the main reasons we use cirros there are 1. we have limited disk space on the workers (as little as 80gb on some) and full cloud images take up a lot of space, 2. we have limited ram in the worker (8gb) and a lot of full distros need lots of ram to run effectively at all, 3. it's unaccelerated qemu inside a virtual machine there so you don't18:14
fungiwant to wait 30 minutes for the image to boot18:14
jeblairit just now cleaned out the noop queue18:14
openstackgerrityolanda.robla proposed openstack-infra/puppet-diskimage_builder: Add lvm2 package to diskimage builder
fungipkoniszewski: i think heat has one or two jobs which boot a minimal fedora image to test some sort of guest client, and it takes some minutes for that to boot18:15
sdagueyeh, gate queue seems to have started to get nodes now18:16
*** zeih has quit IRC18:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable reno on liberty for keystoneauth/keystonemiddleware
AJaegerstevemar: ^18:16
fungijeblair: not sure if you saw my comment late last night, but the replacement went well and we've since done pypi uploads with distro wheel/twine no problem18:16
jeblairfungi: yay!18:16
fungiso that was a great idea18:17
jeblairfungi: i can't believe you stayed up to do that, but thanks! :)18:17
aspiersooh, shiny new gerrit! :)18:17
fungijeblair: well, i was worried people were going to try to push more releases and i didn't want to have to do more manual cleanup than necessary18:17
*** apuimedo_ has quit IRC18:17
jeblairsdague: i _think_ it's caught up now18:17
*** annegentle has quit IRC18:17
sdagueyeh, it seems like it's chugging18:18
fungiif it's caught up, i'll go ahead and retrigger that missing bandit release now18:18
jeblairfungi: ++18:18
*** annegentle has joined #openstack-infra18:18
*** ybathia has joined #openstack-infra18:18
fungiwell, reenqueue i mean18:18
sdaguenow, if we could only get the pypi wheel mirror code landed we'd shave 10 - 30% off the run time of the jobs :)18:18
jeblairor even enqueue for the first time ever :)18:18
fungiright, that18:18
fungitmcpeak must have been either really unlucky and oblivious, or really in a hurry yesterday18:19
*** esker has joined #openstack-infra18:20
jeblairwhile ! git push; do sleep 1; done18:21
jeblairwhat could possibly go wrong18:21
*** maishsk has joined #openstack-infra18:21
AJaegeraspiers: thanks for the enthusiasm and not complaining18:21
*** ayoung has quit IRC18:21
aspiersAJaeger: I haven't found anything to complain about in the first 60 seconds of using it, so congrats ;-)18:22
*** bgmccollum has joined #openstack-infra18:22
sdaguethere are many nice new features as well18:22
*** apuimedo_ has joined #openstack-infra18:22
sdagueConflicts With is one of my favorites18:23
aspiersI read a while back and remember getting excited18:23
sdagueas it helps you find duplicate patches for the same thing18:23
*** abregman|mtg is now known as abregman18:23
anteayaI'm getting duplicate emails for the same patch18:23
anteayathis is new for me18:23
fungisdague: oh that's awesome--i hadn't even looked at that yet18:23
sdaguefungi: yeh, it's one of the column 3 sets18:23
anteayaAJaeger: did you add me as a reviewer to 259137?18:23
boris-42_AJaeger: hi there18:24
boris-42_AJaeger: could you please add me to osprofiler-core and osprofiler-release ACL group?)18:24
AJaegeranteaya: no, I added stevemar - after you reviewed it18:24
AJaegerboris-42_: I don't have those permissions, infra-root can help you18:25
boris-42_AJaeger: ok18:25
boris-42_SergeyLukjanov: could you please add me to osprofiler-core group?)18:25
boris-42_SergeyLukjanov: hi18:25
*** eharney has joined #openstack-infra18:26
*** aopincar has joined #openstack-infra18:27
anteayaAJaeger: I'm trying to figure out why I started getting two emails for each event on that patch18:27
*** harlowja has joined #openstack-infra18:27
pleia2boris-42_: I'll have a look, do you have a link to the change that added those groups to gerrit?18:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable reno on liberty for keystoneauth/keystonemiddleware
anteayaAJaeger: I actually can't find where I turn email off, I know there used to be a place18:28
AJaegeranteaya: Haven't seen such an option18:29
anteayano? perhaps I imagined it18:30
zaroAJaeger: could you please tell me again how i can disable bashate? on change
AJaegerzaro: let me try patching it for you...18:31
*** tjones has joined #openstack-infra18:31
zaroohh even better  :)18:31
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix docstring of downstream-ext publisher
anteayaI still don't understand why I have two email threads for
pleia2boris-42_: ah, found it and I've added you to both18:31
*** sfinucan has quit IRC18:32
*** amuller is now known as amuller_afk18:33
openstackgerritMerged openstack/os-testr: Fail if no tests were successfully executed
*** ayoung has joined #openstack-infra18:33
openstackgerritMerged openstack/os-testr: Add support to ostestr to use subunit-trace color
openstackgerritMerged openstack-infra/glean: Open /etc/hosts file in append mode, not write
jesusauruswhat's the process for removing a jenkins master from nodepool? if I simply remove the target section from nodepool.yaml, would nodepool safely stop adding slaves? would zuul continue to use up the remaining nodes?18:34
zaroanteaya: do you mean you are getting double emails?18:34
anteayazaro: yes, that is what I mean18:34
anteayaI can forward you both threads if you like18:35
zaroanteaya: do you have it starred?18:35
zaroit's only for that change?18:35
anteayayes so far18:36
*** devananda has quit IRC18:36
anteayaI haven't done a whole lot of reviewing since yesterday though18:36
anteayathis may be the only change I have reviewed since the restart18:36
zaroanteaya: check settings,
*** apuimedo_ has quit IRC18:37
anteayasorry I did review this:
zaroanteaya: is CC selected?18:37
anteayaone email thread18:37
anteayazaro: I looked in preferences, CC is not selected18:37
zarowhat? 2 or 1 email?18:38
anteayatwo threads18:38
*** zeih has joined #openstack-infra18:38
zarodo you have multiple things selected on watches?18:38
anteayanow one thread18:39
anteayaI think something confused my thunderbird18:39
anteayaand now thunderbird has gotten itself figured out18:39
anteayaas I look now there is one thread18:39
anteayabut there were two18:39
zarouhh, me not use thunder, but cool18:39
anteayaokay so blip18:39
anteayathanks for listening and sorry for the noise18:40
*** sambetts is now known as sambetts-afk18:41
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Add automated database config retrieval
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add support for querying role assignments.
AJaegerzaro: fixed ^18:41
*** jpr_ has joined #openstack-infra18:41
AJaegerzaro: but long lines are only warnings for bashate, the errors are other stuff like function delcaration or indent not multiple of 418:42
*** apuimedo_ has joined #openstack-infra18:42
*** zeih has quit IRC18:43
zaroah, thanks.  you've made my day  :)18:43
AJaegerzaro: let me update the script instead and you can compare which version you prefer ;)18:44
boris-42_pleia2: thanks18:44
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Add automated database config retrieval
AJaegerzaro: this ^ passes bashate locally, check the diff, please18:45
*** e0ne has joined #openstack-infra18:46
*** trown|lunch is now known as trown18:47
*** apuimedo has quit IRC18:47
*** esker has quit IRC18:47
*** esker has joined #openstack-infra18:47
*** ayoung has quit IRC18:47
*** bharathm has joined #openstack-infra18:48
*** apuimedo has joined #openstack-infra18:48
*** apuimedo_ has quit IRC18:49
jpr_morning...I'm trying to set up an instance of openstack-infra at our site so I can emmulate the management philosophy of, e.g. "systems as code with a defined ci/cd fabric and best-practice workflows"18:49
zaroAJaeger: me likes!18:49
anteayajpr_: morning, have you seen
*** otter768 has joined #openstack-infra18:50
anteayajpr_: that was created by one of our third party operators to create a system that consumes the gerrit event stream via ssh and report back to our gerrit again via ssh18:51
jpr_yes...I think...  I've been working with
*** piet has quit IRC18:51
anteayaah that is a tad dated18:51
jpr_yes  I sense that but felt there was some structural understanding emebedded... at least from steps 1  2 and 318:52
anteayaalways helps to have the big picture18:52
jpr_i'm assuming that basic sturcture is still valid.  is that fair?18:53
*** aopincar has quit IRC18:53
anteayaand this is what it would look like from the dev's point of view:
*** devananda has joined #openstack-infra18:53
* AJaeger gets lots of mails for gerrit events ;)18:53
anteayawell I don't know as I have ever read that document, but if it says we use gerrit zuul gearman jenkins and nodepool then that part is valid18:54
AJaegerfungi, do you want to send an all clear again?18:54
*** gyee has quit IRC18:54
anteayaAJaeger: I bet you do18:54
*** tzn has quit IRC18:54
fungiAJaeger: well, it would be another notice since i didn't use alert for this situation. i was considering waiting until the backlog burned down a bit before making another announcement18:54
anteayathe only part openstackci doesn't set up is gerrit, since the use case for which that puppet module was designed was to listen to our gerrit18:55
asselinHi, I'm not able to login to gerrit anymore. I keep getting "Server Error"18:55
*** otter768 has quit IRC18:55
*** apuimedo_ has joined #openstack-infra18:55
anteayajpr_: but there is a puppet module availabe that we use to set up our gerrit18:55
fungiasselin: when did that start?18:55
anteayaasselin: :(18:55
anteayaI just logged in via chrome18:56
asselinfungi, this morning18:56
AJaegerfungi, fine with me....18:56
fungiasselin: which url is returning the server error? gerrit or launchpad/ubuntu?18:56
*** eharney has quit IRC18:56
asselinsorry...didn't realize it was so long18:56
*** eharney has joined #openstack-infra18:56
anteayatoday I learned18:56
asselinactually...perhaps that's my 'dashboard'...let me try without it18:57
*** esker has quit IRC18:57
fungiasselin: looks like you might be trying to go to a custom dashboard whose url is not parsed properly by 2.1118:57
*** _nadya__ has quit IRC18:57
sdagueasselin: dashboards that reference self fail unless you are logged in18:57
fungiyeah, good idea18:57
asselinyup, ok now I logged in.18:58
*** _nadya_ has joined #openstack-infra18:58
asselinand my dashboard works too. ok thanks, sorry for the noise.18:58
jpr_ok.  so I'm working under the assumption that i still need to 1) set a vm a puppetmaster 2) follow the puppetmaster setup (with the puppet install, module install and system-config as production clone) 3) merge in my changes from my own system-config clone18:58
openstackgerritMerged openstack-infra/project-config: networking-ovn: Make tempest job voting again.
asselinanteaya, jpr_ there's this spec which is trying to do more:
*** tjones has quit IRC19:00
anteayajpr_: our use of puppetmaster has changed substantially in the past year19:00
smarcetfungi: afternoon, have one question, which process triggers tarball creation here
anteayajpr_: I wouldn't go by that document with regards to how to set up your puppet19:01
*** burgerk has quit IRC19:01
*** _nadya_ has quit IRC19:02
*** apuimedo_ has quit IRC19:02
openstackgerritBen Nemec proposed openstack-infra/reviewstats: Gerrit 2.11 compatibility
*** tjones has joined #openstack-infra19:02
jpr_anteaya: is puppetmaster still the primary controller for deploying the services in a domain?19:03
*** tzn has joined #openstack-infra19:03
* anteaya defers that question to someone who can answer it19:03
anteayajpr_: honestly I can't keep track anymore19:04
*** esker has joined #openstack-infra19:04
fungismarcet: the tarballs like openstackid- and openstackid-latest.tar.gz appear to be created any time a change merges, so if you just had a change merge to the openstackci repo then there's probably a pending tarball job for that commit in zuul's post pipeline. we had a bit of an incident earlier today and so we're working through a pretty lengthy job backlog right now which means19:04
fungiit may take a while to complete19:04
jpr_let me sketch  my goal...19:05
smarcetfungi: oh ok19:05
smarcetthx for the info :)19:05
fungismarcet: if so, you can likely check its status at (search for openstackid there)19:06
asselinjpr_, if you like, you can add comments to the spec It intends to update the link you referenced:
jpr_I like the model that the infra project has for managing infrastructure as an openely accessible code base with a review and deploy process.  I'd like to "clone" that model within our domain adding local systems as relevent for our environment19:06
smarcetfungi: oh ok thx :)19:07
*** esker has quit IRC19:07
fungiyeah, looks like there's one queued for it from about an hour ago19:07
jpr_asselin: will certainly engage there19:07
fungismarcet: oh, actually the tarball job looks like it's already running, if you click on the blue box there to expand the job details for that commit19:08
*** apuimedo_ has joined #openstack-infra19:08
fungithat openstackid-release-branch job looks like it will probably complete in a few minutes19:08
*** sabeen1 has joined #openstack-infra19:08
fungior faster19:08
jpr_regarding the goal,  there is a lot of common logic to setting up a code review and ci/cd framework, work that is hard get off the ground in smaller shops19:09
*** u_glide1 has joined #openstack-infra19:09
*** ihrachys has quit IRC19:09
*** kashyap has left #openstack-infra19:10
fungijpr_: yep, asselin's spec is a great opportunity for us to simplify that running-your-own document which has bitrotted pretty badly over the past couple years19:10
openstackgerritMerged openstack-infra/project-config: osops-tools-generic: add bashate job
jpr_so if i can build on top of the mature process outlined by openstack-infra then i can save some effort here.  kinda like saying "we follow the openstack-infra workflow with these addtional local systems"19:11
asselinthis one's phschwartz's spec (which extends my original smaller scoped) spec19:11
fungiusually our documentation is kept up to date best when people are following it frequently and pointing out/fixing inaccuracies which creep in as things change over time, but the scope of that particular document is such that it's very rarely used, and so falls out of date a lot more easily19:11
*** tzn has quit IRC19:11
fungiasselin: d'oh, right, phschwartz's spec19:11
*** _nadya_ has joined #openstack-infra19:12
phschwartz:( I need to finish the last pass on that spec. Been supper busy which has delayed it19:13
jpr_i'll review the spec and follow up with questions19:13
anteayawelcome to the team, jpr_19:13
anteayajpr_: asselin is going to be a great resource as he has done most of what you want to do, as in consuming infra tools19:14
phschwartzjpr_: it sounds like your looking to do exactly what we are working towards with my spec.19:14
anteayajpr_: and phschwartz just recently went through setting up openstackci himself19:15
*** apuimedo_ has quit IRC19:15
phschwartzanteaya: correct19:15
anteayaso the location of many of the trip wires are fresh in his mind19:16
*** jaosorior has quit IRC19:16
jpr_anteaya: now I know who to bug. ;)19:17
anteayaalways helpful19:17
phschwartzjpr_: I have a bouncer running so if I am not at the computer I will get notified and be able to look at scroll back19:18
*** vilobhmm11 has joined #openstack-infra19:18
vilobhmm11AJaeger : ping19:18
vilobhmm11fungii : ping19:19
vilobhmm11wanted to know what I might be missing here as Jenkins seem to be unhappy every time :(19:19
AJaegervilobhmm11: please don't use naked pings, tell us what you want...19:19
openstackgerritBrant Knudson proposed openstack-infra/project-config: Keystone eventlet gate only in keystone
openstackgerritBrant Knudson proposed openstack-infra/project-config: Change postgres job to use standard deployment
vilobhmm11Ajaeger : was typing…19:19
AJaegervilobhmm11: zuul/layout.yaml defines *when* to run the job.19:20
*** _nadya_ has quit IRC19:20
*** apuimedo has quit IRC19:20
asselinvilobhmm11, fyi, you can run those tests locally using tox19:20
openstackgerritMerged openstack-infra/system-config: Add docs for renaming an IRC channel
AJaegerYou also need to declare the job itself - see jenkins/jobs/projects.yaml19:20
*** abregman has quit IRC19:20
*** _nadya_ has joined #openstack-infra19:21
*** apuimedo has joined #openstack-infra19:21
*** apuimedo_ has joined #openstack-infra19:21
vilobhmm11AJaeger : ok19:21
vilobhmm11lemme try19:21
*** dslev has quit IRC19:22
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command
jpr_phschwartz: so what is the best first step if the old how to is out of date?  the spec helps but isn't very how-to oriented.  i assume i still need to get the puppet master set up and operational so it can manage the other nodes19:24
*** zul has quit IRC19:25
*** _nadya_ has quit IRC19:25
openstackgerritKhai Do proposed openstack-infra/nodepool: Remove duplicate python-jenkins code from nodepool
*** sabeen1 has quit IRC19:25
phschwartzjpr_: the docs pertaining to it all are very outdated and working to be updated. Your best resource will be to look at openstack-infra/puppet-openstackci19:25
phschwartzjpr_: in the contrib dir there is a very good readme that asselin put together19:26
*** apuimedo_ has quit IRC19:26
*** gordc has quit IRC19:26
*** apuimedo has quit IRC19:26
* asselin needs to get that readme into the manual19:26
phschwartzjpr_: Also note the focus has been on masterless puppet so there is no more effort to setup a puppet master and puppet-openstackci is designed around this. (you can use a puppet master, but not requried)19:26
*** apuimedo has joined #openstack-infra19:27
*** zul has joined #openstack-infra19:27
dimsAJaeger : anteaya : got +1 from lifeless on this review - enables to deal with the oslo.db[fixtures] syntax in our *requirements.txt19:28
dimsAJaeger : anteaya : please take a look when you get a chance. thanks!19:29
jpr_phschwartz: puppetmaster is just the term i'm using from the old docs.  i've been working with puppet apply as described so I assume i'm in the correct mode19:30
*** asselin_ has joined #openstack-infra19:31
openstackgerritVilobh Meshram proposed openstack-infra/project-config: Add Consul Gate Job
anteayadims: is this script used by every project that runs the check-requirements job?19:32
*** apuimedo_ has joined #openstack-infra19:32
*** bharathm has quit IRC19:32
*** asselin__ has quit IRC19:33
dimsanteaya : from 'gate-{name}-requirements'19:33
jpr_phswartz: as follow up, i've noticed there are the git.openstack repos and the  repos.  They appear identical.  is one preferred from the clone perspective?19:33
*** e0ne has quit IRC19:34
*** ddieterly has joined #openstack-infra19:34
asselin git.openstack19:35
anteayadims: right, okay so putting aside my ability or lack thereof to read code for the moment, this is a change that will affect many repos gate jobs and is going straight into production, without an experimental or non-voting stage19:35
*** ddieterl_ has joined #openstack-infra19:35
anteayadims: I'm really hesitant to do that right now as we have a big backlog to get through and I dont' want to cause a fire if I don't have to19:35
anteayajpr_: the canonical git repos are under gerrit management, we run the git mirrors for our use for cloning and such, the code is mirrored to github as a courtesy19:37
jpr_phschwartz: thx. following your contrib/README.md19:37
dimsanteaya : when would be a good time? friday evening?19:37
jpr_anteaya: thx for clarifying19:37
anteayadims: I can be comfortable with that19:37
anteayadims: ensuring we have time to watch it in action and revert if it is making us sad19:38
anteayadims: is that fair?19:38
anteayajpr_: welcome, you are asking good questions19:38
dimsanteaya : yep. can you please leave a note in there?19:38
*** ddieterly has quit IRC19:38
anteayadims: I can do so, mind if I also add a -1 with my comment, just to draw attention to my comment?19:39
dimsanteaya : sounds good19:39
*** gsagie_ has joined #openstack-infra19:39
anteayadims: thanks19:39
rajinirThirdpart Ci: I'm testing but the commits to ci-sandbox is not triggering a jenkins build. Got an email ( silent check).  Any help?19:40
lifelessanteaya: dims: friday evening would be high risk. I suggest monday morning19:41
lifelessbetter *not* to have weekend fires.19:41
asselinrajinir, what's in the e-mail?19:41
rajinirasselin: Here is the review
*** apuimedo_ has quit IRC19:41
asselinrajinir, also, here's a sample repo with configs. is that what you're using?
rajinirasselin: exactly19:42
*** jed56 has quit IRC19:43
anteayalifeless: why would it be high risk?19:43
AJaegeranteaya: I agree with lifeless. I can +A tomorrow morning my time....19:43
asselinrajinir, can you paste the e-mail response?
anteayaAJaeger: well feel free to disagree with me19:43
AJaegerthanks, anteaya19:44
*** sridhar_ram has joined #openstack-infra19:44
AJaegeranteaya: what happens after friday evening? Who will be around?19:44
*** pfalleno1 has quit IRC19:44
anteayaI am saying I will19:44
anteayabut if you want to do it, feel free19:44
AJaegerFriday morning looks better to me...19:45
anteayasorry I wasn't clear19:45
dimslifeless : if it does not work then, we'd just revert it19:45
anteayaI'm not going to argue with you for fire duty19:45
anteayaif you want it take it19:45
*** u_glide1 has left #openstack-infra19:45
AJaegeranteaya: If you're around to babysit it, I'm fine...19:45
anteayathat's what I was saying19:45
rajinirasselin: email#1
dimsAJaeger : anteaya : so we'd merge that patch and yank it at first sign of trouble19:45
* AJaeger has enough firedrills in the European morning when all of you are sleeping ;)19:45
anteayasorry I was unclear that I was willing to take responsiblity19:45
*** pfallenop has joined #openstack-infra19:46
anteayadims: thanks, glad you understand19:46
anteayaAJaeger: exactly19:46
AJaegeranteaya: then I'm fine19:46
anteayaAJaeger: okay great19:46
anteayasorry to cause confusion19:46
AJaegeranteaya: no worries...19:46
asselinrajinir, noop-check-communication : NOT_REGISTERED.19:46
dimsthanks anteaya : AJaeger : lifeless19:46
*** tzn has joined #openstack-infra19:47
AJaegeranteaya: let me +2 it - and I suggest that you either -2 or WIP instead of the -1...19:47
*** apuimedo_ has joined #openstack-infra19:47
anteayaokay I can do that19:47
anteayathank you19:47
rajinirasselin: Email#2
flaper87question: Is it possible to grant "core access" to a group on a specific subdir ?19:48
AJaegerflaper87: no19:48
* flaper87 is not sure why he prefixed his last message with "question:"19:48
flaper87AJaeger: cool, thanks!19:48
anteayaflaper87: no, acls understand repos and branches19:48
flaper87anteaya: thank you19:48
anteayaflaper87: they don't understand subdirs19:48
AJaegerflaper87: that's social engineering ;)19:48
openstackgerritGal Sagie proposed openstack-infra/project-config: Add Dragonflow fullstack job
AJaegerflaper87: neutron does something like that with social engineering AFAIK19:49
asselinrajinir, did you follow all these steps?
asselinrajinir, especially line 32019:50
*** piet has joined #openstack-infra19:50
*** ybathia has quit IRC19:50
rajinirasselin: Line 320 was successful.19:50
rhedlindasselin: what version of zuul is your ci running?19:51
anteayadims: so if you can watch the gate queue and come find me when you think we have a space, that would be great19:51
dimsanteaya : you mean tomorrow evening?19:51
anteayadims: yes19:51
asselinrajinir, are you using master or a jenkins slave?19:51
asselinrhedlind, master19:51
dimsanteaya : sounds like a plan. thanks19:51
anteayadims: thank you19:52
jeblairfungi, zaro: apparently the change edit endpoints are "unstable" and "can be changed in backwards incompatible way any time without notice."  yet, the previously stable commit message endpoint was removed as deprecated.19:52
fungianteaya: AJaeger: flaper87: i believe there are ways to limit label permissions (voting) based on specific file paths though i can't recall whether that requires custom prolog processing. regardless, it's still an open question as to whether that is a reasonable additional complexity over our current repo/branch granularity19:52
*** apuimedo_ has quit IRC19:52
jeblairfungi, zaro: <sigh>19:52
fungijeblair: wow19:52
rhedlindasselin: is that 2.1.1.dev109 ?19:52
asselinrajinir, dsvm-tempest-full is likely a jenkins slave. do you see any in your jenkins UI?19:53
AJaegerfungi: now I remember, there was some dsicussion in the past and indeed required custom prolog - and we rejected that.19:53
*** amuller_afk is now known as amuller19:53
anteayaI believe it was rally that asked about that19:53
flaper87fungi: here's my use case: I'm seeking for maintainers to help with glance_store drivers. The swift team has agreed to help with the swift driver and the best way to do that is granting them permissions. The glance team is more than happy to give them access to glance_store but, if there's a way to limit the access, we'd probably do it.19:53
asselinrhedlind, not sure...whatever master is here:
boris-42_anteaya: not only Rally19:54
flaper87not because we don't trust swift's team but just to keep it organized.19:54
boris-42_flaper87: I know how to do that19:54
anteayaboris-42_: okay thank you19:54
zarojeblair: are those quotes from code or docs?19:54
boris-42_flaper87: you can write prolog rules19:54
flaper87we'd have to do the same with cinder, probably. This part needs to be discussed19:54
flaper87boris-42_: teach me19:54
boris-42_flaper87: but it was not supported19:54
flaper87boris-42_: ah19:54
boris-42_flaper87: I mean idea by infra team19:54
rajinirasselin: jenkins is on the master with zuul on the same node19:54
boris-42_flaper87: so I just did promise way* (which I don't like)19:54
asselinrajinir, do you have nodepool working?19:55
fungiflaper87: i think the neutron model of expecting people to avoid approving code they aren't comfortable/familiar with is working for them (last i heard), though i understand the concerns especially from a dunning-kruger effect perspective19:55
flaper87yup, I'm sure the trust model will work for us as well.19:56
rajinirasselin: nodepool had ssh issues which I couldn't sort out in time( will figure out later). So we added slave machines manually to the jenkins to run jobs19:56
*** lucasagomes is now known as lucas-dinner19:57
fungiflaper87: also it avoids the chilling effect on trust you otherwise get by micromanaging permissions19:57
jeblairfungi: but good news, 2.8.4 at least has the version endpoint19:57
jeblair(i ran a local gerrit to test that)19:58
asselinrajinir, then you probably didn't add them correctly....they need to have the right label, and need to be online. but I highly recommend you get nodepool working......19:58
zarojeblair: never noticed it before, but i see it now.  that's crazy19:58
*** apuimedo_ has joined #openstack-infra19:58
*** triggerwarning has joined #openstack-infra19:58
fungijeblair: cool, so gertty's behavior guessing can at least be somewhat sane19:58
mtreinishfungi, clarkb, jeblair, nibalizer, pleia2: if you get a sec: just needs a +A, jhesketh held off on the approval until someone was around to watch it (which I am now)19:59
zaroreally need to ask pursehouse about that craziness.19:59
jeblairfungi: yep.  it'll be 3 http requests though: (get change-edit, put change-edit, publish change-edit)19:59
rajinirasselin: The jobs showed up in jenkins. I can manually trigger them. But when the events come from gerrit, they are not triggered. Don't know how to debug that?19:59
rajinirasselin: Is there a log file that can give clues?19:59
*** FallenPegasus has joined #openstack-infra20:00
*** rfolco has quit IRC20:00
*** salv-orl_ has quit IRC20:01
*** aysyd has quit IRC20:01
*** sridhar_ram has quit IRC20:01
ianwmordred: have you tried to get the "vhd-util convert" patch merged, and was it rejected for reasons other than being written in some of the most out-there style i've ever seen (and i say that having taught CS101 courses in C)20:02
ianwmordred: the style i could fix ... if people hate it for other reasons that might be harder20:02
rajinirasselin: The only error is ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.            Cannot get account information.20:02
rajinirasselin: No other error messages, just INFO messages of various events20:02
*** gsagie_ has quit IRC20:03
fungittx: the bandit 0.17.0 release is up at now (has been for a little while)20:03
asselinrajinir, telnet localhost 4730 and type 'status'. ...need to find where that's documented20:04
*** sridhar_ram has joined #openstack-infra20:04
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Add dedicated acls for glance_store
zarojeblair: have you noticed any other backwards incompatabilities besides Change commit message endpoint?20:05
openstackgerritfumihiko kakuma proposed openstack/requirements: Add pylint to upper-constraints.txt
jeblairzaro: not yet20:05
rajinirasselin: Didn't see that instruction anywhere. Here is the output
asselinrajinir, so you don't have any slaves there20:09
rajinirasselin: Is that because we added the slaves manually bypassing nodepool?20:10
*** AJaeger has quit IRC20:10
asselinrajinir, yes, possibly20:10
asselinhere's what I found in my notes:20:11
asselin#output is the name of the job followed by three numbers: queued, running and workers that can run job20:11
asselin# if you see the name there with the last number (workers that can run job) > 0, it should work.20:11
asselinGearman updates on Node changes (added or removed), so a static system may not see new jobs. Workaround is to delete a 'ready' node: nodepool delete <node-id>20:11
*** apuimedo_ has quit IRC20:12
rajinirasselin: looks like the jobs require nodes from nodepool. Back to nodepool. I'll try to see if I can get that up and running. I'll be back. Thanks for help today20:15
*** apuimedo has quit IRC20:16
*** _nadya_ has joined #openstack-infra20:16
pietHi Folks20:17
pietThe OpenStack UX team is conducting a series of interviews focused on hardware provisioning with the intent of using customer feedback to help drive the development of the OpenStack Ironic project.   Individuals using MaaS, Cobbler and Foreman are also welcome to participate in the study.20:17
*** apuimedo has joined #openstack-infra20:17
anteayahey piet20:17
pietIf interested, please select a date/time from the following link
pietanteaya: hiya20:17
*** apuimedo_ has joined #openstack-infra20:17
anteayapiet: um have you given any thought to your communication framework for inviting folks to your studies?20:18
anteayathe infra channel isn't really meant as a bulletin board for announcements of this sort20:18
pietanteaya: It's a bit tricky20:19
*** flepied1 has quit IRC20:19
anteayabut we would like to support a communication framework which does meet your needs20:19
* anteaya listens20:19
pietanteaya: Limesurvey would help with that20:19
anteayafair enough20:19
fungipresumably it's at least being posted to the operators ml, since that seems like the primary audience for this particular study20:19
pietanteaya: we would like to create a screener to identify everyone's interest.  At that point, their would be a very focused email blast 1-2 times a year20:20
anteayaby screener you are thinking of a tool like lime survey?20:21
*** maishsk has quit IRC20:22
*** maishsk has joined #openstack-infra20:22
erlonall: guys is there somewhere we can give feedback about the new gerrit interface?20:23
*** tonytan4ever has joined #openstack-infra20:24
pietanteaya: Screeners are surveys intended to create databases of folks based on their interests, specialities, etc.  At that point, you can review the database and identify folks that meet specific recruitment criteria for a study and just send emails to them.20:24
*** markvoelker has quit IRC20:24
fungierlon:!forum/repo-discuss and
erlonfungi: thank you20:25
fungierlon: though i expect it's pretty frozen at this point while they work on the design for their newer "polygerrit" interface20:26
*** apuimedo has quit IRC20:26
anteayapiet: okay great, so that is the long term vision, and it sounds like a good one20:26
openstackgerritJames E. Blair proposed openstack/gertty: Fix commit message editing in >= 2.11
jeblairfungi, zaro: ^20:26
zarojeblair: i noticed that change edit endpoints are new in 2.11, wasn't there in 2.8.  So i'm guessing what broke is submit change endpoint?20:26
erlonfungi: polygerrit??20:26
anteayapiet: what can be done in the interm while the long term vision takes shape?20:26
*** zeih has joined #openstack-infra20:26
*** e0ne has joined #openstack-infra20:27
*** ybathia has joined #openstack-infra20:27
jeblairzaro: pursehouse removed the 'edit commit message' endpoint in 2.11.20:27
*** apuimedo_ has quit IRC20:27
*** apuimedo has joined #openstack-infra20:28
pietanteaya: The hard answer is that we'll need to keep doing this until we set-up a database.  We don't send invites all that often given the amount of research that is needed to improve the experience20:28
*** binbincong_ has quit IRC20:28
openstackgerritJames E. Blair proposed openstack/gertty: Fix commit message editing in >= 2.11
erlonfungi: hmmm, so, this new interface is temporary? say yes please :)20:28
jeblair(reviewed my own code)20:29
erlonfungi: lol20:29
fungierlon: in software, everything is temporary20:29
erlonfungi: yes, the question 'how long'20:29
zarojeblair: ahh i see it now.20:29
pietanteaya: want the brutal answer?20:29
erlonfungi: but thanks, Ill try to reach guys there20:30
*** kzaitsev_mb has joined #openstack-infra20:30
anteayapiet: what I am saying is you have made a decision about how you want to use the infra channel without any input from the infra team20:30
anteayapiet: I'm suggesting that getting feedback from the infra team about your choices might be in order here20:30
*** tjones has quit IRC20:30
*** matrohon has joined #openstack-infra20:30
jeblairerlon: my guess is 2 years.  with a big error margin.20:31
*** zeih has quit IRC20:31
fungierlon: essentially, the current interface is the second (third? depends on how you count them) major revision of gerrit's web ui. they're working on yet another one (written in polymer) and they also provide a rest interface in gerrit which is quite suited to building new user interfaces on top (for example is one jeblair wrote for console-based reviewing)20:31
jeblairerlon: at least it's not "weeks" or "months" temporary.20:31
*** apuimedo_ has joined #openstack-infra20:32
*** esker has joined #openstack-infra20:32
*** esker has quit IRC20:32
*** abregman has joined #openstack-infra20:32
erlonjeblair: o_O, well that's a long time20:32
*** tjones has joined #openstack-infra20:33
*** esker has joined #openstack-infra20:33
*** regXboi has joined #openstack-infra20:33
erlonfungi: I remember this interface, was optional, I have used it once20:33
*** kzaitsev_mb has quit IRC20:36
pietanteaya: That's fair.  I guess my feedback is that there is a culture that folks should be contributing to OpenStack.  I would propose that part of the contribution should include participating in the occasional studies.20:36
anteayathanks for understanding20:36
anteayaand you make a reasonable follow up point20:36
anteayaI think perhaps if you were to put an item on the infra meeting agenda we could discuss this20:37
*** regXboi has quit IRC20:37
anteayaand we might be able to come up with ways to support your communication strategy20:37
fungierlon: yes, the "new" interface used to be optional, but as of this version of gerrit it's ceased being a preview or optional in any way and the previous interface is no longer included as an option20:37
pietanteaya: Learned the lesson from David Lyle.  I was constantly whining about Horizon and he basically told me to contribute or shut up (in a nice way...)20:37
anteayaand I am fine with your definition of contribution, I think it is reasonable20:38
anteayadavid-lyle says things in the nicest way :)20:38
anteayaso yes contributing is important, I agree20:38
*** kzaitsev_mb has joined #openstack-infra20:38
fungipiet: as a reminder (hopefully you can already take advantage of this) the openstack weekly newsletter can mention things like ui/ux studies if you give them a heads up20:38
rhedlindhow do I install the master revision of zuul? I have tried pip install -U zuul, but it doesn't give me the latest. I assume there some environment setting I am missing. Any pointers?20:39
pietfungi:'ve mentioned few times and I keep forgetting to reach-out through the newsletter.  My bad...20:39
fungialso i wonder if places like "superuser magazine" might be open to community "advertisements" about things like research studies20:39
fungicould be an interesting way to integrate more temporary calls for help/feedback20:40
*** vgridnev has quit IRC20:40
*** dkranz has quit IRC20:41
pietanteaya: But would appreciate recommendations on recruiting for studies because it's one of my biggest challenges. Helping with LimeSurvey is a big step in the right direction.20:41
pietanteaya: brb20:41
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Add containers job to CI status
anteayapiet: awesome, can you add an item to next week's infra meeting and perhaps we can brainstorm then?20:41
anteayapiet: I think our agenda will be light anyway so we could use a topic to discuss :)20:42
anteayapiet: and fungi suggested a couple of good ideas in backscroll20:42
*** avarner has joined #openstack-infra20:42
anteayaweekly newsletter, superuser magazine20:43
fungino need to wait a week to solicit suggestions on strategies for reaching various corners of the openstack community when recruiting for things like openstack-focused research studies and surveys. a thread on one of our more active mailing lists might be a good way to get more ideas on how to reach people20:43
*** sridhar_ram has quit IRC20:43
*** mudassirlatif has quit IRC20:44
*** placko has quit IRC20:44
*** triggerwarning has quit IRC20:44
*** triggerwarning has joined #openstack-infra20:44
*** triggerwarning has quit IRC20:44
*** triggerwarning has joined #openstack-infra20:44
*** sridhar_ram has joined #openstack-infra20:44
*** mudassirlatif has joined #openstack-infra20:45
*** Swami_ has quit IRC20:45
*** mudassirlatif has quit IRC20:45
fungiespecially if the thread starts out as a "how can we reach the right audience for this sort of thing" rather than starting out as a call for participation and ending up instead as a discussion on where would be better to advertise it20:46
anteayarhedlind: we use this module for deploying zuul:
*** _nadya_ has quit IRC20:48
*** gordc has joined #openstack-infra20:48
*** apuimedo has quit IRC20:49
anteayarhedlind: now this part installs zuul:
*** hashar has joined #openstack-infra20:49
anteayarhedlind: but it uses vcsrepo to git clone the latest version first:
*** apuimedo_ has quit IRC20:50
*** apuimedo has joined #openstack-infra20:51
*** otter768 has joined #openstack-infra20:51
openstackgerritMerged openstack/gertty: Fix syncing changes with comments on a missing file
openstackgerritMerged openstack/gertty: Separate search and refine search commands
openstackgerritMerged openstack/gertty: Add a simple kill ring
openstackgerritMerged openstack/gertty: Add interactive search to diff view
openstackgerritMerged openstack/gertty: Add navigation to interactive search
openstackgerritMerged openstack/gertty: Add help entries for kill, yank, isearch
openstackgerritMerged openstack/gertty: Support multiple key input
*** aeng has joined #openstack-infra20:53
anteayajeblair: you've been busy20:53
*** piet has quit IRC20:53
jeblairoh they've been sitting there for weeks20:53
fungijeblair: awesome! i've been using those for a while without issue. i'll switch to master tip now and pull in the commit message editor20:54
jeblairfungi: that one is in the queue still20:54
fungiyep, i meant pull in that fix separately20:54
*** apuimedo_ has joined #openstack-infra20:55
*** otter768 has quit IRC20:56
*** daemontool has quit IRC20:57
rhedlindanteaya: thanks!20:57
*** piet has joined #openstack-infra20:58
*** daemontool has joined #openstack-infra20:58
openstackgerritMerged openstack/gertty: Fix commit message editing in >= 2.11
anteayarhedlind: welcome20:58
anteayarhedlind: thanks for coming back :)20:59
fungioh, didn't realize 259173 had also gotten approved. switching to plain master tip now20:59
*** bharathm has joined #openstack-infra21:00
pietanteaya; Yep, caught Fungi comment about newsletter. Are you referring to Tuesday's meeting?21:00
*** mudassirlatif has joined #openstack-infra21:00
anteayaI am but fungi makes a very good point about considering starting a thread on the mailing list21:01
anteayaand makes an additional good point about the topic suggestion21:01
pietanteaya: Distribution list?21:01
anteayawhat list were you thinking fungi?21:01
*** mudassirlatif has quit IRC21:01
*** apuimedo_ has quit IRC21:02
*** maishsk_ has joined #openstack-infra21:02
rhedlindanteaya: . This is the zuul log from my CI, where it keeps posting comments on the same change over and over. It fails to post the job to jenkins, posts a comment, and retries that sequence over and over.21:02
fungia thread on the -dev ml is probably likely to get more people involved in the discussion (just because it's got more people subscribed) though a thread on the -community ml may actually have a more diverse base for ideas around reaching different parts of the community21:03
*** maishsk has quit IRC21:03
*** maishsk_ is now known as maishsk21:03
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Better duplicate detection for project requirements check
rhedlindthe jenkins process had crash because it could not allocate memory. zuul had eaten up all available memory.21:03
pietanteaya: fungi: Stepping away for some Cajun food21:03
*** lykinsbd has quit IRC21:04
pietanteaya: It would also be helpful to have a conversation around why the studies are bothering folks.  Is it the email blasts?21:04
anteayapiet: no it is just you could use your time better to reach your audience21:05
*** lykinsbd has joined #openstack-infra21:05
anteayapiet: we would like to help you reach your goal21:05
rhedlindThis is the zuul.layout I'm using. . I would appreciate if someone can look it over and call out what I am doing wrong in there.21:05
anteayapiet: right now it looks like you could use your time better21:06
*** salv-orl_ has joined #openstack-infra21:06
*** esker has quit IRC21:06
pietanteaya: agreed...and I don't really like blasting folks because it's annoying for folks receiving the emails and takes alot of my time, but no other options yet...21:06
*** binbincong has joined #openstack-infra21:06
anteayarhedlind: okay glad you found the log that has the loop, hopefully some folks in addition to me can look and we can see what is happening21:06
*** apuimedo_ has joined #openstack-infra21:06
anteayapiet: right well we would like to help expand the options21:07
*** jswarren has left #openstack-infra21:08
*** jswarren has joined #openstack-infra21:08
anteayarhedlind: in the log is this the first instance of this line? 2015-12-17 06:28:15,059 INFO zuul.Gerrit: Updating information for 252250,1321:08
openstackgerritMerged openstack-infra/storyboard: Workaround 500 error when moving items in worklists
rhedlindanteaya: no, probably not. I grabbed a snippet. I will find the first instance and paste it.21:09
anteayaso this is the gearman error: GearmanError: Unable to submit job to any connected servers21:09
anteayarhedlind: okay and if you can about 50 to 100 lines preceeding21:10
*** kzaitsev_mb has quit IRC21:10
openstackgerritMerged openstack-infra/storyboard-webclient: Fix various style bugs with cards
*** kzaitsev_mb has joined #openstack-infra21:12
*** Swami_ has joined #openstack-infra21:12
*** mudassirlatif has joined #openstack-infra21:15
*** rbrndt has quit IRC21:16
*** matrohon has quit IRC21:16
*** mudassirlatif has quit IRC21:16
rhedlindanteaya: a much longer log:
* anteaya looks21:18
anteayaso line 205: 2015-12-16 22:07:33,232 ERROR zuul.Scheduler: Exception in run handler:21:20
anteayathat is our first occurance, yes?21:20
rhedlindanteaya: yes21:21
anteayabut before that we had a whole bunch of these: 2015-12-16 22:07:33,014 ERROR zuul.Gearman: Unable to find build 77f3a573f4ea46b0bd426e59282c650521:23
*** kgiusti has quit IRC21:23
anteayanow I am far from a zuul expert21:23
anteayaare the unable to find build errors normal?21:23
*** rossella_s has quit IRC21:23
anteayarhedlind: have you seen them in prior logs?21:23
*** rossella_s has joined #openstack-infra21:23
*** krtaylor has quit IRC21:24
rhedlindno, I have never seen those before. I think that is because the jenkins process crashed.21:24
*** markvoelker has joined #openstack-infra21:25
*** apuimedo_ has quit IRC21:25
anteayaso two ways to go about this and I think you should consider both21:26
anteayaone, protect yourself21:26
anteayaif a jenkins crashes you want your system to recognize that somehow and stop posting to infra's gerrit21:27
anteayaan alert to you is also an idea but up to you21:27
openstackgerritMerged openstack-infra/reviewstats: Gerrit 2.11 compatibility
anteayatwo, help figure out in upstream zuul how to recognize when a loop is occuring and provide a mechanism that escapes the loop21:28
anteayaprovide a patch and see you reviewers agree that it would be helpful and also if they agree with your approach21:28
*** markvoelker has quit IRC21:30
*** apuimedo_ has joined #openstack-infra21:30
rhedlindanteaya: I agree with both. I will take a look at the zuul source to see if I can identify the loop (if it still exists in the latest version). I was running 2.1.1.dev15 when this happened and now running the latest (2.1.1.dev109).21:30
*** mudassirlatif has joined #openstack-infra21:30
anteayaokay great21:30
anteayadid you get 2.1.1.dev109 installed okay?21:31
*** mudassirlatif has quit IRC21:31
rhedlindanteaya: yes, it installed without any complains.21:31
rhedlindwhat comments did the CI post? was it FAILED or something else?21:31
anteayaI never tried to open the file21:32
anteayathe size would have brought down my browser21:32
anteayaso I don't know21:32
pleia2mtreinish: still about to watch the health dashboard if I approve?21:32
*** piet has quit IRC21:32
anteayaif and infra-root still has the comment content in logs hopefully they will post to you, rhedlind21:32
anteayaif an*21:32
mtreinishpleia2: yep21:33
openstackgerritBrant Knudson proposed openstack-infra/project-config: Keystone eventlet gate only in keystone
openstackgerritBrant Knudson proposed openstack-infra/project-config: Change postgres job to use standard deployment
pleia2mtreinish: ok, lgtm, I'm around for another 2.5 hours if you need anything else (then STAR WARS!)21:34
*** bhunter71 has quit IRC21:34
anteayarhedlind: also if when you reply to the email on third party announce do include a complete debrief with as many details as possible21:34
rhedlindanteaya: that could be useful to see what the comment content was.21:34
rhedlindanteaya: ok21:34
mtreinishpleia2: ooh lucky. I'm not seeing it until tomorrow21:34
anteayarhedlind: hopefully this will provide information to others so they check their systems and don't have the same issue21:35
mtreinishpleia2: ok, once that merges I'll approve the o-h side changes21:35
anteayarhedlind: I do think we may have had to clear gerrit's db in order to get zuul moving again21:35
*** smarcet has quit IRC21:35
anteayarhedlind: it is possible we don't have access to the comment content anymore, it was a decision we had to make quickly for the benefit of our system21:35
pleia2mtreinish: great21:35
anteayapleia2: STAR WARS!21:36
pleia2anteaya :D21:36
openstackgerritDavide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job
*** apuimedo_ has quit IRC21:37
*** krtaylor has joined #openstack-infra21:37
*** avarner_ has joined #openstack-infra21:37
*** flepied has joined #openstack-infra21:37
rhedlindanteaya: ok, understood21:38
anteayarhedlind: thank you, get some rest tonight, and come back tomorrow21:38
anteayawe all make mistakes21:38
pabelangerfungi: jeblair: Do either of you mind cutting a new release of JJB? I don't think we've released the latest version to include the new template macro21:39
anteayaI can tell you the story sometime of how as a crew member who knew noone had to keep 60 crew members waiting for about 3 minutes while I loaded a film magazine (when I was a camera assistant)21:39
anteayathe first camera assistant was the president of the camera union21:40
rhedlindanteaya: thanks for your help. I will respond on the mailing list.21:40
anteayaI was more than a bit queasy walking that mag into a silent studio I can tell you21:40
anteayarhedlind: sounds good, thanks21:40
*** avarner has quit IRC21:41
anteayait wasn't my fault, but it was my responsibility21:41
*** apuimedo has quit IRC21:42
*** apuimedo has joined #openstack-infra21:42
*** apuimedo_ has joined #openstack-infra21:43
openstackgerritDavide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job
fungirhedlind: anteaya: the comments it was posting were "Build failed.  For information on how to proceed, see"21:44
anteayafungi: thank you21:45
fungithis is according to our (copious) gerrit ssh logs21:45
*** jpr_ has quit IRC21:45
*** mudassirlatif has joined #openstack-infra21:45
*** mudassirlatif has quit IRC21:46
fungirhedlind: anteaya: the account seems to have left that message >8000 times starting a few minutes after 05:00 utc21:46
fungiand ending when we deactivated the account and disconnected its ssh connection21:46
*** apuimedo has quit IRC21:47
anteayarhedlind: so the server that created these logs is not on utc time?
anteayamorning jhesketh21:48
*** apuimedo_ has quit IRC21:48
phschwartzJhesketh: morning21:48
*** apuimedo has joined #openstack-infra21:49
fungipabelanger: new jjb release should be discussed with zaro, pelix and zxiiro, but i'm in favor of early and often21:49
fungiwe've also had a recent request for a glean release21:49
pabelangerfungi: Ya, there is a request on ML for one but think it was missed21:50
fungimordred: greghaynes: SpamapS: ideas on whether we should release glean soon?21:50
mgagnejeblair you around? could you try spawning a node at internap? discount and aggregate are in place. I wish to test its proper working21:50
* zxiiro would like to see a new release soon. It would help our users who are having issues with the current 1.3.0 version21:50
fungimordred: greghaynes: SpamapS: ian mentioned some useful fixes since the last release21:50
fungiianw that was21:51
greghaynesI like releases, we should do them21:51
greghaynesI lack perms to release glean I think21:51
greghaynesbut no objection from me21:51
fungigreghaynes is always easy to please21:51
*** apuimedo_ has joined #openstack-infra21:51
greghaynesand ya, I there are some fixes. There is one I just +A'd too let me see if it merged...21:52
openstackgerritDavide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job
clarkbgreghaynes: looks much better now21:52
rhedlindanteaya: no, it is on MST21:52
fungipabelanger: did you get a chance to try out the openstack_project:openstackid_dev class on ubuntu trusty and figure out what needs tweaking/adding (if anything)?21:53
greghayneslooks like the fix I wanted just merged, so ya release whenever21:53
anteayarhedlind: if you do decide to move it to utc, it may help match up timestamps with infra events in future21:53
greghaynesclarkb: w00t21:53
anteayarhedlind: all our servers are on utc21:54
rhedlindanteaya: ok, I will look into that. It makes sense.21:54
fungijeblair: jhesketh: was there a fix proposed for that zuul logging performance regression you were discussing? need reviews?21:54
clarkbgreghaynes: there are even logs that don't say much21:54
anteayathanks, thought I would mention it as something to consider, it isn't required21:54
openstackgerritDavide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job
greghaynesclarkb: derp, that should be logging at debug21:55
openstackgerritJan Hruban proposed openstack-infra/zuul: Clarify zuul-cloner message about missing branch
greghaynesclarkb: so my logging config is messed up somehow21:55
*** dkranz has joined #openstack-infra21:55
pabelangerfungi: I had to push it to tomorrow21:55
clarkbgreghaynes: ok, based on the nodepoold logs I am pretty sure its doing stuff21:55
clarkbso ya wouldn't surprise me if the logging is just not right21:55
fungipabelanger: no worries, just curious, and thanks!21:56
pabelangerfungi: I should know by lunch EST21:56
jheskethfungi, jeblair: not sure if it's a logging performance, but I was looking at a performance regression due to the cache management. Unfortunately it's tightly coupled and I haven't finished yet. I'll poke more at that today and this weekend.21:56
*** flepied has quit IRC21:56
fungipabelanger: cool--i'm excited to hear what you find out, and i'll plan the server replacement once you get time to check it out21:56
jheskethfungi, jeblair: I had it down to just one test failing21:57
*** alivigni has quit IRC21:57
*** gordc has quit IRC21:57
fungijhesketh: ahh, okay. i think jeblair had said something about a bug with changes to logging behavior ended up moving it into the main thread (leading to a possible explanation for why the event flood earlier today was more devastating than we've seen in the past)21:58
clarkbgreghaynes: the double logging in nodepoold's log is really annoying/weird too I should track that down21:58
fungijhesketh: though it's possible i misread what he was suggesting there21:58
*** piet has joined #openstack-infra21:59
onovysorry, i'm idiot (i know). how can i remove branch? [remote rejected] debian/unstable (cannot delete references)21:59
*** lykinsbd has quit IRC21:59
fungionovy: needs gerrit admin assistance (or did in the past, though maybe it's easier to fine-tune the acls now that we're on 2.11 so we should revisit that policy at some point)21:59
jheskethfungi: hmm okay. It's possible the cache regression causes more logs as it has more to figure out. Otherwise it could be something else22:00
*** mudassirlatif has joined #openstack-infra22:00
onovyfungi, so should i write email to mailist?22:00
*** mudassirlatif has quit IRC22:01
fungijhesketh: at 16:40 utc in scrollback22:02
*** triggerwarning is now known as dcentral22:02
* jhesketh looks 22:02
fungionovy: hopefully someone in infra-root with a spare moment can delete the "debian/unstable" branch on... what repo? you're working on openstack/swauth?22:03
onovyfungi, yes, openstack/swauth22:03
mgagnejeblair ok, tested aggregate, works fine22:04
fungijhesketh: i should have clarified, _connection_ logging22:05
openstackgerritMerged openstack-infra/puppet-openstack_health: Update default api configurations
*** spzala has quit IRC22:06
pleia2fungi: is there work to do on the git servers for that, or is it just deleting in gerrit?22:07
*** thorst_ has quit IRC22:07
*** alivigni has joined #openstack-infra22:07
mtreinishok, that landed. Let me push through the other changes22:08
openstackgerritJoshua Harlow proposed openstack/requirements: Ensure python 3.3 is also listed for enum34
pleia2mtreinish: huzzah22:08
fungijhesketh: so maybe the broken connection logging was not directly related to the connections bug he discussed with you (which is what seems to be unnecessarily burdening the main thread)22:08
fungipleia2: just deleting it in gerrit. helps to echo the final git sha of the branch into the channel here for posterity22:08
pleia2fungi: right, on it22:09
*** pigmej has left #openstack-infra22:09
pleia2onovy: deleted debian/unstable with final git sha of 008237924b5e008421193596e1db86e62ceda50a22:09
onovypleia2, perfect, thanks a lot and sorry for it22:09
jeblairjhesketh, fungi: logging was separate: logger names changed -- the potential regression is very related to what jhesketh is already working on, but with the additional observation that we have inadvertently moved some gerrit query work into the main thread;  jhesketh's fix should still fix that22:09
pleia2onovy: you're welcome22:09
jeblairmgagne: i'm back; need me to do anything?22:10
jeblairpabelanger: did anyone make that jjb release?22:10
*** abregman is now known as abregman|ZzZzZz22:10
*** trown is now known as trown|outttypeww22:10
mgagnejeblair I guess you could try create a nodepool node just to make sure you still can and it works ok. I'm working on the quota22:10
fungijeblair: oh, great thanks! i'll keep an eye on that one22:11
*** dcentral has quit IRC22:11
*** dcentral has joined #openstack-infra22:11
* jhesketh finishes scrollback. 22:11
pabelangerjeblair: I don't think so22:11
jheskethYep, what jeblair said22:12
*** flepied has joined #openstack-infra22:12
*** xyang1 has quit IRC22:12
openstackgerritJames E. Blair proposed openstack-infra/project-config: Increase internap to 1 instance
fungijeblair: nobody's tagged it yet. i was going to see what semver-y number we should be increasing to, so was hoping zaro/pelix/zxiiro could summarize the extent of changes since the last release22:12
pabelangerjeblair: 1.3.0 the last one, from aug 2622:12
jeblairmgagne: at this point, that's the easiest way to do that ^
openstackgerritDrew Varner proposed openstack-infra/project-config: Experimental bandit job for glance
jeblairfungi: ok, that's about what i was going to do too :)22:12
jeblairfungi: could i convince you to take a look at ?22:13
*** gyee has joined #openstack-infra22:13
mgagnejeblair haha. quota is now 30. I think it would be safe to set max-servers to 25.22:13
jeblairi'm about to release gertty and wondered if that should go in.22:13
fungijeblair: sure22:14
zxiirofungi: There's been quite a lot of changes since 1.3.0 so I think it should be a 1.4.0 release.22:15
jeblairmgagne: let's set it to 1 for now to make sure it's working first; if there's a problem, nodepool might end up trying to put too many instances there22:15
mgagnejeblair right, good call22:15
jeblairmgagne: we can bump it to 25 quickly after that22:15
*** zeih has joined #openstack-infra22:15
*** mudassirlatif has joined #openstack-infra22:15
mgagnejeblair and I'll be off for ~2 weeks starting tonight :D22:15
*** mudassirlatif has quit IRC22:16
*** flepied has quit IRC22:17
ianwgreghaynes: here's another quick one before a glean release ->
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command
ianwgreghaynes: stops a bunch of warnings in the logs22:17
openstackgerritClark Boylan proposed openstack-infra/nodepool: Remove duplicate log entries
clarkbgreghaynes: ^ I think that may remove duplicate logs22:18
kzaitsev_mbhey, can I ask someone to help me better understand how i18n in OS works?22:18
kzaitsev_mbI mean the automation — who does what? =)22:18
openstackgerritK Jonathan Harker proposed openstack-infra/log_processor: Let the workers shut down gracefully
kzaitsev_mbGuessed this would be a good place to ask. =)22:18
*** eharney has quit IRC22:18
fungikzaitsev_mb: pleia2 is probably the most awake person now who's had a recent hand in that22:19
fungimaybe also StevenK22:19
pleia2we have a page about that, sec22:19
kzaitsev_mboh, that'd be great. =)22:19
pleia2kzaitsev_mb: this is a high level view:
pleia2kzaitsev_mb: the i18n team maintains another, I'll dig that up now22:20
*** zeih has quit IRC22:20
*** bdemers has quit IRC22:20
pleia2kzaitsev_mb: here you go:
*** avarner__ has joined #openstack-infra22:21
*** rossella_s has quit IRC22:21
anteayazaro: beside the reply button in a patch is a button marked workflow+1, it is a blue button, why is it there?22:21
anteayazaro: I set workflow +1 using the radio button in a reply22:21
clarkbnibalizer: crinkle for it has several lists of packages to isntall, is there some way to automagically install all the packages we have listed in jenkins_params.pp ?22:21
*** rossella_s has joined #openstack-infra22:21
kzaitsev_mbI've finally convinced most of the guys on our team & PTL, that we should start i18n murano and we have some volounteers ready to do the actual translation, so I'm in the process of figuring out the machinery we miss for that =)22:21
kzaitsev_mbthanks =)22:22
*** avarner__ is now known as avarner22:22
anteayazaro: is leaving a reply and setting a radio button considered too much effort?22:22
clarkbanteaya: ya its just a shortcut, gerrit knows you have perms to add that flag and it knows the change needs it to merge22:22
clarkbso its like "hey here is easy mode button"22:22
pleia2kzaitsev_mb: that's great, this defines specifically how we do translatiosn on the back end, if you want translator-level docs for the actual translating work you probably want to start with
*** [1]Thelo has joined #openstack-infra22:23
ianwclarkb: i think i wrote that ages ago :
anteayaclarkb: okey dokey, thanks22:23
pleia2kzaitsev_mb: they'll also want to join the i18n team:
clarkbianw: woo22:23
anteayaclarkb: the +A encouragement22:23
clarkbianw: actually we do want it to propogate don't we?22:24
clarkbbut maybe I am wrong for some reason I was thinking that it affected below nodepool as well22:24
clarkblet me check the lgos and can abandon mine if yours works22:24
*** avarner_ has quit IRC22:24
*** bpokorny_ has joined #openstack-infra22:25
kzaitsev_mbpleia2: no, I actually want to figure out the backend stuff at first. Like who updates pot files (if they're updated automatically at all) and how to register in zatana and other stuff in the links you posted, thanks =)22:25
clarkbya I must've read logging docs wrong as that seems to work22:25
* ianw just contexting switch back to what the original problem was .... 22:25
*** Thelo has quit IRC22:25
*** [1]Thelo is now known as Thelo22:25
clarkbianw: I actually think both methods work but yours is more flexible ifwe switch handlers between them22:26
*** spzala has joined #openstack-infra22:26
clarkbso will approve yours and abandon mine22:26
pleia2kzaitsev_mb: great, feel free to reach out if you have questions, if I'm not around StevenK and AJaeger are also well-versed in all of this, and there's always the i18n mailing list too22:27
mtreinishpleia2: well puppet picked up the conf file change and the api server still works. Still waiting on the code to use those options to work it's way through the gate22:27
ianwclarkb: now i remember being surprised by the root logger not filtering out the lower level messages.  you'd think it would filter it, but really it juts goes straight to the handlers22:28
pleia2mtreinish: not breaking it ^522:28
asselinany cores want to review zaro's gearman-plugin patch?
*** bpokorny has quit IRC22:28
*** rhallisey has quit IRC22:28
openstackgerritMerged openstack/gertty: Match links by url instead of domain.
mtreinishpleia2: heh, well the real test is when something that uses the options gets installed22:29
pleia2mtreinish: nods22:29
openstackgerritMerged openstack-infra/project-config: Increase internap to 1 instance
ianwgreghaynes: so ... i have a f23 image.vhd ... i'm going to try uploading it into swift on my rax account.  is it likely i am going to be able to boot it?22:29
clarkbianw: if you made it with that vhd-utils patch it should boot22:30
clarkbthat was the only thing we needed to get boot working on rax. Setup of working networking was different story but I think you are familiar with that22:30
ianwclarkb: well, i did, but i also munged that patch to work on blktap on fedora ... so yay for adding new variables :)22:30
*** dcentral has quit IRC22:31
*** mudassirlatif has joined #openstack-infra22:31
ianwclarkb: do you also know anything about that patch?  i really think we should beat it into shape and get it upstream, cause it's a mess22:31
*** mudassirlatif has quit IRC22:31
*** dcentral has joined #openstack-infra22:31
*** spzala has quit IRC22:31
ianwnot the least of which is spelling GLOBEL everywhere22:31
*** rbrndt has joined #openstack-infra22:31
clarkbianw: aiui upstream said go away22:32
vilobhmm11clarb, AJaeger, fungii : can you please review this when free22:32
vilobhmm11clarkb : ^^22:32
openstackgerritMerged openstack-infra/nodepool: Don't double-log in nodepool/dsvm log output
clarkbianw: but that may have jus tbeen because the patch is actually terrible in its current state22:32
fungiianw: i can't recall specifics, but seems like mordred had some trouble interfacing with xen devs22:32
*** peristeri has quit IRC22:32
clarkbianw: I personally dislike the use of digits for input and output formats22:32
anteayamtreinish: you are welcome to join our holiday party if you wish22:32
*** markvoelker has joined #openstack-infra22:33
fungiianw: seems like at one point he was looking at getting that patch into more acceptable shape i just don't recall where that ended up22:33
ianwyeah, that stuff is easy changed.  it seems like Bob Ball has some of the last commits in the area, and i'd like to think we're on good terms, he does a lot of devstack xen stuff22:33
fungiso instead we have an ubuntu ppa with more or less a xen fork to be able to have that package22:33
clarkbthere is also rumor that qemu-img might actually work now22:33
clarkbwith a new enough version22:33
clarkbbut mordred was unable to get that working22:34
ianwalright, i'll take it as an action item to follow up on this22:34
mtreinishanteaya: holiday party?22:34
mtreinishoh I see the ML post22:34
anteayamtreinish: the etherpad you were on22:34
anteayamtreinish: yeah22:34
ianwafter i figure out how to upload my possibly working image :)22:34
fungialso, it seems generally terrible to have to build xen to be able to get a simple command-line utility to do image format conversions, but i'm probably underestimating the complexities involved22:35
pleia2mtreinish: I promise not to knowledge transfer anything about the new star wars movie (unless you want me to)22:35
*** Swami_ has quit IRC22:35
anteayaha ha ha22:35
anteayayou can make subtle light saber movements as you type22:35
clarkbianw: yes that is indeed the hard thing, if you want to write a little python shade should have it all sorted out for you22:35
pleia2anteaya: hehe22:36
clarkbianw: but shade has no cli so would be a few lines of python to instantiate and call the right methods22:36
mtreinishwell if it's tomorrow I'm only working in the morning, but I'll try to remember to idle on -sprint22:36
anteayamtreinish: cool22:36
mtreinishstar wars is the afternoon for me :)22:36
anteayaof course22:36
clarkbno spoilers22:36
kzaitsev_mbpleia2: so from what I'm reading — we basically don't need to do much, just mark the project with 'options: - translate' and we'll have us registered in zanata and the pot files would get generated by scripts, right?22:36
clarkbthis shall remain a spoiler free zone22:37
anteayait will be March before I see it, I'm sure22:37
anteayait might even be out on dvd before I see it22:37
kzaitsev_mb(since we already mark all strings for translation and have babel in requirements)22:37
anteayaclarkb: I honour your spoiler free zone22:37
mtreinishpleia2: it seems to work:
mtreinishthat list is the filtered list22:37
mtreinishgot to wait for puppet to run on status.o.o to pull in the new js that knows how to use the list22:38
pleia2mtreinish: cool22:38
* pleia2 nods22:38
clarkbkzaitsev_mb: you need to add the jobs to your project22:38
mtreinishpleia2: the group_by drop down is there22:38
kzaitsev_mboh, yeah, right22:39
mtreinishanteaya: ^^^ you were looking for that right?22:39
anteayagroup_by! wooooo22:39
anteayathank you22:39
pleia2kzaitsev_mb: I'll dig up an example of another project that just added jobs, sec22:39
anteayatimothyb89: thank you for your help with group_by22:39
pleia2kzaitsev_mb: you want a patch like this for your project:
anteayamtreinish: so build_branch is interesting22:40
pleia2kzaitsev_mb: also, it would help us out for the PTL to +1 the change, so know there's centralized support for the proposal22:40
*** dcentral has quit IRC22:40
anteayamtreinish: what would it take for the stable branches to filter to the top22:40
timothyb89anteaya: glad I could help!22:40
pleia2kzaitsev_mb: or at least a link to the discussion/agreement from the team (mailing list, irc logs)22:40
mtreinishanteaya: it's by failure percentage just like the other views22:41
*** dcentral has joined #openstack-infra22:41
anteayamtreinish: the UNKNOWN build branch as the first entry doesn't hold much fasination for me22:41
mtreinishthat's just periodic qa22:41
*** alivigni has quit IRC22:41
pleia2mtreinish: oh, that's nice :)22:41
anteayaall lumped together?22:41
mtreinishit's not triggered by a change so there isn't a branch set in the metadata22:41
mtreinishit might also have periodic stable in there, I'd have to check22:42
fungiokay, unlike last night i think i'm going to kick off early and have a non-openstack evening. night all!22:42
*** zeih has joined #openstack-infra22:42
anteayaokay thanks22:42
anteayafungi: enjoy your evening22:42
pleia2fungi: good idea, take care22:43
anteayamtreinish: periodic-stable is in the build_queue view22:43
jeblairanyone want to double check my gertty release announcement?
*** abregman|ZzZzZz has quit IRC22:44
kzaitsev_mbpleia2: sure, I'll ask him to +1 the change. =) One more question — do I need to do anything to have log-translations split (info/warn/critical, etc.) Or is it done automatically just based on how we mark the strings? (didn't find an explanation in the docs)22:44
*** dkranz has quit IRC22:44
pleia2jeblair: lgtm22:44
mtreinishanteaya: yeah the group_by is a metadata key and it shows the results grouped by the value. UNKNOWN is what the gearman client sets the build_branch value to if it doesn't have one set22:45
mtreinishwhich I know it definitely does for periodic-qa queue jobs22:45
*** alivigni has joined #openstack-infra22:45
*** mudassirlatif has joined #openstack-infra22:45
pleia2kzaitsev_mb: I don't know about that one, AJaeger would know better (might be a good question for the mailing list)22:45
mtreinishanteaya: it also does it for periodic-stable jobs:
anteayajeblair: looks good to me22:45
*** bgmccollum has left #openstack-infra22:46
*** gildub has joined #openstack-infra22:46
*** mudassirlatif has quit IRC22:46
kzaitsev_mbpleia2: ok, will take a note to ask that on ML22:46
anteayamtreinish: ah ha22:46
jeblairi'll tag jjb 1.4.022:46
*** e0ne has quit IRC22:47
mtreinishthe group by will be real nice when someone restarts the logstash gearman client so we start setting the node_region in the metadata22:47
*** zeih has quit IRC22:47
mtreinishthen we can group based on where things run22:47
*** dtardivel has quit IRC22:47
clarkbgah would've been a good time to do that hwne gerrit was off22:48
mtreinishheh, yeah I realized that after it was too late22:49
kzaitsev_mbpleia2: just one more question — we have murano-dashboard, that is a dashboard for horizon and is tho most visible thing, which we'd also like to translate of course. But since it's horizon-based it doesn't use oslo.i18n. Would there be anything special we'd have to do to have it hooked up?22:49
openstackgerritJames E. Blair proposed openstack-infra/log_processor: Change node_region to node_provider
anteayamtreinish: okey dokey22:49
jeblairclarkb, mtreinish: since apparently that isn't the actual code.. where is the actual code now?22:49
jeblairso i can make that change.  again.22:49
kzaitsev_mbpleia2: I've noticed there is a designate-dashboard geting translations, probably gotta look at how it's configured22:50
clarkbjeblair: the actual code is in puppet-log_processor because the change to switch over needs reviews22:50
pleia2kzaitsev_mb: good idea :)22:50
jeblairclarkb: okay, i will copy it there22:50
*** triggerwarning has joined #openstack-infra22:50
clarkbjeblair: well more the dep of the change over needs reviews22:50
pleia2kzaitsev_mb: I don't know re: your first question22:50
*** piet has quit IRC22:51
clarkb(my +1 there was due to a group messup should be fixed now so  Iwill bump to a +2)22:51
kzaitsev_mbk. as 1st step I'm gonna add translations to the main repository and ping our PTL to have a word there, next step — I'll add other projects too (we have client and agent and dashboard =))22:51
openstackgerritJames E. Blair proposed openstack-infra/puppet-log_processor: Change node_region to node_provider
jeblairclarkb, mtreinish: ^ let's merge that before we restart :)22:52
*** otter768 has joined #openstack-infra22:52
jeblairpabelanger: jjb 1.4.0 is released22:53
jeblairgerrtty 1.3.0 is released22:53
jeblairarrrrr   gerrrrrrty22:53
*** pradk has quit IRC22:53
*** triggerwarning has quit IRC22:53
*** triggerwarning has joined #openstack-infra22:53
jesusaurusjeblair: can you also push up a change on the opnstack-infra/log_processor project? you will need to base it off of 248868 because I'm not getting many reviews on the change that makes the tests pass22:53
jeblairjesusaurus: i did that first :)22:54
jeblairjesusaurus: but yeah, i just rebased it22:54
*** spzala has joined #openstack-infra22:54
*** hashar has quit IRC22:55
jesusaurusoh, /me notices that was just discussed in backscroll :)22:55
*** mudassirlatif has joined #openstack-infra22:55
jeblairjesusaurus: you copied the 'select' line in tox.ini that makes hacking _only_ look at H231 and F violations and ignore all pep8.22:56
jeblairjesusaurus: i'm fine with that so i approved it.  :)22:56
jeblairbut if someone who likes pep8 wants to fix that, fair warning. ;)22:56
ianwVERBOSE 0.43 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com22:56
ianwVERBOSE 127.77 1 < - <class 'socket.error'> [Errno 110] Connection timed out22:56
ianw^ is that a thing anyone has seen?22:56
*** otter768 has quit IRC22:56
jesusaurusto be honest, I didn't understand that particular line22:56
jeblairjesusaurus: neither did the author ;)22:57
pabelangerjeblair: danke, for both22:57
jeblairjesusaurus: it actually means "*only* report failures of things on this line".22:57
jeblairjesusaurus: the author thought it meant "*also* report failures of things on this line which are otherwise excluded by the previous line".22:57
jeblairand it's been copy-pastad around ever since22:58
openstackgerritKirill Zaitsev proposed openstack-infra/project-config: Add translation jobs to murano
clarkbjesusaurus: did you push an unrevert for the change that needed the boilerplate to be merged?22:59
clarkbjesusaurus: that should be step after getting the delta changes merged22:59
jesusaurusclarkb: yeah, 254992 is an identical change, but this time with a Depends-On23:00
*** EricGonczer_ has quit IRC23:00
openstackgerritMerged openstack-infra/log_processor: Add project boilerplate
*** SumitNaiksatam has quit IRC23:00
*** triggerwarning has quit IRC23:02
zarofungi, pabelanger : i can do that jjb release by end of this week.23:04
zaroanteaya: you've discovered the quick approve button.  when there is only one more vote required to apprv then that button will appear.23:05
anteayazaro: ah23:05
zaroi actually don't think it's necessary.23:05
*** hurgleburgler has joined #openstack-infra23:05
anteayame either23:05
rhedlindanteaya: I found the infinte loop in the zuul scheduler. it is still there in the latest version. who's familiar with the zuul implementation that I could bounce ideas for a fix with?23:05
jeblairSpamapS: I released gertty 1.3.0
anteayajust post to channel23:06
anteayarhedlind: what did you find?23:06
jeblairzaro: i released jjb 1.4.023:06
*** pahuang_ has joined #openstack-infra23:06
anteayarhedlind: zuul experts are around and will follow along as they can23:06
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output
rhedlindanteaya: ok23:07
anteayarhedlind: thanks23:07
*** spzala has quit IRC23:08
zaropabelanger: jeblair says he did it ^, does it not have what you want?23:09
anteayarhedlind: I'm aslo making dinner at the same time so just keep posting links to code or logs as they are relevant to your point23:09
*** zeih has joined #openstack-infra23:09
jeblairwell, i only did it 20 minutes ago23:10
pabelangerzaro: indeed.23:11
pabelangerthanks for the follow up23:11
*** yuanying has joined #openstack-infra23:12
*** zeih has quit IRC23:14
*** hichihara has joined #openstack-infra23:15
*** apuimedo_ has quit IRC23:15
*** apuimedo has quit IRC23:15
*** bpokorny_ has quit IRC23:17
*** apuimedo has joined #openstack-infra23:17
*** bpokorny has joined #openstack-infra23:17
SpamapSjeblair: oh very nice23:18
kzaitsev_mbk, I see that horizon and magnum-ui are configured separately in zuul for message extraction — gotta find people who configured those, to ask about dashboard23:19
clarkbkzaitsev_mb: I really want them to get with the program fwiw23:20
clarkbkzaitsev_mb: its not necessary that they be so different23:20
clarkbbut since horizon gets most of the translation effort we havne't addressed that23:20
*** rhallisey has joined #openstack-infra23:20
kzaitsev_mboh, I see.23:20
clarkbwe support multiple docs for each project which is what they need aiui23:21
*** apuimedo_ has joined #openstack-infra23:21
clarkbbut also something with django iirc, which I think isn't actually necessary23:21
clarkbsince gettext is pretty universal23:21
kzaitsev_mbhorizon seems to use custom extractor for angular and django templates and everything.23:21
clarkbthey use direct gettext iirc, which should just work everywhere23:22
kzaitsev_mbclarkb: last time I saw some movement in horizon regarding string extraction it was related to moving to django_babel23:22
kzaitsev_mband something related to js23:22
clarkbone thing I have contemplated is using a tox target for extraction and updates23:23
*** Daisy has joined #openstack-infra23:23
clarkbso either make the tooling consistent or change the interface to be more Make like23:23
pleia2jeblair: :)23:23
clarkbbut either way addressing that would be good23:23
*** dingyichen has joined #openstack-infra23:23
kzaitsev_mboh, yeah. That was the 1st place I looked to =) but apparently we don't use tox for that =/23:23
jeblairpleia2: diffy the cukoo tweets!23:23
pleia2jeblair: and diffy thinks gertty is amazing!23:24
clarkbTIL 1080 pixels is not wide enough for gerrit23:26
anteayaclarkb: how many does gerrit expect?23:26
pleia2I've long said that google engineers need smaller monitors, so they know how the rest of us live23:26
jeblairall of the pixels23:26
clarkbanteaya: I dunno but my monitor isn't wide enough for the 3 wide column to render properly23:26
anteayaperhaps mine isn't rendering properly23:26
anteayaI don't know23:26
clarkbI get a scrollbar and lots of line wrapping23:27
pleia2I have 1680 width on my screen and that's enough23:27
pleia2still too much whitespace though23:27
pleia2I think one of the patches up makes that better, right zaro?23:27
anteayaon patches I have open it now gives me an updates box: Update from Jeremy Stanley, James E. Blair, Jenkins, Gerrit Code Review Show Ignore23:27
clarkbya sdague has one I believe23:27
* pleia2 nods23:27
anteayalower right corner in a patch I haven't refreshed that has activity23:28
clarkbpleia2: they probably also use much smaller fonts than I do23:28
jeblairanteaya: yeah, it polls periodically to find that23:28
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command
anteayaclarkb: I don't have a scroll bar, I guess it is supposed to look like this23:28
pleia2clarkb: I guess you have dad eyes now :D23:28
anteayajeblair: okay, interesting23:28
*** tonytan4ever has quit IRC23:28
clarkbpleia2: yes optometrist confirmed23:28
clarkb"I am going to change your prescription again" and "you need high index lenses now"23:29
pleia2hah, oh dear23:29
zaroohh, cool.  didn't know Gerrit was on twitter.23:29
zarocan you follow without having an account?23:29
clarkboh the best prat was "do you wear contacts?" "No" "Good, because that would be tricky"23:29
pleia2zaro: not really :\23:29
zaronice summary by Luca there.23:29
pleia2zaro: yeah, that was a great post, along with jeblair's summary I almost felt like I was there!23:30
anteayarhedlind: don't disappear too far, we are all here, hanging out, now is a perfect time (timing wise) to share your findings23:30
anteayarhedlind: you don't have to fix it yourself, we are grateful you found something23:30
zarough, i guess i just need to open web pag to follow23:30
*** rbrndt has quit IRC23:31
*** eil397 has joined #openstack-infra23:31
*** apuimedo_ has quit IRC23:35
*** Swami_ has joined #openstack-infra23:35
anteayapleia2: have a link to the referenced tweet?23:35
*** apuimedo has quit IRC23:35
*** Daisy has quit IRC23:36
jeblairoh look, flepied wrote a thing about us:
*** apuimedo has joined #openstack-infra23:36
jeblair(found it from the gerrit twitterstream)23:37
*** zeih has joined #openstack-infra23:37
*** Daisy has joined #openstack-infra23:37
*** ayoung has joined #openstack-infra23:37
*** rhallisey has quit IRC23:37
pleia2anteaya: I was thinking of the one that links to this article
pleia2anteaya: but now I'm not so sure, since I don't think that was written by luca :)23:38
pleia2but it's still a good post23:38
*** dprince has quit IRC23:39
* anteaya reads jeblair and pleia2's links, while continuing to hope rhedlind returns23:40
*** apuimedo_ has joined #openstack-infra23:41
*** Daisy has quit IRC23:41
*** gildub has quit IRC23:41
*** zeih has quit IRC23:41
*** apuimedo has quit IRC23:41
*** apuimedo has joined #openstack-infra23:43
*** darrenc is now known as darrenc_afk23:44
*** apuimedo_ has quit IRC23:45
openstackgerritMerged openstack-infra/project-config: Add Consul Gate Job
*** erlon has quit IRC23:46
*** Qiming has joined #openstack-infra23:46
openstackgerritMerged openstack-infra/system-config: [WIP] Add puppetlabs-apache 1.8.0?
*** jaypipes has quit IRC23:47
*** sflanigan has joined #openstack-infra23:48
*** esker has joined #openstack-infra23:48
*** apuimedo_ has joined #openstack-infra23:51
*** sdake has joined #openstack-infra23:51
docaedoI have a question about periodic jobs and infra, probably need just a little pointer :) Using propose_projects_yaml_update as my model, I made a script that finds dead links in app-catalog assets.yaml, and proposes an update23:53
docaedoI have a question about periodic jobs and infra, probably need just a little pointer :) Using propose_projects_yaml_update as my model, I made a script that finds dead links in app-catalog assets.yaml, and proposes an update23:54
*** apuimedo has quit IRC23:55
*** apuimedo_ has quit IRC23:55
*** rbrndt has joined #openstack-infra23:56
*** apuimedo has joined #openstack-infra23:57
zaropleia2: i'm pretty sure it was luca.  at least i remember him taking those pics23:59
zaroand gerritforge is his company23:59

Generated by 2.14.0 by Marius Gedminas - find it at!