| mikal | What does "not registered" mean as a test result? | 00:00 |
|---|---|---|
| zaro | clarkb: fungi is seeing timeouts with some of the diffs | 00:00 |
| fungi | jeblair: we'll likely want to deploy a replacement server, but it should be trivial | 00:00 |
| *** alexsyip has joined #openstack-infra | 00:00 | |
| Sam-I-Am | anteaya: how are you this week? | 00:00 |
| jeblair | of course i'm uploading a patch at utc midnight | 00:01 |
| fungi | hello database backup time! | 00:01 |
| *** tjones has quit IRC | 00:01 | |
| zaro | actually diff cache memory is defaulted to 10M, maybe we should just double it? | 00:01 |
| docaedo | jeblair: huh .. I get it every time, triggers immediately for me - I'll try to recreate somewhere else before I pester any more :) | 00:02 |
| jeblair | zaro, clarkb: perhaps "rate" or "count" of timeouts could be measurable | 00:02 |
| asselin_ | fungi, thanks for answering my question. didn't even have to ask. | 00:02 |
| *** baoli has quit IRC | 00:02 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command https://review.openstack.org/258725 | 00:03 |
| fungi | zaro: yes, i have no trouble believing we could be totally blowing out a 10mb diff cache and getting a lot more cache misses than intended | 00:03 |
| *** sabeen3 has joined #openstack-infra | 00:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release https://review.openstack.org/258726 | 00:04 |
| jeblair | clarkb, fungi, greghaynes, mordred: https://review.openstack.org/258726 is a companion to https://review.openstack.org/258722 but merits more thought/discussion | 00:04 |
| anteaya | Sam-I-Am: I'm good thanks | 00:04 |
| anteaya | Sam-I-Am: thanks for asking | 00:04 |
| zaro | jeblair: do you an idea how to measure, besides greps logs i guess? | 00:05 |
| Sam-I-Am | anteaya: cold up there? | 00:05 |
| docaedo | jeblair: oh so sorry .. that's the new SNI warning you just fixed! | 00:05 |
| fungi | i still find it odd that it would take 5 seconds to do a diff, but i'm sure gerrit is plenty busy and maybe jgit is not well optimized? | 00:05 |
| anteaya | Sam-I-Am: no, I'm disappointed | 00:05 |
| anteaya | Sam-I-Am: september weather | 00:05 |
| anteaya | just a skift of snow today | 00:05 |
| Sam-I-Am | anteaya: had 8" of snow here... went snow-biking for lunch. | 00:05 |
| anteaya | no frost in the ground | 00:05 |
| jeblair | neat, the ps2 i uploaded at 00:00 disappeared | 00:05 |
| anteaya | Sam-I-Am: nice work | 00:05 |
| *** sabeen has quit IRC | 00:06 | |
| *** thorst has quit IRC | 00:06 | |
| Sam-I-Am | rode for an hour at full power... average speed: 3 | 00:06 |
| anteaya | jeblair: :\ | 00:06 |
| jeblair | let me make a trivial change to it | 00:06 |
| jlvillal | So I seem to be unable to push a rebased patch. It keeps saying, "! [remote rejected] HEAD -> refs/publish/master (no new changes)" | 00:06 |
| *** spzala has joined #openstack-infra | 00:06 | |
| anteaya | Sam-I-Am: well done | 00:06 |
| jeblair | jlvillal: did you start trying to do that 6 minutes ago? | 00:06 |
| anteaya | jlvillal: which patch? | 00:06 |
| jlvillal | The patch is the same, but I did rebase it so it is based off tip of tree. | 00:06 |
| jlvillal | jeblair, About then | 00:06 |
| jlvillal | :) | 00:06 |
| jlvillal | anteaya, https://review.openstack.org/#/c/253974/ | 00:07 |
| jeblair | jlvillal: yeah, same thing happened to me; the patch is in gerrit's git tree, but not in the database because db backups happened then and i think the query timed out | 00:07 |
| jlvillal | jeblair, First time I pushed it gave a different error | 00:07 |
| jlvillal | jeblair, " ! [remote rejected] HEAD -> refs/publish/master (internal error) | 00:07 |
| jlvillal | " | 00:07 |
| jlvillal | Was first error | 00:08 |
| *** Qiming has quit IRC | 00:08 | |
| jlvillal | jeblair, Do I just wait? | 00:08 |
| anteaya | jlvillal: well if we are losing patches during a db backup that is new and isn't great | 00:08 |
| jlvillal | anteaya, I agree! | 00:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Switch to using twine from packages https://review.openstack.org/258722 | 00:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release https://review.openstack.org/258726 | 00:09 |
| anteaya | jlvillal: I'm guessing jeblair is checking something | 00:09 |
| *** harlowja has joined #openstack-infra | 00:09 | |
| jlvillal | Okay. I'll be patient. Thanks | 00:09 |
| jeblair | jlvillal: no, i doubt they will ever show up. i made insubstantial changes to my commit messages for gerrit to accept them as new patchsets. | 00:09 |
| nibalizer | clarkb: hey you about? my nodepool isn't cooperating | 00:09 |
| *** amuller has quit IRC | 00:09 | |
| nibalizer | i'm getting Authentication (publickey) failed. | 00:09 |
| jeblair | jlvillal: so if you look at https://review.openstack.org/#/c/258722/ you'll see PS1 and then PS3 | 00:09 |
| jlvillal | jeblair, Okay. I will do something similar | 00:09 |
| nibalizer | this is from snapshot, not dib | 00:09 |
| clarkb | nibalizer: ya just sat back in front of laptop | 00:09 |
| nibalizer | but when i use the provided credentials i can ssh in just fine | 00:09 |
| clarkb | nibalizer: nodepool will use the pubkey for hte user it runs as by default | 00:10 |
| clarkb | nibalizer: er key | 00:10 |
| clarkb | so you have to put that pubkey on the hosts | 00:10 |
| nibalizer | it is | 00:10 |
| nibalizer | under the jenkins user | 00:10 |
| jlvillal | jeblair, I did that and I jumped for Patch set 4 to 6. https://review.openstack.org/#/c/253974/ | 00:10 |
| jlvillal | s/for/from/ | 00:10 |
| nibalizer | so i ssh jenkins@template-host just fine from the nodepool user | 00:11 |
| nibalizer | but when image-update tries it fails | 00:11 |
| clarkb | nibalizer: and when you say you do it by hand are you logging in as jenkins using that same key or ar eyou using cloudinit/glean stuff? | 00:11 |
| jeblair | jlvillal: nice! you jumped twice as far as i did! | 00:11 |
| jlvillal | :) | 00:11 |
| clarkb | nibalizer: image-update doesn't use that key, it makes a temp key iirc | 00:11 |
| nibalizer | clarkb: using jenkins | 00:11 |
| Sam-I-Am | anteaya: do you often spy in #openstack ? | 00:11 |
| nibalizer | clarkb: oh | 00:11 |
| anteaya | Sam-I-Am: I click around | 00:11 |
| nibalizer | clarkb: then why does it have a key? | 00:11 |
| anteaya | Sam-I-Am: I see you working openstack a lot and leave you to it | 00:11 |
| clarkb | nibalizer: the key on the image is the key to use once image is created | 00:12 |
| Sam-I-Am | anteaya: front line support! | 00:12 |
| anteaya | most questions in there I can't see how anything I have to offer can help (and make me happy for doing so) | 00:12 |
| Sam-I-Am | but yields lots of happy openstack users | 00:12 |
| clarkb | nibalizer: but in order to bootstrap image creation it keygens and passes that along via metadata/config-drive | 00:12 |
| nibalizer | omfra-cloud doesn't have key support | 00:12 |
| anteaya | Sam-I-Am: complete with safety glasses | 00:12 |
| nibalizer | er metadata/config-drive support | 00:12 |
| anteaya | Sam-I-Am: indeed | 00:12 |
| Sam-I-Am | plus i think its a really good feedback mechanism for docs, among other things | 00:12 |
| clarkb | nibalizer: it needs that | 00:12 |
| clarkb | nibalizer: config-drive is probably preferable | 00:12 |
| *** sabeen3 has quit IRC | 00:12 | |
| anteaya | Sam-I-Am: you contribute a lot to the happy users bottom line | 00:12 |
| Sam-I-Am | also makes me realize i could work on docs 24/7 for years and not make a dent :/ | 00:12 |
| nibalizer | ok | 00:12 |
| clarkb | metadata server is less reliable | 00:13 |
| jeblair | wait | 00:13 |
| anteaya | Sam-I-Am: true that | 00:13 |
| * anteaya waits | 00:13 | |
| jeblair | we don't need that for omfra | 00:13 |
| jeblair | nibalizer, clarkb: ^ | 00:13 |
| greghaynes | Just need dhcp | 00:13 |
| clarkb | jeblair: the way nodepool works we do | 00:14 |
| clarkb | we can change how nodepool works if we want | 00:14 |
| jeblair | clarkb: we can just upload images | 00:14 |
| greghaynes | Why? | 00:14 |
| anteaya | okay I'm going to evening for a few hours | 00:14 |
| clarkb | oh you are saying no snapshots | 00:14 |
| clarkb | ya we can do that instead | 00:14 |
| jeblair | clarkb: yes | 00:14 |
| greghaynes | ++ | 00:14 |
| Sam-I-Am | anteaya: evening before it becomes nighting | 00:14 |
| anteaya | see y'all later or tomorrow | 00:14 |
| clarkb | greghaynes: nodepool needs one or the other when building snapshot images to set up keys | 00:14 |
| anteaya | Sam-I-Am: exactly | 00:14 |
| jeblair | i've been following the no-snapshots policy for new clouds | 00:14 |
| *** Daisy has quit IRC | 00:14 | |
| Sam-I-Am | enjoy | 00:14 |
| zaro | anteaya: have a good evening, thanks for helping | 00:14 |
| clarkb | jeblair: roger | 00:14 |
| *** tjones1 has quit IRC | 00:14 | |
| anteaya | Sam-I-Am: thanks you too | 00:14 |
| clarkb | nibalizer: so then ignore snapshots and just do dib | 00:14 |
| greghaynes | clarkb: ya, just bake in keys | 00:14 |
| anteaya | zaro: thanks, awesome work today | 00:14 |
| anteaya | zaro: talk to you tomorrow | 00:14 |
| clarkb | thta said, config-drive being an option is probably reasonable | 00:15 |
| jeblair | so if we want to add it as a learning experience or for some other thing, cool, just saying we don't need to block on it for nodepool | 00:15 |
| crinkle | our infra-cloud doc says "The cloud will have config drive enabled because we believe it to be more robust than the EC2-style metadata service.", we'll still need that i think? | 00:15 |
| clarkb | then you can opt into it | 00:15 |
| jeblair | clarkb: right | 00:15 |
| zaro | yep, may see a 'ha ha ha' patch from me | 00:15 |
| *** zz_dimtruck is now known as dimtruck | 00:15 | |
| *** Daisy has joined #openstack-infra | 00:15 | |
| clarkb | oh I know what we need it for | 00:15 |
| clarkb | hostnames | 00:15 |
| clarkb | so its actually required | 00:15 |
| greghaynes | Dhcp? | 00:15 |
| zaro | clarkb: https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/config-gerrit.html#cache | 00:15 |
| clarkb | greghaynes: why doesn't that work on clouds that dhcp then? | 00:16 |
| greghaynes | Good question | 00:16 |
| clarkb | is it because cloud-init/glean overwrite? | 00:16 |
| crinkle | the way it is now the nodes take a while to boot because they're waiting to time out finding the metadata service, so i'd like to fix that | 00:16 |
| greghaynes | It shouldn't, but it might | 00:16 |
| zaro | clarkb: am i reading that right? it says cache "changes" is disabled by default? | 00:16 |
| greghaynes | clarkb: it could also be dib setting a fail hostname that doesn't get overwritten | 00:17 |
| clarkb | zaro: ya its disabled by default | 00:17 |
| clarkb | greghaynes: or neutron/nova don't provide hostname via dhcp | 00:17 |
| greghaynes | crinkle: use non cloud-init nodes | 00:17 |
| openstackgerrit | Ben Kero proposed openstack/diskimage-builder: Replace sfdisk partitioning with parted https://review.openstack.org/258721 | 00:17 |
| zaro | clarkb: ohh, i think that should be enabled. | 00:18 |
| greghaynes | clarkb: possible, we should look into that | 00:18 |
| greghaynes | Sounds like a useful thing to do | 00:18 |
| clarkb | jeblair: +2 on system twine | 00:20 |
| *** masber has joined #openstack-infra | 00:20 | |
| clarkb | jeblair: for the wheel change I can look at wheel's changelog and see if there is anything important there we would miss | 00:21 |
| greghaynes | The wheel thing doesn't have implications for constraints, correct? | 00:21 |
| fungi | as soon as i get settled i'll approve the twine change and deploy a replacement release.s.o.o, then retrigger the upload jobs for dims | 00:21 |
| clarkb | greghaynes: not for the release slave since it doesn't run install code | 00:21 |
| clarkb | greghaynes: its straight get file upload file | 00:21 |
| *** Daisy has quit IRC | 00:22 | |
| greghaynes | Ah, awesome | 00:22 |
| pabelanger | puppetdb.o.o seems quiet. 111nodes unreported in the last 1.5 hours | 00:22 |
| clarkb | trusty has 0.24 | 00:22 |
| pabelanger | something going on with ansible? | 00:22 |
| clarkb | 0.26 looks like latest | 00:22 |
| dims | fungi : thanks for reference https://review.openstack.org/#/c/258606/ and https://review.openstack.org/#/c/258618/ did not make it to pypi | 00:22 |
| clarkb | pabelanger: we turned it off for gerrit upgrade | 00:22 |
| *** apuimedo_ has quit IRC | 00:22 | |
| pabelanger | clarkb: Ah, when do you suspect it will come back online? | 00:23 |
| clarkb | pabelanger: I am not sure I had to pop out to visit the eye doctor and am not up to date on upgrade state | 00:23 |
| *** oomichi has joined #openstack-infra | 00:23 | |
| pabelanger | ack | 00:23 |
| *** Daisy has joined #openstack-infra | 00:23 | |
| clarkb | jeblair: https://pypi.python.org/pypi/wheel based on that I htink if we don't use it with python3 we will be ok | 00:23 |
| pabelanger | fungi: when you break away from the pizza ^ | 00:23 |
| jeblair | do we use wheel at all there? | 00:24 |
| clarkb | jeblair: the only thing that we may want is the 'arch' tag support | 00:24 |
| clarkb | jeblair: I htink it may be used to grok the wheels we upload | 00:24 |
| jeblair | ah, i'll grep around | 00:24 |
| clarkb | jeblair: but at that point the wheels are already generated | 00:24 |
| fungi | oh, we haven't reenabled puppet. any objections to turning it back on now? | 00:24 |
| fungi | pabelanger: thanks for the reminder | 00:24 |
| clarkb | jeblair: so its use would be minimal | 00:24 |
| fungi | our maintenance plan had a disable puppet step, no reenable step :/ | 00:24 |
| fungi | i should have caught that | 00:24 |
| jeblair | clarkb: yes in jenkins/scripts/pypi-extract-name.py | 00:25 |
| jeblair | wheel.bdist_wheel.safer_name(distname) | 00:25 |
| jeblair | that's all we use it for | 00:25 |
| jeblair | so cool | 00:25 |
| jeblair | i'll amend the commit message for posterity | 00:25 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command https://review.openstack.org/258725 | 00:26 |
| *** thorst has joined #openstack-infra | 00:26 | |
| jeblair | https://review.openstack.org/258726 | 00:26 |
| jeblair | not sure why no gerritbot | 00:26 |
| *** dimtruck is now known as zz_dimtruck | 00:27 | |
| *** Daisy has quit IRC | 00:27 | |
| mattoliverau | fyi, the new gerrit's reply (review) doesn't mention workflow -1 is WIP, so this could get confusing for new users. | 00:27 |
| *** apuimedo_ has joined #openstack-infra | 00:28 | |
| jeblair | mattoliverau: unless you click on it, you mean? | 00:28 |
| *** zz_dimtruck is now known as dimtruck | 00:29 | |
| *** dims has quit IRC | 00:29 | |
| mattoliverau | don't know just saw it, so thought I'd mention it | 00:29 |
| *** thorst has quit IRC | 00:29 | |
| fungi | i haven't seen any objections to reenabling puppet, so doing that now | 00:29 |
| jeblair | oh, gertty failed to update the commit message | 00:30 |
| *** thorst has joined #openstack-infra | 00:30 | |
| jeblair | 2015-12-16 16:30:59,102 Received: Not found: message | 00:30 |
| openstackgerrit | Apoorva Deshpande proposed openstack-infra/ciwatch: Separate votes from Jenkins queues https://review.openstack.org/258125 | 00:30 |
| jeblair | hrm. | 00:30 |
| jeblair | it totally sent "message" | 00:30 |
| jeblair | the endpoint changed? | 00:31 |
| jeblair | PUT /changes/{change-id}/edit:message | 00:32 |
| jeblair | that used to just be "/changes/{change-id}/message" before | 00:32 |
| fungi | api break | 00:32 |
| * fungi shakes fist ineffectually | 00:32 | |
| jeblair | great... can we query the version? | 00:32 |
| fungi | zaro: ^ is the gerrit api versioned? | 00:33 |
| fungi | (rest api) | 00:33 |
| fungi | pabelanger: if puppetboard is still quiet half an hour from now, let me know | 00:34 |
| sdague | zaro: ok, so I've got 3 days left in the "office" this year. I'd love to be part of the design conversation for that plugin from the UX side as I hacked a lot of the current interface. But I won't be able to really do anything about that until 2016 | 00:36 |
| *** apuimedo has quit IRC | 00:36 | |
| *** apuimedo_ has quit IRC | 00:37 | |
| *** apuimedo has joined #openstack-infra | 00:38 | |
| jeblair | i just checked headers... nothing there. | 00:38 |
| docaedo | jeblair: I think my issue has to do with ubuntu and openssl (the ssl v3 handshake thing), just for my sanity, what OS did you try under? | 00:38 |
| *** thorst has quit IRC | 00:38 | |
| cloudnull | New gerrit looks nice! well done everybody. | 00:39 |
| jeblair | docaedo: Ubuntu 14.04.3 LTS | 00:39 |
| * cloudnull goes to push buttons | 00:39 | |
| clarkb | jeblair I think you can check gerrit version | 00:39 |
| clarkb | but not api version | 00:39 |
| jeblair | clarkb: that would work well enough; any clues as to where? | 00:39 |
| docaedo | jeblair: unpossible! drat! but thanks, I'll try in a clean VM and see what happens :) | 00:40 |
| clarkb | gerrit version is ssh command | 00:40 |
| jeblair | clarkb: that doesn't work so well for gertty :( | 00:40 |
| clarkb | well it may be hhtp too | 00:40 |
| clarkb | I just dont know what that is | 00:41 |
| jeblair | right, i'm looking and not finding | 00:41 |
| jeblair | aha | 00:41 |
| jeblair | https://review.openstack.org/Documentation/rest-api-config.html#get-version | 00:41 |
| jeblair | found it! | 00:41 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Skip dsvm jobs for ironic-python-agent if code doesn't change https://review.openstack.org/258735 | 00:41 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Add reno jobs for ironic-python-agent https://review.openstack.org/258736 | 00:41 |
| *** apuimedo_ has joined #openstack-infra | 00:42 | |
| *** otter768 has joined #openstack-infra | 00:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Switch to using wheel from distro on release https://review.openstack.org/258726 | 00:43 |
| jeblair | i'll fix gertty tomorrow | 00:43 |
| fungi | i take it that works | 00:43 |
| *** dimtruck is now known as zz_dimtruck | 00:43 | |
| fungi | yay! | 00:43 |
| fungi | approving | 00:43 |
| jeblair | nah, i updated that in the web ui | 00:43 |
| fungi | oh. poo | 00:43 |
| fungi | but good that there's a path forward | 00:44 |
| jeblair | fungi: yep. (though i don't have a 2.8 server anymore!) | 00:44 |
| jeblair | editing in the ui is a little weird... | 00:44 |
| jeblair | you open the file, then click "edit"... that's normal enough | 00:44 |
| jeblair | when you're done, you click save, then go back to the change screen, then click "done editing", then click "publish edit" | 00:45 |
| openstackgerrit | Tianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support https://review.openstack.org/258739 | 00:45 |
| jeblair | i was not quite expecting so many "yes i'm done" steps. | 00:45 |
| fungi | that does seem more cumbersome than necessary | 00:45 |
| fungi | anyway, once those two merge i'll launch a new release.s.o.o | 00:45 |
| jeblair | curl https://review.openstack.org/config/server/version -> 2.11.4-11-ga14450f | 00:45 |
| jeblair | so yeah, looks like a path forward | 00:46 |
| fungi | not what i'd call a "versioned api" but it's better than "no clue" i guess | 00:46 |
| jeblair | right | 00:46 |
| *** apuimedo_ has quit IRC | 00:46 | |
| fungi | i suppose you'll also want to know which gerrit version changed that method | 00:47 |
| *** gildub has quit IRC | 00:47 | |
| fungi | oh, and of course you have no way to confirm that the version lookup even works the same way on 2.8 | 00:47 |
| *** otter768 has quit IRC | 00:47 | |
| jeblair | yeah, i was going to go with "try GET; except: assume 2.8" :) | 00:47 |
| jeblair | we can always refine later | 00:48 |
| *** mgrosz has joined #openstack-infra | 00:48 | |
| jeblair | (i should probably check the version everytime gertty switches back into online mode) | 00:49 |
| jeblair | (since, after all, like today, it might change) | 00:49 |
| *** mgrosz has quit IRC | 00:50 | |
| *** rcernin has quit IRC | 00:50 | |
| fungi | indeed it might | 00:51 |
| *** ayoung has quit IRC | 00:51 | |
| zaro | ok. i've met my quota of excitement for the day. going to decompress | 00:51 |
| *** apuimedo_ has joined #openstack-infra | 00:52 | |
| fungi | it was a very good day zaro--thanks for working on the upgrade! | 00:52 |
| pleia2 | yes, thanks zaro! have a nice evening | 00:53 |
| *** masber has quit IRC | 00:53 | |
| *** pvaneck has quit IRC | 00:53 | |
| olaph | nice work, zaro! | 00:53 |
| *** Swami_ has quit IRC | 00:54 | |
| *** angdraug has quit IRC | 00:55 | |
| *** xarses has quit IRC | 00:55 | |
| *** Qiming has joined #openstack-infra | 00:56 | |
| *** MarkAtwood has quit IRC | 00:56 | |
| *** ajmiller has quit IRC | 00:57 | |
| *** chenli has joined #openstack-infra | 00:58 | |
| chenli | hello, the page show for https://review.openstack.org/ is different today ?? | 00:59 |
| chenli | anyone know how to change it back ??? | 01:00 |
| *** eil397 has quit IRC | 01:00 | |
| clarkb | chenli: for the change screen? the version we upgraded to today no longer has the old screen | 01:00 |
| pleia2 | chenli: yes, it upgraded it to version 2.11 today, the old dashboard is no longer available | 01:01 |
| chenli | ... I can't find the line I get comment... | 01:01 |
| *** apuimedo_ has quit IRC | 01:01 | |
| *** apuimedo has quit IRC | 01:01 | |
| pleia2 | chenli: I recommend trying it out for a little while, it gets easier :) | 01:01 |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Handle markers to support sdist on pip < 6 https://review.openstack.org/253289 | 01:01 |
| clarkb | chenli: can you link to an example change where you can't see the comment? | 01:02 |
| *** thorst has joined #openstack-infra | 01:02 | |
| chenli | pleia2: oo.... it is a very big change | 01:02 |
| *** dguitarbite has joined #openstack-infra | 01:02 | |
| *** apuimedo has joined #openstack-infra | 01:02 | |
| chenli | clarkb: I just find it.... enter into files, the same as the old one... | 01:02 |
| pleia2 | there you go | 01:03 |
| openstackgerrit | Merged openstack-infra/system-config: Switch to using twine from packages https://review.openstack.org/258722 | 01:03 |
| *** bharathm has quit IRC | 01:03 | |
| *** bpokorny has quit IRC | 01:04 | |
| openstackgerrit | Tianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support https://review.openstack.org/258739 | 01:04 |
| fungi | there's one | 01:04 |
| chenli | pleia2: clarkb : thanks ! | 01:04 |
| *** masber has joined #openstack-infra | 01:05 | |
| *** bpokorny has joined #openstack-infra | 01:05 | |
| *** apuimedo_ has joined #openstack-infra | 01:06 | |
| *** chlong has joined #openstack-infra | 01:07 | |
| *** apuimedo_ has quit IRC | 01:11 | |
| *** gyee has quit IRC | 01:11 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: narrow the 3 columns progressively with screen width https://review.openstack.org/258744 | 01:12 |
| *** thorst has quit IRC | 01:12 | |
| dguitarbite | Hello, I need some help with regards to releasing for training-labs | 01:13 |
| *** thorst has joined #openstack-infra | 01:13 | |
| dguitarbite | fungi: Can you help me with releasing kilo for training-labs? | 01:13 |
| fungi | dguitarbite: probably not at this time of my night. my dance card is still pretty full. might help though to describe what you mean by "releasing kilo for training-labs" | 01:14 |
| *** alexsyip has quit IRC | 01:16 | |
| *** apuimedo_ has joined #openstack-infra | 01:16 | |
| dguitarbite | fungi: I just want to add tags for Juno and Kilo for training-labs. Which I assume acts as releases. | 01:16 |
| dguitarbite | fungi: I remember doing the same for Icehouse (pinging you for it). Is there a better way? | 01:17 |
| *** ParsectiX has joined #openstack-infra | 01:17 | |
| *** ashleighfarnham has quit IRC | 01:17 | |
| fungi | dguitarbite: i honestly don't remember what we did for the icehouse training-labs release, but that sounds possible | 01:17 |
| *** tjones has joined #openstack-infra | 01:17 | |
| dguitarbite | fungi: Ok, how do I release? By that I want to add git tags. | 01:18 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Use separated ACL group for OSprofiler https://review.openstack.org/258182 | 01:18 |
| ianw | greghaynes / anyone : so is configdrive the default on RAX? or do you have to be special to get it? | 01:18 |
| clarkb | ianw: its a nova boot flag | 01:18 |
| jroll | ianw: it is not default for virt | 01:19 |
| jroll | --config-drive '' | 01:19 |
| fungi | dguitarbite: http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release is our documented process. no idea if the training-labs repo is typical of most projects though | 01:19 |
| ianw | jroll: ok ... it is not exposed via the web interface at all? | 01:19 |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job https://review.openstack.org/258749 | 01:19 |
| jroll | ianw: not that I know of - I think if it allows passing "user data" that will trigger configdrive | 01:19 |
| jroll | but not sure | 01:20 |
| ianw | ok, i have never started a rax node using !web ... let me see here | 01:20 |
| ianw | i know horizon has a tick box for it | 01:20 |
| jroll | nova boot --image bla --flavor bla name | 01:20 |
| * jroll has to run; good luck! | 01:20 | |
| *** thorst has quit IRC | 01:21 | |
| *** mudassirlatif has quit IRC | 01:22 | |
| dguitarbite | fungi: thanks, let me take a look. | 01:22 |
| *** skath_ has quit IRC | 01:23 | |
| *** skath_ has joined #openstack-infra | 01:24 | |
| *** etoews has quit IRC | 01:24 | |
| *** bgmccollum has joined #openstack-infra | 01:25 | |
| *** zhiyan has quit IRC | 01:25 | |
| *** mgkwill has quit IRC | 01:25 | |
| *** etoews has joined #openstack-infra | 01:25 | |
| *** zxiiro has quit IRC | 01:25 | |
| fungi | still waiting on two devstack-centos7 nodes for 258726 to get through check | 01:26 |
| *** maximov has quit IRC | 01:26 | |
| *** mgkwill has joined #openstack-infra | 01:28 | |
| *** maximov has joined #openstack-infra | 01:28 | |
| *** zhiyan has joined #openstack-infra | 01:28 | |
| *** ningy has joined #openstack-infra | 01:28 | |
| *** zxiiro has joined #openstack-infra | 01:28 | |
| fungi | rackspace is taking upwards of an hour to boot instances again | 01:28 |
| fungi | | 6625225 | rax-ord | None | devstack-centos7 | jenkins06 | devstack-centos7-rax-ord-6625225 | devstack-centos7-rax-ord-6625225 | 9b605aba-17be-4512-85cc-2342dae94f69 | None | building | 00:00:59:15 | | 01:28 |
| ningy | @here could anyone help me taking a look at this patch to make sure I'm doing the right thing? https://review.openstack.org/#/c/258749/ | 01:30 |
| *** apuimedo_ has quit IRC | 01:30 | |
| ningy | I'm trying to only enable certain services for devstack install and disable all other services that i don't need to have. any help is much appreciated! | 01:31 |
| *** apuimedo_ has joined #openstack-infra | 01:35 | |
| *** zhenguo has joined #openstack-infra | 01:38 | |
| openstackgerrit | Tianpeng Wang proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support https://review.openstack.org/258739 | 01:38 |
| fungi | it looks like failure to boot in rackspace is starving out most of our devstack-centos7 capacity. we've got a few in used state in hpcloud, and a bunch building forever in rackspace | 01:40 |
| *** apuimedo_ has quit IRC | 01:40 | |
| *** mgrosz has joined #openstack-infra | 01:42 | |
| *** mgrosz has quit IRC | 01:43 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/system-config: Fix top element in hideci.js for new change screen https://review.openstack.org/258751 | 01:44 |
| mgagne | zaro this fixes hideci.js which isn't hiding the correct element anymore ^ | 01:44 |
| *** apuimedo_ has joined #openstack-infra | 01:45 | |
| mgagne | jeblair about hideci.js ^ | 01:45 |
| *** rvasilets___ has quit IRC | 01:45 | |
| mgagne | oh it's 8:45pm EST already :-/ will go home then | 01:46 |
| *** annegentle has joined #openstack-infra | 01:48 | |
| *** mriedem has joined #openstack-infra | 01:48 | |
| *** hdd has quit IRC | 01:48 | |
| *** signed8bit has quit IRC | 01:49 | |
| *** apuimedo_ has quit IRC | 01:50 | |
| *** apuimedo_ has joined #openstack-infra | 01:55 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job https://review.openstack.org/258749 | 01:57 |
| *** tjones has quit IRC | 01:57 | |
| *** masber has quit IRC | 01:58 | |
| *** mtanino has joined #openstack-infra | 01:59 | |
| *** apuimedo_ has quit IRC | 02:00 | |
| *** aeng has quit IRC | 02:02 | |
| mriedem | anyone noticed that the multinode jobs are failing as NOT_REGISTERED? | 02:03 |
| mriedem | something up with test nodes? | 02:03 |
| *** amotoki has joined #openstack-infra | 02:03 | |
| *** apuimedo_ has joined #openstack-infra | 02:04 | |
| *** amotoki_ has joined #openstack-infra | 02:05 | |
| greghaynes | mriedem: there were/are issues with rax going on | 02:06 |
| *** thorst has joined #openstack-infra | 02:08 | |
| *** amotoki has quit IRC | 02:09 | |
| *** thorst has quit IRC | 02:09 | |
| *** thorst has joined #openstack-infra | 02:09 | |
| *** chlong has quit IRC | 02:10 | |
| *** triggerwarning has joined #openstack-infra | 02:11 | |
| *** bpokorny_ has joined #openstack-infra | 02:12 | |
| openstackgerrit | Joey D. proposed openstack/diskimage-builder: Closes bug: 1486689 https://review.openstack.org/258757 | 02:12 |
| openstack | bug 1486689 in diskimage-builder "Ramdisk should consider the size unit when inspecting the amount of RAM" [Medium,In progress] https://launchpad.net/bugs/1486689 - Assigned to Brad Lodgen (lodgenbd) | 02:12 |
| *** yamamoto has joined #openstack-infra | 02:14 | |
| *** bpokorny has quit IRC | 02:16 | |
| *** apuimedo_ has quit IRC | 02:16 | |
| *** bpokorny_ has quit IRC | 02:16 | |
| mriedem | greghaynes: ok, thanks, rechecked | 02:18 |
| *** thorst has quit IRC | 02:18 | |
| *** sridhar_ram has quit IRC | 02:19 | |
| *** yamahata has quit IRC | 02:19 | |
| *** baoli has joined #openstack-infra | 02:20 | |
| *** baoli has quit IRC | 02:22 | |
| *** apuimedo_ has joined #openstack-infra | 02:22 | |
| *** chlong has joined #openstack-infra | 02:22 | |
| *** baoli has joined #openstack-infra | 02:22 | |
| *** [1]Thelo has joined #openstack-infra | 02:23 | |
| fungi | 17 devstack-centos7 nodes building in rackspace many for close to an hour. none ready or used at this point anywhere | 02:25 |
| *** Thelo has quit IRC | 02:25 | |
| *** [1]Thelo is now known as Thelo | 02:25 | |
| * fungi sighs | 02:25 | |
| fungi | so still waiting for 258726 to pass check | 02:26 |
| *** apuimedo_ has quit IRC | 02:26 | |
| fungi | it looks like it may just be a problem with the centos7 images, so likely we've merged something that's broken them in rackspace but not in hpcloud | 02:29 |
| fungi | ianw: ^ any ideas? | 02:29 |
| fungi | i'm going to roll back the devstack-centos7 images in rax regions to see whether that works around it for now | 02:30 |
| ianw | fungi: not immediately ... | 02:30 |
| ianw | my monitoring shows the nodepool builds all seem ok... | 02:31 |
| *** sputnik13 has quit IRC | 02:32 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make check for image-should-fail quiet https://review.openstack.org/258762 | 02:32 |
| *** apuimedo_ has joined #openstack-infra | 02:32 | |
| greghaynes | bkero: ^ | 02:32 |
| bkero | tyty | 02:33 |
| fungi | ianw: i mean they build fine, but aren't able to boot in rackspace | 02:33 |
| *** yamamoto has quit IRC | 02:33 | |
| ianw | yeah, so ... well, no idea, any logs will help :) | 02:33 |
| fungi | at this point i just want that pending system-config change to finish getting tested and merge so i can replace the release worker and finish fixing the stuck releases | 02:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: install systemd service as 0644 https://review.openstack.org/258763 | 02:34 |
| greghaynes | ianw: monitoring? | 02:35 |
| *** baoli has quit IRC | 02:35 | |
| ianw | greghaynes: https://github.com/ianw/nodechecker -> "monitoring" ... at least lets me see big failiures | 02:35 |
| *** baoli has joined #openstack-infra | 02:36 | |
| greghaynes | ianw: so, I have a dream of something like that up to an irc bot | 02:36 |
| ianw | in build, but doesn't help with boot | 02:36 |
| ianw | greghaynes: well, i can easily put you on the cc list if you want it sent every day | 02:36 |
| *** mgrosz has joined #openstack-infra | 02:36 | |
| greghaynes | haha | 02:36 |
| fungi | right, we're mostly blind to problems (like this) with failing to boot an instance from a successfully built image | 02:36 |
| greghaynes | It wouldn't be hard to do a similar thing to the dib CI where we qemu boot an image | 02:37 |
| greghaynes | and just ensure we ping it | 02:37 |
| fungi | well, it seems to be provider-specific | 02:37 |
| *** piet has joined #openstack-infra | 02:37 | |
| greghaynes | :( | 02:37 |
| fungi | so in this case we upload one image to two providers, it's booting in one and not the other | 02:38 |
| greghaynes | It is stuck in building? | 02:38 |
| fungi | oh, though these are probably snapshot images not dib | 02:38 |
| *** mgrosz has quit IRC | 02:38 | |
| greghaynes | or they just dont get networking? | 02:38 |
| greghaynes | ah, I cant help you there ;) | 02:38 |
| fungi | greghaynes: nodepool shows devstack-centos7 nodes "building" in rackspace for about an hour and then switches them to delete | 02:39 |
| fungi | and since it counts building nodes as satisfying demand, we basically just wind up with no capacity and accumulate nodes stuck in a building state in the broken provider | 02:39 |
| *** dguitarbite has quit IRC | 02:40 | |
| * greghaynes tries to remember when nodepool considers something done building | 02:40 | |
| *** dguitarbite has joined #openstack-infra | 02:41 | |
| fungi | according to "nova show" they're in status BUILD the entire time | 02:42 |
| greghaynes | ok | 02:42 |
| fungi | so nova finally gives up waiting for them to go active | 02:42 |
| fungi | which in cases like this usually means some sort of boot-time problem where the instance never fully boots up | 02:42 |
| greghaynes | If they are dib I have seen things like that happen if you dont set hvm on the images | 02:43 |
| greghaynes | also saw similar things in hpcloud when we didnt set qcow2 compat | 02:43 |
| fungi | i don't think devstack-centos7 is dib in rax yet, but i'll confirm | 02:43 |
| *** otter768 has joined #openstack-infra | 02:43 | |
| *** gongysh_ has joined #openstack-infra | 02:44 | |
| greghaynes | Ya, I think it isnt | 02:44 |
| fungi | yeah, not dib | 02:44 |
| fungi | snapshots still | 02:44 |
| * greghaynes points fingers at rax | 02:44 | |
| *** apuimedo_ has quit IRC | 02:44 | |
| fungi | meh, we could still easily be setting something on them that's not working as expected | 02:44 |
| fungi | for example, the mem= kernel command line change we made in the last 24 hours | 02:45 |
| greghaynes | right, that | 02:45 |
| *** otter768 has quit IRC | 02:48 | |
| pabelanger | ianw: greghaynes: I want to get grafana tracking dib builds too. I should see what needs to be added into nodepool to publish into to statsd | 02:50 |
| pabelanger | we have some info, but I don't think enough | 02:50 |
| *** apuimedo_ has joined #openstack-infra | 02:50 | |
| greghaynes | I just want something to tell me when they failed, and I am bad at email | 02:50 |
| pabelanger | ianw: script does that today :) | 02:51 |
| pabelanger | show up in my inbox at noon every day | 02:51 |
| *** piet has quit IRC | 02:52 | |
| *** jyuso1 has joined #openstack-infra | 02:52 | |
| *** gongysh_ has quit IRC | 02:53 | |
| zaro | move forward []; | 02:53 |
| pabelanger | So, how is OVH holding up to nodepool? | 02:53 |
| fungi | okay, the last job for 258726 finally got a worker in the gate | 02:54 |
| *** apuimedo_ has quit IRC | 02:54 | |
| zaro | move right []; | 02:55 |
| ianw | fungi: so obviously centos is priority, not sure what i can do but willing to help. on f23 ... i'm quite out of ideas about why it's not working. manual testing of glean on rax (with configdrive) seems to show it configuring interfaces ok | 02:56 |
| fungi | put your hands on your hips, then pull your knees in tight? | 02:56 |
| greghaynes | Ian | 02:56 |
| greghaynes | Phone derp | 02:57 |
| fungi | ianw: yeah, i'm not really sure about my centos7 theory either. even rolling back to yesterday's images the only ones i see coming online still are booting in hpcloud not rax | 02:57 |
| greghaynes | ianw: did you get a change in nodepool to use glean from git? | 02:57 |
| *** amotoki_ has quit IRC | 02:57 | |
| zaro | o | 02:57 |
| openstackgerrit | Merged openstack-infra/system-config: Switch to using wheel from distro on release https://review.openstack.org/258726 | 02:57 |
| *** harlowja has quit IRC | 02:58 | |
| ianw | greghaynes: yes, the f23 images are installing glean from git | 02:58 |
| greghaynes | ianw: oh, and they are still broken? | 02:58 |
| ianw | greghaynes: seemingly, yes. NOT_REGISTERED when i try a job. could be specific ... might even be related to centos issues? | 02:59 |
| greghaynes | Ya, I wouldn't be surprised | 02:59 |
| ianw | greghaynes: best i can do is bring up a f22 node in rax, uninstall cloud-init, remove the interface stuff and reboot with glean | 02:59 |
| ianw | that *seems* to work | 02:59 |
| *** baoli has quit IRC | 03:00 | |
| greghaynes | I can resurrect my set up to upload vhds to rax if it turns out to not be that | 03:00 |
| *** apuimedo_ has joined #openstack-infra | 03:00 | |
| ianw | greghaynes: do you need special permissions to do that? | 03:00 |
| ianw | because it would be super duper handy to be able to test the image without nodepool reaping it | 03:00 |
| greghaynes | I don't think so, but you need vhd-util | 03:00 |
| *** masber has joined #openstack-infra | 03:01 | |
| greghaynes | And you need to go through the motions of uploading to swift, importing l, setting hvm | 03:01 |
| *** elo has quit IRC | 03:01 | |
| *** apoorvad has quit IRC | 03:02 | |
| *** piet has joined #openstack-infra | 03:02 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job https://review.openstack.org/258749 | 03:05 |
| *** apuimedo_ has quit IRC | 03:05 | |
| ianw | fungi: i wonder, can you "share" the f23 image? http://www.rackspace.com/cloud/images | 03:08 |
| fungi | ianw: i could stick a copy of it in the web root, i bet | 03:10 |
| fungi | ianw: hrm, nope. we're tight on space in that filesystem | 03:10 |
| *** apuimedo_ has joined #openstack-infra | 03:10 | |
| fungi | would require some tweaking to expose another location, and i'm still trying to get the release.slave replacement online | 03:11 |
| *** EricGonczer_ has joined #openstack-infra | 03:11 | |
| fungi | and desperately want to finish fixing the missing releases so i can call it a night | 03:11 |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job https://review.openstack.org/258749 | 03:12 |
| *** mriedem has quit IRC | 03:13 | |
| *** yonglihe has joined #openstack-infra | 03:14 | |
| *** yamamoto has joined #openstack-infra | 03:15 | |
| *** thorst has joined #openstack-infra | 03:15 | |
| ianw | greghaynes: sooo... the fedora vhd-util doesn't seem to "convert" | 03:15 |
| *** hdd has joined #openstack-infra | 03:17 | |
| *** xarses has joined #openstack-infra | 03:18 | |
| *** ashtokolov has quit IRC | 03:18 | |
| *** apuimedo_ has quit IRC | 03:20 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Minimize devstack installed packages for monasca job https://review.openstack.org/258749 | 03:20 |
| *** ashtokolov has joined #openstack-infra | 03:20 | |
| fungi | anybody know why we're trying to install etcd on trusty? | 03:23 |
| *** thorst has quit IRC | 03:23 | |
| fungi | E: Unable to locate package etcd | 03:23 |
| fungi | so sayeth apt-get | 03:23 |
| fungi | when trying to build this replacement server | 03:23 |
| *** unicell1 has quit IRC | 03:24 | |
| fungi | Error: /Stage[main]/Openstack_project::Thick_slave/Package[etcd]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install etcd' returned 100: Reading package lists... | 03:24 |
| *** apuimedo_ has joined #openstack-infra | 03:25 | |
| fungi | i sort of feel like i don't need yet another thing to dig into tonight, so i'm just going to hope it's benign | 03:26 |
| ianw | hmm, smells like dlm | 03:26 |
| *** woodster_ has quit IRC | 03:26 | |
| *** jamielennox is now known as jamielennox|away | 03:26 | |
| ianw | greghaynes: "If using the VHD output format you need to have a patched version of vhd-util installed for the image" so yeah ... you just gotta read :) | 03:26 |
| fungi | yeah, seems unnecessary on our release slave | 03:26 |
| *** links has joined #openstack-infra | 03:28 | |
| fungi | ugh, someone approved addition of etcd package on our slaves without confirming it actually existed in those distros?!? | 03:29 |
| fungi | i couldn't help but look | 03:30 |
| *** mgrosz has joined #openstack-infra | 03:31 | |
| lifeless | fungi: in that vein you'll *adore* the consul driver setup script for tooz tests | 03:31 |
| *** tzn has joined #openstack-infra | 03:31 | |
| lifeless | fungi: https://review.openstack.org/#/c/245362/17/setup-consul-env.sh | 03:31 |
| *** jamielennox|away is now known as jamielennox | 03:31 | |
| *** apuimedo_ has quit IRC | 03:32 | |
| *** mgrosz has quit IRC | 03:32 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Install etcd on nodes for tooz unit tests" https://review.openstack.org/258779 | 03:32 |
| fungi | yeah, i'm sure | 03:32 |
| fungi | not really in the mood i'm afraid | 03:33 |
| lifeless | fungi: can I help? | 03:33 |
| fungi | lifeless: nah, it's just been a longish process trying to get to the point i can retrigger the broken release jobs on a rebuilt slave that's no longer installing twine and requests from pypi | 03:34 |
| lifeless | pypi twine is bust? Sorry, I don't have context. | 03:34 |
| fungi | didn't want to go to sleep before it's settled and i can not worry about people pushing more tags that end up requiring more manual intervention | 03:35 |
| fungi | lifeless: twine is suffering from a bug in requests 2.9.0 at the momnt | 03:35 |
| fungi | so we're switching to distro packages for it | 03:35 |
| *** tzn has quit IRC | 03:35 | |
| *** docaedo has quit IRC | 03:36 | |
| fungi | don't really need bleeding-edge python lib releases injected into the middle of our release automation | 03:36 |
| fungi | when there are distro packages we can use | 03:36 |
| *** zhurong has joined #openstack-infra | 03:36 | |
| lifeless | yah | 03:36 |
| lifeless | we could of course pin it, but distro should be approximately good enough | 03:37 |
| *** apuimedo_ has joined #openstack-infra | 03:37 | |
| fungi | right, and pinning pypi package versions in puppet can be a bit fiddly | 03:39 |
| fungi | since the problem lib is being pulled in as a dependency of the tool we had puppet installing | 03:40 |
| *** apuimedo_ has quit IRC | 03:42 | |
| *** _denisra has joined #openstack-infra | 03:45 | |
| *** njirap has quit IRC | 03:47 | |
| fungi | okay, replacement slave is built now and checks out. once i give dns a few minutes to settle i'll get it registered in jenkins properly to replace the old one | 03:47 |
| *** docaedo has joined #openstack-infra | 03:47 | |
| *** apuimedo_ has joined #openstack-infra | 03:47 | |
| *** amotoki has joined #openstack-infra | 03:47 | |
| greghaynes | fungi: I am betting etcd is for tooz | 03:48 |
| greghaynes | ianw: yep, you need a patched vhd-util which you get the patch off some random ml | 03:49 |
| fungi | greghaynes: yes, the change for which i proposed the revert (258779) said as much | 03:49 |
| greghaynes | ianw: mordred made a ppa | 03:49 |
| *** EricGonczer_ has quit IRC | 03:49 | |
| ianw | greghaynes: yeah, doesn't help when everything i'm building on is fedora :) | 03:49 |
| fungi | greghaynes: i'm going to guess they only wanted to install it on fedora, or maybe just assumed it also existed in current ubuntu lts without actually checking | 03:50 |
| *** amotoki has quit IRC | 03:50 | |
| greghaynes | fungi: they also can install it on their own rather than by default | 03:51 |
| fungi | yeah | 03:52 |
| fungi | which is what we prefer at this point | 03:52 |
| *** ddieterly has quit IRC | 03:53 | |
| *** ddieterly has joined #openstack-infra | 03:54 | |
| *** apuimedo_ has quit IRC | 03:54 | |
| *** amotoki has joined #openstack-infra | 03:55 | |
| *** ddieterly has quit IRC | 03:58 | |
| *** apuimedo_ has joined #openstack-infra | 04:00 | |
| *** amotoki_ has joined #openstack-infra | 04:02 | |
| *** hdd has quit IRC | 04:02 | |
| *** apuimedo_ has quit IRC | 04:04 | |
| fungi | now i'm mildly concerned that the oslo.db tag ref i've reenqueued is just sitting there in the release pipeline getting no workers assigned to any of its jobs | 04:04 |
| fungi | 6 minutes and counting | 04:05 |
| *** amotoki has quit IRC | 04:05 | |
| bkero | any idle workers? | 04:08 |
| *** dguitarbite has quit IRC | 04:08 | |
| ianw | greghaynes: that patch for vhd-convert is ... interesting | 04:09 |
| *** apuimedo_ has joined #openstack-infra | 04:09 | |
| greghaynes | ianw: yep. It is extremely annoying that we need to use it. | 04:09 |
| fungi | yay, it worked! | 04:10 |
| fungi | just took a bit to pick up | 04:10 |
| fungi | http://logs.openstack.org/0b/0b68786e735ebabb2f58fc1d264c42ec6760e2d4/release/oslo.db-pypi-both-upload/2f8960b/console.html | 04:10 |
| fungi | jeblair: ^ distro packaged twine in use | 04:10 |
| fungi | i've got the django_openstack_auth release reenqueued now as well | 04:11 |
| greghaynes | w00t | 04:13 |
| fungi | i already took care of swauth when it was pointed out earlier today and manually completed its upload job, but from https://jenkins.openstack.org/computer/release.slave.openstack.org/builds it looks like we're also short uploads for releases of oslo.utils, gnocchi and tripleo-image-elements | 04:13 |
| fungi | i'll take care of those three before i call it a night | 04:15 |
| greghaynes | whoa, t-i-e released | 04:15 |
| *** apuimedo_ has quit IRC | 04:16 | |
| greghaynes | first time in 6mo. and they hit this ;) | 04:16 |
| *** EricGonczer_ has joined #openstack-infra | 04:16 | |
| *** EricGonczer_ has quit IRC | 04:17 | |
| fungi | yeah, just their luck | 04:17 |
| *** maishsk has quit IRC | 04:20 | |
| *** thorst has joined #openstack-infra | 04:20 | |
| *** abregman has joined #openstack-infra | 04:20 | |
| *** apuimedo_ has joined #openstack-infra | 04:21 | |
| fungi | as proven by the fact that retriggering the job also failed | 04:21 |
| *** abregman has quit IRC | 04:22 | |
| fungi | oh, their wheel builds are noncomforming | 04:22 |
| *** mahatic has joined #openstack-infra | 04:22 | |
| fungi | tripleo_image_elements-0.9.7-cp27-none-linux_x86_64.whl | 04:23 |
| *** maishsk has joined #openstack-infra | 04:23 | |
| *** chenli has quit IRC | 04:23 | |
| fungi | and the pypi upload job is looking for tripleo_image_elements-0.9.7-py2.py3-none-any.whl | 04:23 |
| fungi | presumably they've been broken for a while | 04:23 |
| *** mgrosz has joined #openstack-infra | 04:25 | |
| ianw | greghaynes: that *should* work for fedora ... let's see ... https://copr.fedoraproject.org/coprs/iwienand/blktap/build/148350/ | 04:25 |
| *** apuimedo_ has quit IRC | 04:25 | |
| *** piet has quit IRC | 04:26 | |
| fungi | gnocchi's tarball job takes a while to complete | 04:26 |
| *** mgrosz has quit IRC | 04:26 | |
| *** thorst has quit IRC | 04:28 | |
| *** apuimedo_ has joined #openstack-infra | 04:30 | |
| *** mriedem has joined #openstack-infra | 04:32 | |
| fungi | okay, pypi uploads have now completed for all releases broken by the requests release yesterday | 04:32 |
| fungi | i'm going to hit the hay--night all! | 04:33 |
| *** EricGonczer_ has joined #openstack-infra | 04:35 | |
| *** apuimedo_ has quit IRC | 04:35 | |
| olaph | 'night fungi! | 04:35 |
| *** Ramanjaneya has joined #openstack-infra | 04:36 | |
| *** chenli has joined #openstack-infra | 04:37 | |
| *** spzala has quit IRC | 04:37 | |
| *** apuimedo_ has joined #openstack-infra | 04:39 | |
| *** tsg has joined #openstack-infra | 04:41 | |
| *** EricGonczer_ has quit IRC | 04:43 | |
| *** tlian has quit IRC | 04:44 | |
| *** otter768 has joined #openstack-infra | 04:44 | |
| *** apuimedo_ has quit IRC | 04:44 | |
| *** hichihara has joined #openstack-infra | 04:44 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Publish per-branch astara-appliance images https://review.openstack.org/256076 | 04:46 |
| *** rguillebert has quit IRC | 04:47 | |
| *** gongysh_ has joined #openstack-infra | 04:48 | |
| *** zhurong has quit IRC | 04:48 | |
| *** otter768 has quit IRC | 04:49 | |
| *** apuimedo_ has joined #openstack-infra | 04:49 | |
| *** Nakato has quit IRC | 04:51 | |
| *** Nakato has joined #openstack-infra | 04:51 | |
| *** gongysh_ has quit IRC | 04:52 | |
| *** gsagie_ has joined #openstack-infra | 04:53 | |
| *** boris-42_ has quit IRC | 04:53 | |
| *** apuimedo_ has quit IRC | 04:53 | |
| gsagie_ | Where in the new gerrit look, can i see the line that fetch me the change? | 04:57 |
| gsagie_ | ohh i saw | 04:58 |
| *** apuimedo_ has joined #openstack-infra | 04:58 | |
| *** ramineni1 has joined #openstack-infra | 04:59 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Disable tempest plugin for stable branches https://review.openstack.org/258799 | 04:59 |
| *** gongysh_ has joined #openstack-infra | 05:01 | |
| *** ramineni has joined #openstack-infra | 05:02 | |
| *** ramineni1 has quit IRC | 05:03 | |
| *** ramineni1 has joined #openstack-infra | 05:04 | |
| *** gus has quit IRC | 05:04 | |
| *** zhurong has joined #openstack-infra | 05:05 | |
| *** ramineni_ has joined #openstack-infra | 05:06 | |
| *** ramineni has quit IRC | 05:07 | |
| *** mtreinish has quit IRC | 05:08 | |
| *** bpokorny has joined #openstack-infra | 05:08 | |
| *** ramineni1 has quit IRC | 05:08 | |
| *** apuimedo_ has quit IRC | 05:10 | |
| adam_g | gsagie, where did you find it? im lookin for the copy-able line for a quick local cherry-pick, with no luck | 05:10 |
| adam_g | oh nvm. there it is, in that drop-down :p | 05:11 |
| *** ningy has quit IRC | 05:14 | |
| *** apuimedo_ has joined #openstack-infra | 05:15 | |
| *** masber has quit IRC | 05:15 | |
| *** mtreinish has joined #openstack-infra | 05:17 | |
| *** hdd has joined #openstack-infra | 05:18 | |
| *** apuimedo_ has quit IRC | 05:20 | |
| *** signed8bit has joined #openstack-infra | 05:20 | |
| *** maishsk has quit IRC | 05:20 | |
| *** gsagie_ has quit IRC | 05:22 | |
| *** apuimedo_ has joined #openstack-infra | 05:25 | |
| *** thorst has joined #openstack-infra | 05:25 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: test https://review.openstack.org/258805 | 05:27 |
| *** tsg has quit IRC | 05:27 | |
| *** apuimedo_ has quit IRC | 05:29 | |
| *** mtanino has quit IRC | 05:30 | |
| *** yonglihe has quit IRC | 05:31 | |
| *** thorst has quit IRC | 05:33 | |
| *** apuimedo_ has joined #openstack-infra | 05:35 | |
| *** MarkAtwood has joined #openstack-infra | 05:36 | |
| *** apuimedo has quit IRC | 05:36 | |
| *** maishsk has joined #openstack-infra | 05:37 | |
| *** apuimedo has joined #openstack-infra | 05:37 | |
| *** mgrosz has joined #openstack-infra | 05:38 | |
| *** maishsk has quit IRC | 05:38 | |
| *** apuimedo_ has quit IRC | 05:39 | |
| *** abregman has joined #openstack-infra | 05:42 | |
| *** gongysh_ has quit IRC | 05:43 | |
| mgrosz | Good Morning | 05:44 |
| *** apuimedo has quit IRC | 05:44 | |
| *** apuimedo has joined #openstack-infra | 05:44 | |
| *** apuimedo_ has joined #openstack-infra | 05:45 | |
| *** apuimedo has quit IRC | 05:48 | |
| *** bpokorny has quit IRC | 05:49 | |
| *** gus has joined #openstack-infra | 05:49 | |
| *** piet has joined #openstack-infra | 05:50 | |
| *** apuimedo_ has quit IRC | 05:50 | |
| *** apuimedo has joined #openstack-infra | 05:51 | |
| *** jerryz has quit IRC | 05:52 | |
| *** apuimedo_ has joined #openstack-infra | 05:54 | |
| *** apuimedo has quit IRC | 05:55 | |
| *** apuimedo has joined #openstack-infra | 05:56 | |
| *** camunoz has quit IRC | 05:57 | |
| *** apuimedo_ has quit IRC | 05:58 | |
| *** aeng has joined #openstack-infra | 06:00 | |
| *** harlowja_at_home has joined #openstack-infra | 06:01 | |
| *** apuimedo_ has joined #openstack-infra | 06:03 | |
| *** apuimedo has quit IRC | 06:03 | |
| *** apuimedo has joined #openstack-infra | 06:04 | |
| *** piet has quit IRC | 06:04 | |
| *** harlowja_still_a has joined #openstack-infra | 06:05 | |
| *** apuimedo_ has quit IRC | 06:08 | |
| *** aeng has quit IRC | 06:08 | |
| *** harlowja_at_home has quit IRC | 06:08 | |
| *** apuimedo has quit IRC | 06:09 | |
| *** camunoz has joined #openstack-infra | 06:09 | |
| *** jswarren_ has joined #openstack-infra | 06:09 | |
| *** piet has joined #openstack-infra | 06:09 | |
| *** apuimedo has joined #openstack-infra | 06:09 | |
| *** asselin_ has quit IRC | 06:10 | |
| *** ryanpetrello has quit IRC | 06:10 | |
| *** asselin_ has joined #openstack-infra | 06:11 | |
| *** davidlenwell has quit IRC | 06:12 | |
| *** jswarren has quit IRC | 06:12 | |
| *** xarses has quit IRC | 06:13 | |
| *** xarses has joined #openstack-infra | 06:14 | |
| *** davidlenwell has joined #openstack-infra | 06:14 | |
| *** apuimedo_ has joined #openstack-infra | 06:14 | |
| *** xarses has quit IRC | 06:16 | |
| *** calebb has quit IRC | 06:16 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 06:16 | |
| *** xarses has joined #openstack-infra | 06:16 | |
| *** calebb has joined #openstack-infra | 06:16 | |
| *** zhurong has quit IRC | 06:16 | |
| *** apuimedo has quit IRC | 06:16 | |
| *** xarses has quit IRC | 06:17 | |
| *** apuimedo has joined #openstack-infra | 06:17 | |
| *** xarses has joined #openstack-infra | 06:18 | |
| *** zhurong has joined #openstack-infra | 06:18 | |
| *** mudassirlatif has joined #openstack-infra | 06:18 | |
| *** xarses has quit IRC | 06:18 | |
| *** xarses has joined #openstack-infra | 06:19 | |
| *** apuimedo_ has quit IRC | 06:19 | |
| *** xarses has quit IRC | 06:19 | |
| *** ryanpetrello has joined #openstack-infra | 06:20 | |
| *** xarses has joined #openstack-infra | 06:20 | |
| *** gsagie_ has joined #openstack-infra | 06:20 | |
| *** xarses has quit IRC | 06:20 | |
| *** xarses has joined #openstack-infra | 06:20 | |
| gsagie_ | Hello, i am trying to use docker client (python binding) in the gate , but the connection is aborted: http://logs.openstack.org/79/258379/5/check/gate-kuryr-dsvm-fullstack-nv/f46ebdb/console.html (this works locally) is there anything inn the gate blocking this? and is there any solution? | 06:21 |
| *** piet has quit IRC | 06:22 | |
| *** apuimedo_ has joined #openstack-infra | 06:24 | |
| *** xarses has quit IRC | 06:24 | |
| *** xarses has joined #openstack-infra | 06:24 | |
| *** camunoz has quit IRC | 06:25 | |
| *** maishsk has joined #openstack-infra | 06:27 | |
| *** apuimedo_ has quit IRC | 06:28 | |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Add Dragonflow fullstack job https://review.openstack.org/258821 | 06:29 |
| *** calebb has quit IRC | 06:30 | |
| *** signed8bit_ZZZzz has quit IRC | 06:30 | |
| *** thorst has joined #openstack-infra | 06:31 | |
| mgrosz | Hey... I got lost with the new Gerrit UI... I have pushed a new patch set to https://review.openstack.org/#/c/255431/, number 18. For some reason I got Code-Review -1 from Iury Gregory Melo Ferreira - but there is no comment about that. I am waiting for more than 50 minutes for Jenkins to start.... Anyone? | 06:32 |
| mgrosz | I believe this guy is even sleeping at the moment (Brazil time) | 06:33 |
| mgrosz | Bug? | 06:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/258281 | 06:34 |
| *** apuimedo_ has joined #openstack-infra | 06:34 | |
| *** xarses has quit IRC | 06:34 | |
| *** xarses has joined #openstack-infra | 06:35 | |
| *** xarses has quit IRC | 06:36 | |
| *** xarses has joined #openstack-infra | 06:36 | |
| *** ramineni_ has quit IRC | 06:36 | |
| *** xarses has quit IRC | 06:37 | |
| *** jffische has quit IRC | 06:38 | |
| *** thorst has quit IRC | 06:38 | |
| *** hdd has quit IRC | 06:38 | |
| *** camunoz has joined #openstack-infra | 06:38 | |
| *** xarses has joined #openstack-infra | 06:38 | |
| *** maishsk has quit IRC | 06:40 | |
| *** maishsk_ has joined #openstack-infra | 06:40 | |
| *** calebb has joined #openstack-infra | 06:42 | |
| *** apuimedo_ has quit IRC | 06:43 | |
| *** otter768 has joined #openstack-infra | 06:45 | |
| *** apuimedo_ has joined #openstack-infra | 06:48 | |
| *** mgrosz has quit IRC | 06:49 | |
| *** camunoz has quit IRC | 06:49 | |
| *** otter768 has quit IRC | 06:50 | |
| *** apuimedo has quit IRC | 06:51 | |
| *** apuimedo has joined #openstack-infra | 06:51 | |
| *** aopincar has joined #openstack-infra | 06:52 | |
| *** apuimedo_ has quit IRC | 06:53 | |
| *** jerryz has joined #openstack-infra | 06:54 | |
| *** AJaeger has joined #openstack-infra | 06:57 | |
| *** maishsk_ has quit IRC | 06:57 | |
| *** apuimedo_ has joined #openstack-infra | 06:58 | |
| *** mgrosz has joined #openstack-infra | 06:58 | |
| *** isaacb has joined #openstack-infra | 07:00 | |
| *** apuimedo_ has quit IRC | 07:02 | |
| *** HeOS has quit IRC | 07:05 | |
| *** apuimedo has quit IRC | 07:05 | |
| *** apuimedo has joined #openstack-infra | 07:05 | |
| *** isaacb has quit IRC | 07:07 | |
| *** apuimedo_ has joined #openstack-infra | 07:08 | |
| *** apuimedo has quit IRC | 07:10 | |
| *** apuimedo has joined #openstack-infra | 07:11 | |
| *** apuimedo_ has quit IRC | 07:12 | |
| *** gsagie_ has quit IRC | 07:13 | |
| lifeless | sss | 07:13 |
| lifeless | bah, sorry | 07:13 |
| *** triggerwarning has quit IRC | 07:14 | |
| *** apuimedo has quit IRC | 07:15 | |
| *** apuimedo has joined #openstack-infra | 07:17 | |
| *** apuimedo_ has joined #openstack-infra | 07:18 | |
| *** apuimedo has quit IRC | 07:21 | |
| *** apuimedo_ has quit IRC | 07:23 | |
| AJaeger | jhesketh: please check https://review.openstack.org/#/c/258515/ - the commit message job fails since the script cannot be found. | 07:23 |
| *** apuimedo has joined #openstack-infra | 07:24 | |
| *** irenab has quit IRC | 07:24 | |
| mgrosz | AJaeger: Good Morning.Hey... I got lost with the new Gerrit UI... I have pushed a new patch set to https://review.openstack.org/#/c/255431/, number 18. For some reason I got Code-Review -1 from Iury Gregory Melo Ferreira - but there is no comment about that. I am waiting for more than 2 hours ago for Jenkins to start.... | 07:26 |
| mgrosz | Maybe a bug? | 07:27 |
| *** apuimedo_ has joined #openstack-infra | 07:28 | |
| *** apuimedo has quit IRC | 07:28 | |
| mgrosz | You can see in the history of the change that this guy Gregory last -1was a week ago. Since then he already gave +1... | 07:29 |
| *** apuimedo has joined #openstack-infra | 07:30 | |
| * mkoderer is also very confused about the new Gerrit UI | 07:30 | |
| *** Guest6859 has quit IRC | 07:32 | |
| *** apuimedo_ has quit IRC | 07:33 | |
| AJaeger | mgrosz: was your upload just a rebase or something like that? Then -1s stay... | 07:34 |
| *** thorst has joined #openstack-infra | 07:35 | |
| AJaeger | mgrosz: this looks strange indeed - but I can't help. | 07:35 |
| mgrosz | Just did what I use to... git commit amend and then git review. I am sure this guy is sleeping now (Brazil time) | 07:35 |
| AJaeger | zaro: are you still around and can have a look at 255431? | 07:36 |
| mgrosz | AJaeger: then how helps here? | 07:36 |
| *** chlong has quit IRC | 07:36 | |
| mgrosz | AJaeger: do you advice to send an email infra mailing list? | 07:36 |
| AJaeger | mgrosz: Our CI systems are quite busy, I see on http://status.openstack.org/zuul/ that your job is still in the queue... | 07:37 |
| AJaeger | mgrosz: either ask on openstack-infra mailing list or come back here in the US morning - or wait and in most cases infra-root reads the backscroll for 255431 | 07:37 |
| jhesketh | AJaeger: yep, that's because the nodes need to be rebuilt | 07:38 |
| jhesketh | that'll happen overnight sometime | 07:38 |
| *** apuimedo_ has joined #openstack-infra | 07:38 | |
| AJaeger | mgrosz: you can approve a change even if there's a -1 review comment... | 07:38 |
| jhesketh | *node images | 07:38 |
| AJaeger | jhesketh: ah, that problem again. Thanks | 07:38 |
| jhesketh | AJaeger, mgrosz: so no jobs for 255431 is the problem? | 07:39 |
| mgrosz | AJaeger: I am saying that the -1 was there at first place... You don't see any entry in the history list nest to my latest "Uploaded patch set 18." entry. | 07:40 |
| jhesketh | mgrosz: looks like jenkins just reported on it | 07:41 |
| jhesketh | but yes, the -1 is odd | 07:41 |
| openstackgerrit | David Pursehouse proposed openstack-infra/lodgeit: Remove unused image file https://review.openstack.org/258849 | 07:41 |
| openstackgerrit | David Pursehouse proposed openstack-infra/lodgeit: Update i18n files to remove unused translations https://review.openstack.org/258850 | 07:41 |
| jhesketh | mgrosz: diffing patchset 18 to 13 there looks like nothing is changed | 07:41 |
| mgrosz | AJaeger: No... there should be several of jobs (I had 17 patches so far so I know that...).OK... saw the email from Jenkins coming... 2 hour and 10 minutes after submitting.. I think things are not working good.... | 07:42 |
| *** apuimedo has quit IRC | 07:42 | |
| jhesketh | mgrosz: so that vote has carried over because Gregorgy hasn't updated it along the line | 07:42 |
| jhesketh | actually I lie, I can see his newer vote | 07:43 |
| *** thorst has quit IRC | 07:43 | |
| *** apuimedo has joined #openstack-infra | 07:44 | |
| AJaeger | jhesketh: do you see a vote on patch set 17? | 07:46 |
| jhesketh | AJaeger: from who? | 07:46 |
| AJaeger | the -1 from Iury Gregory... | 07:47 |
| jhesketh | AJaeger: I see a +1 from lury | 07:47 |
| *** apuimedo_ has quit IRC | 07:47 | |
| AJaeger | but the summary on the top shows a -1 from him for me | 07:48 |
| * AJaeger is confused | 07:48 | |
| jhesketh | AJaeger: right, yes, I see that in the summary.. I see a +1 in the comment stream | 07:48 |
| *** apuimedo has quit IRC | 07:48 | |
| AJaeger | jhesketh: I see that on patch 17 but not on 18 which is the most recent | 07:48 |
| jhesketh | yeah I don't see anything on 18 | 07:49 |
| *** apuimedo has joined #openstack-infra | 07:50 | |
| jhesketh | AJaeger: so ps18 is identical to ps13 (although 17 and others were different) | 07:50 |
| jhesketh | so what's happened is that the -1 from ps13 has been applied | 07:50 |
| jhesketh | since the patches match | 07:50 |
| AJaeger | jhesketh: AH! Good spotting, that explains it... | 07:50 |
| jhesketh | lury will need to put a new vote in on ps18 now | 07:50 |
| jhesketh | and gerrit expands his -1 from ps13 in the comment section | 07:50 |
| *** apuimedo_ has joined #openstack-infra | 07:52 | |
| AJaeger | mgrosz: see the explanation above ^ | 07:52 |
| *** mgrosz has quit IRC | 07:54 | |
| *** mgrosz has joined #openstack-infra | 07:55 | |
| *** apuimedo_ has quit IRC | 07:57 | |
| *** k4n0 has joined #openstack-infra | 07:58 | |
| *** k4n0 has quit IRC | 07:58 | |
| *** apuimedo has quit IRC | 07:59 | |
| *** sflanigan has quit IRC | 07:59 | |
| *** apuimedo has joined #openstack-infra | 08:01 | |
| *** jerryz has quit IRC | 08:01 | |
| mgrosz | AJaeger: OK, now I understand. this guy, lury, has reverted his code he did here https://review.openstack.org/#/c/256570/. My patch 13 was not compliance with this changes, and 14 was. Now, he reverted his code here https://review.openstack.org/258518 , So now patch 13 should be good... I will need to wait for him when he wakes up... | 08:01 |
| *** apuimedo_ has joined #openstack-infra | 08:02 | |
| mgrosz | AJaeger: And also patch 18 should be good | 08:02 |
| *** harlowja_still_a has quit IRC | 08:03 | |
| *** jcoufal has joined #openstack-infra | 08:04 | |
| *** rvasilets___ has joined #openstack-infra | 08:05 | |
| *** zeih has joined #openstack-infra | 08:06 | |
| AJaeger | ;) | 08:06 |
| *** jtomasek has joined #openstack-infra | 08:06 | |
| *** scheuran has joined #openstack-infra | 08:08 | |
| *** apuimedo_ has quit IRC | 08:09 | |
| *** zeih has quit IRC | 08:10 | |
| *** Liuqing has joined #openstack-infra | 08:10 | |
| *** apuimedo has quit IRC | 08:10 | |
| *** apuimedo has joined #openstack-infra | 08:12 | |
| *** irenab has joined #openstack-infra | 08:12 | |
| *** unicell has joined #openstack-infra | 08:13 | |
| *** hashar has joined #openstack-infra | 08:14 | |
| *** apuimedo_ has joined #openstack-infra | 08:14 | |
| *** thorst has joined #openstack-infra | 08:16 | |
| *** unicell1 has joined #openstack-infra | 08:17 | |
| *** zeih has joined #openstack-infra | 08:18 | |
| *** unicell has quit IRC | 08:18 | |
| *** apuimedo_ has quit IRC | 08:18 | |
| *** apuimedo has quit IRC | 08:19 | |
| openstackgerrit | Denis Laxalde proposed openstack-infra/jenkins-job-builder: Fix docstring of downstream-ext publisher https://review.openstack.org/258863 | 08:19 |
| *** apuimedo has joined #openstack-infra | 08:19 | |
| *** shardy has joined #openstack-infra | 08:22 | |
| *** thorst has quit IRC | 08:23 | |
| *** zeih has quit IRC | 08:23 | |
| *** apuimedo_ has joined #openstack-infra | 08:23 | |
| *** yaume has joined #openstack-infra | 08:23 | |
| *** e0ne has joined #openstack-infra | 08:23 | |
| *** egonzalez has joined #openstack-infra | 08:23 | |
| *** apuimedo has quit IRC | 08:24 | |
| *** apuimedo has joined #openstack-infra | 08:25 | |
| *** mudassirlatif has quit IRC | 08:26 | |
| *** rcernin has joined #openstack-infra | 08:26 | |
| *** Hal has joined #openstack-infra | 08:27 | |
| *** Hal is now known as Guest84703 | 08:27 | |
| *** markus_z has joined #openstack-infra | 08:27 | |
| *** ifarkas has joined #openstack-infra | 08:29 | |
| *** apuimedo_ has quit IRC | 08:32 | |
| *** apuimedo_ has joined #openstack-infra | 08:37 | |
| *** apuimedo has quit IRC | 08:37 | |
| *** MarkAtwood has quit IRC | 08:38 | |
| *** apuimedo has joined #openstack-infra | 08:38 | |
| *** e0ne has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| *** rcernin has quit IRC | 08:40 | |
| *** rossella_s has joined #openstack-infra | 08:40 | |
| *** rcernin has joined #openstack-infra | 08:41 | |
| *** apuimedo_ has quit IRC | 08:42 | |
| *** _nadya_ has joined #openstack-infra | 08:43 | |
| *** otter768 has joined #openstack-infra | 08:46 | |
| *** apuimedo_ has joined #openstack-infra | 08:46 | |
| *** zeih has joined #openstack-infra | 08:48 | |
| *** FallenPegasus has joined #openstack-infra | 08:48 | |
| *** otter768 has quit IRC | 08:50 | |
| *** apuimedo has quit IRC | 08:52 | |
| *** zeih has quit IRC | 08:52 | |
| *** ashtokolov has quit IRC | 08:53 | |
| *** e0ne has quit IRC | 08:53 | |
| *** apuimedo_ has quit IRC | 08:53 | |
| *** apuimedo has joined #openstack-infra | 08:54 | |
| *** patrickeast has quit IRC | 08:56 | |
| *** ashtokolov has joined #openstack-infra | 08:57 | |
| *** apuimedo_ has joined #openstack-infra | 08:58 | |
| *** dingyichen has quit IRC | 08:59 | |
| *** larainema has quit IRC | 08:59 | |
| *** sshnaidm has joined #openstack-infra | 09:00 | |
| *** matrohon has joined #openstack-infra | 09:01 | |
| *** apuimedo_ has quit IRC | 09:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish per-branch astara-appliance images https://review.openstack.org/256076 | 09:03 |
| *** larainema has joined #openstack-infra | 09:03 | |
| *** apuimedo has quit IRC | 09:04 | |
| *** briancurtin has quit IRC | 09:04 | |
| *** ashtokolov has quit IRC | 09:04 | |
| *** apuimedo has joined #openstack-infra | 09:05 | |
| *** patrickeast has joined #openstack-infra | 09:06 | |
| *** fhubik has joined #openstack-infra | 09:06 | |
| *** amotoki_ has quit IRC | 09:06 | |
| *** apuimedo_ has joined #openstack-infra | 09:07 | |
| *** fhubik is now known as fhubik_brb | 09:08 | |
| *** fhubik_brb is now known as fhubik | 09:08 | |
| *** derekh has joined #openstack-infra | 09:09 | |
| *** mgrosz has quit IRC | 09:09 | |
| *** jistr has joined #openstack-infra | 09:09 | |
| *** HeOS has joined #openstack-infra | 09:09 | |
| *** fhubik has quit IRC | 09:09 | |
| *** zhurong has quit IRC | 09:10 | |
| *** ParsectiX has quit IRC | 09:11 | |
| *** ashtokolov has joined #openstack-infra | 09:11 | |
| *** ParsectiX has joined #openstack-infra | 09:12 | |
| *** briancurtin has joined #openstack-infra | 09:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Use separated ACL group for OSprofiler https://review.openstack.org/258182 | 09:14 |
| *** Guest84703 has quit IRC | 09:14 | |
| *** apuimedo_ has quit IRC | 09:14 | |
| *** patrickeast has quit IRC | 09:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-tempest-dsvm-neutron-dvr job voting in the check queue https://review.openstack.org/255325 | 09:15 |
| openstackgerrit | Merged openstack-infra/project-config: Skip dsvm jobs for ironic-python-agent if code doesn't change https://review.openstack.org/258735 | 09:16 |
| openstackgerrit | Merged openstack-infra/project-config: Make python34 job voting for Congress https://review.openstack.org/255621 | 09:16 |
| *** placko_ has joined #openstack-infra | 09:17 | |
| *** briancurtin has quit IRC | 09:17 | |
| *** betherly has quit IRC | 09:19 | |
| *** apuimedo_ has joined #openstack-infra | 09:19 | |
| *** mgrosz has joined #openstack-infra | 09:20 | |
| *** thorst has joined #openstack-infra | 09:20 | |
| *** ashtokolov has quit IRC | 09:21 | |
| *** rossella_s has quit IRC | 09:21 | |
| *** rossella_s has joined #openstack-infra | 09:23 | |
| *** apuimedo_ has quit IRC | 09:24 | |
| *** jlanoux has joined #openstack-infra | 09:25 | |
| *** zigo has quit IRC | 09:25 | |
| *** zigo_ has joined #openstack-infra | 09:26 | |
| *** FallenPegasus has quit IRC | 09:26 | |
| *** zeih has joined #openstack-infra | 09:26 | |
| *** ParsectiX has quit IRC | 09:27 | |
| *** thorst has quit IRC | 09:27 | |
| *** salv-orlando has joined #openstack-infra | 09:28 | |
| *** larainema has quit IRC | 09:28 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Correct typo in diskimage-builder job https://review.openstack.org/258896 | 09:29 |
| *** hichihara has quit IRC | 09:29 | |
| *** apuimedo_ has joined #openstack-infra | 09:29 | |
| *** Hal has joined #openstack-infra | 09:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Creation of the Coda Repo https://review.openstack.org/258194 | 09:31 |
| *** Hal is now known as Guest98282 | 09:31 | |
| *** chenli has quit IRC | 09:31 | |
| *** amotoki has joined #openstack-infra | 09:33 | |
| *** apuimedo has quit IRC | 09:33 | |
| *** placko_ has quit IRC | 09:33 | |
| *** apuimedo_ has quit IRC | 09:34 | |
| *** apuimedo has joined #openstack-infra | 09:34 | |
| *** ashtokolov has joined #openstack-infra | 09:37 | |
| *** betherly has joined #openstack-infra | 09:38 | |
| *** apuimedo_ has joined #openstack-infra | 09:39 | |
| *** patrickeast has joined #openstack-infra | 09:39 | |
| openstackgerrit | Merged openstack-infra/project-config: [sahara] job for sahara-dashboard integration tests https://review.openstack.org/255232 | 09:39 |
| *** larainema has joined #openstack-infra | 09:40 | |
| *** apuimedo_ has quit IRC | 09:43 | |
| *** amotoki has quit IRC | 09:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove check-requirements for Solar. https://review.openstack.org/258378 | 09:44 |
| *** briancurtin has joined #openstack-infra | 09:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix for sahara grenade job https://review.openstack.org/258504 | 09:45 |
| *** maishsk has joined #openstack-infra | 09:46 | |
| *** apuimedo has quit IRC | 09:46 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Build a Debian simple-init guest image when DIB changes merge https://review.openstack.org/254892 | 09:46 |
| *** apuimedo has joined #openstack-infra | 09:47 | |
| *** openstackgerrit has quit IRC | 09:47 | |
| *** openstackgerrit has joined #openstack-infra | 09:47 | |
| *** rguillebert has joined #openstack-infra | 09:49 | |
| *** apuimedo_ has joined #openstack-infra | 09:49 | |
| *** placko_ has joined #openstack-infra | 09:51 | |
| *** apuimedo has quit IRC | 09:52 | |
| *** apuimedo has joined #openstack-infra | 09:54 | |
| *** apuimedo_ has quit IRC | 09:54 | |
| *** e0ne has joined #openstack-infra | 09:55 | |
| *** sfinucan has joined #openstack-infra | 09:57 | |
| *** dizquierdo has joined #openstack-infra | 09:58 | |
| *** apuimedo_ has joined #openstack-infra | 09:59 | |
| *** mgrosz has quit IRC | 10:02 | |
| *** placko_ has quit IRC | 10:02 | |
| *** ihrachys has joined #openstack-infra | 10:03 | |
| gsagie | Hello, i am trying to use docker client (python binding) in the gate , but the connection is aborted: http://logs.openstack.org/79/258379/5/check/gate-kuryr-dsvm-fullstack-nv/f46ebdb/console.html (this works locally) is there anything inn the gate blocking this? and is there any solution? | 10:03 |
| *** apuimedo has quit IRC | 10:03 | |
| gsagie | requests.exceptions.ConnectionError: ('Connection aborted.', error(13, 'Permission denied')) | 10:04 |
| *** apuimedo_ has quit IRC | 10:04 | |
| *** apuimedo has joined #openstack-infra | 10:05 | |
| *** e0ne has quit IRC | 10:05 | |
| *** apuimedo_ has joined #openstack-infra | 10:08 | |
| *** bexelbie has joined #openstack-infra | 10:10 | |
| *** apuimedo has quit IRC | 10:12 | |
| *** Liuqing has quit IRC | 10:12 | |
| *** apuimedo has joined #openstack-infra | 10:12 | |
| *** electrofelix has joined #openstack-infra | 10:13 | |
| *** Liuqing has joined #openstack-infra | 10:13 | |
| *** apuimedo_ has quit IRC | 10:13 | |
| *** pigmej has left #openstack-infra | 10:15 | |
| *** apuimedo_ has joined #openstack-infra | 10:18 | |
| *** apuimedo has quit IRC | 10:19 | |
| *** armax has quit IRC | 10:20 | |
| *** apuimedo has joined #openstack-infra | 10:20 | |
| *** Qiming has quit IRC | 10:20 | |
| *** unicell1 has quit IRC | 10:20 | |
| *** unicell has joined #openstack-infra | 10:20 | |
| *** ihrachys has quit IRC | 10:21 | |
| *** apuimedo_ has quit IRC | 10:22 | |
| *** ihrachys has joined #openstack-infra | 10:22 | |
| *** ihrachys has quit IRC | 10:23 | |
| *** armax has joined #openstack-infra | 10:23 | |
| *** dizquierdo has quit IRC | 10:24 | |
| *** ihrachys has joined #openstack-infra | 10:24 | |
| *** Liuqing has quit IRC | 10:25 | |
| *** thorst has joined #openstack-infra | 10:26 | |
| *** apuimedo_ has joined #openstack-infra | 10:27 | |
| *** apuimedo has quit IRC | 10:29 | |
| *** mpavone has joined #openstack-infra | 10:30 | |
| *** apuimedo has joined #openstack-infra | 10:31 | |
| *** mpavone has quit IRC | 10:31 | |
| *** apuimedo_ has quit IRC | 10:32 | |
| *** thorst has quit IRC | 10:32 | |
| *** zhurong has joined #openstack-infra | 10:35 | |
| *** apuimedo has quit IRC | 10:35 | |
| *** apuimedo has joined #openstack-infra | 10:37 | |
| *** apuimedo_ has joined #openstack-infra | 10:37 | |
| *** apuimedo has quit IRC | 10:41 | |
| *** unicell has quit IRC | 10:42 | |
| *** apuimedo_ has quit IRC | 10:42 | |
| *** unicell has joined #openstack-infra | 10:42 | |
| *** apuimedo has joined #openstack-infra | 10:43 | |
| *** masber has joined #openstack-infra | 10:43 | |
| *** e0ne has joined #openstack-infra | 10:43 | |
| *** yolanda has quit IRC | 10:46 | |
| *** otter768 has joined #openstack-infra | 10:47 | |
| *** apuimedo_ has joined #openstack-infra | 10:47 | |
| *** apuimedo has quit IRC | 10:48 | |
| *** rfolco_ has quit IRC | 10:48 | |
| *** yamahata has joined #openstack-infra | 10:48 | |
| *** apuimedo has joined #openstack-infra | 10:49 | |
| *** rfolco has joined #openstack-infra | 10:50 | |
| *** Liuqing has joined #openstack-infra | 10:50 | |
| *** otter768 has quit IRC | 10:51 | |
| *** yolanda has joined #openstack-infra | 10:51 | |
| *** zhurong has quit IRC | 10:52 | |
| *** zhurong has joined #openstack-infra | 10:53 | |
| *** apuimedo has quit IRC | 10:54 | |
| *** apuimedo_ has quit IRC | 10:54 | |
| *** apuimedo has joined #openstack-infra | 10:55 | |
| *** hashar has quit IRC | 10:56 | |
| *** hashar has joined #openstack-infra | 10:57 | |
| *** apuimedo_ has joined #openstack-infra | 10:59 | |
| *** apuimedo_ has quit IRC | 11:04 | |
| *** apuimedo has quit IRC | 11:05 | |
| *** apuimedo has joined #openstack-infra | 11:05 | |
| *** aysyd has joined #openstack-infra | 11:07 | |
| *** dims has joined #openstack-infra | 11:07 | |
| *** lucas-dinner is now known as lucasagomes | 11:07 | |
| *** sambetts-afk is now known as sambetts | 11:08 | |
| *** Qiming has joined #openstack-infra | 11:08 | |
| *** salv-orlando has quit IRC | 11:08 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add UI for complex permissions on boards and worklists https://review.openstack.org/235476 | 11:08 |
| *** apuimedo_ has joined #openstack-infra | 11:09 | |
| *** salv-orlando has joined #openstack-infra | 11:11 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: DO NOT MERGE changes needed for delorean promotion https://review.openstack.org/258140 | 11:11 |
| *** salv-orlando has quit IRC | 11:12 | |
| *** salv-orlando has joined #openstack-infra | 11:12 | |
| *** apuimedo_ has quit IRC | 11:14 | |
| *** Liuqing has quit IRC | 11:14 | |
| *** salv-orl_ has joined #openstack-infra | 11:15 | |
| *** yamahata has quit IRC | 11:18 | |
| *** _nadya_ has quit IRC | 11:18 | |
| *** salv-orlando has quit IRC | 11:18 | |
| *** apuimedo_ has joined #openstack-infra | 11:19 | |
| *** mgrosz has joined #openstack-infra | 11:20 | |
| *** _nadya_ has joined #openstack-infra | 11:23 | |
| *** christiana_ has quit IRC | 11:23 | |
| *** apuimedo_ has quit IRC | 11:24 | |
| *** apuimedo has quit IRC | 11:26 | |
| *** apuimedo has joined #openstack-infra | 11:27 | |
| *** sfinucan has quit IRC | 11:28 | |
| *** sfinucan has joined #openstack-infra | 11:28 | |
| *** apuimedo_ has joined #openstack-infra | 11:29 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 11:30 |
| *** thorst has joined #openstack-infra | 11:31 | |
| *** apuimedo has quit IRC | 11:31 | |
| *** apuimedo has joined #openstack-infra | 11:32 | |
| freerunner | AJaeger: Hey! Are you here? kzaitsev_ws and me have a little question about caps in requirements. Is this possible to cap our muranoclient in stable/liberty requirements branch? Or we should to cap it directly in murano requirements? | 11:33 |
| *** christiana_ has joined #openstack-infra | 11:36 | |
| *** zeih has quit IRC | 11:37 | |
| *** apuimedo has quit IRC | 11:37 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:37 | |
| *** jlanoux has quit IRC | 11:37 | |
| *** apuimedo_ has quit IRC | 11:38 | |
| *** apuimedo has joined #openstack-infra | 11:38 | |
| *** thorst has quit IRC | 11:38 | |
| kzaitsev_mb | freerunner: probably gotta ping the guys later today | 11:39 |
| kzaitsev_mb | wonder what sahara folks did in the same situation | 11:39 |
| *** betherly has quit IRC | 11:39 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make kanban cards be the correct width https://review.openstack.org/256457 | 11:40 |
| *** claudiub has joined #openstack-infra | 11:41 | |
| *** mika has quit IRC | 11:41 | |
| *** larainema has quit IRC | 11:42 | |
| *** patrickeast has quit IRC | 11:42 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** apuimedo_ has joined #openstack-infra | 11:42 | |
| *** ashtokolov has quit IRC | 11:43 | |
| *** betherly has joined #openstack-infra | 11:44 | |
| *** apuimedo has quit IRC | 11:45 | |
| *** apuimedo has joined #openstack-infra | 11:45 | |
| *** mika has joined #openstack-infra | 11:46 | |
| *** mika has joined #openstack-infra | 11:46 | |
| *** mgrosz has quit IRC | 11:48 | |
| *** masber has quit IRC | 11:51 | |
| *** apuimedo_ has quit IRC | 11:52 | |
| *** mgrosz has joined #openstack-infra | 11:52 | |
| *** apuimedo has quit IRC | 11:52 | |
| *** jswarren_ is now known as jswarren | 11:53 | |
| *** briancurtin has quit IRC | 11:53 | |
| *** betherly has quit IRC | 11:53 | |
| *** apuimedo has joined #openstack-infra | 11:53 | |
| *** irenab has quit IRC | 11:56 | |
| *** ashtokolov has joined #openstack-infra | 11:56 | |
| lennyb_ | Hi, is someone using lastcomment-scoreboard ( by jogo )? It stopped working after gerrit upgrade. I did not debug it yet. | 11:57 |
| *** apuimedo_ has joined #openstack-infra | 11:57 | |
| *** abregman is now known as abregman|afk | 12:00 | |
| *** rcernin has quit IRC | 12:00 | |
| *** apuimedo_ has quit IRC | 12:02 | |
| *** zeih has joined #openstack-infra | 12:03 | |
| *** ashtokolov has quit IRC | 12:04 | |
| *** tzn has joined #openstack-infra | 12:05 | |
| *** julim has joined #openstack-infra | 12:05 | |
| *** patrickeast has joined #openstack-infra | 12:05 | |
| *** _nadya_ has quit IRC | 12:06 | |
| *** apuimedo_ has joined #openstack-infra | 12:08 | |
| *** zeih has quit IRC | 12:08 | |
| AJaeger | freerunner: now I'm here... | 12:11 |
| AJaeger | freerunner: for liberty you can cap directly in requirements. | 12:11 |
| *** apuimedo_ has quit IRC | 12:12 | |
| AJaeger | gsagie: does it need root access? | 12:12 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-diskimage_builder: Add lvm2 package to diskimage builder https://review.openstack.org/258974 | 12:14 |
| *** rcernin has joined #openstack-infra | 12:15 | |
| *** patrickeast has quit IRC | 12:15 | |
| *** u_glide1 has joined #openstack-infra | 12:16 | |
| *** apuimedo has quit IRC | 12:17 | |
| *** apuimedo has joined #openstack-infra | 12:17 | |
| u_glide1 | Hello folks! | 12:18 |
| u_glide1 | Could anyone help me to investigate why gate-manila-buildimage-generic job is NOT_REGISTERED ? | 12:18 |
| u_glide1 | #link https://review.openstack.org/#/c/232142/ | 12:18 |
| *** apuimedo_ has joined #openstack-infra | 12:18 | |
| *** tsg has joined #openstack-infra | 12:21 | |
| *** weshay_xchat has joined #openstack-infra | 12:21 | |
| *** dprince has joined #openstack-infra | 12:22 | |
| *** bexelbie has quit IRC | 12:22 | |
| *** [1]Thelo has joined #openstack-infra | 12:23 | |
| *** yamamoto has joined #openstack-infra | 12:23 | |
| *** maishsk has quit IRC | 12:24 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Bump the lower boundary of pysnmp to 4.2.3 https://review.openstack.org/246564 | 12:25 |
| *** Thelo has quit IRC | 12:25 | |
| *** [1]Thelo is now known as Thelo | 12:25 | |
| *** rvasilets___ has quit IRC | 12:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Bump the lower boundary of pysnmp to 4.2.3 https://review.openstack.org/246564 | 12:26 |
| *** jaypipes has joined #openstack-infra | 12:26 | |
| *** apuimedo has quit IRC | 12:26 | |
| *** patrickeast has joined #openstack-infra | 12:26 | |
| *** irenab has joined #openstack-infra | 12:28 | |
| *** briancurtin has joined #openstack-infra | 12:29 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Tidy up project groups states https://review.openstack.org/253049 | 12:29 |
| *** _nadya_ has joined #openstack-infra | 12:29 | |
| *** amuller has joined #openstack-infra | 12:30 | |
| ekarlso | would it be an idea to have the zuul status thing push SSE events to the browser instead of pulling ? | 12:30 |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add pylint to upper-constraints.txt https://review.openstack.org/258980 | 12:30 |
| *** apuimedo has joined #openstack-infra | 12:32 | |
| *** larainema has joined #openstack-infra | 12:38 | |
| *** apuimedo has quit IRC | 12:39 | |
| *** ashtokolov has joined #openstack-infra | 12:40 | |
| *** betherly has joined #openstack-infra | 12:41 | |
| *** smarcet has joined #openstack-infra | 12:43 | |
| *** amotoki has joined #openstack-infra | 12:43 | |
| *** Daisy has joined #openstack-infra | 12:44 | |
| *** apuimedo has joined #openstack-infra | 12:44 | |
| *** tsg has quit IRC | 12:44 | |
| *** amotoki has quit IRC | 12:46 | |
| *** tsg has joined #openstack-infra | 12:47 | |
| *** otter768 has joined #openstack-infra | 12:47 | |
| *** abregman|afk is now known as abregman | 12:48 | |
| *** apuimedo has quit IRC | 12:48 | |
| *** apuimedo_ has quit IRC | 12:49 | |
| *** smarcet has quit IRC | 12:50 | |
| *** apuimedo has joined #openstack-infra | 12:50 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Tidy up project groups states https://review.openstack.org/253049 | 12:51 |
| *** thorst has joined #openstack-infra | 12:52 | |
| *** otter768 has quit IRC | 12:52 | |
| *** apuimedo_ has joined #openstack-infra | 12:53 | |
| *** thorst_ has joined #openstack-infra | 12:54 | |
| *** baoli_ has joined #openstack-infra | 12:54 | |
| *** smarcet has joined #openstack-infra | 12:55 | |
| *** apuimedo has quit IRC | 12:55 | |
| *** apuimedo has joined #openstack-infra | 12:55 | |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Adds networking-hyperv to projects.txt https://review.openstack.org/258987 | 12:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Make flavor-creation failure non-fatal https://review.openstack.org/251552 | 12:56 |
| *** thorst has quit IRC | 12:56 | |
| *** links has quit IRC | 12:58 | |
| *** smarcet has quit IRC | 12:59 | |
| *** smarcet has joined #openstack-infra | 12:59 | |
| *** apuimedo_ has quit IRC | 13:00 | |
| *** bdemers has quit IRC | 13:01 | |
| *** jyuso1 has quit IRC | 13:01 | |
| *** apuimedo has quit IRC | 13:02 | |
| *** Daisy has quit IRC | 13:02 | |
| *** apuimedo has joined #openstack-infra | 13:03 | |
| *** Daisy has joined #openstack-infra | 13:03 | |
| *** salv-orl_ has quit IRC | 13:04 | |
| *** apuimedo_ has joined #openstack-infra | 13:06 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds check-requirements job in networking-hyperv https://review.openstack.org/258992 | 13:06 |
| *** yamamoto has quit IRC | 13:08 | |
| *** salv-orlando has joined #openstack-infra | 13:09 | |
| smarcet | fungi:clarkb: morning, i am gonna to merge https://review.openstack.org/#/c/198458/ | 13:10 |
| smarcet | OIDC implementation | 13:10 |
| *** zeih has joined #openstack-infra | 13:11 | |
| *** apuimedo has quit IRC | 13:12 | |
| *** odlD2 has quit IRC | 13:12 | |
| *** apuimedo_ has quit IRC | 13:13 | |
| *** apuimedo has joined #openstack-infra | 13:13 | |
| *** tsg has quit IRC | 13:14 | |
| *** rhallisey has joined #openstack-infra | 13:15 | |
| *** odlD2 has joined #openstack-infra | 13:16 | |
| *** fhubik has joined #openstack-infra | 13:17 | |
| *** briancurtin has quit IRC | 13:17 | |
| *** briancurtin has joined #openstack-infra | 13:17 | |
| *** apuimedo_ has joined #openstack-infra | 13:17 | |
| *** gordc has joined #openstack-infra | 13:19 | |
| *** fhubik is now known as fhubik_brb | 13:19 | |
| lucasagomes | the font size of the new gerrit interface is very small, there's nothing in settings to change it either | 13:20 |
| *** yamamoto has joined #openstack-infra | 13:20 | |
| * lucasagomes is using firefox | 13:21 | |
| *** rcernin has quit IRC | 13:21 | |
| *** ujuc_ has joined #openstack-infra | 13:21 | |
| *** flepied1 has quit IRC | 13:21 | |
| *** kgiusti has joined #openstack-infra | 13:22 | |
| *** vgridnev has joined #openstack-infra | 13:22 | |
| *** apuimedo has quit IRC | 13:23 | |
| *** briancurtin_ has joined #openstack-infra | 13:23 | |
| *** rossella_s has quit IRC | 13:23 | |
| *** alivigni has joined #openstack-infra | 13:23 | |
| *** apuimedo has joined #openstack-infra | 13:23 | |
| *** rossella_s has joined #openstack-infra | 13:23 | |
| *** rlandy has joined #openstack-infra | 13:26 | |
| *** apuimedo has quit IRC | 13:28 | |
| lucasagomes | and the ctrl+f javascript regex thing is so bad! | 13:28 |
| *** ashtokolov has quit IRC | 13:28 | |
| *** ashtokolov has joined #openstack-infra | 13:29 | |
| *** betherly has quit IRC | 13:29 | |
| *** betherly has joined #openstack-infra | 13:29 | |
| *** apuimedo_ has quit IRC | 13:29 | |
| *** apuimedo has joined #openstack-infra | 13:30 | |
| *** patrickeast has quit IRC | 13:30 | |
| *** patrickeast has joined #openstack-infra | 13:30 | |
| *** pradk has joined #openstack-infra | 13:31 | |
| *** ashtokolov_ has joined #openstack-infra | 13:32 | |
| *** betherly_ has joined #openstack-infra | 13:32 | |
| *** ayoung has joined #openstack-infra | 13:32 | |
| *** ashtokolov has quit IRC | 13:33 | |
| *** ashtokolov_ is now known as ashtokolov | 13:33 | |
| *** betherly has quit IRC | 13:33 | |
| *** betherly_ is now known as betherly | 13:33 | |
| *** links has joined #openstack-infra | 13:33 | |
| *** apuimedo_ has joined #openstack-infra | 13:34 | |
| *** mgrosz has quit IRC | 13:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Adds networking-hyperv to projects.txt https://review.openstack.org/258987 | 13:35 |
| *** flepied has joined #openstack-infra | 13:35 | |
| *** flepied1 has joined #openstack-infra | 13:36 | |
| *** trown|outttypeww is now known as trown | 13:37 | |
| *** apuimedo_ has quit IRC | 13:38 | |
| *** apuimedo has quit IRC | 13:39 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:39 | |
| *** flepied has quit IRC | 13:40 | |
| *** mgrosz has joined #openstack-infra | 13:40 | |
| *** apuimedo has joined #openstack-infra | 13:40 | |
| *** ashtokolov_ has joined #openstack-infra | 13:40 | |
| *** betherly_ has joined #openstack-infra | 13:41 | |
| *** Ramanjaneya has quit IRC | 13:42 | |
| *** AJaeger has quit IRC | 13:42 | |
| *** hdd has joined #openstack-infra | 13:44 | |
| *** apuimedo_ has joined #openstack-infra | 13:44 | |
| *** patrickeast_ has joined #openstack-infra | 13:45 | |
| *** fhubik_brb is now known as fhubik | 13:45 | |
| ttx | fungi: when up, could you investigate why the bandit 0.17.0 tag did apparently not result in a tarball build or Pypi upload ? | 13:45 |
| ttx | afaict the post job didn't run | 13:46 |
| *** yamamoto has quit IRC | 13:46 | |
| *** Daisy_ has joined #openstack-infra | 13:47 | |
| ttx | hmm, maybe some jobs were purged, the zuul graph looks funny around that hour | 13:47 |
| *** Daisy has quit IRC | 13:48 | |
| yolanda | infra-root, what is best way to remove a jenkins target from nodepool? for dropping a provider, there is that min-ready=-1 trick, but for dropping a jenkins master? | 13:50 |
| yolanda | (i need to do that downstream) | 13:50 |
| *** yamamoto has joined #openstack-infra | 13:50 | |
| *** aopincar has quit IRC | 13:53 | |
| *** tlian has joined #openstack-infra | 13:53 | |
| *** placko has joined #openstack-infra | 14:00 | |
| *** apuimedo_ has quit IRC | 14:00 | |
| *** lykinsbd has joined #openstack-infra | 14:01 | |
| openstackgerrit | Evgeniy L proposed openstack-infra/project-config: Create openstack/bareon-specs repository https://review.openstack.org/259026 | 14:02 |
| *** nmagnezi has joined #openstack-infra | 14:04 | |
| ujuc_ | hey #openstack-meeting bot is problem..... how to... | 14:04 |
| *** apuimedo_ has joined #openstack-infra | 14:05 | |
| *** mattt has joined #openstack-infra | 14:06 | |
| ujuc_ | ah, relived. :) | 14:08 |
| *** xyang1 has joined #openstack-infra | 14:09 | |
| *** hdd has quit IRC | 14:09 | |
| *** apuimedo_ has quit IRC | 14:10 | |
| *** apuimedo has quit IRC | 14:11 | |
| *** Daisy_ has quit IRC | 14:11 | |
| *** automagically has joined #openstack-infra | 14:12 | |
| *** apuimedo has joined #openstack-infra | 14:12 | |
| Sam-I-Am | anteaya: in #openstack its as if people sense when i wake up | 14:12 |
| *** jcoufal_ has joined #openstack-infra | 14:13 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/devstack-gate: Bump ironic test timeout to 900 https://review.openstack.org/259035 | 14:16 |
| *** fhubik is now known as fhubik_brb | 14:16 | |
| *** apuimedo_ has joined #openstack-infra | 14:16 | |
| *** jcoufal has quit IRC | 14:16 | |
| *** spzala has joined #openstack-infra | 14:16 | |
| *** lucasagomes is now known as lucas-hungry | 14:17 | |
| *** lykinsbd has quit IRC | 14:17 | |
| automagically | FYI - looks like post Gerrit upgrade, the https://review.openstack.org/#/settings/new-agreement fails with a 500 error “Cannot store contact information" | 14:17 |
| automagically | Unclear if this a good place to report that or if another location is preferable | 14:18 |
| *** jlanoux has joined #openstack-infra | 14:19 | |
| *** markvoelker has quit IRC | 14:21 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: narrow the 3 columns progressively with screen width https://review.openstack.org/258744 | 14:23 |
| *** apuimedo has quit IRC | 14:24 | |
| bgmccollum | anyone seen this before? -- ERROR: Publisher be.certipost.hudson.plugin.SCPRepositoryPublisher aborted due to exception | 14:24 |
| *** dkranz has joined #openstack-infra | 14:24 | |
| *** apuimedo has joined #openstack-infra | 14:25 | |
| *** bhunter71 has quit IRC | 14:26 | |
| *** apuimedo_ has quit IRC | 14:27 | |
| sdague | the inline fix is pretty nice for gerrit | 14:28 |
| sdague | especially for simple stuff | 14:28 |
| *** markvoelker has joined #openstack-infra | 14:29 | |
| *** vkramskikh has joined #openstack-infra | 14:29 | |
| *** peristeri has joined #openstack-infra | 14:30 | |
| *** ilyashakhat has joined #openstack-infra | 14:30 | |
| *** EricGonczer_ has joined #openstack-infra | 14:31 | |
| *** ddieterly has joined #openstack-infra | 14:31 | |
| *** apuimedo_ has joined #openstack-infra | 14:32 | |
| *** bdemers has joined #openstack-infra | 14:32 | |
| *** zhurong has quit IRC | 14:34 | |
| *** thiagop has joined #openstack-infra | 14:34 | |
| *** piet has joined #openstack-infra | 14:34 | |
| *** placko has quit IRC | 14:35 | |
| *** ociuhandu has joined #openstack-infra | 14:35 | |
| *** fhubik_brb is now known as fhubik | 14:35 | |
| *** zhurong has joined #openstack-infra | 14:37 | |
| dansmith | does anyone have a pointer to something like a doc that explains what query string differences there may be in the new version of gerrit? | 14:40 |
| dansmith | the query I've been using for a couple years is suddenly giving me a 400 | 14:40 |
| sdague | dansmith: what is your query string | 14:40 |
| *** jlanoux has quit IRC | 14:40 | |
| dansmith | ((owner:danms OR is:starred) OR (project:openstack/nova)) AND status:open | 14:40 |
| sdague | https://review.openstack.org/Documentation/user-search.html | 14:40 |
| *** pradk has quit IRC | 14:41 | |
| sdague | is there an encoding issue | 14:41 |
| *** pradk has joined #openstack-infra | 14:41 | |
| dansmith | well, that's the pre-encoded version | 14:41 |
| dansmith | https://review.openstack.org/changes/?q=%28%28owner%3Adanms%20OR%20is%3Astarred%29%20OR%20%28project%3Aopenstack/nova%29%29%20AND%20status%3Aopen | 14:41 |
| *** apuimedo_ has quit IRC | 14:41 | |
| *** apuimedo has quit IRC | 14:41 | |
| *** ChuckC has joined #openstack-infra | 14:42 | |
| sdague | hmmm,,, that seems to work for me, though you need to be signed in | 14:42 |
| dansmith | hmm, well, I cut part of it off: | 14:43 |
| dansmith | https://review.openstack.org/changes/?q=%28%28owner%3Adanms%20OR%20is%3Astarred%29%20OR%20%28project%3A%27openstack/nova%27%29%29%20AND%20status%3Aopen&o=DETAILED_ACCOUNTS | 14:43 |
| *** apuimedo has joined #openstack-infra | 14:43 | |
| dansmith | but the link above does not work for me in a browser | 14:43 |
| dansmith | where I am signed in | 14:43 |
| *** dizquierdo has joined #openstack-infra | 14:43 | |
| sdague | &o=DETAILED_ACCOUNTS ? | 14:43 |
| dansmith | er, wait | 14:43 |
| *** hdd has joined #openstack-infra | 14:44 | |
| dansmith | I dunno, I had to add that some long time ago for some reason I guess | 14:44 |
| dansmith | okay without that it does work for me in a browser if I'm signed in, but I'm doing http auth with the http password in dash | 14:44 |
| dansmith | maybe my password changed with the upgrade? | 14:45 |
| *** redixin has left #openstack-infra | 14:45 | |
| dansmith | nope, didn't change | 14:45 |
| *** mgrosz has quit IRC | 14:45 | |
| jroll | dansmith: this works for me https://review.openstack.org/#/q/%28%28owner:danms++OR+is:starred%29+OR+%28project:openstack/nova%29%29+AND+status:open | 14:46 |
| *** yamamoto has quit IRC | 14:46 | |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| jroll | is the result of pasting your query in | 14:46 |
| dansmith | jroll: yeah, me too, but not with the http auth and json output for some reason | 14:46 |
| jroll | dansmith: ohhhhh. | 14:46 |
| jroll | right, because # is a browser thing | 14:47 |
| *** apuimedo_ has joined #openstack-infra | 14:47 | |
| jroll | this seems to make my browser give me json https://review.openstack.org/changes/?q=%28%28owner:danms+OR+is:starred%29+OR+%28project:openstack/nova%29%29+AND+status:open | 14:47 |
| jroll | try that with http auth? | 14:48 |
| *** jlanoux has joined #openstack-infra | 14:48 | |
| *** eharney has joined #openstack-infra | 14:48 | |
| dansmith | that's what I'm running except %20 for spaces instead of + | 14:48 |
| *** otter768 has joined #openstack-infra | 14:48 | |
| gsagie | Hello, i am trying to use docker client (python binding) in the gate , but the connection is aborted: http://logs.openstack.org/79/258379/5/check/gate-kuryr-dsvm-fullstack-nv/f46ebdb/console.html (this works locally) is there anything inn the gate blocking this? and is there any solution? | 14:49 |
| gsagie | am i doing something wrong? | 14:49 |
| jroll | dansmith: well, does it work with + ? :P | 14:50 |
| dansmith | jroll: well, I'm using urllib.quote, but I guess I'll try | 14:50 |
| * jroll suddenly wonders if gertty has a dashboards feature | 14:51 | |
| dansmith | so, if I try to load your line with curl and my http auth creds, I get "400 not signed in" | 14:51 |
| dansmith | so something seems weird with the http auth | 14:52 |
| *** asselin has quit IRC | 14:52 | |
| *** otter768 has quit IRC | 14:53 | |
| *** ashtokolov_ has quit IRC | 14:54 | |
| *** betherly_ has quit IRC | 14:54 | |
| *** patrickeast_ has quit IRC | 14:54 | |
| *** bogdando has quit IRC | 14:55 | |
| sdague | ummm... what happened to all our test nodes? | 14:55 |
| sdague | fungi / yolanda ? | 14:55 |
| sdague | nodepool seems empty | 14:56 |
| yolanda | sdague, don't know... let me see | 14:56 |
| gsagie | the gate might be blocking docker client somehow? | 14:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add ansible-role-functional-jobs-(centos|trusty)-nv https://review.openstack.org/258682 | 14:56 |
| *** apuimedo_ has quit IRC | 14:56 | |
| *** fhubik is now known as fhubik_brb | 14:56 | |
| sdague | yolanda: seems like zuul is just wedged as well | 14:57 |
| sdague | we've got a completely completed gate queue just sitting there | 14:57 |
| yolanda | checking logs | 14:58 |
| yolanda | looks more a zuul problem than nodepool, nodepool shows no queue | 14:58 |
| *** tjones has joined #openstack-infra | 14:58 | |
| yolanda | sdague... 2015-12-17 14:59:30,305 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information. | 14:59 |
| pabelanger | question about signal, if you kill -1 a process that doesn't support SIGHUP, what is the expected outcome? I would assume, the process stops since you killed it | 15:00 |
| yolanda | infra-root, you know about this error? | 15:00 |
| *** odlD2 has quit IRC | 15:00 | |
| *** asselin has joined #openstack-infra | 15:00 | |
| *** briancurtin_ has quit IRC | 15:01 | |
| *** nmagnezi has quit IRC | 15:01 | |
| *** apuimedo_ has joined #openstack-infra | 15:01 | |
| *** dtardivel has joined #openstack-infra | 15:01 | |
| *** dslev has joined #openstack-infra | 15:01 | |
| SotK | is there a way to see more than the 10 most recent reports in puppetboard? | 15:01 |
| pabelanger | SotK: you need to drill down to the specific node | 15:02 |
| pabelanger | eg: http://puppetdb.openstack.org/node/afs01.dfw.openstack.org | 15:03 |
| SotK | pabelanger: I'm looking at http://puppetboard.openstack.org/node/storyboard.openstack.org , but it still only shows the most recent 10 reports | 15:03 |
| pabelanger | SotK: Oh, sorry. I missed the more then part | 15:03 |
| pabelanger | ya, it only displays 10 | 15:03 |
| pabelanger | not sure why | 15:03 |
| yolanda | sdague, how long has it been happening, you know? | 15:04 |
| SotK | heh, thanks anyway :) | 15:04 |
| sdague | yolanda: no idea, I just see that it's all jammed up | 15:04 |
| *** HeOS has quit IRC | 15:05 | |
| *** rvasilets___ has joined #openstack-infra | 15:05 | |
| *** piet has quit IRC | 15:05 | |
| sdague | my guess is there are new events from gerrit after the upgrade, and zuul doesn't understand some of them | 15:06 |
| *** automagically_ has joined #openstack-infra | 15:06 | |
| *** automagically has quit IRC | 15:06 | |
| *** automagically_ is now known as automagically | 15:06 | |
| yolanda | i see several errors in zuul, i don't know if they are critical right now | 15:06 |
| yolanda | several errors about replication | 15:06 |
| *** Mmike has quit IRC | 15:06 | |
| sdague | ok, well nothing is progressing in the gate | 15:07 |
| sdague | http://status.openstack.org/zuul/ | 15:07 |
| yolanda | yes, saw that :( | 15:07 |
| *** Mmike has joined #openstack-infra | 15:08 | |
| SotK | fungi: is there any chance you could provide me with some storyboard logs? | 15:08 |
| *** Mmike has quit IRC | 15:09 | |
| *** Mmike has joined #openstack-infra | 15:09 | |
| pabelanger | yolanda: sdague: Here are the 2 I am seeing: | 15:09 |
| pabelanger | 2015-12-17 08:10:21,703 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information. | 15:09 |
| *** yaume_ has joined #openstack-infra | 15:09 | |
| pabelanger | 2015-12-17 08:10:49,191 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information. | 15:09 |
| yolanda | ye | 15:09 |
| yolanda | ndepool queue is not receiving any demand | 15:09 |
| yolanda | zuul seems to be receiving comment added, patchset events.. but translating that to demand | 15:10 |
| *** apuimedo_ has quit IRC | 15:10 | |
| *** vgridnev has quit IRC | 15:10 | |
| *** apuimedo has quit IRC | 15:11 | |
| *** apuimedo has joined #openstack-infra | 15:12 | |
| *** yaume has quit IRC | 15:12 | |
| *** odlD2 has joined #openstack-infra | 15:13 | |
| *** lucas-hungry is now known as lucasagomes | 15:13 | |
| *** piet has joined #openstack-infra | 15:13 | |
| yolanda | sdague, i hope that this can be solved without downgrading gerrit, but it looks so bad now | 15:14 |
| *** apuimedo_ has joined #openstack-infra | 15:16 | |
| *** ayoung has quit IRC | 15:17 | |
| *** apuimedo has quit IRC | 15:19 | |
| *** apuimedo has joined #openstack-infra | 15:20 | |
| *** xarses_ has joined #openstack-infra | 15:20 | |
| *** xarses_ has quit IRC | 15:20 | |
| *** xarses_ has joined #openstack-infra | 15:21 | |
| *** apuimedo_ has quit IRC | 15:21 | |
| *** xarses has quit IRC | 15:21 | |
| *** jlanoux has quit IRC | 15:21 | |
| *** hashar is now known as hasharAway | 15:22 | |
| *** kzaitsev_mb has quit IRC | 15:23 | |
| fungi | ttx: i'll have a look. at the bandit release issue once i'm done looking into nodepool. we had some release issues related to pypi uploads but i thought i had that solved before i went to bed (it was exercised by rather a number of rerun release jobs) | 15:24 |
| *** jlanoux has joined #openstack-infra | 15:24 | |
| sdague | yolanda: who do we need to look into the issue deeper? | 15:24 |
| yolanda | hi fungi, i restarted nodepool and now i don't see activity in its logs | 15:24 |
| fungi | yolanda: sdague: nodepool doesn't know about gerrit, so in theory this is unrelated. usually it's a sign that nodepool has lost its connection to zuul's gearman server | 15:24 |
| yolanda | fungi, i thought that and i rebooted, but it doesn't come up normally | 15:24 |
| yolanda | i was thiking in running my command line... | 15:25 |
| ttx | fungi: the release was yesterday so maybe that was the same issue | 15:25 |
| yolanda | doh, not rebooted, but restarted... bad words | 15:25 |
| fungi | ttx: oh, well it wasn't in the list of pypi upload jobs that got triggered yesterday, so probably it died earlier, like during tarball/wheel generation | 15:25 |
| yolanda | fungi, are you looking at it? | 15:26 |
| yolanda | i don't want to overlap | 15:26 |
| fungi | yolanda: ahh, reading more of scrollback yes it's possible this is a zuul bug with it receiving additional gerrit events it doesn't recognize | 15:26 |
| fungi | yolanda: starting to look, yes | 15:26 |
| *** apuimedo_ has joined #openstack-infra | 15:27 | |
| yolanda | fungi, i'm afraid of some nodepool problem, that unactivity in logs is bad | 15:27 |
| *** daemontool has joined #openstack-infra | 15:27 | |
| fungi | yolanda: well, nodepoold isn't running. did you stop it? | 15:27 |
| yolanda | was going to ask you the same | 15:28 |
| fungi | heh, then perhaps nodepool crashed? | 15:28 |
| *** BobBall has quit IRC | 15:28 | |
| yolanda | fungi yep, but i restarted a pair of times | 15:28 |
| yolanda | and after creating jenkins masters, there is no activity in logs | 15:28 |
| u_glide1 | Hello folks! Could anyone help me to investigate why gate-manila-buildimage-generic job is NOT_REGISTERED ? #link https://review.openstack.org/#/c/232142/ | 15:29 |
| fungi | dmesg has rather a lot of errors about buffer issues on the loopback | 15:29 |
| fungi | ahh, that looks like dib-related errors in dmesg maybe | 15:29 |
| yolanda | checked at disk , 96% of disk space usage, but we are not full | 15:29 |
| fungi | we seem to get those daily, so probably unrelated | 15:30 |
| fungi | nodepoold is running again. did you start it? | 15:30 |
| yolanda | yep | 15:30 |
| yolanda | but look at debug.log | 15:30 |
| yolanda | no activity | 15:30 |
| *** bdemers has quit IRC | 15:30 | |
| yolanda | process is there, but at least... doesn't log | 15:30 |
| yolanda | we can strace | 15:30 |
| *** angdraug has joined #openstack-infra | 15:31 | |
| fungi | perhaps there have been changes merged to nodepool since the last time we restarted. i'll see about rolling back the install a bit | 15:31 |
| yolanda | ok | 15:32 |
| yolanda | fungi, changes in nodepool are not normally applied after they land? i mean, restart nodepool after a change that can affects it lands | 15:32 |
| *** jaosorior has joined #openstack-infra | 15:34 | |
| fungi | yolanda: no, we manually restart nodepoold when convenient | 15:34 |
| fungi | so i'm not sure what version of nodepool was running yesterday | 15:34 |
| fungi | presumably something crashed it and now it's trying to run a newer broken version (at least that's one theory i have) | 15:35 |
| fungi | i'm going to try rolling back to 39b306ac3b4cdd0e7982527ec2d216ef5855039c from a couple weeks ago and see what happens | 15:35 |
| yolanda | strace looks bad.. | 15:35 |
| yolanda | strace -p 2414 | 15:35 |
| yolanda | Process 2414 attached | 15:35 |
| yolanda | pause( | 15:35 |
| yolanda | ok | 15:35 |
| *** bdemers has joined #openstack-infra | 15:35 | |
| *** mtanino has joined #openstack-infra | 15:36 | |
| *** bhunter71 has joined #openstack-infra | 15:36 | |
| fungi | okay, i've restarted it at that commit installed | 15:37 |
| *** sridhar_ram has joined #openstack-infra | 15:37 | |
| yolanda | fungi... i saw some activity in zuu logs, with jobs merged | 15:37 |
| yolanda | still no activity in debug.log, however | 15:37 |
| fungi | looks like it's possibly still not logging anything after the jenkins masters | 15:37 |
| yolanda | it should log stuff, even just the regular events | 15:38 |
| *** zeih has quit IRC | 15:39 | |
| fungi | next theory is that the requests 2.9.0 release from a couple days ago is to blame. i'm going to put nodepool back to origin/master but downgrade requests to 2.8.1 | 15:39 |
| yolanda | but zuul log shouws better now | 15:39 |
| bgmccollum | so, anecdotal evidence...but i have a change in the gate check that failed...but hasn't popped off, nor updated gerrit for almost 3 hours | 15:39 |
| fungi | well, it looks like nodepool is actually building nodes based on the recent ages of some in its node table | 15:39 |
| sdague | bgmccollum: yes, that's what fungi and yolanda are looking at | 15:39 |
| fungi | but no idea why it's not logging anything | 15:40 |
| bgmccollum | sdague: cool... | 15:40 |
| sdague | fungi: so, is this actually a nodepool issue or a zuul issue? | 15:40 |
| fungi | well, i still haven't stepped back and looked at the big picture either... when you say the change has failed, did it run all jobs? | 15:40 |
| sdague | because the nodepool graph actually just looks like it's not being fed any work | 15:40 |
| fungi | i just started looking at nodepool because that's where yolanda ended up | 15:40 |
| sdague | ok, sorry, I started with that | 15:41 |
| sdague | sometime around 3 hours ago, we stopped processing work | 15:41 |
| bgmccollum | fungi: yes...all have run | 15:41 |
| fungi | that sounds like a zuul problem | 15:41 |
| fungi | i'm going to put nodepool back like i found it, step back and start troubleshooting this from the top | 15:41 |
| openstackgerrit | Evgeniy L proposed openstack-infra/project-config: Create openstack/bareon-specs repository https://review.openstack.org/259026 | 15:42 |
| fungi | going to move to a different computer where i have a better view of everything. brb | 15:43 |
| *** placko has joined #openstack-infra | 15:43 | |
| *** zhurong has quit IRC | 15:45 | |
| yolanda | fungi, unrelated ... but i saw logs in nodepool saying jenkins02 is offline. And i cannot reach it. Is that known? | 15:45 |
| *** AJaeger has joined #openstack-infra | 15:45 | |
| *** fhubik_brb is now known as fhubik | 15:45 | |
| *** ningy has joined #openstack-infra | 15:46 | |
| fungi | yolanda: not known to me, jenkins02 likely crashed and needs to be cleaned up/restarted | 15:47 |
| fungi | if you can't reach it via ssh? then maybe kernel panic or something | 15:47 |
| yolanda | by ui | 15:47 |
| bgmccollum | one other bit...not sure if related, jenkins scp publisher exception -- http://paste.openstack.org/show/482198/ | 15:47 |
| yolanda | i can reach by ssh | 15:47 |
| dims | fungi : thanks for the re-run of those 2 pypi uploads last night! | 15:47 |
| fungi | yolanda: yeah, then you'll probably find evidence of the oom killer's work in dmesg | 15:47 |
| fungi | dims: sure thing. apparently oslo.utils release was bitten by it too and you were unaware, but i retriggered that as well (along with gnocchi) | 15:48 |
| dims | fungi : ah. thanks! | 15:48 |
| *** links has quit IRC | 15:50 | |
| *** mriedem has quit IRC | 15:50 | |
| *** cfarquhar has quit IRC | 15:51 | |
| fungi | okay, so zuul load in the nodepool graphs looks like it started to trail off around 14:00 utc | 15:51 |
| fungi | image updates start at 14:14 but that could simply be coincidence | 15:51 |
| *** xarses_ is now known as xarses | 15:51 | |
| *** ddieterly has quit IRC | 15:51 | |
| fungi | also the fact that nodepoold isn't writing to its debug log could simply be an unrelated nuisance blinding us to the nodepool end of the problem | 15:52 |
| evgenyl | AJaeger: hi, thanks for the review https://review.openstack.org/#/c/259026/ , I have a small question, do I need to additionally define "bareon group" somewhere? Or it's enough to write "group: - bareon" for the project? | 15:52 |
| *** apuimedo_ has quit IRC | 15:52 | |
| fungi | however changes sitting in pipelines unserviced by zuul even though their jobs are complete makes me think the issue is zuul side | 15:52 |
| fungi | making image updates timing and possibly lack of nodepool debug logging seem much more like a coincidence | 15:53 |
| yolanda | fungi, last thing i saw in nodepool logs, is a queue 0 for all node types | 15:53 |
| yolanda | i did a telnet to gear server, and status showed jobs there | 15:53 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Set delay-release for gnocchiclient https://review.openstack.org/259085 | 15:54 |
| fungi | i'm going to start with a zuul thread dump, though it's a little like reading tea leaves for me | 15:54 |
| *** esker has joined #openstack-infra | 15:54 | |
| fungi | sort of wondering if someone merged a zuul config change around 14:00 and that was maybe the first time it attempted a config reload since the restart yesterday | 15:55 |
| clarkb | the fact that not even noop jobs are making it through zuul right now indicates a zuul issue | 15:55 |
| *** zeih has joined #openstack-infra | 15:55 | |
| *** david-lyle_ has joined #openstack-infra | 15:55 | |
| *** esker has quit IRC | 15:56 | |
| fungi | not seeing any candidates for that theory either | 15:56 |
| *** esker has joined #openstack-infra | 15:56 | |
| fungi | https://review.openstack.org/258378 merging at 09:44 seems to be the most recent thing to hit zuul configuration, unless i'm missing something | 15:57 |
| *** david-lyle_ has quit IRC | 15:57 | |
| *** esker has quit IRC | 15:57 | |
| *** apuimedo has quit IRC | 15:57 | |
| fungi | clarkb: agreed | 15:57 |
| fungi | Last reconfigured: Thu Dec 17 2015 10:06:51 GMT+0000 (UTC) | 15:58 |
| *** apuimedo has joined #openstack-infra | 15:58 | |
| clarkb | 2015-12-17 14:01:59,919 DEBUG zuul.Scheduler: Project None not found there is a lot of that in the log | 15:58 |
| fungi | and the nodes graph shows evidence nodepoold was still getting demand info for some hours | 15:58 |
| fungi | clarkb: yep, is that the first incidence? | 15:58 |
| *** rbrndt has joined #openstack-infra | 15:58 | |
| clarkb | let me dig back | 15:58 |
| *** david-lyle has quit IRC | 15:59 | |
| *** apuimedo_ has joined #openstack-infra | 15:59 | |
| fungi | looks like we've had that further back | 15:59 |
| *** dizquierdo has quit IRC | 15:59 | |
| *** ChuckC has quit IRC | 15:59 | |
| *** luqas is now known as lezbar | 15:59 | |
| fungi | so probably unrelated | 15:59 |
| fungi | first hit is 20:49:42 yesterday | 16:00 |
| *** hasharAway is now known as hashar | 16:00 | |
| clarkb | which is when we turned gerrit on | 16:00 |
| fungi | likely something introduced in zuul before yesterday's restart but after the prior restart | 16:00 |
| clarkb | so probably something in an event it doesn't understand and is getting ignored | 16:00 |
| AJaeger | fungi, yolanda approved a few changes this morning (European morning), zuul should have been reconfigurd there | 16:00 |
| fungi | oh, or that | 16:00 |
| *** zeih has quit IRC | 16:00 | |
| fungi | yeah could be we're seeing events with an empty project? | 16:01 |
| yolanda | i approved changes for project-config yes | 16:01 |
| *** rhedlind has left #openstack-infra | 16:01 | |
| AJaeger | fungi: and status../zuul says "Last reconfigured: Thu Dec 17 2015 11:06:51 GMT+0100 (CET)" | 16:01 |
| sdague | fungi: well earlier yolanda linked the following message | 16:01 |
| sdague | <yolanda> sdague... 2015-12-17 14:59:30,305 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information. | 16:01 |
| AJaeger | that's 10:06 UTC | 16:01 |
| yolanda | but those don't look critical | 16:01 |
| yolanda | just some event ignored | 16:01 |
| *** kmartin has quit IRC | 16:02 | |
| fungi | right, we have those all the way back to when zuul restarted for new gerrit | 16:02 |
| *** kmartin has joined #openstack-infra | 16:02 | |
| fungi | while the problem seems to have started around 14:00 utc | 16:02 |
| AJaeger | is it time to send out a notice that we have a stuck zuul before everybody asks? | 16:02 |
| *** tjones has left #openstack-infra | 16:04 | |
| sdague | AJaeger: yeh, that's probably a good idea | 16:04 |
| anteaya | Sam-I-Am: they know your schedule | 16:04 |
| *** david-lyle has joined #openstack-infra | 16:05 | |
| yolanda | nodepool started logging now | 16:06 |
| yolanda | but queue is still to zero | 16:06 |
| *** egonzalez has quit IRC | 16:06 | |
| yolanda | so logging problem looks as a red herring | 16:06 |
| pabelanger | yolanda: possible somebody tried to SIGINT nodepool? Cause, it won't shutdown properly just using that. | 16:06 |
| clarkb | yolanda: I think that nodepool is trying to connect to jenkins02 and timing out | 16:07 |
| clarkb | that causes the delay in logging | 16:07 |
| fungi | yeah, that explains the lengthy pause in the logs | 16:07 |
| pabelanger | I can reproduce an issue with SIGING and nodepool hanging | 16:07 |
| yolanda | clarkb, so that's one of the things i fixed then. I restarted jenkins02 | 16:07 |
| yolanda | and it started logging again | 16:07 |
| jeblair | this is the first restart since jhesketh's branch merged | 16:07 |
| yolanda | but still no demand received | 16:07 |
| yolanda | so lack of debug was caused by jenkins02 beng down.. problem solved. But we have the no queue problem now | 16:08 |
| *** sridhar_ram has quit IRC | 16:09 | |
| e0ne | hi. is any news about CI status? looks like we don't recieve response to gerrit | 16:09 |
| AJaeger | e0ne: we're debugging.. | 16:09 |
| e0ne | I checkt few patches to cinder in zuul: all tests passed but results not publiished | 16:09 |
| e0ne | thanks | 16:09 |
| fungi | okay, there are a pair of zuul stack dumps one minute apart along with the thread stats info in ~fungi/stack.dump on zuul | 16:10 |
| *** vgridnev has joined #openstack-infra | 16:10 | |
| fungi | i'm seeing what i cam make of that now, but other infra-root feel free to open that file and see what jumps out at you | 16:10 |
| jeblair | the second one makes me suspicious that line numbers have changed since the restart | 16:11 |
| *** yamahata has joined #openstack-infra | 16:11 | |
| fungi | oh, like we've merged more zuul changes today? | 16:11 |
| jeblair | yeah, which, if so, makes them a little hard to read | 16:11 |
| *** boris-42_ has joined #openstack-infra | 16:11 | |
| jeblair | or at least merged more since...what, last week? | 16:12 |
| jeblair | no, i guess since yesterday | 16:12 |
| *** ihrachys has quit IRC | 16:12 | |
| *** apuimedo has quit IRC | 16:12 | |
| jeblair | but the git log doesn't seem to back that up | 16:12 |
| jeblair | File "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py", line 285, in query | 16:12 |
| jeblair | return data | 16:12 |
| jeblair | i just don't expect the traceback to go into 'pprint' after that. | 16:12 |
| fungi | yeah, that's more than strange | 16:13 |
| *** cfarquhar has joined #openstack-infra | 16:14 | |
| *** cfarquhar has quit IRC | 16:14 | |
| *** cfarquhar has joined #openstack-infra | 16:14 | |
| *** Qiming has quit IRC | 16:14 | |
| fungi | PrettyPrinter seems to have a lot of process time | 16:14 |
| *** odlD2 has quit IRC | 16:15 | |
| jeblair | both times you took thread dumps the main thread was pprinting | 16:17 |
| *** apuimedo has joined #openstack-infra | 16:17 | |
| fungi | bad timing on my part? i can try a couple more | 16:17 |
| *** rebase has joined #openstack-infra | 16:18 | |
| clarkb | couod be inefficient handling of our data structure too? | 16:18 |
| *** rebase has quit IRC | 16:18 | |
| jeblair | fungi: or perhaps it's just sinking a lot of time into that | 16:18 |
| fungi | seems more likely, but i'm doing two more 90 seconds apart to see if we catch it at something else | 16:18 |
| jeblair | aha | 16:19 |
| jeblair | the logger for connections is wrong | 16:19 |
| jeblair | so we aren't logging that | 16:19 |
| jeblair | (this at least explains why we aren't seeing the results of all this pretty printing) | 16:20 |
| *** apuimedo_ has quit IRC | 16:20 | |
| *** apuimedo_ has joined #openstack-infra | 16:20 | |
| *** markus_z has quit IRC | 16:22 | |
| *** tonytan4ever has joined #openstack-infra | 16:23 | |
| fungi | okay, second pair of stack dumps and accompanying stats are in ~fungi/stack.dump.1 | 16:24 |
| *** apuimedo has quit IRC | 16:24 | |
| *** pigmej has joined #openstack-infra | 16:24 | |
| fungi | a lot of PrettyPrinter in the gerrit query function here too | 16:26 |
| *** yamamoto has quit IRC | 16:26 | |
| *** matrohon has quit IRC | 16:26 | |
| *** bexelbie has joined #openstack-infra | 16:27 | |
| *** sshnaidm has quit IRC | 16:28 | |
| *** odlD2 has joined #openstack-infra | 16:28 | |
| jeblair | i'm tempted to try altering the logging config and sighuping | 16:29 |
| *** bpokorny has joined #openstack-infra | 16:29 | |
| fungi | we do have rather a lot of varied tracebacks in zuul's logs too, which look like bugs in need of fixing, though not necessarily corresponding to when this seems to have started falling apart on us | 16:29 |
| *** Swami_ has joined #openstack-infra | 16:30 | |
| *** apuimedo has joined #openstack-infra | 16:30 | |
| jeblair | from what i can tell, it is not stuck | 16:30 |
| jeblair | it is just moving very, very slowly | 16:31 |
| fungi | so slowly that it outwardly appears to have been standing almost still for something like 2.5 hours | 16:32 |
| *** tmcpeak has joined #openstack-infra | 16:32 | |
| stevemar | how can i find out the keyboard shortcuts for the new gerrit ui? | 16:32 |
| tmcpeak | hey guys, I've got the automatic push to PyPI enabled on Bandit. I pushed a new version of Bandit yesterday but it didn't show up in PyPI. Any idea why and what I should do about it? | 16:33 |
| sdague | stevemar: "?" | 16:33 |
| stevemar | it used to be that when i pressed "r" it would take me to the revie form | 16:33 |
| *** asselin__ has joined #openstack-infra | 16:33 | |
| sdague | stevemar: it's 'a' now | 16:33 |
| stevemar | sdague: ugh, i bet it says it somewhere on the page huh | 16:33 |
| stevemar | Press '?' to view keyboard shortcuts | 16:33 |
| stevemar | gdi | 16:33 |
| yolanda | infra-root, if that helps, the only queue that nodepool seems to be receiving, is nodes for rax-iad | 16:33 |
| sdague | the "?" for help thing is a standard google product interface, I don't know if it's documented though | 16:34 |
| sdague | oh, yeh, it is there on the bottom (but it's pretty light font). So not surprised you didn't catch it. | 16:34 |
| *** abregman is now known as abregman|nb | 16:34 | |
| jeblair | it's taking about 15 seconds to process each event | 16:34 |
| anteaya | stevemar: I think you made a typo on thank you | 16:34 |
| stevemar | didn't know that about google interfaces | 16:34 |
| *** apuimedo_ has quit IRC | 16:34 | |
| stevemar | anteaya: yes i did! | 16:34 |
| stevemar | sdague: thanks :) | 16:35 |
| *** apuimedo has quit IRC | 16:35 | |
| *** asselin has quit IRC | 16:35 | |
| *** scheuran has quit IRC | 16:35 | |
| *** ningy has quit IRC | 16:35 | |
| sdague | jeblair: hmmm... any theory about why it downshifted 100 gears? | 16:35 |
| stevemar | anteaya: and thanks to you for keeping me in check | 16:35 |
| anteaya | stevemar: :) | 16:36 |
| fungi | tmcpeak: yeah, ttx mentioned it earlier. we're dealing with a bigger issue right now but i'll take a look once the ci system is moving at a typical pace again. most likely the tarball job log will have the answers | 16:36 |
| *** apuimedo has joined #openstack-infra | 16:36 | |
| tmcpeak | fungi: ok sounds good. Thank you | 16:36 |
| anteaya | stevemar: thank you just has such a lovely ring to it | 16:36 |
| jeblair | sdague: signs point to spending a lot of time in pprint | 16:36 |
| sdague | jeblair: because the new event structures are more complicated? | 16:37 |
| jeblair | sdague: i don't know yet | 16:37 |
| fungi | status Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating. | 16:39 |
| fungi | ^? | 16:39 |
| jeblair | aha | 16:39 |
| anteaya | lgtm | 16:39 |
| *** aorourke has joined #openstack-infra | 16:40 | |
| *** apuimedo_ has joined #openstack-infra | 16:40 | |
| fungi | also this graph makes it look like it didn't slow down all at once, but was a little more gradual | 16:40 |
| fungi | http://graphite.openstack.org/render/?from=-4hours&height=480&until=-2hours&width=640&bgcolor=ffffff&fgcolor=000000&areaMode=stacked&target=color(alias(sumSeries(stats.gauges.nodepool.nodes.building),%20%27Building%27),%20%27ffbf52%27)&target=color(alias(sumSeries(stats.gauges.nodepool.nodes.ready),%20%27Available%27),%20%2700c868%27)&target=color(alias(sumSeries(stats.gauges.nodepool.nodes.used),%20 | 16:40 |
| jeblair | the bug in connections i pointed out to jhesketh the other day not only means that we're now potentially operating on outdated information, we've effectively moved quite a lot of gerrit query processing from a separate thread into the main thread | 16:40 |
| fungi | %27In%20Use%27),%20%276464ff%27)&target=color(alias(sumSeries(stats.gauges.nodepool.nodes.delete),%20%27Deleting%27),%20%27c864ff%27)&title=Test%20Nodes | 16:40 |
| fungi | ugh | 16:40 |
| fungi | #status notice Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating. | 16:41 |
| openstackstatus | fungi: sending notice | 16:41 |
| *** automagically has left #openstack-infra | 16:41 | |
| *** _nadya_ has quit IRC | 16:41 | |
| jeblair | that would explain things being slower, but i would not expect that to cause it to slow down gradually | 16:41 |
| *** _nadya_ has joined #openstack-infra | 16:41 | |
| *** marcusvrn_ has joined #openstack-infra | 16:42 | |
| -openstackstatus- NOTICE: Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating. | 16:42 | |
| *** ujuc_ has quit IRC | 16:42 | |
| fungi | it at least looks like nodepoold may have gotten some demand numbers after 13:30 but spaced further out | 16:42 |
| jeblair | fungi: is there a coincident nodepool problem? | 16:43 |
| yolanda | i still saw some demand, but very sporadic | 16:43 |
| openstackstatus | fungi: finished sending notice | 16:43 |
| fungi | jeblair: possible there's also something going on with nodepoold, but hard to say until zuul is back up to speed | 16:44 |
| jeblair | fungi: why? | 16:44 |
| jeblair | i mean, why is it hard to say? | 16:44 |
| fungi | oh, actually i guess this is easily explainable by the current zuul situation | 16:45 |
| yolanda | fungi, i don't think there is a nodepool problem now. The lack of logging was likely caused by jenkins02 not reacting | 16:45 |
| fungi | there are no demand numbers to be had | 16:45 |
| jeblair | 2015-12-17 16:04:42,424 DEBUG nodepool.NodePool: Demand from gearman: devstack-centos7: 1 | 16:45 |
| fungi | according to statsd, zuul doesn't think it has a job backlog at the moment | 16:45 |
| fungi | so there's little reason for nodepool to see any demand | 16:45 |
| jeblair | right, it's correctly seeing no demand every ~10 seconds though | 16:46 |
| jeblair | so i believe that means it's operating normally | 16:46 |
| *** HeOS has joined #openstack-infra | 16:46 | |
| jeblair | (almost no demand -- it wants centos7) | 16:46 |
| *** tmcpeak has left #openstack-infra | 16:46 | |
| fungi | we had some separate confusion earlier with nodepoold. it seems to have died of its own accord before yolanda and i began looking into it, and when it was restarted jenkins02 was apparently in a crashed state so it was timing out trying to connect to it and leaving the debug log silend for a very long period at startup | 16:47 |
| *** apuimedo_ has quit IRC | 16:47 | |
| jeblair | i just wanted to make sure that we weren't looking at a gearman problem | 16:47 |
| yolanda | fungi, yep, a red herring caused by jenkins02 | 16:47 |
| jeblair | don't want to ignore evidence. at least by accidenty. | 16:47 |
| fungi | agreed | 16:48 |
| fungi | yolanda: so anyway, double-checking that, it looks like you went ahead and cleaned up any pending slaves in the jenkins02 config and then started it up around 16:00 utc? | 16:48 |
| *** Guest98282 has quit IRC | 16:48 | |
| jeblair | okay, back to zuul... i'm going to try to figure out one of the queries the main thread is doing and then pprint it | 16:49 |
| yolanda | i restarted safely using the play | 16:49 |
| jeblair | if that doesn't tell us anything, i'm going to try to live-reconfigure logging | 16:49 |
| fungi | i'll take a quick look and see if i can tell when/why nodepoold died | 16:49 |
| fungi | jeblair: that sounds reasonable | 16:49 |
| *** otter768 has joined #openstack-infra | 16:49 | |
| *** burgerk has joined #openstack-infra | 16:49 | |
| fungi | if mildly dangerous, but then again we don't have much to lose (grab a snapshot of check and gate pipelines first, presumably?) | 16:50 |
| jeblair | ya | 16:50 |
| *** davideag_ has joined #openstack-infra | 16:50 | |
| jeblair | (which will be of little overall use considering the 3k unprocessed events) | 16:50 |
| clarkb | jeblair: sounds good | 16:50 |
| fungi | right | 16:50 |
| *** davideagnello has quit IRC | 16:51 | |
| jeblair | these tracebacks don't make any sense. | 16:52 |
| *** apuimedo has quit IRC | 16:52 | |
| fungi | the ones in the zuul log? yeah, they're a little random and strange | 16:52 |
| jeblair | self._isMerged(change) | 16:52 |
| jeblair | that doesn't call any of the methods underneath it | 16:53 |
| anteaya | is it safe to share any of them in a paste? | 16:53 |
| *** apuimedo has joined #openstack-infra | 16:53 | |
| *** annegentle has quit IRC | 16:53 | |
| fungi | like it's skipping around in the source when mapping up the lines | 16:53 |
| fungi | similar to the line number oddities in the stack dump? | 16:53 |
| *** apuimedo_ has joined #openstack-infra | 16:53 | |
| jeblair | anteaya: http://paste.openstack.org/show/482207/ | 16:53 |
| anteaya | thank you | 16:53 |
| * anteaya goes to stare at that a while | 16:54 | |
| stevemar | anteaya: release note jobs should not run on stable branches for clients right? | 16:54 |
| fungi | oh, you meant in the stack dump | 16:54 |
| *** otter768 has quit IRC | 16:55 | |
| jeblair | actually... | 16:55 |
| jeblair | the *second* tracebacks don't make any sense | 16:55 |
| jeblair | the first ones do | 16:55 |
| jeblair | clarkb: could there be something funny about the way our signal handler is doing that? | 16:55 |
| fungi | in both pairs of stack dumps? | 16:56 |
| jeblair | fungi: yep | 16:56 |
| fungi | if so, seems the most likely place for that to be happening | 16:56 |
| jeblair | 2 is a pattern :) | 16:56 |
| * jeblair ignores second tracebacks and gets back on track | 16:56 | |
| *** davideag_ has quit IRC | 16:56 | |
| clarkb | jeblair: possibly? maybe if there is a context switch while dumping tracebacks it races the code that is running | 16:56 |
| jeblair | clarkb: or maybe the profiler is mucking things up? | 16:57 |
| clarkb | I am not sure if the GIL locks up around traceback writing | 16:57 |
| fungi | or we're somehow reusing the first context in the second dump | 16:57 |
| *** placko has quit IRC | 16:58 | |
| anteaya | stevemar: um check with ttx and dhellmann when doug is back from vacation but you sound like you believe the answer is no | 16:58 |
| *** claudiub has quit IRC | 16:58 | |
| clarkb | I would be surprised if the profiler could do things that confuse where stack pointers are in threads | 16:58 |
| *** placko has joined #openstack-infra | 16:58 | |
| clarkb | the profiler can add things to the stack but the stack pointer is pretty fundamental to operation. | 16:58 |
| clarkb | though maybe its just messing up the recording of such things | 16:59 |
| fungi | we've got ip allocation over quota errors for bluebox-sjc1 (not tat it's important for now) | 16:59 |
| *** ddieterly has joined #openstack-infra | 16:59 | |
| *** ihrachys has joined #openstack-infra | 16:59 | |
| fungi | also we've got ports leaked in hpcloud preventing node launches there | 16:59 |
| *** annegentle has joined #openstack-infra | 17:00 | |
| *** abregman|nb is now known as abregman | 17:00 | |
| *** rhallisey_ has joined #openstack-infra | 17:00 | |
| *** changbl has quit IRC | 17:00 | |
| fungi | and possibly also floating ips leaked too since there are a lot of http 500 unknown error responses from nova there in addition to the port overlimit errors | 17:01 |
| clarkb | jeblair: for the trace you pasted I think data = change._data calls connection.gerrit query() | 17:01 |
| clarkb | and htats trying to render? | 17:01 |
| *** rhallisey_ has quit IRC | 17:02 | |
| clarkb | oh maybe I am wrong | 17:02 |
| pleia2 | good morning | 17:02 |
| jeblair | clarkb: did someone make that a property? | 17:02 |
| *** maishsk has joined #openstack-infra | 17:02 | |
| clarkb | jeblair: trying to see if that is the case now, it is odd that the source updates change's _data | 17:02 |
| *** maishsk has quit IRC | 17:03 | |
| *** fhubik has quit IRC | 17:03 | |
| clarkb | isMerged() updates _data | 17:03 |
| anteaya | morning pleia2 | 17:04 |
| *** lykinsbd has joined #openstack-infra | 17:04 | |
| anteaya | pleia2: we have a bit of an issue with zuul, it is moving slowly, we don't know why yet | 17:04 |
| clarkb | and isMerged() runs query() but I don't see it getting called | 17:05 |
| jeblair | clarkb: _isMerged does not | 17:05 |
| anteaya | pleia2: fungi has some thread dumps on his zuul user if you want to catch up by looking there | 17:05 |
| jeblair | (with the tail) | 17:05 |
| clarkb | jeblair: right | 17:05 |
| *** eharney has quit IRC | 17:05 | |
| *** yaume_ has quit IRC | 17:05 | |
| pleia2 | anteaya: thanks, I saw the notice and have been catching up :) | 17:06 |
| anteaya | pleia2: wonderful | 17:06 |
| jeblair | to start with, the info for 252250 is 4MB. | 17:07 |
| jeblair | i believe we're looking at a looping third party ci as the root cause | 17:08 |
| anteaya | :( | 17:09 |
| *** eharney has joined #openstack-infra | 17:09 | |
| fungi | oh, is zuul getting flooded with events? | 17:09 |
| sdague | x-io seems to be tight looping there | 17:10 |
| sdague | every 5 - 10 seconds | 17:10 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Update merge status after merge:merge is submitted https://review.openstack.org/128921 | 17:10 |
| anteaya | I think we have close connection on gerrit 2.11 yay | 17:10 |
| anteaya | if we can find out which ci it is | 17:10 |
| sdague | anteaya: x-io | 17:10 |
| reed | Your voter key is invalid. You should have received a correct URL by email. | 17:11 |
| jeblair | it takes 5 seconds to query gerrit for 252250 and another 1.5 to parse and pretty-print it. so that's about half of our time. | 17:11 |
| reed | known issue? trying to vote for O | 17:11 |
| anteaya | https://review.openstack.org/Documentation/cmd-close-connection.html | 17:11 |
| *** hashar has quit IRC | 17:11 | |
| anteaya | reed: you need to find mordred | 17:11 |
| clarkb | probably need to disable the account first to prevent it reconnecting | 17:11 |
| reed | anteaya, thanks | 17:11 |
| anteaya | reed: welcome | 17:11 |
| sdague | yeh, it's still posting | 17:11 |
| anteaya | clarkb: good point | 17:11 |
| pleia2 | reed: as a data point some folks had problems the day it was sent out, but for the most part they sorted themselves out, so yeah, ask mordred :) | 17:12 |
| *** mudassirlatif has joined #openstack-infra | 17:12 | |
| sdague | it started going crazy 12 hrs ago | 17:12 |
| jeblair | openstack-ci@x-io.com | 17:12 |
| sdague | from what I can see in the logs | 17:12 |
| clarkb | I am setting that email addr inactive now | 17:12 |
| anteaya | sdague: oh you told me before and I didn't understand what you were saying | 17:13 |
| clarkb | ok thats done, now tofigure out the session | 17:13 |
| anteaya | as x-io means nothing to me | 17:13 |
| fungi | yeah, i'm getting the connections list now | 17:13 |
| sdague | anteaya: oh, osrry | 17:13 |
| *** ayoung has joined #openstack-infra | 17:13 | |
| anteaya | sdague: no worries, our brainwaves diverge | 17:13 |
| fungi | it takes a while, i think because it tries reverse dns on all the ip addresses | 17:13 |
| fungi | there we go | 17:13 |
| jeblair | fungi: there's an -n option | 17:14 |
| fungi | i always forget about that | 17:14 |
| *** automagically_ has joined #openstack-infra | 17:14 | |
| fungi | what's the username associated with openstack-ci@x-io.com? | 17:14 |
| anteaya | yeah I can't get them by email | 17:14 |
| zaro | morning | 17:15 |
| clarkb | fungi: not sure may have to ask the db | 17:15 |
| pleia2 | morning zaro | 17:15 |
| *** apoorvad has joined #openstack-infra | 17:15 | |
| fungi | yep, digging in the db now | 17:15 |
| fungi | just didn't know if someone already knew | 17:15 |
| anteaya | zaro: can you help me here, this command used to work and is failing for me today: http://paste.openstack.org/show/482208/ | 17:15 |
| anteaya | zaro: has the syntax changed? | 17:15 |
| *** salv-orl_ has joined #openstack-infra | 17:15 | |
| anteaya | fungi: was trying to find it, my usual method is failing | 17:16 |
| zaro | looking | 17:16 |
| jeblair | of course there is no cinder-ci group to make this convenient | 17:16 |
| anteaya | fungi: xio-ise-iscsi-ci | 17:16 |
| anteaya | fungi: according to their wikipage | 17:16 |
| anteaya | https://wiki.openstack.org/wiki/ThirdPartySystems/X-IO_technologies_CI | 17:16 |
| fungi | yep, they have two accounts... x-io-ise-ci and xio-ise-iscsi-ci | 17:17 |
| zaro | can you get your own account? | 17:17 |
| zaro | case sensitive? | 17:17 |
| fungi | i unfortunately don't see either of those in the gerrit connection list | 17:17 |
| fungi | ahh | 17:18 |
| clarkb | fungi: maybe it went away as part of the inactivation? | 17:18 |
| zaro | mine works ok, curl https://review.openstack.org/accounts/zaro0508@gmail.com | 17:18 |
| *** ayoung has quit IRC | 17:18 | |
| fungi | show-connections -n also doesn't lookup usernames | 17:18 |
| *** automagically has joined #openstack-infra | 17:18 | |
| *** salv-orlando has quit IRC | 17:18 | |
| jeblair | just ids | 17:18 |
| clarkb | fungi: does it return account id because that is numeric? | 17:18 |
| fungi | yep | 17:18 |
| clarkb | ya | 17:18 |
| zaro | ahh case sensitvie, curl https://review.openstack.org/accounts/Zaro0508@gmail.com, does not work | 17:18 |
| sdague | it looks like it stopped 4 minutes ago | 17:18 |
| anteaya | zaro: thanks | 17:18 |
| anteaya | sdague: someone is watching the channel mayhap? | 17:18 |
| sdague | so the deactivation | 17:19 |
| *** automagically_ has quit IRC | 17:19 | |
| jeblair | clarkb: disabled it, so perhaps they were also reconnecting | 17:19 |
| fungi | we also have an emc-xio-ci conencting a couple of times but presumably they're not involved | 17:19 |
| jeblair | fungi: what 2 account numbers did you find? | 17:19 |
| sdague | anyway, that change is no longer getting updated | 17:19 |
| *** jistr has quit IRC | 17:19 | |
| *** changbl has joined #openstack-infra | 17:19 | |
| fungi | 13142 and 14969 the latter of which has an open connection so i'm killing that now | 17:20 |
| fungi | 75a0b1ad 20:39:00 00:00:04 a/14969 207.250.72.10 | 17:20 |
| jeblair | xio-ise-iscsi-ci a/14969 | 17:20 |
| xyang1 | fungi: what do you mean | 17:20 |
| anteaya | xyang1: hi | 17:20 |
| *** hemna has joined #openstack-infra | 17:20 | |
| xyang1 | anteaya: hi | 17:20 |
| zaro | nice to see new gerrit haters coming out of the woodwork. | 17:20 |
| jeblair | fungi: yeah that's from the sshd log, that was the one doing it | 17:20 |
| xyang1 | what is the problem | 17:20 |
| fungi | closing connection 75a0b1ad... | 17:20 |
| hemna | lots of us don't like the new UI | 17:20 |
| fungi | closed connection 75a0b1ad | 17:20 |
| *** unicell has quit IRC | 17:20 | |
| anteaya | xyang1: is emc-xio-ci at all associated with https://wiki.openstack.org/wiki/ThirdPartySystems/X-IO_technologies_CI | 17:20 |
| xyang1 | no | 17:21 |
| anteaya | xyang1: thank you | 17:21 |
| xyang1 | anteaya: that is a different company | 17:21 |
| anteaya | fungi: the answer to your question about emc-xio-ci is no | 17:21 |
| fungi | xio-ise-iscsi-ci was the one connecting | 17:21 |
| anteaya | xyang1: thank you | 17:21 |
| zaro | hemna: yes, nice hearing from you. | 17:21 |
| fungi | thanks xyang1 | 17:21 |
| hemna | :) | 17:21 |
| jeblair | fungi, clarkb: sshd log is probably a good way to confirm these in the future. | 17:21 |
| xyang1 | welcome | 17:21 |
| anteaya | xyang1: we had a ci auto generating comments and it took out our system | 17:22 |
| *** yamahata has quit IRC | 17:22 | |
| xyang1 | oh | 17:22 |
| anteaya | xyang1: you alread know not to auto generate comments, I'm sure | 17:22 |
| *** rhedlind has joined #openstack-infra | 17:22 | |
| xyang1 | I think so | 17:22 |
| anteaya | xyang1: yes wanted to make sure your system wasn't related | 17:22 |
| sdague | jeblair: so is the belief now that this just needs to burn down the event queue and we'll be back to normal? | 17:23 |
| *** bradjones has quit IRC | 17:23 | |
| anteaya | xyang1: can you check with your operators and make sure they know, no auto generated comments please | 17:23 |
| xyang1 | anteaya: sure, thanks | 17:23 |
| clarkb | I have to step out for a bit have an appointment in just over half an hour I need to get ready for | 17:23 |
| jeblair | fungi, clarkb: if we weren't swamped with irrelevant log entries for the new gerrit events (i thought someone fixed that?), we might have noticed that zuul is actually querying the same change repeatedly. | 17:23 |
| anteaya | xyang1: thank you | 17:23 |
| *** trown is now known as trown|lunch | 17:23 | |
| *** rossella_s has quit IRC | 17:23 | |
| xyang1 | anteaya: I'll pass that along to make sure | 17:23 |
| anteaya | clarkb: I hope it goes well | 17:23 |
| *** rossella_s has joined #openstack-infra | 17:23 | |
| anteaya | xyang1: thank you | 17:23 |
| xyang1 | anteaya: thank you | 17:23 |
| *** bradjones has joined #openstack-infra | 17:24 | |
| *** bradjones has quit IRC | 17:24 | |
| *** bradjones has joined #openstack-infra | 17:24 | |
| jeblair | sdague: sort-of. i suspect there are N events in the queue from that change, and it'll take 15 seconds for each. we should probably edit the gerrit db, delete the comments from that system on that change, and then restart zuul, taking the data loss hit to get moving again. | 17:24 |
| anteaya | I will email the third party announce mailing list regarding xio-ise-iscsi-ci | 17:25 |
| jeblair | i could do the math, but my gut says hours or days to get through that. | 17:25 |
| sdague | jeblair: coolio | 17:25 |
| rhedlind | anteaya: what's up with the X-IO CI? | 17:25 |
| jeblair | clarkb, fungi: i will edit the db now. | 17:25 |
| sdague | yeh, at this point I think whatever gets work flowing again fastest is cool | 17:25 |
| fungi | jeblair: thanks | 17:26 |
| sdague | once we get a zuul restart I'll go and see what's in approved state and stuck | 17:26 |
| jeblair | (also, i believe we can *vastly* improve this in zuul; i believe i'm going to hit some of this code in my v3 work, i'll try to keep this case in mind) | 17:26 |
| anteaya | rhedlind: it took out our zuul with auto generated comments | 17:26 |
| fungi | and yeah, i'm playing with analyzing the gerrit ssh logs now for a good way to spot these | 17:27 |
| anteaya | rhedlind: it is disabled and disconnected | 17:27 |
| *** jlanoux has quit IRC | 17:27 | |
| rhedlind | anteaya: huh? how could it do that? | 17:27 |
| *** elo has joined #openstack-infra | 17:27 | |
| anteaya | rhedlind: auto generated comments | 17:27 |
| sdague | rhedlind: it was reporting every 5 - 10 seconds | 17:27 |
| sdague | on review 252250 | 17:27 |
| sdague | 8455 comments in the last 12 hours | 17:28 |
| fungi | so these are the 10 most active accounts commenting via ssh connection during the utc 16:00 hour http://paste.openstack.org/show/482210/ | 17:28 |
| *** jcoufal_ has quit IRC | 17:28 | |
| jeblair | actually, it's possible that removing the comment data might make zuul fast enough to process the backlog in a reasonable time | 17:29 |
| jeblair | so i will wait to asses performance before restarting zuul | 17:29 |
| sdague | jeblair: that's worth trying | 17:29 |
| fungi | looks like it was averaging just shy of 10 a minute | 17:29 |
| rhedlind | sdague: anyone has a theory on how it could do that? My zuul server is eating up 7.17 gig of memory and jenkins crashed due to out of memory | 17:30 |
| *** leeantho has joined #openstack-infra | 17:30 | |
| *** HeOS has quit IRC | 17:30 | |
| rhedlind | why would it keep commenting on the same change? | 17:31 |
| jeblair | select count(*) from change_messages where change_id=252250 and author_id=14969; | 17:31 |
| jeblair | 8463 | 17:31 |
| jeblair | turning that into a delete | 17:31 |
| *** ddieterly has quit IRC | 17:31 | |
| *** eharney has quit IRC | 17:32 | |
| jeblair | delete from change_messages where change_id=252250 and author_id=14969; | 17:32 |
| jeblair | UPDATE 8463; 133 ms | 17:32 |
| jeblair | zuul's cycle is about 1s per event now | 17:32 |
| jeblair | a lot of those events are ignorable | 17:32 |
| jeblair | so we may just be able to let it run | 17:33 |
| anteaya | rhedlind: it has an auto generated commenter which loops on something | 17:33 |
| *** apuimedo_ has quit IRC | 17:33 | |
| anteaya | rhedlind: such as if it posts a message with recheck in the comment and triggers on the word recheck | 17:33 |
| fungi | looks like it started at 05:06:38, so i guess zuul wan't really crippled by that until load picked up elsewhere? | 17:33 |
| *** automagically has left #openstack-infra | 17:34 | |
| sdague | yeh, the event queue seems to be dropping | 17:34 |
| jeblair | fungi: it would have had to leave enough comments to generate enough data to slow things down | 17:34 |
| sdague | pretty quickly | 17:34 |
| jeblair | fungi: this matches the 'gradually got worse' profile you observed | 17:34 |
| fungi | xio-ise-iscsi-ci posted 1634 comments during the 05:00 hour when it was first starting to get going | 17:34 |
| anteaya | is today the first appearance of that ci system sshing into gerrit? | 17:35 |
| sdague | so, I'm not sure if it's helpful or not, but given that gerrit 2.11 does merge conflict checking internally, those merge conflict checks could be disabled now in zuul I think | 17:35 |
| fungi | and yeah, it looks like its rate slowed over time (as the performance of our systems began to degrade?) | 17:35 |
| sdague | you can even query for mergable | 17:35 |
| *** ayoung has joined #openstack-infra | 17:36 | |
| jeblair | sdague: yeah, i think we'd want to have zuul consider gerrit mergability as a gate pipeline requirement first, but once that's in place, i think we can drop the merge-check pipeline | 17:36 |
| rhedlind | anteaya: our CI has been online for quite some time (since this spring) and I have not changed the zuul scripts recently. | 17:36 |
| anteaya | this is a ui style nit: https://review.openstack.org/#/c/252250 I can't see what project the patch is for before I scroll down | 17:37 |
| jeblair | sdague: i'm not sure whether the needed bits for that are in zuul already and therefore it would just be a layout.yaml config change, or if we'd need to add something to zuul first to pick up mergeability | 17:37 |
| anteaya | rhedlind: which ci is yours? | 17:37 |
| electrofelix | zaro waynr: wondering if either of you have looked at https://review.openstack.org/252335 it's to remove the need to having to update setup.cfg for ever new addition to JJB. Only if it's a brand new module. | 17:37 |
| rhedlind | x-io | 17:37 |
| fungi | rhedlind: oh! you're the one we just disabled | 17:37 |
| anteaya | rhedlind: oh well time to take a look at its logs then | 17:37 |
| rhedlind | the one causing all the trouble | 17:37 |
| rhedlind | and making me sick right now | 17:38 |
| rhedlind | :( | 17:38 |
| AJaeger | stevemar: release notes job should run on stable branches - why do you ask? | 17:38 |
| anteaya | rhedlind: well take a walk around | 17:38 |
| AJaeger | stevemar: but we publish from master the content including all branches you configured | 17:38 |
| anteaya | rhedlind: we need to decompress ourselves and assure ourselves that our backlog can clear | 17:38 |
| *** apuimedo_ has joined #openstack-infra | 17:39 | |
| anteaya | rhedlind: your system isn't doing any more harm at the moment | 17:39 |
| stevemar | AJaeger: i'm asking cause of this: https://review.openstack.org/#/c/256039/ | 17:39 |
| electrofelix | waynr: I like your JJB CLI changes proposed. I would however like if we could make a push to land a lot of the other work that is in flight at the moment, do a release and then merge your CLI rework. | 17:39 |
| anteaya | rhedlind: after you have had a walk and a cup of tea, and we have assured ourselves our backlog will clear | 17:39 |
| *** ddieterly has joined #openstack-infra | 17:39 | |
| stevemar | AJaeger: i was under the impression that we don't include reno for liberty for libraries and clients | 17:39 |
| anteaya | rhedlind: then you can go through some of your event logs, and we can look too if you wish to share | 17:40 |
| anteaya | rhedlind: and we can figure out what happened | 17:40 |
| anteaya | rhedlind: let us decompress though | 17:40 |
| *** ifarkas has quit IRC | 17:40 | |
| stevemar | AJaeger: the release note fail, not the python test fail :P | 17:40 |
| sdague | the event queue has drained to 0, but the integrated gate still seems stuck | 17:40 |
| jeblair | indeed | 17:41 |
| rhedlind | anteaya:ok. I am really sorry for all this. as far as I knew it was running along fine. | 17:41 |
| stevemar | AJaeger: actually... that is really old, sorry | 17:41 |
| anteaya | rhedlind: okay thanks, let's take a break for a bit and then come back to this and figure out a way forward | 17:41 |
| anteaya | rhedlind: thanks for contacting us so quickly | 17:41 |
| jeblair | it looks like it's reporting changes in check... | 17:41 |
| stevemar | AJaeger: cause this one, same repo and branch doesn't include the releasenotes job: https://review.openstack.org/#/c/258284/ | 17:41 |
| jeblair | sdague: oh | 17:42 |
| jeblair | sdague: it's only just now processing the queues for the first time since the dos started | 17:42 |
| zaro | electrofelix: i have not but that's very cool. | 17:42 |
| jeblair | sdague: so it's got to go through all of the pipelines now | 17:43 |
| sdague | jeblair: ok, is there a known order for that? | 17:43 |
| *** shardy has quit IRC | 17:43 | |
| jeblair | sdague: yeah, check then gate | 17:43 |
| zaro | electrofelix: i've been busy with gerrit lately so haven't gotten around to reviewing much. now i we to fix stuff after upgrading. | 17:44 |
| sdague | I guess I am surprised the gate doesn't have priority on that | 17:44 |
| anteaya | fungi: also when you get to email, while all the fun was going on, ovh posted to the infra list with a suggestion of how to help us more | 17:44 |
| openstackgerrit | Apoorva Deshpande proposed openstack-infra/ciwatch: Separate votes from Jenkins queues https://review.openstack.org/258125 | 17:44 |
| jeblair | sdague: i don't think it's come up before :) but we could probably sort them in priority order | 17:44 |
| anteaya | fungi: I replied saying we will discuss the suggestion as a group and get back to him and we appreciate all his help and his commitment to supporting us | 17:45 |
| dmellado | Hi here, is zuul still having issues? I'm seeing a gate queued like...forever | 17:45 |
| dmellado | I guess so, but just wanted to double check ;) | 17:45 |
| anteaya | dmellado: yes we just identified the issue and it is dealt with | 17:45 |
| anteaya | dmellado: it will take some time for the backlog to clear | 17:45 |
| dmellado | thanks anteaya ;) | 17:45 |
| anteaya | dmellado: thank you | 17:45 |
| *** kushal has joined #openstack-infra | 17:46 | |
| *** kushal has quit IRC | 17:46 | |
| *** kushal has joined #openstack-infra | 17:46 | |
| *** bexelbie has quit IRC | 17:46 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Workaround 500 error when moving items in worklists https://review.openstack.org/259128 | 17:47 |
| fungi | ttx: i can't find any evidence that zuul ever saw the bandit 0.17.0 tag upload (according to its logs). tmcpeak doesn't seem to stick around in channel or i'd ask him, but my guess is that he pushed it the _moment_ gerrit came back online, a few minutes before we started zuul up after the upgrade maintenance | 17:47 |
| jeblair | (so, before we announced the maint window was over) | 17:47 |
| fungi | i'll just reenqueue it here in a bit once things calm down | 17:47 |
| fungi | jeblair: no idea but that's my guess | 17:47 |
| fungi | i could check the gerrit logs actually | 17:48 |
| *** ddieterly has quit IRC | 17:48 | |
| *** yamahata has joined #openstack-infra | 17:48 | |
| *** asselin_ has quit IRC | 17:48 | |
| *** eharney has joined #openstack-infra | 17:49 | |
| *** ociuhandu has quit IRC | 17:49 | |
| fungi | [2015-12-16 20:42:16,138 +0000] ecb50f4c travis-mcpeak a/11861 git-receive-pack./openstack/bandit.git 0ms 436ms 0 | 17:49 |
| fungi | so... yep | 17:49 |
| *** gyee has joined #openstack-infra | 17:49 | |
| sdague | gate queue finally moving again | 17:49 |
| sdague | no longer staring at that tempest change at the top of it | 17:50 |
| AJaeger | stevemar: let's discuss on #openstack-keystone, I'll join you there to get noise out of here.. | 17:50 |
| *** apuimedo_ has quit IRC | 17:50 | |
| *** electrofelix has quit IRC | 17:51 | |
| *** tonytan4ever has quit IRC | 17:52 | |
| jaypipes | fungi, clarkb, jeblair: sorry if stupid question, but in the latest gerrit, is there a way to go back to the old code review screen? | 17:52 |
| *** e0ne has quit IRC | 17:53 | |
| olaph | unfortunately, no | 17:53 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Fix various style bugs with cards https://review.openstack.org/259131 | 17:53 |
| *** u_glide1 has quit IRC | 17:54 | |
| sdague | jaypipes: see - http://lists.openstack.org/pipermail/openstack-dev/2015-October/076962.html | 17:54 |
| fungi | jaypipes: the old review screen was finally dropped completely from gerrit upstream in 2.11. the "new" (not so new as it's been available as an option for years) review screen is now the only web interface. they're working on an even newer review screen to replace it in upcoming gerrit versions, but it's still in the design stages | 17:55 |
| *** apuimedo_ has joined #openstack-infra | 17:55 | |
| *** abregman is now known as abregman|mtg | 17:55 | |
| pkoniszewski | hi, is there an option to start (implement) a test in gate with, let's say, custom image? I need something that will prevent live migration from finishing... | 17:56 |
| *** asselin has joined #openstack-infra | 17:56 | |
| anteaya | jaypipes: and jeblair readily hands out links to gertty to all who ask | 17:57 |
| jaypipes | sdague, fungi: OK, thanks guys. | 17:57 |
| anteaya | pkoniszewski: what do you mean when you say custom image | 17:58 |
| anteaya | pkoniszewski: do you mean use your image instead of bare-trusty? | 17:58 |
| *** dangers_away has quit IRC | 17:58 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:59 | |
| *** apuimedo_ has quit IRC | 17:59 | |
| pkoniszewski | anteaya: it can be bare-trusty if I can modify it a bit somehow, e.g., install and start additional software during test | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| openstackgerrit | Merged openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 18:00 |
| *** ashleighfarnham has joined #openstack-infra | 18:00 | |
| fungi | pkoniszewski: the jobs have full root access on the machines where they run and can do pretty much anything. you don't need a custom image for what you're describing | 18:00 |
| *** thiagop has quit IRC | 18:01 | |
| pkoniszewski | fungi: so there is an option to start a VM with stress installed on so it will dirty memory and therefore prevent live migration from finishing? | 18:01 |
| *** bgmccollum has left #openstack-infra | 18:01 | |
| pkoniszewski | fungi: or just install a stress tool post-boot | 18:01 |
| fungi | pkoniszewski: why does it need to be installed before booting? | 18:02 |
| fungi | yes, you can run any commands you want in a job | 18:02 |
| fungi | including installing additional packages | 18:02 |
| SumitNaiksatam | hi, the horizon team is trying to end the meeting on openstack-meeting-3 but the bot does not seem to be responding | 18:04 |
| david-lyle | I think it ended | 18:04 |
| david-lyle | but the channel status is stuck | 18:04 |
| david-lyle | other meetings have been started and stopped since | 18:04 |
| sdague | we still don't seem to be demanding nodes as fast as I'd have guessed | 18:05 |
| *** esker has joined #openstack-infra | 18:05 | |
| *** sputnik13 has joined #openstack-infra | 18:05 | |
| *** apuimedo_ has joined #openstack-infra | 18:05 | |
| fungi | david-lyle: which channel? | 18:05 |
| sdague | that might be a pathological issue in the work that zuul does before it asks for any nodes | 18:05 |
| SumitNaiksatam | david-lyle: thats interesting | 18:06 |
| david-lyle | fungi: openstack-meeting-3 | 18:06 |
| SumitNaiksatam | fungi: meeting-3 | 18:06 |
| fungi | looks like meeting is also in a similar state | 18:06 |
| fungi | probably the meetbot got netsplit or restarted in the middle of those meetings | 18:06 |
| *** kdas_ has joined #openstack-infra | 18:06 | |
| *** kdas_ has quit IRC | 18:06 | |
| *** dangers_away has joined #openstack-infra | 18:07 | |
| jeblair | sdague: i don't think zuul has fully worked through everything yet | 18:07 |
| sdague | jeblair: that could be | 18:07 |
| *** kushal has quit IRC | 18:07 | |
| jeblair | sdague: the 'queued' jobs are actually probably mostly still in merge-pending state | 18:08 |
| SumitNaiksatam | fungi: i just started a new meeting, the topic changed | 18:08 |
| *** unicell has joined #openstack-infra | 18:08 | |
| SumitNaiksatam | fungi: so i think we are good :-) | 18:08 |
| *** _nadya_ has quit IRC | 18:08 | |
| jeblair | sdague: which means they're waiting on results from mergers before the actual jobs get launched | 18:08 |
| sdague | ah | 18:08 |
| sdague | I guess I'm surprised that's true even for the gate | 18:09 |
| *** _nadya_ has joined #openstack-infra | 18:09 | |
| fungi | SumitNaiksatam: david-lyle: yeah, it's just the default channel topic which is getting set back to an incorrect string at the end of meetings now. i need to catch that channel between meetings to fix it safely (like i just did for #openstack-meeting) | 18:09 |
| SumitNaiksatam | fungi: okay thanks | 18:09 |
| jeblair | sdague: gate does get priority there, but there's actually still a backlog of _results_ it's working through now | 18:10 |
| *** davideagnello has joined #openstack-infra | 18:11 | |
| sdague | ok, sure, there were just 12 patches in the gate above the current work set, I thought as soon as the sliding window was open we started running things, based on the same speculation criteria as before | 18:11 |
| david-lyle | thanks fungi | 18:11 |
| *** jerryz has joined #openstack-infra | 18:11 | |
| sdague | but I guess that was my misunderstanding | 18:11 |
| *** zeih has joined #openstack-infra | 18:11 | |
| pkoniszewski | fungi: just one more thing, by default devstack uploads only a cirros image to glance during installation, do I need to use image builder to build trusty image or is there a store that contains other images than cirros? | 18:11 |
| *** eharney has quit IRC | 18:11 | |
| jeblair | sdague: this is "as soon as" when zuul hasn't caught up with 5 hours of backlog :) | 18:11 |
| *** esker has quit IRC | 18:12 | |
| sdague | ok, I guess I don't have the model of what gets processed fifo before we get there :) | 18:12 |
| *** ihrachys has quit IRC | 18:12 | |
| *** _nadya__ has joined #openstack-infra | 18:13 | |
| *** ihrachys has joined #openstack-infra | 18:13 | |
| jeblair | i think it just launched a bunch of jobs because nodepool is now hyperactive | 18:13 |
| *** _nadya_ has quit IRC | 18:13 | |
| fungi | pkoniszewski: do you expect to actually boot the images? the main reasons we use cirros there are 1. we have limited disk space on the workers (as little as 80gb on some) and full cloud images take up a lot of space, 2. we have limited ram in the worker (8gb) and a lot of full distros need lots of ram to run effectively at all, 3. it's unaccelerated qemu inside a virtual machine there so you don't | 18:14 |
| fungi | want to wait 30 minutes for the image to boot | 18:14 |
| jeblair | it just now cleaned out the noop queue | 18:14 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-diskimage_builder: Add lvm2 package to diskimage builder https://review.openstack.org/258974 | 18:15 |
| fungi | pkoniszewski: i think heat has one or two jobs which boot a minimal fedora image to test some sort of guest client, and it takes some minutes for that to boot | 18:15 |
| sdague | yeh, gate queue seems to have started to get nodes now | 18:16 |
| *** zeih has quit IRC | 18:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Disable reno on liberty for keystoneauth/keystonemiddleware https://review.openstack.org/259137 | 18:16 |
| AJaeger | stevemar: ^ | 18:16 |
| fungi | jeblair: not sure if you saw my comment late last night, but the release.slave.openstack.org replacement went well and we've since done pypi uploads with distro wheel/twine no problem | 18:16 |
| jeblair | fungi: yay! | 18:16 |
| fungi | so that was a great idea | 18:17 |
| jeblair | fungi: i can't believe you stayed up to do that, but thanks! :) | 18:17 |
| aspiers | ooh, shiny new gerrit! :) | 18:17 |
| fungi | jeblair: well, i was worried people were going to try to push more releases and i didn't want to have to do more manual cleanup than necessary | 18:17 |
| *** apuimedo_ has quit IRC | 18:17 | |
| jeblair | sdague: i _think_ it's caught up now | 18:17 |
| *** annegentle has quit IRC | 18:17 | |
| sdague | yeh, it seems like it's chugging | 18:18 |
| fungi | if it's caught up, i'll go ahead and retrigger that missing bandit release now | 18:18 |
| jeblair | fungi: ++ | 18:18 |
| *** annegentle has joined #openstack-infra | 18:18 | |
| *** ybathia has joined #openstack-infra | 18:18 | |
| fungi | well, reenqueue i mean | 18:18 |
| sdague | now, if we could only get the pypi wheel mirror code landed we'd shave 10 - 30% off the run time of the jobs :) | 18:18 |
| jeblair | or even enqueue for the first time ever :) | 18:18 |
| fungi | right, that | 18:18 |
| fungi | tmcpeak must have been either really unlucky and oblivious, or really in a hurry yesterday | 18:19 |
| *** esker has joined #openstack-infra | 18:20 | |
| jeblair | while ! git push; do sleep 1; done | 18:21 |
| jeblair | what could possibly go wrong | 18:21 |
| *** maishsk has joined #openstack-infra | 18:21 | |
| AJaeger | aspiers: thanks for the enthusiasm and not complaining | 18:21 |
| *** ayoung has quit IRC | 18:21 | |
| aspiers | AJaeger: I haven't found anything to complain about in the first 60 seconds of using it, so congrats ;-) | 18:22 |
| *** bgmccollum has joined #openstack-infra | 18:22 | |
| sdague | there are many nice new features as well | 18:22 |
| *** apuimedo_ has joined #openstack-infra | 18:22 | |
| sdague | Conflicts With is one of my favorites | 18:23 |
| aspiers | I read https://archive.fosdem.org/2015/schedule/event/editors_gerrit_code_review/ a while back and remember getting excited | 18:23 |
| sdague | as it helps you find duplicate patches for the same thing | 18:23 |
| *** abregman|mtg is now known as abregman | 18:23 | |
| anteaya | I'm getting duplicate emails for the same patch | 18:23 |
| anteaya | this is new for me | 18:23 |
| fungi | sdague: oh that's awesome--i hadn't even looked at that yet | 18:23 |
| sdague | fungi: yeh, it's one of the column 3 sets | 18:23 |
| anteaya | AJaeger: did you add me as a reviewer to 259137? | 18:23 |
| boris-42_ | AJaeger: hi there | 18:24 |
| boris-42_ | AJaeger: could you please add me to osprofiler-core and osprofiler-release ACL group?) | 18:24 |
| AJaeger | anteaya: no, I added stevemar - after you reviewed it | 18:24 |
| AJaeger | boris-42_: I don't have those permissions, infra-root can help you | 18:25 |
| boris-42_ | AJaeger: ok | 18:25 |
| boris-42_ | SergeyLukjanov: could you please add me to osprofiler-core group?) | 18:25 |
| boris-42_ | SergeyLukjanov: hi | 18:25 |
| *** eharney has joined #openstack-infra | 18:26 | |
| *** aopincar has joined #openstack-infra | 18:27 | |
| anteaya | AJaeger: I'm trying to figure out why I started getting two emails for each event on that patch | 18:27 |
| *** harlowja has joined #openstack-infra | 18:27 | |
| pleia2 | boris-42_: I'll have a look, do you have a link to the change that added those groups to gerrit? | 18:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Disable reno on liberty for keystoneauth/keystonemiddleware https://review.openstack.org/259137 | 18:28 |
| anteaya | AJaeger: I actually can't find where I turn email off, I know there used to be a place | 18:28 |
| AJaeger | anteaya: Haven't seen such an option | 18:29 |
| anteaya | no? perhaps I imagined it | 18:30 |
| zaro | AJaeger: could you please tell me again how i can disable bashate? on change https://review.openstack.org/#/c/258088/ | 18:30 |
| AJaeger | zaro: let me try patching it for you... | 18:31 |
| *** tjones has joined #openstack-infra | 18:31 | |
| zaro | ohh even better :) | 18:31 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix docstring of downstream-ext publisher https://review.openstack.org/258863 | 18:31 |
| anteaya | I still don't understand why I have two email threads for https://review.openstack.org/#/c/259137/ | 18:31 |
| pleia2 | boris-42_: ah, found it https://review.openstack.org/#/c/258182/ and I've added you to both | 18:31 |
| *** sfinucan has quit IRC | 18:32 | |
| *** amuller is now known as amuller_afk | 18:33 | |
| openstackgerrit | Merged openstack/os-testr: Fail if no tests were successfully executed https://review.openstack.org/246980 | 18:33 |
| *** ayoung has joined #openstack-infra | 18:33 | |
| openstackgerrit | Merged openstack/os-testr: Add support to ostestr to use subunit-trace color https://review.openstack.org/256133 | 18:33 |
| openstackgerrit | Merged openstack-infra/glean: Open /etc/hosts file in append mode, not write https://review.openstack.org/258405 | 18:34 |
| jesusaurus | what's the process for removing a jenkins master from nodepool? if I simply remove the target section from nodepool.yaml, would nodepool safely stop adding slaves? would zuul continue to use up the remaining nodes? | 18:34 |
| zaro | anteaya: do you mean you are getting double emails? | 18:34 |
| anteaya | zaro: yes, that is what I mean | 18:34 |
| anteaya | I can forward you both threads if you like | 18:35 |
| zaro | anteaya: do you have it starred? | 18:35 |
| anteaya | no | 18:35 |
| zaro | it's only for that change? | 18:35 |
| anteaya | yes so far | 18:36 |
| *** devananda has quit IRC | 18:36 | |
| anteaya | I haven't done a whole lot of reviewing since yesterday though | 18:36 |
| anteaya | this may be the only change I have reviewed since the restart | 18:36 |
| zaro | anteaya: check settings, https://review.openstack.org/#/settings/preferences | 18:37 |
| *** apuimedo_ has quit IRC | 18:37 | |
| anteaya | sorry I did review this: https://review.openstack.org/#/c/258672/ | 18:37 |
| zaro | anteaya: is CC selected? | 18:37 |
| anteaya | one email thread | 18:37 |
| anteaya | zaro: I looked in preferences, CC is not selected | 18:37 |
| zaro | what? 2 or 1 email? | 18:38 |
| anteaya | two threads | 18:38 |
| *** zeih has joined #openstack-infra | 18:38 | |
| zaro | do you have multiple things selected on watches? | 18:38 |
| zaro | https://review.openstack.org/#/settings/projects | 18:38 |
| anteaya | no | 18:38 |
| anteaya | now one thread | 18:39 |
| anteaya | I think something confused my thunderbird | 18:39 |
| anteaya | and now thunderbird has gotten itself figured out | 18:39 |
| anteaya | as I look now there is one thread | 18:39 |
| anteaya | but there were two | 18:39 |
| zaro | uhh, me not use thunder, but cool | 18:39 |
| anteaya | okay so blip | 18:39 |
| anteaya | thanks for listening and sorry for the noise | 18:40 |
| zaro | np | 18:40 |
| *** sambetts is now known as sambetts-afk | 18:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Add automated database config retrieval https://review.openstack.org/258088 | 18:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for querying role assignments. https://review.openstack.org/259147 | 18:41 |
| AJaeger | zaro: fixed ^ | 18:41 |
| *** jpr_ has joined #openstack-infra | 18:41 | |
| AJaeger | zaro: but long lines are only warnings for bashate, the errors are other stuff like function delcaration or indent not multiple of 4 | 18:42 |
| *** apuimedo_ has joined #openstack-infra | 18:42 | |
| *** zeih has quit IRC | 18:43 | |
| zaro | ah, thanks. you've made my day :) | 18:43 |
| AJaeger | zaro: let me update the script instead and you can compare which version you prefer ;) | 18:44 |
| boris-42_ | pleia2: thanks | 18:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Add automated database config retrieval https://review.openstack.org/258088 | 18:45 |
| AJaeger | zaro: this ^ passes bashate locally, check the diff, please | 18:45 |
| *** e0ne has joined #openstack-infra | 18:46 | |
| *** trown|lunch is now known as trown | 18:47 | |
| *** apuimedo has quit IRC | 18:47 | |
| *** esker has quit IRC | 18:47 | |
| *** esker has joined #openstack-infra | 18:47 | |
| *** ayoung has quit IRC | 18:47 | |
| *** bharathm has joined #openstack-infra | 18:48 | |
| *** apuimedo has joined #openstack-infra | 18:48 | |
| *** apuimedo_ has quit IRC | 18:49 | |
| jpr_ | morning...I'm trying to set up an instance of openstack-infra at our site so I can emmulate the management philosophy of openstack.org, e.g. "systems as code with a defined ci/cd fabric and best-practice workflows" | 18:49 |
| zaro | AJaeger: me likes! | 18:49 |
| anteaya | jpr_: morning, have you seen http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/ | 18:50 |
| *** otter768 has joined #openstack-infra | 18:50 | |
| anteaya | jpr_: that was created by one of our third party operators to create a system that consumes the gerrit event stream via ssh and report back to our gerrit again via ssh | 18:51 |
| jpr_ | yes...I think... I've been working with http://docs.openstack.org/infra/system-config/running-your-own.html | 18:51 |
| *** piet has quit IRC | 18:51 | |
| anteaya | ah that is a tad dated | 18:51 |
| jpr_ | yes I sense that but felt there was some structural understanding emebedded... at least from steps 1 2 and 3 | 18:52 |
| anteaya | sure | 18:52 |
| anteaya | always helps to have the big picture | 18:52 |
| jpr_ | i'm assuming that basic sturcture is still valid. is that fair? | 18:53 |
| *** aopincar has quit IRC | 18:53 | |
| anteaya | and this is what it would look like from the dev's point of view: http://docs.openstack.org/infra/manual/developers.html#quick-reference | 18:53 |
| *** devananda has joined #openstack-infra | 18:53 | |
| * AJaeger gets lots of mails for gerrit events ;) | 18:53 | |
| jpr_ | yes | 18:53 |
| anteaya | well I don't know as I have ever read that document, but if it says we use gerrit zuul gearman jenkins and nodepool then that part is valid | 18:54 |
| AJaeger | fungi, do you want to send an all clear again? | 18:54 |
| *** gyee has quit IRC | 18:54 | |
| anteaya | AJaeger: I bet you do | 18:54 |
| *** tzn has quit IRC | 18:54 | |
| fungi | AJaeger: well, it would be another notice since i didn't use alert for this situation. i was considering waiting until the backlog burned down a bit before making another announcement | 18:54 |
| anteaya | the only part openstackci doesn't set up is gerrit, since the use case for which that puppet module was designed was to listen to our gerrit | 18:55 |
| asselin | Hi, I'm not able to login to gerrit anymore. I keep getting "Server Error" | 18:55 |
| *** otter768 has quit IRC | 18:55 | |
| *** apuimedo_ has joined #openstack-infra | 18:55 | |
| anteaya | jpr_: but there is a puppet module availabe that we use to set up our gerrit | 18:55 |
| fungi | asselin: when did that start? | 18:55 |
| anteaya | asselin: :( | 18:55 |
| jeblair | wfm | 18:56 |
| anteaya | I just logged in via chrome | 18:56 |
| asselin | fungi, this morning | 18:56 |
| AJaeger | fungi, fine with me.... | 18:56 |
| fungi | asselin: which url is returning the server error? gerrit or launchpad/ubuntu? | 18:56 |
| asselin | fungi, https://review.openstack.org/OpenID?gerrit.mode=SIGN_IN&gerrit.token=%23%2Fdashboard%2F%3Fforeach%3Dstatus%253Aopen%2BNOT%2Blabel%253AWorkflow%3C%253D%252D1%2BNOT%2Blabel%253ACode%252DReview%3C%253D%252D2%2BNOT%2Bowner%253Aself%2BNOT%2Blabel%253ACode%252DReview%3C%253D%252D1%252cself%2BNOT%2Blabel%253ACode%252DReview%3E%253D1%252cself%26title%3DRamy%27s%2BGerrit%2BDashboard%26Elastic%2BRecheck%3D%28project%253Aopenstac | 18:56 |
| asselin | k%252Dinfra%252Felastic%252Drecheck%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fpuppet%252Delastic_recheck%29%26ThirdParty%2BCI%2BCORE%3D%28project%253Aopenstack%252Fthird%252Dparty%252Dci%252Dtools%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fciwatch%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fpuppet%252Dciwatch%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fproject%252Dconfig%252Dexample%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fp | 18:56 |
| asselin | uppet%252Dopenstackci%29%26ThirdParty%2BCI%3D%28project%253Aopenstack%252Dinfra%252Fpuppet%252Dnodepool%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fpuppet%252Dzuul%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fpuppet%252Dproject_config%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fpuppet%252Djenkins%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fgearman%252Dplugin%2BOR%0Aproject%253Aopenstack%252Dinfra%252Fzmq%252Devent%252Dpublisher%2B | 18:56 |
| *** eharney has quit IRC | 18:56 | |
| asselin | OR%0Aproject%253Aopenstack%252Fpython%252Djenkins%29%26My%2Btopics%3D%28topic%253Adownstream%252Dpuppet%29%26MISC%3D%28project%253Aopenstack%252Fgerrit%252Ddash%252Dcreator%29%26Infra%2Bdocs%2B%2526%2Bspecs%3D%28project%253Aopenstack%252Dinfra%252Finfra%252Dmanual%2BOR%2Bproject%253Aopenstack%252Dinfra%252Finfra%252Dspecs%29 | 18:56 |
| asselin | sorry...didn't realize it was so long | 18:56 |
| *** eharney has joined #openstack-infra | 18:56 | |
| pabelanger | eep | 18:56 |
| anteaya | today I learned | 18:56 |
| fungi | wowie | 18:56 |
| asselin | actually...perhaps that's my 'dashboard'...let me try without it | 18:57 |
| AJaeger | ;) | 18:57 |
| *** esker has quit IRC | 18:57 | |
| fungi | asselin: looks like you might be trying to go to a custom dashboard whose url is not parsed properly by 2.11 | 18:57 |
| *** _nadya__ has quit IRC | 18:57 | |
| sdague | asselin: dashboards that reference self fail unless you are logged in | 18:57 |
| fungi | yeah, good idea | 18:57 |
| asselin | yup, ok now I logged in. | 18:58 |
| *** _nadya_ has joined #openstack-infra | 18:58 | |
| asselin | and my dashboard works too. ok thanks, sorry for the noise. | 18:58 |
| jpr_ | ok. so I'm working under the assumption that i still need to 1) set a vm a puppetmaster 2) follow the puppetmaster setup http://docs.openstack.org/infra/system-config/puppet.html#id2 (with the puppet install, module install and system-config as production clone) 3) merge in my changes from my own system-config clone | 18:58 |
| openstackgerrit | Merged openstack-infra/project-config: networking-ovn: Make tempest job voting again. https://review.openstack.org/258595 | 18:59 |
| asselin | anteaya, jpr_ there's this spec which is trying to do more: https://review.openstack.org/#/c/239810/ | 19:00 |
| *** tjones has quit IRC | 19:00 | |
| anteaya | jpr_: our use of puppetmaster has changed substantially in the past year | 19:00 |
| smarcet | fungi: afternoon, have one question, which process triggers tarball creation here http://tarballs.openstack.org/openstackid? | 19:01 |
| anteaya | jpr_: I wouldn't go by that document with regards to how to set up your puppet | 19:01 |
| *** burgerk has quit IRC | 19:01 | |
| *** _nadya_ has quit IRC | 19:02 | |
| *** apuimedo_ has quit IRC | 19:02 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats: Gerrit 2.11 compatibility https://review.openstack.org/259151 | 19:02 |
| *** tjones has joined #openstack-infra | 19:02 | |
| jpr_ | anteaya: is puppetmaster still the primary controller for deploying the services in a domain? | 19:03 |
| *** tzn has joined #openstack-infra | 19:03 | |
| * anteaya defers that question to someone who can answer it | 19:03 | |
| anteaya | jpr_: honestly I can't keep track anymore | 19:04 |
| *** esker has joined #openstack-infra | 19:04 | |
| jpr_ | :) | 19:04 |
| fungi | smarcet: the tarballs like openstackid-1.0.7.5.1ee300d.tar.gz and openstackid-latest.tar.gz appear to be created any time a change merges, so if you just had a change merge to the openstackci repo then there's probably a pending tarball job for that commit in zuul's post pipeline. we had a bit of an incident earlier today and so we're working through a pretty lengthy job backlog right now which means | 19:04 |
| fungi | it may take a while to complete | 19:04 |
| jpr_ | let me sketch my goal... | 19:05 |
| smarcet | fungi: oh ok | 19:05 |
| smarcet | np | 19:05 |
| smarcet | thx for the info :) | 19:05 |
| fungi | smarcet: if so, you can likely check its status at http://status.openstack.org/zuul/ (search for openstackid there) | 19:06 |
| asselin | jpr_, if you like, you can add comments to the spec https://review.openstack.org/#/c/239810/9/specs/openstackci-extension.rst. It intends to update the link you referenced: http://docs.openstack.org/infra/system-config/running-your-own.html | 19:06 |
| jpr_ | I like the model that the infra project has for managing infrastructure as an openely accessible code base with a review and deploy process. I'd like to "clone" that model within our domain adding local systems as relevent for our environment | 19:06 |
| smarcet | fungi: oh ok thx :) | 19:07 |
| *** esker has quit IRC | 19:07 | |
| fungi | yeah, looks like there's one queued for it from about an hour ago | 19:07 |
| jpr_ | asselin: will certainly engage there | 19:07 |
| fungi | smarcet: oh, actually the tarball job looks like it's already running, if you click on the blue box there to expand the job details for that commit | 19:08 |
| *** apuimedo_ has joined #openstack-infra | 19:08 | |
| fungi | that openstackid-release-branch job looks like it will probably complete in a few minutes | 19:08 |
| *** sabeen1 has joined #openstack-infra | 19:08 | |
| fungi | or faster | 19:08 |
| jpr_ | regarding the goal, there is a lot of common logic to setting up a code review and ci/cd framework, work that is hard get off the ground in smaller shops | 19:09 |
| *** u_glide1 has joined #openstack-infra | 19:09 | |
| *** ihrachys has quit IRC | 19:09 | |
| *** kashyap has left #openstack-infra | 19:10 | |
| fungi | jpr_: yep, asselin's spec is a great opportunity for us to simplify that running-your-own document which has bitrotted pretty badly over the past couple years | 19:10 |
| openstackgerrit | Merged openstack-infra/project-config: osops-tools-generic: add bashate job https://review.openstack.org/229043 | 19:10 |
| jpr_ | so if i can build on top of the mature process outlined by openstack-infra then i can save some effort here. kinda like saying "we follow the openstack-infra workflow with these addtional local systems" | 19:11 |
| asselin | this one's phschwartz's spec (which extends my original smaller scoped) spec | 19:11 |
| fungi | usually our documentation is kept up to date best when people are following it frequently and pointing out/fixing inaccuracies which creep in as things change over time, but the scope of that particular document is such that it's very rarely used, and so falls out of date a lot more easily | 19:11 |
| *** tzn has quit IRC | 19:11 | |
| fungi | asselin: d'oh, right, phschwartz's spec | 19:11 |
| *** _nadya_ has joined #openstack-infra | 19:12 | |
| phschwartz | :( I need to finish the last pass on that spec. Been supper busy which has delayed it | 19:13 |
| jpr_ | i'll review the spec and follow up with questions | 19:13 |
| anteaya | welcome to the team, jpr_ | 19:13 |
| jpr_ | thanks | 19:14 |
| anteaya | jpr_: asselin is going to be a great resource as he has done most of what you want to do, as in consuming infra tools | 19:14 |
| phschwartz | jpr_: it sounds like your looking to do exactly what we are working towards with my spec. | 19:14 |
| anteaya | jpr_: and phschwartz just recently went through setting up openstackci himself | 19:15 |
| *** apuimedo_ has quit IRC | 19:15 | |
| phschwartz | anteaya: correct | 19:15 |
| anteaya | so the location of many of the trip wires are fresh in his mind | 19:16 |
| *** jaosorior has quit IRC | 19:16 | |
| jpr_ | anteaya: ah...so now I know who to bug. ;) | 19:17 |
| anteaya | always helpful | 19:17 |
| anteaya | :) | 19:17 |
| phschwartz | jpr_: I have a bouncer running so if I am not at the computer I will get notified and be able to look at scroll back | 19:18 |
| *** vilobhmm11 has joined #openstack-infra | 19:18 | |
| vilobhmm11 | AJaeger : ping | 19:18 |
| vilobhmm11 | fungii : ping | 19:19 |
| vilobhmm11 | wanted to know what I might be missing here https://review.openstack.org/#/c/254488/4/zuul/layout.yaml as Jenkins seem to be unhappy every time :( | 19:19 |
| AJaeger | vilobhmm11: please don't use naked pings, tell us what you want... | 19:19 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone eventlet gate only in keystone https://review.openstack.org/257999 | 19:19 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Change postgres job to use standard deployment https://review.openstack.org/258006 | 19:19 |
| vilobhmm11 | Ajaeger : was typing… | 19:19 |
| AJaeger | vilobhmm11: zuul/layout.yaml defines *when* to run the job. | 19:20 |
| *** _nadya_ has quit IRC | 19:20 | |
| *** apuimedo has quit IRC | 19:20 | |
| asselin | vilobhmm11, fyi, you can run those tests locally using tox | 19:20 |
| openstackgerrit | Merged openstack-infra/system-config: Add docs for renaming an IRC channel https://review.openstack.org/244860 | 19:20 |
| AJaeger | You also need to declare the job itself - see jenkins/jobs/projects.yaml | 19:20 |
| *** abregman has quit IRC | 19:20 | |
| *** _nadya_ has joined #openstack-infra | 19:21 | |
| *** apuimedo has joined #openstack-infra | 19:21 | |
| *** apuimedo_ has joined #openstack-infra | 19:21 | |
| vilobhmm11 | AJaeger : ok | 19:21 |
| vilobhmm11 | lemme try | 19:21 |
| *** dslev has quit IRC | 19:22 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command https://review.openstack.org/258725 | 19:24 |
| jpr_ | phschwartz: so what is the best first step if the old how to is out of date? the spec helps but isn't very how-to oriented. i assume i still need to get the puppet master set up and operational so it can manage the other nodes | 19:24 |
| *** zul has quit IRC | 19:25 | |
| *** _nadya_ has quit IRC | 19:25 | |
| openstackgerrit | Khai Do proposed openstack-infra/nodepool: Remove duplicate python-jenkins code from nodepool https://review.openstack.org/259157 | 19:25 |
| *** sabeen1 has quit IRC | 19:25 | |
| phschwartz | jpr_: the docs pertaining to it all are very outdated and working to be updated. Your best resource will be to look at openstack-infra/puppet-openstackci | 19:25 |
| phschwartz | jpr_: in the contrib dir there is a very good readme that asselin put together | 19:26 |
| *** apuimedo_ has quit IRC | 19:26 | |
| *** gordc has quit IRC | 19:26 | |
| *** apuimedo has quit IRC | 19:26 | |
| * asselin needs to get that readme into the manual | 19:26 | |
| phschwartz | jpr_: Also note the focus has been on masterless puppet so there is no more effort to setup a puppet master and puppet-openstackci is designed around this. (you can use a puppet master, but not requried) | 19:26 |
| *** apuimedo has joined #openstack-infra | 19:27 | |
| *** zul has joined #openstack-infra | 19:27 | |
| dims | AJaeger : anteaya : got +1 from lifeless on this review https://review.openstack.org/#/c/238871/ - enables project-requirements-change.py to deal with the oslo.db[fixtures] syntax in our *requirements.txt | 19:28 |
| dims | AJaeger : anteaya : please take a look when you get a chance. thanks! | 19:29 |
| jpr_ | phschwartz: puppetmaster is just the term i'm using from the old docs. i've been working with puppet apply as described so I assume i'm in the correct mode | 19:30 |
| *** asselin_ has joined #openstack-infra | 19:31 | |
| openstackgerrit | Vilobh Meshram proposed openstack-infra/project-config: Add Consul Gate Job https://review.openstack.org/254488 | 19:31 |
| anteaya | dims: is this script used by every project that runs the check-requirements job? | 19:32 |
| *** apuimedo_ has joined #openstack-infra | 19:32 | |
| *** bharathm has quit IRC | 19:32 | |
| *** asselin__ has quit IRC | 19:33 | |
| dims | anteaya : from 'gate-{name}-requirements' | 19:33 |
| jpr_ | phswartz: as follow up, i've noticed there are the git.openstack repos and the github.com repos. They appear identical. is one preferred from the clone perspective? | 19:33 |
| *** e0ne has quit IRC | 19:34 | |
| *** ddieterly has joined #openstack-infra | 19:34 | |
| asselin | git.openstack | 19:35 |
| anteaya | dims: right, okay so putting aside my ability or lack thereof to read code for the moment, this is a change that will affect many repos gate jobs and is going straight into production, without an experimental or non-voting stage | 19:35 |
| *** ddieterl_ has joined #openstack-infra | 19:35 | |
| anteaya | dims: I'm really hesitant to do that right now as we have a big backlog to get through and I dont' want to cause a fire if I don't have to | 19:35 |
| anteaya | jpr_: the canonical git repos are under gerrit management, we run the git mirrors for our use for cloning and such, the code is mirrored to github as a courtesy | 19:37 |
| jpr_ | phschwartz: thx. following your contrib/README.md | 19:37 |
| dims | anteaya : when would be a good time? friday evening? | 19:37 |
| jpr_ | anteaya: thx for clarifying | 19:37 |
| anteaya | dims: I can be comfortable with that | 19:37 |
| anteaya | dims: ensuring we have time to watch it in action and revert if it is making us sad | 19:38 |
| anteaya | dims: is that fair? | 19:38 |
| anteaya | jpr_: welcome, you are asking good questions | 19:38 |
| dims | anteaya : yep. can you please leave a note in there? | 19:38 |
| *** ddieterly has quit IRC | 19:38 | |
| anteaya | dims: I can do so, mind if I also add a -1 with my comment, just to draw attention to my comment? | 19:39 |
| dims | anteaya : sounds good | 19:39 |
| *** gsagie_ has joined #openstack-infra | 19:39 | |
| anteaya | dims: thanks | 19:39 |
| rajinir | Thirdpart Ci: I'm testing but the commits to ci-sandbox is not triggering a jenkins build. Got an email ( silent check). Any help? | 19:40 |
| lifeless | anteaya: dims: friday evening would be high risk. I suggest monday morning | 19:41 |
| lifeless | better *not* to have weekend fires. | 19:41 |
| asselin | rajinir, what's in the e-mail? | 19:41 |
| rajinir | asselin: Here is the review https://review.openstack.org/#/c/259158/ | 19:41 |
| *** apuimedo_ has quit IRC | 19:41 | |
| asselin | rajinir, also, here's a sample repo with configs. is that what you're using? https://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/layout.yaml | 19:42 |
| rajinir | asselin: exactly | 19:42 |
| *** jed56 has quit IRC | 19:43 | |
| anteaya | lifeless: why would it be high risk? | 19:43 |
| AJaeger | anteaya: I agree with lifeless. I can +A tomorrow morning my time.... | 19:43 |
| asselin | rajinir, can you paste the e-mail response? http://paste.openstack.org/ | 19:43 |
| anteaya | AJaeger: well feel free to disagree with me | 19:43 |
| AJaeger | thanks, anteaya | 19:44 |
| *** sridhar_ram has joined #openstack-infra | 19:44 | |
| AJaeger | anteaya: what happens after friday evening? Who will be around? | 19:44 |
| *** pfalleno1 has quit IRC | 19:44 | |
| anteaya | I am saying I will | 19:44 |
| anteaya | but if you want to do it, feel free | 19:44 |
| AJaeger | Friday morning looks better to me... | 19:45 |
| anteaya | sorry I wasn't clear | 19:45 |
| dims | lifeless : if it does not work then, we'd just revert it | 19:45 |
| anteaya | I'm not going to argue with you for fire duty | 19:45 |
| anteaya | if you want it take it | 19:45 |
| *** u_glide1 has left #openstack-infra | 19:45 | |
| AJaeger | anteaya: If you're around to babysit it, I'm fine... | 19:45 |
| anteaya | that's what I was saying | 19:45 |
| rajinir | asselin: email#1 http://paste.openstack.org/show/482220/ | 19:45 |
| dims | AJaeger : anteaya : so we'd merge that patch and yank it at first sign of trouble | 19:45 |
| * AJaeger has enough firedrills in the European morning when all of you are sleeping ;) | 19:45 | |
| anteaya | sorry I was unclear that I was willing to take responsiblity | 19:45 |
| *** pfallenop has joined #openstack-infra | 19:46 | |
| anteaya | dims: thanks, glad you understand | 19:46 |
| anteaya | AJaeger: exactly | 19:46 |
| AJaeger | anteaya: then I'm fine | 19:46 |
| anteaya | AJaeger: okay great | 19:46 |
| anteaya | sorry to cause confusion | 19:46 |
| AJaeger | anteaya: no worries... | 19:46 |
| asselin | rajinir, noop-check-communication http://10.148.44.152/58/259158/1/check/noop-check-communication/5d8f664 : NOT_REGISTERED. | 19:46 |
| dims | thanks anteaya : AJaeger : lifeless | 19:46 |
| *** tzn has joined #openstack-infra | 19:47 | |
| AJaeger | anteaya: let me +2 it - and I suggest that you either -2 or WIP instead of the -1... | 19:47 |
| *** apuimedo_ has joined #openstack-infra | 19:47 | |
| anteaya | okay I can do that | 19:47 |
| anteaya | thank you | 19:47 |
| rajinir | asselin: Email#2 http://paste.openstack.org/show/482221/ | 19:47 |
| flaper87 | question: Is it possible to grant "core access" to a group on a specific subdir ? | 19:48 |
| AJaeger | flaper87: no | 19:48 |
| * flaper87 is not sure why he prefixed his last message with "question:" | 19:48 | |
| flaper87 | AJaeger: cool, thanks! | 19:48 |
| anteaya | flaper87: no, acls understand repos and branches | 19:48 |
| flaper87 | anteaya: thank you | 19:48 |
| anteaya | flaper87: they don't understand subdirs | 19:48 |
| anteaya | welcome | 19:48 |
| AJaeger | flaper87: that's social engineering ;) | 19:48 |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Add Dragonflow fullstack job https://review.openstack.org/258821 | 19:48 |
| AJaeger | flaper87: neutron does something like that with social engineering AFAIK | 19:49 |
| asselin | rajinir, did you follow all these steps? https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n307 | 19:49 |
| asselin | rajinir, especially line 320 | 19:50 |
| *** piet has joined #openstack-infra | 19:50 | |
| *** ybathia has quit IRC | 19:50 | |
| rajinir | asselin: Line 320 was successful. | 19:50 |
| rhedlind | asselin: what version of zuul is your ci running? | 19:51 |
| anteaya | dims: so if you can watch the gate queue and come find me when you think we have a space, that would be great | 19:51 |
| dims | anteaya : you mean tomorrow evening? | 19:51 |
| anteaya | dims: yes | 19:51 |
| asselin | rajinir, are you using master or a jenkins slave? | 19:51 |
| asselin | rhedlind, master | 19:51 |
| dims | anteaya : sounds like a plan. thanks | 19:51 |
| anteaya | dims: thank you | 19:52 |
| jeblair | fungi, zaro: apparently the change edit endpoints are "unstable" and "can be changed in backwards incompatible way any time without notice." yet, the previously stable commit message endpoint was removed as deprecated. | 19:52 |
| fungi | anteaya: AJaeger: flaper87: i believe there are ways to limit label permissions (voting) based on specific file paths though i can't recall whether that requires custom prolog processing. regardless, it's still an open question as to whether that is a reasonable additional complexity over our current repo/branch granularity | 19:52 |
| *** apuimedo_ has quit IRC | 19:52 | |
| jeblair | fungi, zaro: <sigh> | 19:52 |
| fungi | jeblair: wow | 19:52 |
| rhedlind | asselin: is that 2.1.1.dev109 ? | 19:52 |
| asselin | rajinir, dsvm-tempest-full is likely a jenkins slave. do you see any in your jenkins UI? | 19:53 |
| AJaeger | fungi: now I remember, there was some dsicussion in the past and indeed required custom prolog - and we rejected that. | 19:53 |
| *** amuller_afk is now known as amuller | 19:53 | |
| anteaya | I believe it was rally that asked about that | 19:53 |
| flaper87 | fungi: here's my use case: I'm seeking for maintainers to help with glance_store drivers. The swift team has agreed to help with the swift driver and the best way to do that is granting them permissions. The glance team is more than happy to give them access to glance_store but, if there's a way to limit the access, we'd probably do it. | 19:53 |
| asselin | rhedlind, not sure...whatever master is here: https://git.openstack.org/cgit/openstack-infra/zuul/ | 19:53 |
| boris-42_ | anteaya: not only Rally | 19:54 |
| flaper87 | not because we don't trust swift's team but just to keep it organized. | 19:54 |
| boris-42_ | flaper87: I know how to do that | 19:54 |
| anteaya | boris-42_: okay thank you | 19:54 |
| zaro | jeblair: are those quotes from code or docs? | 19:54 |
| boris-42_ | flaper87: you can write prolog rules | 19:54 |
| flaper87 | we'd have to do the same with cinder, probably. This part needs to be discussed | 19:54 |
| flaper87 | boris-42_: teach me | 19:54 |
| boris-42_ | flaper87: but it was not supported | 19:54 |
| flaper87 | boris-42_: ah | 19:54 |
| boris-42_ | flaper87: I mean idea by infra team | 19:54 |
| rajinir | asselin: jenkins is on the master with zuul on the same node | 19:54 |
| boris-42_ | flaper87: so I just did promise way* (which I don't like) | 19:54 |
| asselin | rajinir, do you have nodepool working? | 19:55 |
| fungi | flaper87: i think the neutron model of expecting people to avoid approving code they aren't comfortable/familiar with is working for them (last i heard), though i understand the concerns especially from a dunning-kruger effect perspective | 19:55 |
| flaper87 | yup, I'm sure the trust model will work for us as well. | 19:56 |
| rajinir | asselin: nodepool had ssh issues which I couldn't sort out in time( will figure out later). So we added slave machines manually to the jenkins to run jobs | 19:56 |
| *** lucasagomes is now known as lucas-dinner | 19:57 | |
| fungi | flaper87: also it avoids the chilling effect on trust you otherwise get by micromanaging permissions | 19:57 |
| jeblair | fungi: but good news, 2.8.4 at least has the version endpoint | 19:57 |
| jeblair | (i ran a local gerrit to test that) | 19:58 |
| asselin | rajinir, then you probably didn't add them correctly....they need to have the right label, and need to be online. but I highly recommend you get nodepool working...... | 19:58 |
| zaro | jeblair: never noticed it before, but i see it now. that's crazy | 19:58 |
| *** apuimedo_ has joined #openstack-infra | 19:58 | |
| *** triggerwarning has joined #openstack-infra | 19:58 | |
| fungi | jeblair: cool, so gertty's behavior guessing can at least be somewhat sane | 19:58 |
| mtreinish | fungi, clarkb, jeblair, nibalizer, pleia2: if you get a sec: https://review.openstack.org/#/c/247492/ just needs a +A, jhesketh held off on the approval until someone was around to watch it (which I am now) | 19:59 |
| zaro | really need to ask pursehouse about that craziness. | 19:59 |
| jeblair | fungi: yep. it'll be 3 http requests though: (get change-edit, put change-edit, publish change-edit) | 19:59 |
| rajinir | asselin: The jobs showed up in jenkins. I can manually trigger them. But when the events come from gerrit, they are not triggered. Don't know how to debug that? | 19:59 |
| rajinir | asselin: Is there a log file that can give clues? | 19:59 |
| *** FallenPegasus has joined #openstack-infra | 20:00 | |
| asselin | rajinir, https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n245 | 20:00 |
| *** rfolco has quit IRC | 20:00 | |
| *** salv-orl_ has quit IRC | 20:01 | |
| *** aysyd has quit IRC | 20:01 | |
| *** sridhar_ram has quit IRC | 20:01 | |
| ianw | mordred: have you tried to get the "vhd-util convert" patch merged, and was it rejected for reasons other than being written in some of the most out-there style i've ever seen (and i say that having taught CS101 courses in C) | 20:02 |
| ianw | mordred: the style i could fix ... if people hate it for other reasons that might be harder | 20:02 |
| rajinir | asselin: The only error is ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Cannot get account information. | 20:02 |
| rajinir | asselin: No other error messages, just INFO messages of various events | 20:02 |
| *** gsagie_ has quit IRC | 20:03 | |
| fungi | ttx: the bandit 0.17.0 release is up at https://pypi.python.org/pypi/bandit now (has been for a little while) | 20:03 |
| asselin | rajinir, telnet localhost 4730 and type 'status'. ...need to find where that's documented | 20:04 |
| *** sridhar_ram has joined #openstack-infra | 20:04 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Add dedicated acls for glance_store https://review.openstack.org/259168 | 20:04 |
| zaro | jeblair: have you noticed any other backwards incompatabilities besides Change commit message endpoint? | 20:05 |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add pylint to upper-constraints.txt https://review.openstack.org/258980 | 20:05 |
| jeblair | zaro: not yet | 20:05 |
| rajinir | asselin: Didn't see that instruction anywhere. Here is the output http://paste.openstack.org/show/482222/ | 20:06 |
| asselin | rajinir, so you don't have any slaves there | 20:09 |
| rajinir | asselin: Is that because we added the slaves manually bypassing nodepool? | 20:10 |
| *** AJaeger has quit IRC | 20:10 | |
| asselin | rajinir, yes, possibly | 20:10 |
| asselin | here's what I found in my notes: | 20:11 |
| asselin | #output is the name of the job followed by three numbers: queued, running and workers that can run job | 20:11 |
| asselin | # if you see the name there with the last number (workers that can run job) > 0, it should work. | 20:11 |
| asselin | Gearman updates on Node changes (added or removed), so a static system may not see new jobs. Workaround is to delete a 'ready' node: nodepool delete <node-id> | 20:11 |
| *** apuimedo_ has quit IRC | 20:12 | |
| rajinir | asselin: looks like the jobs require nodes from nodepool. Back to nodepool. I'll try to see if I can get that up and running. I'll be back. Thanks for help today | 20:15 |
| *** apuimedo has quit IRC | 20:16 | |
| *** _nadya_ has joined #openstack-infra | 20:16 | |
| piet | Hi Folks | 20:17 |
| piet | The OpenStack UX team is conducting a series of interviews focused on hardware provisioning with the intent of using customer feedback to help drive the development of the OpenStack Ironic project. Individuals using MaaS, Cobbler and Foreman are also welcome to participate in the study. | 20:17 |
| *** apuimedo has joined #openstack-infra | 20:17 | |
| anteaya | hey piet | 20:17 |
| piet | If interested, please select a date/time from the following link http://doodle.com/poll/8a7ftkqe4d2bacts | 20:17 |
| piet | anteaya: hiya | 20:17 |
| *** apuimedo_ has joined #openstack-infra | 20:17 | |
| anteaya | piet: um have you given any thought to your communication framework for inviting folks to your studies? | 20:18 |
| anteaya | the infra channel isn't really meant as a bulletin board for announcements of this sort | 20:18 |
| piet | anteaya: It's a bit tricky | 20:19 |
| *** flepied1 has quit IRC | 20:19 | |
| anteaya | but we would like to support a communication framework which does meet your needs | 20:19 |
| * anteaya listens | 20:19 | |
| piet | anteaya: Limesurvey would help with that | 20:19 |
| anteaya | fair enough | 20:19 |
| fungi | presumably it's at least being posted to the operators ml, since that seems like the primary audience for this particular study | 20:19 |
| piet | anteaya: we would like to create a screener to identify everyone's interest. At that point, their would be a very focused email blast 1-2 times a year | 20:20 |
| anteaya | by screener you are thinking of a tool like lime survey? | 20:21 |
| *** maishsk has quit IRC | 20:22 | |
| *** maishsk has joined #openstack-infra | 20:22 | |
| erlon | all: guys is there somewhere we can give feedback about the new gerrit interface? | 20:23 |
| *** tonytan4ever has joined #openstack-infra | 20:24 | |
| piet | anteaya: Screeners are surveys intended to create databases of folks based on their interests, specialities, etc. At that point, you can review the database and identify folks that meet specific recruitment criteria for a study and just send emails to them. | 20:24 |
| *** markvoelker has quit IRC | 20:24 | |
| fungi | erlon: https://groups.google.com/forum/#!forum/repo-discuss and https://code.google.com/p/gerrit/issues/list | 20:25 |
| erlon | fungi: thank you | 20:25 |
| fungi | erlon: though i expect it's pretty frozen at this point while they work on the design for their newer "polygerrit" interface | 20:26 |
| *** apuimedo has quit IRC | 20:26 | |
| anteaya | piet: okay great, so that is the long term vision, and it sounds like a good one | 20:26 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix commit message editing in >= 2.11 https://review.openstack.org/259173 | 20:26 |
| jeblair | fungi, zaro: ^ | 20:26 |
| zaro | jeblair: i noticed that change edit endpoints are new in 2.11, wasn't there in 2.8. So i'm guessing what broke is submit change endpoint? | 20:26 |
| erlon | fungi: polygerrit?? | 20:26 |
| anteaya | piet: what can be done in the interm while the long term vision takes shape? | 20:26 |
| *** zeih has joined #openstack-infra | 20:26 | |
| *** e0ne has joined #openstack-infra | 20:27 | |
| *** ybathia has joined #openstack-infra | 20:27 | |
| jeblair | zaro: pursehouse removed the 'edit commit message' endpoint in 2.11. | 20:27 |
| *** apuimedo_ has quit IRC | 20:27 | |
| fungi | erlon: https://gerrit.googlesource.com/gerrit/+/refs/heads/master/polygerrit-ui/ | 20:27 |
| *** apuimedo has joined #openstack-infra | 20:28 | |
| piet | anteaya: The hard answer is that we'll need to keep doing this until we set-up a database. We don't send invites all that often given the amount of research that is needed to improve the experience | 20:28 |
| *** binbincong_ has quit IRC | 20:28 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix commit message editing in >= 2.11 https://review.openstack.org/259173 | 20:28 |
| erlon | fungi: hmmm, so, this new interface is temporary? say yes please :) | 20:28 |
| jeblair | (reviewed my own code) | 20:29 |
| erlon | fungi: lol | 20:29 |
| fungi | erlon: in software, everything is temporary | 20:29 |
| erlon | fungi: yes, the question 'how long' | 20:29 |
| zaro | jeblair: ahh i see it now. | 20:29 |
| piet | anteaya: want the brutal answer? | 20:29 |
| erlon | fungi: but thanks, Ill try to reach guys there | 20:30 |
| *** kzaitsev_mb has joined #openstack-infra | 20:30 | |
| anteaya | piet: what I am saying is you have made a decision about how you want to use the infra channel without any input from the infra team | 20:30 |
| anteaya | piet: I'm suggesting that getting feedback from the infra team about your choices might be in order here | 20:30 |
| *** tjones has quit IRC | 20:30 | |
| *** matrohon has joined #openstack-infra | 20:30 | |
| jeblair | erlon: my guess is 2 years. with a big error margin. | 20:31 |
| *** zeih has quit IRC | 20:31 | |
| fungi | erlon: essentially, the current interface is the second (third? depends on how you count them) major revision of gerrit's web ui. they're working on yet another one (written in polymer) and they also provide a rest interface in gerrit which is quite suited to building new user interfaces on top (for example https://pypi.python.org/pypi/gertty is one jeblair wrote for console-based reviewing) | 20:31 |
| jeblair | erlon: at least it's not "weeks" or "months" temporary. | 20:31 |
| *** apuimedo_ has joined #openstack-infra | 20:32 | |
| *** esker has joined #openstack-infra | 20:32 | |
| *** esker has quit IRC | 20:32 | |
| *** abregman has joined #openstack-infra | 20:32 | |
| erlon | jeblair: o_O, well that's a long time | 20:32 |
| *** tjones has joined #openstack-infra | 20:33 | |
| *** esker has joined #openstack-infra | 20:33 | |
| *** regXboi has joined #openstack-infra | 20:33 | |
| erlon | fungi: I remember this interface, was optional, I have used it once | 20:33 |
| *** kzaitsev_mb has quit IRC | 20:36 | |
| piet | anteaya: That's fair. I guess my feedback is that there is a culture that folks should be contributing to OpenStack. I would propose that part of the contribution should include participating in the occasional studies. | 20:36 |
| anteaya | thanks for understanding | 20:36 |
| anteaya | and you make a reasonable follow up point | 20:36 |
| anteaya | I think perhaps if you were to put an item on the infra meeting agenda we could discuss this | 20:37 |
| *** regXboi has quit IRC | 20:37 | |
| anteaya | and we might be able to come up with ways to support your communication strategy | 20:37 |
| fungi | erlon: yes, the "new" interface used to be optional, but as of this version of gerrit it's ceased being a preview or optional in any way and the previous interface is no longer included as an option | 20:37 |
| piet | anteaya: Learned the lesson from David Lyle. I was constantly whining about Horizon and he basically told me to contribute or shut up (in a nice way...) | 20:37 |
| anteaya | and I am fine with your definition of contribution, I think it is reasonable | 20:38 |
| anteaya | david-lyle says things in the nicest way :) | 20:38 |
| anteaya | so yes contributing is important, I agree | 20:38 |
| *** kzaitsev_mb has joined #openstack-infra | 20:38 | |
| fungi | piet: as a reminder (hopefully you can already take advantage of this) the openstack weekly newsletter can mention things like ui/ux studies if you give them a heads up | 20:38 |
| rhedlind | how do I install the master revision of zuul? I have tried pip install -U zuul, but it doesn't give me the latest. I assume there some environment setting I am missing. Any pointers? | 20:39 |
| piet | fungi: Yep...you've mentioned few times and I keep forgetting to reach-out through the newsletter. My bad... | 20:39 |
| fungi | also i wonder if places like "superuser magazine" might be open to community "advertisements" about things like research studies | 20:39 |
| fungi | could be an interesting way to integrate more temporary calls for help/feedback | 20:40 |
| *** vgridnev has quit IRC | 20:40 | |
| *** dkranz has quit IRC | 20:41 | |
| piet | anteaya: But would appreciate recommendations on recruiting for studies because it's one of my biggest challenges. Helping with LimeSurvey is a big step in the right direction. | 20:41 |
| piet | anteaya: brb | 20:41 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Add containers job to tripleo.org CI status https://review.openstack.org/259177 | 20:41 |
| anteaya | piet: awesome, can you add an item to next week's infra meeting and perhaps we can brainstorm then? | 20:41 |
| anteaya | piet: I think our agenda will be light anyway so we could use a topic to discuss :) | 20:42 |
| anteaya | piet: and fungi suggested a couple of good ideas in backscroll | 20:42 |
| *** avarner has joined #openstack-infra | 20:42 | |
| anteaya | weekly newsletter, superuser magazine | 20:43 |
| fungi | no need to wait a week to solicit suggestions on strategies for reaching various corners of the openstack community when recruiting for things like openstack-focused research studies and surveys. a thread on one of our more active mailing lists might be a good way to get more ideas on how to reach people | 20:43 |
| *** sridhar_ram has quit IRC | 20:43 | |
| *** mudassirlatif has quit IRC | 20:44 | |
| *** placko has quit IRC | 20:44 | |
| *** triggerwarning has quit IRC | 20:44 | |
| *** triggerwarning has joined #openstack-infra | 20:44 | |
| *** triggerwarning has quit IRC | 20:44 | |
| *** triggerwarning has joined #openstack-infra | 20:44 | |
| *** sridhar_ram has joined #openstack-infra | 20:44 | |
| *** mudassirlatif has joined #openstack-infra | 20:45 | |
| *** Swami_ has quit IRC | 20:45 | |
| *** mudassirlatif has quit IRC | 20:45 | |
| fungi | especially if the thread starts out as a "how can we reach the right audience for this sort of thing" rather than starting out as a call for participation and ending up instead as a discussion on where would be better to advertise it | 20:46 |
| anteaya | rhedlind: we use this module for deploying zuul: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/ | 20:47 |
| *** _nadya_ has quit IRC | 20:48 | |
| *** gordc has joined #openstack-infra | 20:48 | |
| *** apuimedo has quit IRC | 20:49 | |
| anteaya | rhedlind: now this part installs zuul: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/init.pp#n129 | 20:49 |
| *** hashar has joined #openstack-infra | 20:49 | |
| anteaya | rhedlind: but it uses vcsrepo to git clone the latest version first: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/init.pp#n122 | 20:50 |
| *** apuimedo_ has quit IRC | 20:50 | |
| *** apuimedo has joined #openstack-infra | 20:51 | |
| *** otter768 has joined #openstack-infra | 20:51 | |
| openstackgerrit | Merged openstack/gertty: Fix syncing changes with comments on a missing file https://review.openstack.org/242883 | 20:52 |
| openstackgerrit | Merged openstack/gertty: Separate search and refine search commands https://review.openstack.org/238260 | 20:52 |
| openstackgerrit | Merged openstack/gertty: Add a simple kill ring https://review.openstack.org/238261 | 20:53 |
| openstackgerrit | Merged openstack/gertty: Add interactive search to diff view https://review.openstack.org/238262 | 20:53 |
| openstackgerrit | Merged openstack/gertty: Add navigation to interactive search https://review.openstack.org/238263 | 20:53 |
| openstackgerrit | Merged openstack/gertty: Add help entries for kill, yank, isearch https://review.openstack.org/242884 | 20:53 |
| openstackgerrit | Merged openstack/gertty: Support multiple key input https://review.openstack.org/242885 | 20:53 |
| *** aeng has joined #openstack-infra | 20:53 | |
| anteaya | jeblair: you've been busy | 20:53 |
| *** piet has quit IRC | 20:53 | |
| jeblair | oh they've been sitting there for weeks | 20:53 |
| anteaya | ah | 20:54 |
| fungi | jeblair: awesome! i've been using those for a while without issue. i'll switch to master tip now and pull in the commit message editor | 20:54 |
| fungi | fix | 20:54 |
| jeblair | fungi: that one is in the queue still | 20:54 |
| fungi | yep, i meant pull in that fix separately | 20:54 |
| *** apuimedo_ has joined #openstack-infra | 20:55 | |
| *** otter768 has quit IRC | 20:56 | |
| *** daemontool has quit IRC | 20:57 | |
| rhedlind | anteaya: thanks! | 20:57 |
| *** piet has joined #openstack-infra | 20:58 | |
| *** daemontool has joined #openstack-infra | 20:58 | |
| openstackgerrit | Merged openstack/gertty: Fix commit message editing in >= 2.11 https://review.openstack.org/259173 | 20:58 |
| anteaya | rhedlind: welcome | 20:58 |
| anteaya | rhedlind: thanks for coming back :) | 20:59 |
| fungi | oh, didn't realize 259173 had also gotten approved. switching to plain master tip now | 20:59 |
| *** bharathm has joined #openstack-infra | 21:00 | |
| piet | anteaya; Yep, caught Fungi comment about newsletter. Are you referring to Tuesday's meeting? | 21:00 |
| *** mudassirlatif has joined #openstack-infra | 21:00 | |
| anteaya | I am but fungi makes a very good point about considering starting a thread on the mailing list | 21:01 |
| anteaya | and makes an additional good point about the topic suggestion | 21:01 |
| piet | anteaya: Distribution list? | 21:01 |
| anteaya | what list were you thinking fungi? | 21:01 |
| *** mudassirlatif has quit IRC | 21:01 | |
| *** apuimedo_ has quit IRC | 21:02 | |
| *** maishsk_ has joined #openstack-infra | 21:02 | |
| rhedlind | anteaya: http://paste.openstack.org/show/482232/ . This is the zuul log from my CI, where it keeps posting comments on the same change over and over. It fails to post the job to jenkins, posts a comment, and retries that sequence over and over. | 21:02 |
| fungi | a thread on the -dev ml is probably likely to get more people involved in the discussion (just because it's got more people subscribed) though a thread on the -community ml may actually have a more diverse base for ideas around reaching different parts of the community | 21:03 |
| *** maishsk has quit IRC | 21:03 | |
| *** maishsk_ is now known as maishsk | 21:03 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Better duplicate detection for project requirements check https://review.openstack.org/238871 | 21:03 |
| rhedlind | the jenkins process had crash because it could not allocate memory. zuul had eaten up all available memory. | 21:03 |
| piet | anteaya: fungi: Stepping away for some Cajun food | 21:03 |
| *** lykinsbd has quit IRC | 21:04 | |
| piet | anteaya: It would also be helpful to have a conversation around why the studies are bothering folks. Is it the email blasts? | 21:04 |
| anteaya | piet: no it is just you could use your time better to reach your audience | 21:05 |
| *** lykinsbd has joined #openstack-infra | 21:05 | |
| anteaya | piet: we would like to help you reach your goal | 21:05 |
| rhedlind | This is the zuul.layout I'm using. http://paste.openstack.org/show/482233/ . I would appreciate if someone can look it over and call out what I am doing wrong in there. | 21:05 |
| anteaya | piet: right now it looks like you could use your time better | 21:06 |
| *** salv-orl_ has joined #openstack-infra | 21:06 | |
| *** esker has quit IRC | 21:06 | |
| piet | anteaya: agreed...and I don't really like blasting folks because it's annoying for folks receiving the emails and takes alot of my time, but no other options yet... | 21:06 |
| *** binbincong has joined #openstack-infra | 21:06 | |
| anteaya | rhedlind: okay glad you found the log that has the loop, hopefully some folks in addition to me can look and we can see what is happening | 21:06 |
| *** apuimedo_ has joined #openstack-infra | 21:06 | |
| anteaya | piet: right well we would like to help expand the options | 21:07 |
| *** jswarren has left #openstack-infra | 21:08 | |
| *** jswarren has joined #openstack-infra | 21:08 | |
| anteaya | rhedlind: in the log is this the first instance of this line? 2015-12-17 06:28:15,059 INFO zuul.Gerrit: Updating information for 252250,13 | 21:08 |
| openstackgerrit | Merged openstack-infra/storyboard: Workaround 500 error when moving items in worklists https://review.openstack.org/259128 | 21:09 |
| rhedlind | anteaya: no, probably not. I grabbed a snippet. I will find the first instance and paste it. | 21:09 |
| anteaya | so this is the gearman error: GearmanError: Unable to submit job to any connected servers | 21:09 |
| anteaya | rhedlind: okay and if you can about 50 to 100 lines preceeding | 21:10 |
| *** kzaitsev_mb has quit IRC | 21:10 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fix various style bugs with cards https://review.openstack.org/259131 | 21:12 |
| *** kzaitsev_mb has joined #openstack-infra | 21:12 | |
| *** Swami_ has joined #openstack-infra | 21:12 | |
| *** mudassirlatif has joined #openstack-infra | 21:15 | |
| *** rbrndt has quit IRC | 21:16 | |
| *** matrohon has quit IRC | 21:16 | |
| *** mudassirlatif has quit IRC | 21:16 | |
| rhedlind | anteaya: a much longer log: http://paste.openstack.org/show/482234/ | 21:17 |
| * anteaya looks | 21:18 | |
| anteaya | so line 205: 2015-12-16 22:07:33,232 ERROR zuul.Scheduler: Exception in run handler: | 21:20 |
| anteaya | that is our first occurance, yes? | 21:20 |
| rhedlind | anteaya: yes | 21:21 |
| anteaya | but before that we had a whole bunch of these: 2015-12-16 22:07:33,014 ERROR zuul.Gearman: Unable to find build 77f3a573f4ea46b0bd426e59282c6505 | 21:23 |
| *** kgiusti has quit IRC | 21:23 | |
| anteaya | now I am far from a zuul expert | 21:23 |
| anteaya | are the unable to find build errors normal? | 21:23 |
| *** rossella_s has quit IRC | 21:23 | |
| anteaya | rhedlind: have you seen them in prior logs? | 21:23 |
| *** rossella_s has joined #openstack-infra | 21:23 | |
| *** krtaylor has quit IRC | 21:24 | |
| rhedlind | no, I have never seen those before. I think that is because the jenkins process crashed. | 21:24 |
| *** markvoelker has joined #openstack-infra | 21:25 | |
| *** apuimedo_ has quit IRC | 21:25 | |
| anteaya | okay | 21:26 |
| anteaya | so two ways to go about this and I think you should consider both | 21:26 |
| anteaya | one, protect yourself | 21:26 |
| anteaya | if a jenkins crashes you want your system to recognize that somehow and stop posting to infra's gerrit | 21:27 |
| anteaya | an alert to you is also an idea but up to you | 21:27 |
| openstackgerrit | Merged openstack-infra/reviewstats: Gerrit 2.11 compatibility https://review.openstack.org/259151 | 21:27 |
| anteaya | two, help figure out in upstream zuul how to recognize when a loop is occuring and provide a mechanism that escapes the loop | 21:28 |
| anteaya | provide a patch and see you reviewers agree that it would be helpful and also if they agree with your approach | 21:28 |
| *** markvoelker has quit IRC | 21:30 | |
| *** apuimedo_ has joined #openstack-infra | 21:30 | |
| rhedlind | anteaya: I agree with both. I will take a look at the zuul source to see if I can identify the loop (if it still exists in the latest version). I was running 2.1.1.dev15 when this happened and now running the latest (2.1.1.dev109). | 21:30 |
| *** mudassirlatif has joined #openstack-infra | 21:30 | |
| anteaya | okay great | 21:30 |
| anteaya | did you get 2.1.1.dev109 installed okay? | 21:31 |
| *** mudassirlatif has quit IRC | 21:31 | |
| rhedlind | anteaya: yes, it installed without any complains. | 21:31 |
| anteaya | congratulations | 21:31 |
| rhedlind | what comments did the CI post? was it FAILED or something else? | 21:31 |
| anteaya | I never tried to open the file | 21:32 |
| rhedlind | ok | 21:32 |
| anteaya | the size would have brought down my browser | 21:32 |
| anteaya | so I don't know | 21:32 |
| rhedlind | :/ | 21:32 |
| pleia2 | mtreinish: still about to watch the health dashboard if I approve? | 21:32 |
| *** piet has quit IRC | 21:32 | |
| anteaya | if and infra-root still has the comment content in logs hopefully they will post to you, rhedlind | 21:32 |
| anteaya | if an* | 21:32 |
| mtreinish | pleia2: yep | 21:33 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone eventlet gate only in keystone https://review.openstack.org/257999 | 21:33 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Change postgres job to use standard deployment https://review.openstack.org/258006 | 21:33 |
| pleia2 | mtreinish: ok, lgtm, I'm around for another 2.5 hours if you need anything else (then STAR WARS!) | 21:34 |
| *** bhunter71 has quit IRC | 21:34 | |
| anteaya | rhedlind: also if when you reply to the email on third party announce do include a complete debrief with as many details as possible | 21:34 |
| rhedlind | anteaya: that could be useful to see what the comment content was. | 21:34 |
| rhedlind | anteaya: ok | 21:34 |
| mtreinish | pleia2: ooh lucky. I'm not seeing it until tomorrow | 21:34 |
| anteaya | rhedlind: hopefully this will provide information to others so they check their systems and don't have the same issue | 21:35 |
| mtreinish | pleia2: ok, once that merges I'll approve the o-h side changes | 21:35 |
| anteaya | rhedlind: I do think we may have had to clear gerrit's db in order to get zuul moving again | 21:35 |
| *** smarcet has quit IRC | 21:35 | |
| anteaya | rhedlind: it is possible we don't have access to the comment content anymore, it was a decision we had to make quickly for the benefit of our system | 21:35 |
| pleia2 | mtreinish: great | 21:35 |
| anteaya | pleia2: STAR WARS! | 21:36 |
| pleia2 | anteaya :D | 21:36 |
| openstackgerrit | Davide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job https://review.openstack.org/259188 | 21:36 |
| *** apuimedo_ has quit IRC | 21:37 | |
| *** krtaylor has joined #openstack-infra | 21:37 | |
| *** avarner_ has joined #openstack-infra | 21:37 | |
| *** flepied has joined #openstack-infra | 21:37 | |
| rhedlind | anteaya: ok, understood | 21:38 |
| anteaya | rhedlind: thank you, get some rest tonight, and come back tomorrow | 21:38 |
| anteaya | :) | 21:38 |
| anteaya | we all make mistakes | 21:38 |
| ruagair | o/ | 21:39 |
| pabelanger | fungi: jeblair: Do either of you mind cutting a new release of JJB? I don't think we've released the latest version to include the new template macro | 21:39 |
| anteaya | I can tell you the story sometime of how as a crew member who knew noone had to keep 60 crew members waiting for about 3 minutes while I loaded a film magazine (when I was a camera assistant) | 21:39 |
| anteaya | the first camera assistant was the president of the camera union | 21:40 |
| rhedlind | anteaya: thanks for your help. I will respond on the mailing list. | 21:40 |
| anteaya | I was more than a bit queasy walking that mag into a silent studio I can tell you | 21:40 |
| anteaya | rhedlind: sounds good, thanks | 21:40 |
| *** avarner has quit IRC | 21:41 | |
| anteaya | it wasn't my fault, but it was my responsibility | 21:41 |
| *** apuimedo has quit IRC | 21:42 | |
| *** apuimedo has joined #openstack-infra | 21:42 | |
| *** apuimedo_ has joined #openstack-infra | 21:43 | |
| openstackgerrit | Davide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job https://review.openstack.org/259188 | 21:44 |
| fungi | rhedlind: anteaya: the comments it was posting were "Build failed. For information on how to proceed, see https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures" | 21:44 |
| anteaya | fungi: thank you | 21:45 |
| fungi | this is according to our (copious) gerrit ssh logs | 21:45 |
| *** jpr_ has quit IRC | 21:45 | |
| *** mudassirlatif has joined #openstack-infra | 21:45 | |
| *** mudassirlatif has quit IRC | 21:46 | |
| fungi | rhedlind: anteaya: the account seems to have left that message >8000 times starting a few minutes after 05:00 utc | 21:46 |
| fungi | and ending when we deactivated the account and disconnected its ssh connection | 21:46 |
| *** apuimedo has quit IRC | 21:47 | |
| anteaya | rhedlind: so the server that created these logs is not on utc time? http://paste.openstack.org/show/482234/ | 21:47 |
| jhesketh | Morning | 21:48 |
| anteaya | morning jhesketh | 21:48 |
| *** apuimedo_ has quit IRC | 21:48 | |
| phschwartz | Jhesketh: morning | 21:48 |
| *** apuimedo has joined #openstack-infra | 21:49 | |
| fungi | pabelanger: new jjb release should be discussed with zaro, pelix and zxiiro, but i'm in favor of early and often | 21:49 |
| fungi | we've also had a recent request for a glean release | 21:49 |
| pabelanger | fungi: Ya, there is a request on ML for one but think it was missed | 21:50 |
| fungi | mordred: greghaynes: SpamapS: ideas on whether we should release glean soon? | 21:50 |
| pabelanger | http://lists.openstack.org/pipermail/openstack-infra/2015-December/003530.html | 21:50 |
| mgagne | jeblair you around? could you try spawning a node at internap? discount and aggregate are in place. I wish to test its proper working | 21:50 |
| * zxiiro would like to see a new release soon. It would help our users who are having issues with the current 1.3.0 version | 21:50 | |
| fungi | mordred: greghaynes: SpamapS: ian mentioned some useful fixes since the last release | 21:50 |
| fungi | ianw that was | 21:51 |
| greghaynes | I like releases, we should do them | 21:51 |
| greghaynes | I lack perms to release glean I think | 21:51 |
| greghaynes | but no objection from me | 21:51 |
| fungi | greghaynes is always easy to please | 21:51 |
| *** apuimedo_ has joined #openstack-infra | 21:51 | |
| greghaynes | and ya, I there are some fixes. There is one I just +A'd too let me see if it merged... | 21:52 |
| openstackgerrit | Davide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job https://review.openstack.org/259188 | 21:52 |
| clarkb | greghaynes: https://review.openstack.org/#/c/232251/13..17/nodepool/cmd/builder.py looks much better now | 21:52 |
| rhedlind | anteaya: no, it is on MST | 21:52 |
| fungi | pabelanger: did you get a chance to try out the openstack_project:openstackid_dev class on ubuntu trusty and figure out what needs tweaking/adding (if anything)? | 21:53 |
| greghaynes | looks like the fix I wanted just merged, so ya release whenever | 21:53 |
| anteaya | rhedlind: if you do decide to move it to utc, it may help match up timestamps with infra events in future | 21:53 |
| greghaynes | clarkb: w00t | 21:53 |
| anteaya | rhedlind: all our servers are on utc | 21:54 |
| rhedlind | anteaya: ok, I will look into that. It makes sense. | 21:54 |
| fungi | jeblair: jhesketh: was there a fix proposed for that zuul logging performance regression you were discussing? need reviews? | 21:54 |
| clarkb | greghaynes: http://logs.openstack.org/51/232251/17/check/gate-dsvm-nodepool/c0895f4/logs/screen-nodepool-builder.txt.gz there are even logs that don't say much | 21:54 |
| anteaya | thanks, thought I would mention it as something to consider, it isn't required | 21:54 |
| openstackgerrit | Davide Agnello proposed openstack-infra/project-config: Moving Cue rally test to non-voting check job https://review.openstack.org/259188 | 21:55 |
| greghaynes | clarkb: derp, that should be logging at debug | 21:55 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Clarify zuul-cloner message about missing branch https://review.openstack.org/259193 | 21:55 |
| greghaynes | clarkb: so my logging config is messed up somehow | 21:55 |
| *** dkranz has joined #openstack-infra | 21:55 | |
| pabelanger | fungi: I had to push it to tomorrow | 21:55 |
| clarkb | greghaynes: ok, based on the nodepoold logs I am pretty sure its doing stuff | 21:55 |
| clarkb | so ya wouldn't surprise me if the logging is just not right | 21:55 |
| fungi | pabelanger: no worries, just curious, and thanks! | 21:56 |
| pabelanger | fungi: I should know by lunch EST | 21:56 |
| jhesketh | fungi, jeblair: not sure if it's a logging performance, but I was looking at a performance regression due to the cache management. Unfortunately it's tightly coupled and I haven't finished yet. I'll poke more at that today and this weekend. | 21:56 |
| *** flepied has quit IRC | 21:56 | |
| fungi | pabelanger: cool--i'm excited to hear what you find out, and i'll plan the server replacement once you get time to check it out | 21:56 |
| jhesketh | fungi, jeblair: I had it down to just one test failing | 21:57 |
| *** alivigni has quit IRC | 21:57 | |
| *** gordc has quit IRC | 21:57 | |
| fungi | jhesketh: ahh, okay. i think jeblair had said something about a bug with changes to logging behavior ended up moving it into the main thread (leading to a possible explanation for why the event flood earlier today was more devastating than we've seen in the past) | 21:58 |
| clarkb | greghaynes: the double logging in nodepoold's log is really annoying/weird too I should track that down | 21:58 |
| fungi | jhesketh: though it's possible i misread what he was suggesting there | 21:58 |
| *** piet has joined #openstack-infra | 21:59 | |
| onovy | sorry, i'm idiot (i know). how can i remove branch? [remote rejected] debian/unstable (cannot delete references) | 21:59 |
| *** lykinsbd has quit IRC | 21:59 | |
| fungi | onovy: needs gerrit admin assistance (or did in the past, though maybe it's easier to fine-tune the acls now that we're on 2.11 so we should revisit that policy at some point) | 21:59 |
| jhesketh | fungi: hmm okay. It's possible the cache regression causes more logs as it has more to figure out. Otherwise it could be something else | 22:00 |
| *** mudassirlatif has joined #openstack-infra | 22:00 | |
| onovy | fungi, so should i write email to mailist? | 22:00 |
| *** mudassirlatif has quit IRC | 22:01 | |
| fungi | jhesketh: at 16:40 utc in scrollback | 22:02 |
| *** triggerwarning is now known as dcentral | 22:02 | |
| * jhesketh looks | 22:02 | |
| fungi | onovy: hopefully someone in infra-root with a spare moment can delete the "debian/unstable" branch on... what repo? you're working on openstack/swauth? | 22:03 |
| onovy | fungi, yes, openstack/swauth | 22:03 |
| mgagne | jeblair ok, tested aggregate, works fine | 22:04 |
| fungi | jhesketh: i should have clarified, _connection_ logging | 22:05 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health: Update default api configurations https://review.openstack.org/247492 | 22:06 |
| *** spzala has quit IRC | 22:06 | |
| pleia2 | fungi: is there work to do on the git servers for that, or is it just deleting in gerrit? | 22:07 |
| *** thorst_ has quit IRC | 22:07 | |
| *** alivigni has joined #openstack-infra | 22:07 | |
| mtreinish | ok, that landed. Let me push through the other changes | 22:08 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Ensure python 3.3 is also listed for enum34 https://review.openstack.org/259196 | 22:08 |
| pleia2 | mtreinish: huzzah | 22:08 |
| fungi | jhesketh: so maybe the broken connection logging was not directly related to the connections bug he discussed with you (which is what seems to be unnecessarily burdening the main thread) | 22:08 |
| fungi | pleia2: just deleting it in gerrit. helps to echo the final git sha of the branch into the channel here for posterity | 22:08 |
| pleia2 | fungi: right, on it | 22:09 |
| *** pigmej has left #openstack-infra | 22:09 | |
| pleia2 | onovy: deleted debian/unstable with final git sha of 008237924b5e008421193596e1db86e62ceda50a | 22:09 |
| onovy | pleia2, perfect, thanks a lot and sorry for it | 22:09 |
| jeblair | jhesketh, fungi: logging was separate: logger names changed -- the potential regression is very related to what jhesketh is already working on, but with the additional observation that we have inadvertently moved some gerrit query work into the main thread; jhesketh's fix should still fix that | 22:09 |
| pleia2 | onovy: you're welcome | 22:09 |
| ruagair | o/ | 22:10 |
| jeblair | mgagne: i'm back; need me to do anything? | 22:10 |
| jeblair | pabelanger: did anyone make that jjb release? | 22:10 |
| *** abregman is now known as abregman|ZzZzZz | 22:10 | |
| *** trown is now known as trown|outttypeww | 22:10 | |
| mgagne | jeblair I guess you could try create a nodepool node just to make sure you still can and it works ok. I'm working on the quota | 22:10 |
| fungi | jeblair: oh, great thanks! i'll keep an eye on that one | 22:11 |
| *** dcentral has quit IRC | 22:11 | |
| *** dcentral has joined #openstack-infra | 22:11 | |
| * jhesketh finishes scrollback. | 22:11 | |
| pabelanger | jeblair: I don't think so | 22:11 |
| jhesketh | Yep, what jeblair said | 22:12 |
| *** flepied has joined #openstack-infra | 22:12 | |
| *** xyang1 has quit IRC | 22:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Increase internap to 1 instance https://review.openstack.org/259198 | 22:12 |
| fungi | jeblair: nobody's tagged it yet. i was going to see what semver-y number we should be increasing to, so was hoping zaro/pelix/zxiiro could summarize the extent of changes since the last release | 22:12 |
| pabelanger | jeblair: 1.3.0 the last one, from aug 26 | 22:12 |
| jeblair | mgagne: at this point, that's the easiest way to do that ^ https://review.openstack.org/259198 | 22:12 |
| openstackgerrit | Drew Varner proposed openstack-infra/project-config: Experimental bandit job for glance https://review.openstack.org/258130 | 22:12 |
| jeblair | fungi: ok, that's about what i was going to do too :) | 22:12 |
| fungi | perfect | 22:13 |
| jeblair | fungi: could i convince you to take a look at https://review.openstack.org/252973 ? | 22:13 |
| *** gyee has joined #openstack-infra | 22:13 | |
| mgagne | jeblair haha. quota is now 30. I think it would be safe to set max-servers to 25. | 22:13 |
| jeblair | i'm about to release gertty and wondered if that should go in. | 22:13 |
| fungi | jeblair: sure | 22:14 |
| zxiiro | fungi: There's been quite a lot of changes since 1.3.0 so I think it should be a 1.4.0 release. | 22:15 |
| jeblair | mgagne: let's set it to 1 for now to make sure it's working first; if there's a problem, nodepool might end up trying to put too many instances there | 22:15 |
| mgagne | jeblair right, good call | 22:15 |
| jeblair | mgagne: we can bump it to 25 quickly after that | 22:15 |
| *** zeih has joined #openstack-infra | 22:15 | |
| *** mudassirlatif has joined #openstack-infra | 22:15 | |
| mgagne | jeblair and I'll be off for ~2 weeks starting tonight :D | 22:15 |
| jeblair | heh | 22:15 |
| *** mudassirlatif has quit IRC | 22:16 | |
| *** flepied has quit IRC | 22:17 | |
| ianw | greghaynes: here's another quick one before a glean release -> https://review.openstack.org/#/c/258763/ | 22:17 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command https://review.openstack.org/258725 | 22:17 |
| ianw | greghaynes: stops a bunch of warnings in the logs | 22:17 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Remove duplicate log entries https://review.openstack.org/259199 | 22:18 |
| clarkb | greghaynes: ^ I think that may remove duplicate logs | 22:18 |
| kzaitsev_mb | hey, can I ask someone to help me better understand how i18n in OS works? | 22:18 |
| kzaitsev_mb | I mean the automation — who does what? =) | 22:18 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/log_processor: Let the workers shut down gracefully https://review.openstack.org/259200 | 22:18 |
| kzaitsev_mb | Guessed this would be a good place to ask. =) | 22:18 |
| *** eharney has quit IRC | 22:18 | |
| fungi | kzaitsev_mb: pleia2 is probably the most awake person now who's had a recent hand in that | 22:19 |
| fungi | maybe also StevenK | 22:19 |
| pleia2 | we have a page about that, sec | 22:19 |
| kzaitsev_mb | oh, that'd be great. =) | 22:19 |
| pleia2 | kzaitsev_mb: this is a high level view: http://docs.openstack.org/infra/system-config/translate.html | 22:19 |
| pleia2 | kzaitsev_mb: the i18n team maintains another, I'll dig that up now | 22:20 |
| *** zeih has quit IRC | 22:20 | |
| *** bdemers has quit IRC | 22:20 | |
| pleia2 | kzaitsev_mb: here you go: https://wiki.openstack.org/wiki/Translations/Infrastructure#Workflow | 22:20 |
| *** avarner__ has joined #openstack-infra | 22:21 | |
| *** rossella_s has quit IRC | 22:21 | |
| anteaya | zaro: beside the reply button in a patch is a button marked workflow+1, it is a blue button, why is it there? | 22:21 |
| anteaya | zaro: I set workflow +1 using the radio button in a reply | 22:21 |
| clarkb | nibalizer: crinkle for https://review.openstack.org/#/c/165240/20/modules/openstack_project/manifests/wheel_mirror_slave.pp it has several lists of packages to isntall, is there some way to automagically install all the packages we have listed in jenkins_params.pp ? | 22:21 |
| *** rossella_s has joined #openstack-infra | 22:21 | |
| kzaitsev_mb | I've finally convinced most of the guys on our team & PTL, that we should start i18n murano and we have some volounteers ready to do the actual translation, so I'm in the process of figuring out the machinery we miss for that =) | 22:21 |
| kzaitsev_mb | thanks =) | 22:22 |
| *** avarner__ is now known as avarner | 22:22 | |
| anteaya | zaro: is leaving a reply and setting a radio button considered too much effort? | 22:22 |
| clarkb | anteaya: ya its just a shortcut, gerrit knows you have perms to add that flag and it knows the change needs it to merge | 22:22 |
| clarkb | so its like "hey here is easy mode button" | 22:22 |
| pleia2 | kzaitsev_mb: that's great, this defines specifically how we do translatiosn on the back end, if you want translator-level docs for the actual translating work you probably want to start with https://wiki.openstack.org/wiki/Translations | 22:22 |
| *** [1]Thelo has joined #openstack-infra | 22:23 | |
| ianw | clarkb: i think i wrote that ages ago : https://review.openstack.org/#/c/234500/ | 22:23 |
| anteaya | clarkb: okey dokey, thanks | 22:23 |
| pleia2 | kzaitsev_mb: they'll also want to join the i18n team: https://wiki.openstack.org/wiki/I18nTeam | 22:23 |
| clarkb | ianw: woo | 22:23 |
| anteaya | clarkb: the +A encouragement | 22:23 |
| clarkb | ianw: actually we do want it to propogate don't we? | 22:24 |
| clarkb | but maybe I am wrong for some reason I was thinking that it affected below nodepool as well | 22:24 |
| clarkb | let me check the lgos and can abandon mine if yours works | 22:24 |
| *** avarner_ has quit IRC | 22:24 | |
| *** bpokorny_ has joined #openstack-infra | 22:25 | |
| kzaitsev_mb | pleia2: no, I actually want to figure out the backend stuff at first. Like who updates pot files (if they're updated automatically at all) and how to register in zatana and other stuff in the links you posted, thanks =) | 22:25 |
| clarkb | ya I must've read logging docs wrong as that seems to work | 22:25 |
| * ianw just contexting switch back to what the original problem was .... | 22:25 | |
| *** Thelo has quit IRC | 22:25 | |
| *** [1]Thelo is now known as Thelo | 22:25 | |
| clarkb | ianw: I actually think both methods work but yours is more flexible ifwe switch handlers between them | 22:26 |
| *** spzala has joined #openstack-infra | 22:26 | |
| clarkb | so will approve yours and abandon mine | 22:26 |
| pleia2 | kzaitsev_mb: great, feel free to reach out if you have questions, if I'm not around StevenK and AJaeger are also well-versed in all of this, and there's always the i18n mailing list too | 22:27 |
| mtreinish | pleia2: well puppet picked up the conf file change and the api server still works. Still waiting on the code to use those options to work it's way through the gate | 22:27 |
| ianw | clarkb: now i remember being surprised by the root logger not filtering out the lower level messages. you'd think it would filter it, but really it juts goes straight to the handlers | 22:28 |
| pleia2 | mtreinish: not breaking it ^5 | 22:28 |
| asselin | any cores want to review zaro's gearman-plugin patch? https://review.openstack.org/#/c/252768/ | 22:28 |
| *** bpokorny has quit IRC | 22:28 | |
| *** rhallisey has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack/gertty: Match links by url instead of domain. https://review.openstack.org/252973 | 22:29 |
| mtreinish | pleia2: heh, well the real test is when something that uses the options gets installed | 22:29 |
| pleia2 | mtreinish: nods | 22:29 |
| openstackgerrit | Merged openstack-infra/project-config: Increase internap to 1 instance https://review.openstack.org/259198 | 22:29 |
| ianw | greghaynes: so ... i have a f23 image.vhd ... i'm going to try uploading it into swift on my rax account. is it likely i am going to be able to boot it? | 22:29 |
| clarkb | ianw: if you made it with that vhd-utils patch it should boot | 22:30 |
| clarkb | that was the only thing we needed to get boot working on rax. Setup of working networking was different story but I think you are familiar with that | 22:30 |
| ianw | clarkb: well, i did, but i also munged that patch to work on blktap on fedora ... so yay for adding new variables :) | 22:30 |
| *** dcentral has quit IRC | 22:31 | |
| *** mudassirlatif has joined #openstack-infra | 22:31 | |
| ianw | clarkb: do you also know anything about that patch? i really think we should beat it into shape and get it upstream, cause it's a mess | 22:31 |
| *** mudassirlatif has quit IRC | 22:31 | |
| *** dcentral has joined #openstack-infra | 22:31 | |
| *** spzala has quit IRC | 22:31 | |
| ianw | not the least of which is spelling GLOBEL everywhere | 22:31 |
| *** rbrndt has joined #openstack-infra | 22:31 | |
| clarkb | ianw: aiui upstream said go away | 22:32 |
| vilobhmm11 | clarb, AJaeger, fungii : can you please review this https://review.openstack.org/#/c/254488/ when free | 22:32 |
| vilobhmm11 | clarkb : ^^ | 22:32 |
| openstackgerrit | Merged openstack-infra/nodepool: Don't double-log in nodepool/dsvm log output https://review.openstack.org/234500 | 22:32 |
| clarkb | ianw: but that may have jus tbeen because the patch is actually terrible in its current state | 22:32 |
| fungi | ianw: i can't recall specifics, but seems like mordred had some trouble interfacing with xen devs | 22:32 |
| *** peristeri has quit IRC | 22:32 | |
| clarkb | ianw: I personally dislike the use of digits for input and output formats | 22:32 |
| anteaya | mtreinish: you are welcome to join our holiday party if you wish | 22:32 |
| *** markvoelker has joined #openstack-infra | 22:33 | |
| fungi | ianw: seems like at one point he was looking at getting that patch into more acceptable shape i just don't recall where that ended up | 22:33 |
| ianw | yeah, that stuff is easy changed. it seems like Bob Ball has some of the last commits in the area, and i'd like to think we're on good terms, he does a lot of devstack xen stuff | 22:33 |
| fungi | so instead we have an ubuntu ppa with more or less a xen fork to be able to have that package | 22:33 |
| clarkb | there is also rumor that qemu-img might actually work now | 22:33 |
| clarkb | with a new enough version | 22:33 |
| clarkb | but mordred was unable to get that working | 22:34 |
| ianw | alright, i'll take it as an action item to follow up on this | 22:34 |
| mtreinish | anteaya: holiday party? | 22:34 |
| mtreinish | oh I see the ML post | 22:34 |
| anteaya | mtreinish: the etherpad you were on | 22:34 |
| anteaya | mtreinish: yeah | 22:34 |
| ianw | after i figure out how to upload my possibly working image :) | 22:34 |
| fungi | also, it seems generally terrible to have to build xen to be able to get a simple command-line utility to do image format conversions, but i'm probably underestimating the complexities involved | 22:35 |
| pleia2 | mtreinish: I promise not to knowledge transfer anything about the new star wars movie (unless you want me to) | 22:35 |
| *** Swami_ has quit IRC | 22:35 | |
| anteaya | ha ha ha | 22:35 |
| anteaya | you can make subtle light saber movements as you type | 22:35 |
| anteaya | :) | 22:35 |
| clarkb | ianw: yes that is indeed the hard thing, if you want to write a little python shade should have it all sorted out for you | 22:35 |
| pleia2 | anteaya: hehe | 22:36 |
| clarkb | ianw: but shade has no cli so would be a few lines of python to instantiate and call the right methods | 22:36 |
| mtreinish | well if it's tomorrow I'm only working in the morning, but I'll try to remember to idle on -sprint | 22:36 |
| anteaya | mtreinish: cool | 22:36 |
| mtreinish | star wars is the afternoon for me :) | 22:36 |
| anteaya | of course | 22:36 |
| clarkb | no spoilers | 22:36 |
| kzaitsev_mb | pleia2: so from what I'm reading — we basically don't need to do much, just mark the project with 'options: - translate' and we'll have us registered in zanata and the pot files would get generated by scripts, right? | 22:36 |
| clarkb | this shall remain a spoiler free zone | 22:37 |
| anteaya | it will be March before I see it, I'm sure | 22:37 |
| anteaya | it might even be out on dvd before I see it | 22:37 |
| kzaitsev_mb | (since we already mark all strings for translation and have babel in requirements) | 22:37 |
| anteaya | clarkb: I honour your spoiler free zone | 22:37 |
| mtreinish | pleia2: it seems to work: http://health.openstack.org/runs/metadata/keys | 22:37 |
| mtreinish | that list is the filtered list | 22:37 |
| mtreinish | got to wait for puppet to run on status.o.o to pull in the new js that knows how to use the list | 22:38 |
| pleia2 | mtreinish: cool | 22:38 |
| * pleia2 nods | 22:38 | |
| clarkb | kzaitsev_mb: you need to add the jobs to your project | 22:38 |
| mtreinish | pleia2: http://status.openstack.org/openstack-health/#/ the group_by drop down is there | 22:38 |
| kzaitsev_mb | oh, yeah, right | 22:39 |
| mtreinish | anteaya: ^^^ you were looking for that right? | 22:39 |
| anteaya | group_by! wooooo | 22:39 |
| anteaya | yes! | 22:39 |
| anteaya | thank you | 22:39 |
| pleia2 | kzaitsev_mb: I'll dig up an example of another project that just added jobs, sec | 22:39 |
| anteaya | timothyb89: thank you for your help with group_by | 22:39 |
| pleia2 | kzaitsev_mb: you want a patch like this for your project: https://review.openstack.org/#/c/257780/ | 22:39 |
| anteaya | mtreinish: so build_branch is interesting | 22:40 |
| pleia2 | kzaitsev_mb: also, it would help us out for the PTL to +1 the change, so know there's centralized support for the proposal | 22:40 |
| *** dcentral has quit IRC | 22:40 | |
| anteaya | mtreinish: what would it take for the stable branches to filter to the top | 22:40 |
| timothyb89 | anteaya: glad I could help! | 22:40 |
| pleia2 | kzaitsev_mb: or at least a link to the discussion/agreement from the team (mailing list, irc logs) | 22:40 |
| mtreinish | anteaya: it's by failure percentage just like the other views | 22:41 |
| *** dcentral has joined #openstack-infra | 22:41 | |
| anteaya | mtreinish: the UNKNOWN build branch as the first entry doesn't hold much fasination for me | 22:41 |
| mtreinish | that's just periodic qa | 22:41 |
| *** alivigni has quit IRC | 22:41 | |
| anteaya | ah | 22:41 |
| pleia2 | mtreinish: oh, that's nice :) | 22:41 |
| anteaya | all lumped together? | 22:41 |
| mtreinish | it's not triggered by a change so there isn't a branch set in the metadata | 22:41 |
| mtreinish | it might also have periodic stable in there, I'd have to check | 22:42 |
| fungi | okay, unlike last night i think i'm going to kick off early and have a non-openstack evening. night all! | 22:42 |
| *** zeih has joined #openstack-infra | 22:42 | |
| anteaya | okay thanks | 22:42 |
| anteaya | fungi: enjoy your evening | 22:42 |
| pleia2 | fungi: good idea, take care | 22:43 |
| anteaya | mtreinish: periodic-stable is in the build_queue view | 22:43 |
| jeblair | anyone want to double check my gertty release announcement? https://etherpad.openstack.org/p/HxoGxcazcJ | 22:43 |
| *** abregman|ZzZzZz has quit IRC | 22:44 | |
| kzaitsev_mb | pleia2: sure, I'll ask him to +1 the change. =) One more question — do I need to do anything to have log-translations split (info/warn/critical, etc.) Or is it done automatically just based on how we mark the strings? (didn't find an explanation in the docs) | 22:44 |
| *** dkranz has quit IRC | 22:44 | |
| pleia2 | jeblair: lgtm | 22:44 |
| mtreinish | anteaya: yeah the group_by is a metadata key and it shows the results grouped by the value. UNKNOWN is what the gearman client sets the build_branch value to if it doesn't have one set | 22:45 |
| mtreinish | which I know it definitely does for periodic-qa queue jobs | 22:45 |
| *** alivigni has joined #openstack-infra | 22:45 | |
| *** mudassirlatif has joined #openstack-infra | 22:45 | |
| pleia2 | kzaitsev_mb: I don't know about that one, AJaeger would know better (might be a good question for the mailing list) | 22:45 |
| mtreinish | anteaya: it also does it for periodic-stable jobs: http://status.openstack.org/openstack-health/#/g/build_branch/UNKNOWN | 22:45 |
| anteaya | jeblair: looks good to me | 22:45 |
| *** bgmccollum has left #openstack-infra | 22:46 | |
| *** gildub has joined #openstack-infra | 22:46 | |
| *** mudassirlatif has quit IRC | 22:46 | |
| kzaitsev_mb | pleia2: ok, will take a note to ask that on ML | 22:46 |
| anteaya | mtreinish: ah ha | 22:46 |
| jeblair | i'll tag jjb 1.4.0 | 22:46 |
| *** e0ne has quit IRC | 22:47 | |
| mtreinish | the group by will be real nice when someone restarts the logstash gearman client so we start setting the node_region in the metadata | 22:47 |
| *** zeih has quit IRC | 22:47 | |
| mtreinish | then we can group based on where things run | 22:47 |
| *** dtardivel has quit IRC | 22:47 | |
| clarkb | gah would've been a good time to do that hwne gerrit was off | 22:48 |
| mtreinish | heh, yeah I realized that after it was too late | 22:49 |
| kzaitsev_mb | pleia2: just one more question — we have murano-dashboard, that is a dashboard for horizon and is tho most visible thing, which we'd also like to translate of course. But since it's horizon-based it doesn't use oslo.i18n. Would there be anything special we'd have to do to have it hooked up? | 22:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/log_processor: Change node_region to node_provider https://review.openstack.org/254312 | 22:49 |
| anteaya | mtreinish: okey dokey | 22:49 |
| jeblair | clarkb, mtreinish: since apparently that isn't the actual code.. where is the actual code now? | 22:49 |
| jeblair | so i can make that change. again. | 22:49 |
| kzaitsev_mb | pleia2: I've noticed there is a designate-dashboard geting translations, probably gotta look at how it's configured | 22:50 |
| clarkb | jeblair: the actual code is in puppet-log_processor because the change to switch over needs reviews | 22:50 |
| pleia2 | kzaitsev_mb: good idea :) | 22:50 |
| jeblair | clarkb: okay, i will copy it there | 22:50 |
| *** triggerwarning has joined #openstack-infra | 22:50 | |
| clarkb | jeblair: well more the dep of the change over needs reviews | 22:50 |
| pleia2 | kzaitsev_mb: I don't know re: your first question | 22:50 |
| clarkb | https://review.openstack.org/#/c/248868/ | 22:50 |
| *** piet has quit IRC | 22:51 | |
| clarkb | (my +1 there was due to a group messup should be fixed now so Iwill bump to a +2) | 22:51 |
| kzaitsev_mb | k. as 1st step I'm gonna add translations to the main repository and ping our PTL to have a word there, next step — I'll add other projects too (we have client and agent and dashboard =)) | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-log_processor: Change node_region to node_provider https://review.openstack.org/259210 | 22:52 |
| jeblair | clarkb, mtreinish: ^ let's merge that before we restart :) | 22:52 |
| *** otter768 has joined #openstack-infra | 22:52 | |
| jeblair | pabelanger: jjb 1.4.0 is released | 22:53 |
| jeblair | gerrtty 1.3.0 is released | 22:53 |
| jeblair | arrrrr gerrrrrrty | 22:53 |
| *** pradk has quit IRC | 22:53 | |
| *** triggerwarning has quit IRC | 22:53 | |
| *** triggerwarning has joined #openstack-infra | 22:53 | |
| jesusaurus | jeblair: can you also push up a change on the opnstack-infra/log_processor project? you will need to base it off of 248868 because I'm not getting many reviews on the change that makes the tests pass | 22:53 |
| jeblair | jesusaurus: i did that first :) | 22:54 |
| jeblair | jesusaurus: but yeah, i just rebased it | 22:54 |
| jesusaurus | awesome | 22:54 |
| *** spzala has joined #openstack-infra | 22:54 | |
| *** hashar has quit IRC | 22:55 | |
| jesusaurus | oh, /me notices that was just discussed in backscroll :) | 22:55 |
| *** mudassirlatif has joined #openstack-infra | 22:55 | |
| jeblair | jesusaurus: you copied the 'select' line in tox.ini that makes hacking _only_ look at H231 and F violations and ignore all pep8. | 22:56 |
| jeblair | jesusaurus: i'm fine with that so i approved it. :) | 22:56 |
| jeblair | but if someone who likes pep8 wants to fix that, fair warning. ;) | 22:56 |
| ianw | VERBOSE 0.43 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com | 22:56 |
| ianw | VERBOSE 127.77 1 < - <class 'socket.error'> [Errno 110] Connection timed out | 22:56 |
| ianw | ^ is that a thing anyone has seen? | 22:56 |
| *** otter768 has quit IRC | 22:56 | |
| jesusaurus | to be honest, I didn't understand that particular line | 22:56 |
| jeblair | jesusaurus: neither did the author ;) | 22:57 |
| pabelanger | jeblair: danke, for both | 22:57 |
| jeblair | jesusaurus: it actually means "*only* report failures of things on this line". | 22:57 |
| jeblair | jesusaurus: the author thought it meant "*also* report failures of things on this line which are otherwise excluded by the previous line". | 22:57 |
| jeblair | and it's been copy-pastad around ever since | 22:58 |
| jesusaurus | heh | 22:58 |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config: Add translation jobs to murano https://review.openstack.org/259211 | 22:59 |
| clarkb | jesusaurus: did you push an unrevert for the change that needed the boilerplate to be merged? | 22:59 |
| clarkb | jesusaurus: that should be step after getting the delta changes merged | 22:59 |
| jesusaurus | clarkb: yeah, 254992 is an identical change, but this time with a Depends-On | 23:00 |
| *** EricGonczer_ has quit IRC | 23:00 | |
| openstackgerrit | Merged openstack-infra/log_processor: Add project boilerplate https://review.openstack.org/248868 | 23:00 |
| *** SumitNaiksatam has quit IRC | 23:00 | |
| *** triggerwarning has quit IRC | 23:02 | |
| zaro | fungi, pabelanger : i can do that jjb release by end of this week. | 23:04 |
| zaro | anteaya: you've discovered the quick approve button. when there is only one more vote required to apprv then that button will appear. | 23:05 |
| anteaya | zaro: ah | 23:05 |
| zaro | i actually don't think it's necessary. | 23:05 |
| *** hurgleburgler has joined #openstack-infra | 23:05 | |
| anteaya | me either | 23:05 |
| rhedlind | anteaya: I found the infinte loop in the zuul scheduler. it is still there in the latest version. who's familiar with the zuul implementation that I could bounce ideas for a fix with? | 23:05 |
| jeblair | SpamapS: I released gertty 1.3.0 http://lists.openstack.org/pipermail/openstack-dev/2015-December/082637.html | 23:06 |
| anteaya | just post to channel | 23:06 |
| anteaya | rhedlind: what did you find? | 23:06 |
| jeblair | zaro: i released jjb 1.4.0 | 23:06 |
| *** pahuang_ has joined #openstack-infra | 23:06 | |
| anteaya | rhedlind: zuul experts are around and will follow along as they can | 23:06 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output https://review.openstack.org/234447 | 23:06 |
| rhedlind | anteaya: ok | 23:07 |
| anteaya | rhedlind: thanks | 23:07 |
| *** spzala has quit IRC | 23:08 | |
| zaro | pabelanger: jeblair says he did it ^, does it not have what you want? | 23:09 |
| anteaya | rhedlind: I'm aslo making dinner at the same time so just keep posting links to code or logs as they are relevant to your point | 23:09 |
| *** zeih has joined #openstack-infra | 23:09 | |
| jeblair | well, i only did it 20 minutes ago | 23:10 |
| pabelanger | zaro: indeed. | 23:11 |
| pabelanger | thanks for the follow up | 23:11 |
| *** yuanying has joined #openstack-infra | 23:12 | |
| *** zeih has quit IRC | 23:14 | |
| *** hichihara has joined #openstack-infra | 23:15 | |
| *** apuimedo_ has quit IRC | 23:15 | |
| *** apuimedo has quit IRC | 23:15 | |
| *** bpokorny_ has quit IRC | 23:17 | |
| *** apuimedo has joined #openstack-infra | 23:17 | |
| *** bpokorny has joined #openstack-infra | 23:17 | |
| SpamapS | jeblair: oh very nice | 23:18 |
| kzaitsev_mb | k, I see that horizon and magnum-ui are configured separately in zuul for message extraction — gotta find people who configured those, to ask about dashboard | 23:19 |
| clarkb | kzaitsev_mb: I really want them to get with the program fwiw | 23:20 |
| clarkb | kzaitsev_mb: its not necessary that they be so different | 23:20 |
| clarkb | but since horizon gets most of the translation effort we havne't addressed that | 23:20 |
| *** rhallisey has joined #openstack-infra | 23:20 | |
| kzaitsev_mb | oh, I see. | 23:20 |
| clarkb | we support multiple docs for each project which is what they need aiui | 23:21 |
| *** apuimedo_ has joined #openstack-infra | 23:21 | |
| clarkb | but also something with django iirc, which I think isn't actually necessary | 23:21 |
| clarkb | since gettext is pretty universal | 23:21 |
| kzaitsev_mb | horizon seems to use custom extractor for angular and django templates and everything. | 23:21 |
| clarkb | they use direct gettext iirc, which should just work everywhere | 23:22 |
| kzaitsev_mb | clarkb: last time I saw some movement in horizon regarding string extraction it was related to moving to django_babel | 23:22 |
| kzaitsev_mb | and something related to js | 23:22 |
| clarkb | one thing I have contemplated is using a tox target for extraction and updates | 23:23 |
| *** Daisy has joined #openstack-infra | 23:23 | |
| clarkb | so either make the tooling consistent or change the interface to be more Make like | 23:23 |
| pleia2 | jeblair: https://twitter.com/GerritReview/status/677630100200927235 :) | 23:23 |
| clarkb | but either way addressing that would be good | 23:23 |
| *** dingyichen has joined #openstack-infra | 23:23 | |
| kzaitsev_mb | oh, yeah. That was the 1st place I looked to =) but apparently we don't use tox for that =/ | 23:23 |
| jeblair | pleia2: diffy the cukoo tweets! | 23:23 |
| pleia2 | jeblair: and diffy thinks gertty is amazing! | 23:24 |
| clarkb | TIL 1080 pixels is not wide enough for gerrit | 23:26 |
| anteaya | clarkb: how many does gerrit expect? | 23:26 |
| pleia2 | I've long said that google engineers need smaller monitors, so they know how the rest of us live | 23:26 |
| jeblair | all of the pixels | 23:26 |
| clarkb | anteaya: I dunno but my monitor isn't wide enough for the 3 wide column to render properly | 23:26 |
| anteaya | perhaps mine isn't rendering properly | 23:26 |
| anteaya | I don't know | 23:26 |
| clarkb | I get a scrollbar and lots of line wrapping | 23:27 |
| anteaya | :( | 23:27 |
| pleia2 | I have 1680 width on my screen and that's enough | 23:27 |
| pleia2 | still too much whitespace though | 23:27 |
| pleia2 | I think one of the patches up makes that better, right zaro? | 23:27 |
| anteaya | on patches I have open it now gives me an updates box: Update from Jeremy Stanley, James E. Blair, Jenkins, Gerrit Code Review Show Ignore | 23:27 |
| clarkb | ya sdague has one I believe | 23:27 |
| * pleia2 nods | 23:27 | |
| anteaya | lower right corner in a patch I haven't refreshed that has activity | 23:28 |
| clarkb | pleia2: they probably also use much smaller fonts than I do | 23:28 |
| jeblair | anteaya: yeah, it polls periodically to find that | 23:28 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix elastic-recheck query command https://review.openstack.org/258725 | 23:28 |
| anteaya | clarkb: I don't have a scroll bar, I guess it is supposed to look like this | 23:28 |
| pleia2 | clarkb: I guess you have dad eyes now :D | 23:28 |
| anteaya | jeblair: okay, interesting | 23:28 |
| *** tonytan4ever has quit IRC | 23:28 | |
| clarkb | pleia2: yes optometrist confirmed | 23:28 |
| clarkb | "I am going to change your prescription again" and "you need high index lenses now" | 23:29 |
| pleia2 | hah, oh dear | 23:29 |
| zaro | ohh, cool. didn't know Gerrit was on twitter. | 23:29 |
| zaro | can you follow without having an account? | 23:29 |
| clarkb | oh the best prat was "do you wear contacts?" "No" "Good, because that would be tricky" | 23:29 |
| pleia2 | zaro: not really :\ | 23:29 |
| zaro | nice summary by Luca there. | 23:29 |
| pleia2 | zaro: yeah, that was a great post, along with jeblair's summary I almost felt like I was there! | 23:30 |
| anteaya | rhedlind: don't disappear too far, we are all here, hanging out, now is a perfect time (timing wise) to share your findings | 23:30 |
| anteaya | rhedlind: you don't have to fix it yourself, we are grateful you found something | 23:30 |
| zaro | ugh, i guess i just need to open web pag to follow | 23:30 |
| *** rbrndt has quit IRC | 23:31 | |
| *** eil397 has joined #openstack-infra | 23:31 | |
| *** apuimedo_ has quit IRC | 23:35 | |
| *** Swami_ has joined #openstack-infra | 23:35 | |
| anteaya | pleia2: have a link to the referenced tweet? | 23:35 |
| *** apuimedo has quit IRC | 23:35 | |
| *** Daisy has quit IRC | 23:36 | |
| jeblair | oh look, flepied wrote a thing about us: https://opensource.com/business/15/12/next-gen-ci | 23:36 |
| *** apuimedo has joined #openstack-infra | 23:36 | |
| jeblair | (found it from the gerrit twitterstream) | 23:37 |
| *** zeih has joined #openstack-infra | 23:37 | |
| *** Daisy has joined #openstack-infra | 23:37 | |
| *** ayoung has joined #openstack-infra | 23:37 | |
| *** rhallisey has quit IRC | 23:37 | |
| pleia2 | anteaya: I was thinking of the one that links to this article http://gitenterprise.me/2015/11/16/gerrit-user-summit-hackathon-2015/ | 23:38 |
| pleia2 | anteaya: but now I'm not so sure, since I don't think that was written by luca :) | 23:38 |
| pleia2 | but it's still a good post | 23:38 |
| *** dprince has quit IRC | 23:39 | |
| * anteaya reads jeblair and pleia2's links, while continuing to hope rhedlind returns | 23:40 | |
| *** apuimedo_ has joined #openstack-infra | 23:41 | |
| *** Daisy has quit IRC | 23:41 | |
| *** gildub has quit IRC | 23:41 | |
| *** zeih has quit IRC | 23:41 | |
| *** apuimedo has quit IRC | 23:41 | |
| *** apuimedo has joined #openstack-infra | 23:43 | |
| *** darrenc is now known as darrenc_afk | 23:44 | |
| *** apuimedo_ has quit IRC | 23:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Consul Gate Job https://review.openstack.org/254488 | 23:46 |
| *** erlon has quit IRC | 23:46 | |
| *** Qiming has joined #openstack-infra | 23:46 | |
| openstackgerrit | Merged openstack-infra/system-config: [WIP] Add puppetlabs-apache 1.8.0? https://review.openstack.org/244835 | 23:46 |
| *** jaypipes has quit IRC | 23:47 | |
| *** sflanigan has joined #openstack-infra | 23:48 | |
| *** esker has joined #openstack-infra | 23:48 | |
| *** apuimedo_ has joined #openstack-infra | 23:51 | |
| *** sdake has joined #openstack-infra | 23:51 | |
| docaedo | I have a question about periodic jobs and infra, probably need just a little pointer :) Using propose_projects_yaml_update as my model, I made a script that finds dead links in app-catalog assets.yaml, and proposes an update | 23:53 |
| docaedo | I have a question about periodic jobs and infra, probably need just a little pointer :) Using propose_projects_yaml_update as my model, I made a script that finds dead links in app-catalog assets.yaml, and proposes an update | 23:54 |
| *** apuimedo has quit IRC | 23:55 | |
| *** apuimedo_ has quit IRC | 23:55 | |
| *** rbrndt has joined #openstack-infra | 23:56 | |
| *** apuimedo has joined #openstack-infra | 23:57 | |
| zaro | pleia2: i'm pretty sure it was luca. at least i remember him taking those pics | 23:59 |
| zaro | and gerritforge is his company | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!