| jhesketh | clarkb: in the interest of moving things forward, do you think it'd be better to just trial jenkins07 and if it fails undo it? | 00:00 |
|---|---|---|
| clarkb | jhesketh: yes, and I don't expect it to be a problem because gearman deals with those env vars directly | 00:01 |
| *** unicell has joined #openstack-infra | 00:01 | |
| clarkb | jhesketh: to me the tricky bit were the changes you already wrote and I approved to make the job not use the plugin | 00:01 |
| jhesketh | cool | 00:01 |
| *** wenlock has quit IRC | 00:01 | |
| jhesketh | clarkb: okay, when's a good time to do this? | 00:01 |
| clarkb | jhesketh: I can probably take a stab at that tomorrow | 00:02 |
| clarkb | jhesketh: this evening isn't good beacuse EMELTING | 00:02 |
| clarkb | and I am already babysitting a pvmove and image rebuilds | 00:02 |
| jhesketh | all good | 00:03 |
| jhesketh | thanks clarkb :-) | 00:03 |
| *** david-lyle has quit IRC | 00:05 | |
| *** mmaglana has quit IRC | 00:05 | |
| *** david-lyle has joined #openstack-infra | 00:06 | |
| *** msabramo has quit IRC | 00:06 | |
| clarkb | mordred: I don't know why run_all.sh is skipping review.o.o | 00:06 |
| *** anode has quit IRC | 00:06 | |
| clarkb | mordred: any chance you know why that is happening? | 00:06 |
| *** chuckC has quit IRC | 00:07 | |
| arosen | clarkb: after this merged: https://review.openstack.org/#/c/102697/ I'm not able to push to: ssh://arosen@review.openstack.org:29418/stackforge/python-congressclient.git yet | 00:07 |
| arosen | I guess it might take a little time? | 00:07 |
| clarkb | arosen: yes the project hasn't been created for some reason | 00:07 |
| *** atiwari has quit IRC | 00:07 | |
| clarkb | arosen: because puppet hasn't run on review.openstack.org | 00:07 |
| arosen | ah okay. | 00:07 |
| arosen | thanks | 00:07 |
| jesusaurus | is run_all.sh new? how does that get triggered? | 00:07 |
| clarkb | its the thing I am hoping mordred will look into above because its breaking me now too | 00:07 |
| clarkb | jesusaurus: its new ish been around for a few weeks. and cron | 00:07 |
| clarkb | I really wish puppet agent --disable was a bit more clear about whether or not it is disabled | 00:08 |
| *** msabramo has joined #openstack-infra | 00:08 | |
| jeblair | clarkb: puppet agent --disable is dangerous on any git-related machines because of the ordering | 00:09 |
| jeblair | clarkb: that's why i'm updating the project rename instructions to indicate that the run_all cron should be stopped | 00:10 |
| clarkb | jeblair: ya Ithink I merged that change | 00:10 |
| jeblair | (there is not a good way to do that now, we need to make that better) | 00:10 |
| clarkb | jeblair: but I don't even know if puppet is disabled | 00:10 |
| clarkb | all I know is it has been 48ish minutes since puppet ran on review.o.o | 00:10 |
| *** david-lyle has quit IRC | 00:10 | |
| clarkb | and mordred was the last person to report working on this stuff | 00:10 |
| jeblair | clarkb: check syslog? | 00:10 |
| *** zehicle_at_dell has quit IRC | 00:11 | |
| clarkb | syslog only shows the 2319 catalog run complete message | 00:11 |
| clarkb | puppet is running on other nodes on a 15 minutes interval | 00:12 |
| clarkb | maybe I should just try running it manually with --noop. maybe error reporting is broken? /me does this | 00:12 |
| jeblair | clarkb: it may not be running on review.o.o if it's breaking on a git host | 00:12 |
| clarkb | jeblair: oh | 00:13 |
| clarkb | also --noop on review.o.o was clean /me checks git01 | 00:13 |
| melwitt | clarkb: what does it mean to add the hudson-openstack account to the nova bug team? (https://bugs.launchpad.net/python-novaclient/+bug/1326503) | 00:13 |
| uvirtbot | Launchpad bug 1326503 in python-novaclient "hudson-openstack not working for python-novaclient" [Undecided,New] | 00:13 |
| clarkb | puppet on git01.openstack.org failed with return code: 1 thats it | 00:13 |
| clarkb | melwitt: its a launchpad groups membership permissions thing | 00:14 |
| clarkb | melwitt: some projects have oepn bug and allow gerrit to assign bugs with no issues | 00:15 |
| clarkb | melwitt: but if the project isn't open then you need to explicitly give the user gerrit uses permissions to do operations like that | 00:15 |
| clarkb | err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: File[/etc/httpd/conf/httpd.conf] | 00:15 |
| clarkb | I think we must've udpated apache module? | 00:15 |
| clarkb | maybe I should've been meaner with my vote on that... | 00:15 |
| melwitt | clarkb: okay. I was trying to see if I could add it but I don't see a way. I'm not a admin of nova-bugs but I am a member | 00:16 |
| sdague | hmmm... the requirements fix didn't seem to do a thing - http://logs.openstack.org/60/103960/1/check/gate-oslo-incubator-python33/31f5d50/console.html | 00:16 |
| clarkb | sdague: we don't enforce requirements on unittests | 00:17 |
| clarkb | sdague: so the update has to merge first | 00:17 |
| clarkb | jesusaurus: ^ any chance you have time to debug the apache moduelthing? | 00:17 |
| sdague | clarkb: which update? | 00:17 |
| *** melwitt has quit IRC | 00:17 | |
| clarkb | sdague: the requirements file update in oslo-incubator | 00:17 |
| sdague | oh | 00:18 |
| sdague | gotcha | 00:18 |
| *** dangers is now known as dangers_away | 00:18 | |
| jeblair | clarkb, mordred: okay, so we've been trying to adopt this policy where we approve potentially disruptive puppet patches when their authors are around to monitor and fix changes | 00:19 |
| jeblair | clarkb, mordred: we don't seem to actually be doing that | 00:19 |
| jesusaurus | clarkb: sure, did that "revert" get merged? | 00:20 |
| jesusaurus | ah, so it did | 00:20 |
| clarkb | jeblair: yes I dunno who approved it or when | 00:20 |
| jesusaurus | clarkb: do you have a link of the error for me? | 00:20 |
| clarkb | but I failed at -1ing so :( | 00:20 |
| jeblair | https://review.openstack.org/#/c/103258/ | 00:21 |
| jeblair | mordred: approved it | 00:21 |
| clarkb | jesusaurus only what I pasted above. puppetboard may have more | 00:21 |
| jeblair | and neither monitored it himself nor mode sure that jesusaurus was around to do so :( | 00:21 |
| mtreinish | clarkb: yes, I'd support pacman everywhere :) | 00:22 |
| *** msabramo has quit IRC | 00:22 | |
| marun | in job yaml, I'm assuming there is an 80 char limit? how would you suggest I format the following? | 00:22 |
| marun | sudo -H -u stack OS_SUDO_TESTING=1 OS_ROOTWRAP_CMD='sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf' tox -e functional | 00:22 |
| jeblair | marun: i'm not aware of such a limit -- however, you might consider making a new tox env | 00:23 |
| jeblair | marun: maybe 'sudo-functional' | 00:23 |
| marun | jeblair: ah, fair enough. putting the configuration in the neutron tree likely makes more sense than in config | 00:24 |
| *** Ryan_Lane has joined #openstack-infra | 00:24 | |
| *** gangil has quit IRC | 00:24 | |
| *** gokrokve has quit IRC | 00:24 | |
| jeblair | yeah, seems like a good place for those vars | 00:24 |
| *** dims__ has quit IRC | 00:25 | |
| lifeless | now I have to call a variable pseudo-functional somewhere that sudo is used. | 00:25 |
| jeblair | lifeless: all of our software is sudo-functional ;) | 00:26 |
| marun | heh | 00:26 |
| ianw | mordred: re https://review.openstack.org/#/c/88479/27 what environment do you use for local testing? | 00:28 |
| jeblair | ianw: i think a change in that series made a doc update about how to run it? | 00:29 |
| ianw | jeblair: https://review.openstack.org/#/c/88479/29/modules/openstack_project/files/nodepool/elements/README.rst i assume ubuntu trusty? | 00:31 |
| clarkb | ianw: tools/build-image.sh is what you want | 00:32 |
| jesusaurus | clarkb: okay, it looks like the two templates (one from the cgit module and one from the upgraded apache module) have some significant differences | 00:33 |
| jesusaurus | and theres no simple way to just turn off the one in the apache module | 00:33 |
| jesusaurus | so we should probably revert back to 0.0.4 | 00:33 |
| *** Ryan_Lane has quit IRC | 00:37 | |
| jeblair | ianw: well, i hope it works on precise since that's what nodepool is running on right now :) | 00:39 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Revert "Revert "Downgrade puppetlabs-apache to version 0.0.4."" https://review.openstack.org/104038 | 00:39 |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:40 | |
| *** HACKING-TWITTER has quit IRC | 00:41 | |
| *** dims__ has joined #openstack-infra | 00:43 | |
| mordred | jeblair, clarkb: sorry - my bad - I failed to flag that in my brain as potentially disruptive. | 00:43 |
| *** HACKING-TWITTER has joined #openstack-infra | 00:43 | |
| mordred | although in retrospect, how I could have possibly thought that was non-disruptive is beyond me | 00:44 |
| jeblair | mordred: 'revert' something seems like a nice safe thing to do, but really it should have been "try upgrading apache module again" | 00:44 |
| *** Ryan_Lane has joined #openstack-infra | 00:44 | |
| jeblair | mordred: or "revert revert apache upgrade" | 00:45 |
| jeblair | we're basically one revert off | 00:45 |
| *** msabramo has joined #openstack-infra | 00:45 | |
| *** jp_at_hp has quit IRC | 00:45 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Fix neutron func job to use gate-specific tox env https://review.openstack.org/104039 | 00:45 |
| *** signed8bit_ZZZzz is now known as signed8bit | 00:46 | |
| *** jdahlin has joined #openstack-infra | 00:46 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:46 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 00:46 | |
| lifeless | is the requirements job broken or something? | 00:49 |
| lifeless | I'm seeing patches from 'longgeek' that seem to do exactly what the requirements jobs do | 00:49 |
| lifeless | e.g. 103390 | 00:49 |
| *** dstanek is now known as dstanek_zzz | 00:50 | |
| *** yaguang has joined #openstack-infra | 00:51 | |
| * jesusaurus begins the 30 minute commute home | 00:53 | |
| clarkb | pvmove complete | 00:54 |
| *** xianghui has joined #openstack-infra | 00:58 | |
| *** bdpayne has quit IRC | 00:59 | |
| *** nati_ueno has quit IRC | 00:59 | |
| *** nati_ueno has joined #openstack-infra | 01:01 | |
| *** talluri has joined #openstack-infra | 01:06 | |
| harlowja | anyone aware of a bug for 'Routes is in an unsupported or invalid wheel' issue? | 01:07 |
| clarkb | harlowja: yes they have a bad wheel | 01:08 |
| clarkb | harlowja: a requirements change was pushed and merged to remove that version | 01:08 |
| harlowja | kk, thx | 01:08 |
| *** jp_at_hp has joined #openstack-infra | 01:08 | |
| harlowja | guess thats still filtering out to the other projects, thx clarkb | 01:09 |
| *** nati_ueno has quit IRC | 01:09 | |
| *** nati_ueno has joined #openstack-infra | 01:10 | |
| sdague | harlowja: you need to sync manually for unit tests | 01:13 |
| harlowja | sdague ya, makes sense, just seeing alot of oslo-incubator reviews hitting that; https://review.openstack.org/#/c/104022/ already up | 01:13 |
| sdague | harlowja: yep, I rebased my change on that one | 01:14 |
| harlowja | sdague kk, good idea | 01:14 |
| *** jp_at_hp has quit IRC | 01:14 | |
| *** dstanek_zzz is now known as dstanek | 01:17 | |
| *** nelsnelson has joined #openstack-infra | 01:19 | |
| *** Sukhdev has quit IRC | 01:20 | |
| *** Sukhdev has joined #openstack-infra | 01:22 | |
| *** marcoemorais has quit IRC | 01:22 | |
| *** nelsnelson has quit IRC | 01:23 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726 | 01:24 |
| *** nelsnelson has joined #openstack-infra | 01:24 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 01:25 | |
| *** otherwiseguy has quit IRC | 01:27 | |
| *** moted has quit IRC | 01:27 | |
| *** otherwiseguy has joined #openstack-infra | 01:27 | |
| *** xchu has joined #openstack-infra | 01:31 | |
| *** zz_dimtruck is now known as dimtruck | 01:32 | |
| mordred | oh my. ^^ | 01:32 |
| mordred | jhesketh: what crazy things are you doing now? | 01:32 |
| jhesketh | mordred: trying to earn a beer off jogo_away | 01:33 |
| jhesketh | mordred: the idea is to not retrigger automatic checks when a reviewer has left a -1 | 01:33 |
| mordred | jhesketh: nicely done. that makes me want to actually implement that feature in zuul - so taht you could ask zuul how many beers each person is owed | 01:33 |
| mordred | jhesketh: nice | 01:34 |
| *** nelsnelson has quit IRC | 01:34 | |
| jhesketh | mordred: and a count system that stops code from merging until the beer has been claimed | 01:34 |
| mordred | jhesketh: if we start now, we could have it ready by next april fool's | 01:35 |
| *** yaguang has quit IRC | 01:36 | |
| StevenK | I don't know if I like reviewing turning into a drinking game ... | 01:37 |
| clarkb | StevenK: embrace the liver damage | 01:38 |
| clarkb | jhesketh: if you ever make it to portland I have kegs >_> | 01:38 |
| jhesketh | clarkb: you only say that because of how many drinks I owe you ;-) | 01:38 |
| jhesketh | oh shiny! :-) | 01:38 |
| *** Ryan_Lane has quit IRC | 01:38 | |
| clarkb | jhesketh: maybe I will just make you help me brew the next beer to fill the kegs while we drink the ones already kegged | 01:39 |
| *** MarkAtwood has joined #openstack-infra | 01:39 | |
| jhesketh | I'm all up for learning :-) | 01:39 |
| *** markwash has quit IRC | 01:42 | |
| ianw | mordred / anyone : is this image build script cloning git.openstack.org/* ? | 01:43 |
| ianw | it seems to be downloading *a lot* | 01:44 |
| mordred | ianw: yse | 01:46 |
| mordred | ianw: it clones every single git repo on git.openstack.org | 01:46 |
| mordred | ianw: however, it does local caching | 01:46 |
| mordred | ianw: so once you build once, the _second_ time you build will go MUCH quicker | 01:46 |
| ianw | mordred: hmm, do you know how big that will end up being? | 01:46 |
| mordred | not off the top of my head, no | 01:46 |
| ianw | like 1gb or more like 10? | 01:47 |
| * mordred looks | 01:47 | |
| mordred | ianw: the resulting image will be 2G | 01:48 |
| lifeless | FWIW my image-create cache is 13G | 01:48 |
| mordred | ianw: in the process, it will produce 2.5G of cache | 01:48 |
| lifeless | but I've been building many different things | 01:48 |
| lifeless | and have lots of different root dirs etc | 01:49 |
| mordred | lifeless: you're crazy | 01:49 |
| *** nosnos has joined #openstack-infra | 01:49 | |
| ianw | ok, thanks, i'm just in the land of 200gb/month isp quotas | 01:49 |
| lifeless | mordred: foxlike | 01:49 |
| mordred | ianw: AH. ewww | 01:49 |
| lifeless | ianw: ok so this is why we cache things :) | 01:49 |
| *** gokrokve has joined #openstack-infra | 01:49 | |
| lifeless | ianw: so you don't go over quota | 01:49 |
| ianw | lifeless: yeah, but i've still got to get it the first time :) | 01:50 |
| *** baohua has joined #openstack-infra | 01:50 | |
| *** yaguang has joined #openstack-infra | 01:51 | |
| *** otherwiseguy has quit IRC | 01:51 | |
| * ianw super-checks i'm not accidentally tethered, because that would probably bankrupt me | 01:52 | |
| *** mriedem has quit IRC | 01:53 | |
| *** praneshp has quit IRC | 01:54 | |
| *** r1chardj0n3s is now known as r1chard_afk | 01:54 | |
| *** nosnos has quit IRC | 01:55 | |
| *** nosnos has joined #openstack-infra | 01:55 | |
| *** otter768 has joined #openstack-infra | 01:56 | |
| *** homeless has quit IRC | 01:57 | |
| *** dkranz has quit IRC | 01:59 | |
| *** hdd_ has joined #openstack-infra | 02:02 | |
| *** wenlock has joined #openstack-infra | 02:03 | |
| ianw | $ git clone git://git.openstack.org/stackforge/python-congressclient.git | 02:04 |
| ianw | Cloning into 'python-congressclient'... | 02:04 |
| ianw | fatal: The remote end hung up unexpectedly | 02:04 |
| ianw | ^ is this just me? it seems pretty repeatable | 02:04 |
| *** dstufft|laptop has joined #openstack-infra | 02:06 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 02:06 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 02:06 | |
| *** nati_ueno has quit IRC | 02:07 | |
| clarkb | ianw: its not just you | 02:08 |
| clarkb | puppet isn't running on the git farm right now | 02:08 |
| *** nati_ueno has joined #openstack-infra | 02:10 | |
| *** otherwiseguy has joined #openstack-infra | 02:12 | |
| ianw | clarkb: is that a known issue? | 02:14 |
| clarkb | ya it came up in scrollback around 0000UTC | 02:16 |
| clarkb | I am currently melting so trying to avoid hot computer with keys to do more fixing | 02:16 |
| *** MarkAtwood has quit IRC | 02:18 | |
| *** nati_ueno has quit IRC | 02:21 | |
| *** nati_ueno has joined #openstack-infra | 02:21 | |
| *** Ryan_Lane has joined #openstack-infra | 02:21 | |
| *** zhiyan_ is now known as zhiyan | 02:22 | |
| *** nati_ueno has quit IRC | 02:22 | |
| *** xianghui has quit IRC | 02:24 | |
| *** dstanek is now known as dstanek_zzz | 02:26 | |
| *** arnaud has quit IRC | 02:29 | |
| *** zhiyan is now known as zhiyan_ | 02:30 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 02:31 | |
| *** r1chard_afk is now known as r1chardj0n3s | 02:31 | |
| *** signed8bit has quit IRC | 02:32 | |
| *** signed8bit has joined #openstack-infra | 02:32 | |
| *** signed8bit has quit IRC | 02:32 | |
| *** talluri has quit IRC | 02:34 | |
| *** camunoz is now known as camunoz_away | 02:36 | |
| *** zhiyan_ is now known as zhiyan | 02:42 | |
| *** wenlock has quit IRC | 02:43 | |
| *** changbl has joined #openstack-infra | 02:43 | |
| *** chuckC has joined #openstack-infra | 02:44 | |
| *** penguinRaider has quit IRC | 02:44 | |
| *** pballand has joined #openstack-infra | 02:45 | |
| *** Sukhdev has quit IRC | 02:46 | |
| *** dstanek_zzz is now known as dstanek | 02:48 | |
| *** alexandra has joined #openstack-infra | 02:49 | |
| *** alexandra is now known as asettle | 02:49 | |
| *** pballand has quit IRC | 02:49 | |
| *** harlowja is now known as harlowja_away | 02:50 | |
| *** praneshp has joined #openstack-infra | 02:52 | |
| *** asettle has quit IRC | 02:54 | |
| *** otherwiseguy has quit IRC | 02:58 | |
| *** ArxCruz has joined #openstack-infra | 02:59 | |
| *** yaguang has quit IRC | 03:00 | |
| *** yaguang has joined #openstack-infra | 03:01 | |
| *** nati_ueno has joined #openstack-infra | 03:02 | |
| *** theannegentle has joined #openstack-infra | 03:03 | |
| *** otherwiseguy has joined #openstack-infra | 03:04 | |
| *** theannegentle has quit IRC | 03:04 | |
| *** xianghui has joined #openstack-infra | 03:05 | |
| *** xchu has quit IRC | 03:08 | |
| *** dims__ has quit IRC | 03:09 | |
| *** gyee has quit IRC | 03:10 | |
| *** alexandra_ has joined #openstack-infra | 03:11 | |
| *** alexandra_ is now known as asettle | 03:11 | |
| *** mwagner_ has quit IRC | 03:15 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 03:20 | |
| *** zhiyan is now known as zhiyan_ | 03:22 | |
| *** zhiyan_ is now known as zhiyan | 03:24 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:26 | |
| *** mwagner_ has joined #openstack-infra | 03:27 | |
| *** nosnos has quit IRC | 03:27 | |
| *** zhiyan has quit IRC | 03:30 | |
| *** camunoz_away is now known as camunoz | 03:33 | |
| *** gangil has joined #openstack-infra | 03:35 | |
| *** zhiyan has joined #openstack-infra | 03:37 | |
| *** gothicmindfood_ has joined #openstack-infra | 03:37 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 03:37 | |
| *** zhiyan is now known as zhiyan_ | 03:39 | |
| *** otherwiseguy has quit IRC | 03:39 | |
| *** otherwiseguy has joined #openstack-infra | 03:40 | |
| *** zhiyan_ is now known as zhiyan | 03:40 | |
| *** zhiyan is now known as zhiyan_ | 03:46 | |
| *** talluri has joined #openstack-infra | 03:48 | |
| *** zhiyan_ is now known as zhiyan | 03:50 | |
| *** arnaud has joined #openstack-infra | 03:50 | |
| *** zehicle_at_dell has quit IRC | 03:56 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:57 | |
| *** praneshp_ has joined #openstack-infra | 04:01 | |
| *** asettle has quit IRC | 04:01 | |
| *** pcrews has quit IRC | 04:02 | |
| *** zehicle_at_dell has quit IRC | 04:02 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:02 | |
| *** lcheng has joined #openstack-infra | 04:02 | |
| *** praneshp has quit IRC | 04:04 | |
| *** praneshp_ is now known as praneshp | 04:04 | |
| *** ArxCruz has quit IRC | 04:06 | |
| *** zehicle_at_dell has quit IRC | 04:07 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:08 | |
| *** garyh has quit IRC | 04:09 | |
| *** yfried_ has quit IRC | 04:10 | |
| *** nelsnelson has joined #openstack-infra | 04:11 | |
| *** olaph has quit IRC | 04:12 | |
| *** Sukhdev has joined #openstack-infra | 04:12 | |
| tchaypo | 12 | 04:16 |
| *** gothicmindfood_ has quit IRC | 04:21 | |
| openstackgerrit | Jamie Lennox proposed a change to openstack/requirements: Add requests-mock package to global requirements https://review.openstack.org/104067 | 04:25 |
| *** alexandra_ has joined #openstack-infra | 04:30 | |
| *** alexandra_ is now known as asettle | 04:31 | |
| *** dimtruck is now known as zz_dimtruck | 04:31 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 04:32 | |
| *** crc32 has quit IRC | 04:32 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 04:36 | |
| *** dims__ has joined #openstack-infra | 04:37 | |
| *** nosnos has joined #openstack-infra | 04:42 | |
| *** dims__ has quit IRC | 04:42 | |
| *** arnaud has quit IRC | 04:47 | |
| *** arnaud has joined #openstack-infra | 04:49 | |
| *** yfried_ has joined #openstack-infra | 04:53 | |
| *** otter768 has quit IRC | 04:55 | |
| *** colettecello has joined #openstack-infra | 04:55 | |
| *** gothicmindfood has quit IRC | 04:56 | |
| *** HACKING-TWITTER has quit IRC | 05:02 | |
| *** dims__ has joined #openstack-infra | 05:02 | |
| *** gangil has quit IRC | 05:03 | |
| *** dstanek is now known as dstanek_zzz | 05:04 | |
| *** asettle has quit IRC | 05:06 | |
| *** dims__ has quit IRC | 05:07 | |
| *** otherwiseguy has quit IRC | 05:08 | |
| *** colettecello is now known as gothicmindfood | 05:08 | |
| *** HACKING-TWITTER has joined #openstack-infra | 05:11 | |
| *** alexandra_ has joined #openstack-infra | 05:11 | |
| *** otherwiseguy has joined #openstack-infra | 05:15 | |
| *** marun is now known as marun_afk | 05:16 | |
| *** gothicmindfood has left #openstack-infra | 05:16 | |
| *** Ryan_Lane has quit IRC | 05:21 | |
| *** amitpp has joined #openstack-infra | 05:21 | |
| *** otherwiseguy has quit IRC | 05:21 | |
| *** afazekas_ has joined #openstack-infra | 05:21 | |
| *** HACKING-TWITTER has quit IRC | 05:22 | |
| *** rwsu has quit IRC | 05:24 | |
| *** basha has joined #openstack-infra | 05:25 | |
| *** zul has quit IRC | 05:27 | |
| *** ildikov has joined #openstack-infra | 05:27 | |
| *** cody-somerville has quit IRC | 05:27 | |
| *** cody-somerville has joined #openstack-infra | 05:27 | |
| *** zul has joined #openstack-infra | 05:28 | |
| *** gothicmindfood has joined #openstack-infra | 05:28 | |
| *** HACKING-TWITTER has joined #openstack-infra | 05:29 | |
| openstackgerrit | Chris Buccella proposed a change to openstack/requirements: Update glanceclient to 0.13.1 https://review.openstack.org/104070 | 05:31 |
| *** bdpayne has joined #openstack-infra | 05:33 | |
| *** gema_ is now known as gema | 05:33 | |
| *** amitpp has quit IRC | 05:35 | |
| *** basha has quit IRC | 05:37 | |
| *** otherwiseguy has joined #openstack-infra | 05:38 | |
| *** chandan_kumar has joined #openstack-infra | 05:38 | |
| *** basha has joined #openstack-infra | 05:38 | |
| *** HACKING-TWITTER has quit IRC | 05:40 | |
| *** amitpp has joined #openstack-infra | 05:40 | |
| *** lcheng has quit IRC | 05:41 | |
| *** otherwiseguy has quit IRC | 05:42 | |
| *** basha_ has joined #openstack-infra | 05:45 | |
| *** basha has quit IRC | 05:46 | |
| *** basha_ is now known as basha | 05:46 | |
| *** otherwiseguy has joined #openstack-infra | 05:48 | |
| *** mrmartin has joined #openstack-infra | 05:49 | |
| *** dstanek_zzz is now known as dstanek | 05:51 | |
| *** bdpayne has quit IRC | 05:52 | |
| *** otherwiseguy has quit IRC | 05:52 | |
| *** ildikov has quit IRC | 05:53 | |
| *** hdd_ has quit IRC | 05:58 | |
| *** zz_dimtruck is now known as dimtruck | 05:59 | |
| *** dstanek is now known as dstanek_zzz | 06:01 | |
| *** rcarrillocruz has joined #openstack-infra | 06:02 | |
| *** dims__ has joined #openstack-infra | 06:03 | |
| *** nati_ueno has quit IRC | 06:03 | |
| *** rcarrill` has quit IRC | 06:04 | |
| *** HACKING-TWITTER has joined #openstack-infra | 06:04 | |
| *** dstanek_zzz is now known as dstanek | 06:06 | |
| *** dims__ has quit IRC | 06:08 | |
| *** akuznetsova has joined #openstack-infra | 06:09 | |
| *** mrmartin has quit IRC | 06:10 | |
| *** Sukhdev has quit IRC | 06:11 | |
| clarkb | FYI re gerrit release "There is a major regression in 2.8.6 that causes exceptions in database transactions." | 06:13 |
| StevenK | clarkb: ... Yay? | 06:14 |
| clarkb | StevenK: it probably won't affect us | 06:14 |
| clarkb | because we don't upgrade quickly | 06:14 |
| clarkb | but there was talk at one point of maybe upgraded soonish | 06:14 |
| *** dstanek is now known as dstanek_zzz | 06:15 | |
| *** HACKING-TWITTER has quit IRC | 06:16 | |
| akuznetsova | Hi all, I updated requirement in one repository due to global-requirements from master branch and got this http://logs.openstack.org/08/103908/1/check/gate-mistral-pep8/2016953/console.html | 06:16 |
| *** nati_ueno has joined #openstack-infra | 06:16 | |
| akuznetsova | Can somebody explain me what the problem is? | 06:16 |
| clarkb | akuznetsova: ideally you shouldn't depend on pep8 and flake8 directly. instead hacking will pull it what it needs | 06:17 |
| clarkb | akuznetsova: I believe the problem is the version specified there is different that what hacking specifies | 06:17 |
| clarkb | or flake8 | 06:18 |
| akuznetsova | should I just remove pep8 and flake8 from my requirements? | 06:18 |
| clarkb | yes I think so | 06:18 |
| akuznetsova | clarkb, thanks, I will try | 06:19 |
| *** jdahlin has quit IRC | 06:27 | |
| *** gokrokve has quit IRC | 06:29 | |
| *** HACKING-TWITTER has joined #openstack-infra | 06:32 | |
| *** rcarrill` has joined #openstack-infra | 06:34 | |
| *** ildikov has joined #openstack-infra | 06:35 | |
| *** rcarrillocruz has quit IRC | 06:36 | |
| *** dimtruck is now known as zz_dimtruck | 06:37 | |
| *** rfolco has quit IRC | 06:38 | |
| *** ildikov has quit IRC | 06:40 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 06:41 | |
| *** HACKING-TWITTER has quit IRC | 06:47 | |
| *** doude has joined #openstack-infra | 06:47 | |
| *** alexandra_ has quit IRC | 06:47 | |
| *** dstufft|laptop has quit IRC | 06:49 | |
| *** nati_ueno has quit IRC | 06:51 | |
| *** tkelsey has joined #openstack-infra | 06:51 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Added devstack job for manila https://review.openstack.org/99933 | 06:55 |
| *** GheRivero_afk is now known as GheRivero | 06:56 | |
| *** gokrokve has joined #openstack-infra | 07:00 | |
| *** pcrews has joined #openstack-infra | 07:01 | |
| *** rdopieralski has joined #openstack-infra | 07:02 | |
| *** gokrokve_ has joined #openstack-infra | 07:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixing deprecation warnings https://review.openstack.org/101045 | 07:04 |
| *** gokrokv__ has joined #openstack-infra | 07:04 | |
| *** gokrokve has quit IRC | 07:04 | |
| *** mrmartin has joined #openstack-infra | 07:06 | |
| *** gokrokve_ has quit IRC | 07:07 | |
| *** gokrokv__ has quit IRC | 07:08 | |
| *** rdopieralski has quit IRC | 07:11 | |
| *** mrda is now known as mrda_away | 07:11 | |
| *** pcrews has quit IRC | 07:17 | |
| *** ociuhandu has joined #openstack-infra | 07:19 | |
| *** flaper87|afk is now known as flaper87 | 07:19 | |
| *** jlibosva has joined #openstack-infra | 07:24 | |
| *** mrmartin has quit IRC | 07:24 | |
| *** afazekas_ is now known as afazekas | 07:25 | |
| *** otherwiseguy has joined #openstack-infra | 07:26 | |
| *** andreykurilin_ has joined #openstack-infra | 07:28 | |
| *** mzoeller has joined #openstack-infra | 07:29 | |
| *** arnaud has quit IRC | 07:31 | |
| *** otherwiseguy has quit IRC | 07:31 | |
| *** otherwiseguy has joined #openstack-infra | 07:32 | |
| *** hashar has joined #openstack-infra | 07:33 | |
| mattoliverau | night all | 07:34 |
| mattoliverau | its dinner making time! | 07:34 |
| *** flaper87 is now known as flaper87|afk | 07:35 | |
| *** otherwiseguy has quit IRC | 07:37 | |
| *** pblaho has joined #openstack-infra | 07:38 | |
| openstackgerrit | Radoslav Gerganov proposed a change to openstack-infra/config: Add button that shows/hides CI comments in Gerrit https://review.openstack.org/95743 | 07:38 |
| *** jcoufal has joined #openstack-infra | 07:38 | |
| *** praneshp has quit IRC | 07:38 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Update pbr version https://review.openstack.org/103347 | 07:39 |
| rgerganov | mordred, are you around? | 07:42 |
| *** mkerrin has joined #openstack-infra | 07:45 | |
| *** camunoz is now known as camunoz_gone | 07:47 | |
| *** Hal_ has joined #openstack-infra | 07:50 | |
| *** ildikov has joined #openstack-infra | 07:50 | |
| *** mestery has quit IRC | 07:51 | |
| *** andreykurilin_ has quit IRC | 07:53 | |
| *** lcheng has joined #openstack-infra | 07:53 | |
| *** basha has quit IRC | 07:54 | |
| *** basha has joined #openstack-infra | 07:55 | |
| *** mestery has joined #openstack-infra | 07:55 | |
| *** jpich has joined #openstack-infra | 07:56 | |
| *** penguinRaider has joined #openstack-infra | 07:57 | |
| *** markmc has joined #openstack-infra | 08:00 | |
| tteggel | hpcloud have a nova deploy going on this morning GMT. starting in 30 mins. expect some api calls to fail. | 08:00 |
| hashar | rgerganov: mordred has been idle for 2h30m, he is most probably sleeping by now | 08:03 |
| hashar | SergeyLukjanov: <tteggel> hpcloud have a nova deploy going on this morning GMT. starting in 30 mins. expect some api calls to fail. | 08:04 |
| hashar | SergeyLukjanov: I think you are the only one with access at this time of the day :D | 08:04 |
| rgerganov | hashar, I am looking for someone that can make a very simple change on review-dev.openstack.org | 08:04 |
| rgerganov | hashar, do you know anyone? | 08:04 |
| *** gokrokve has joined #openstack-infra | 08:04 | |
| *** dims__ has joined #openstack-infra | 08:06 | |
| hashar | rgerganov: if you are logged in with your launchpad account you can browse the All-Projects rights https://review-dev.openstack.org/#/admin/projects/All-Projects,access | 08:06 |
| hashar | I can't list members of the Administrators group though | 08:07 |
| SergeyLukjanov | hashar, hey | 08:08 |
| hashar | rgerganov: I guess anyone that did a change recently most probably has admin rights. You might want to ask on the infra mailing list. But maybe everything is puppetized and you can do the change in openstack-infra/config.git | 08:09 |
| *** gokrokve has quit IRC | 08:09 | |
| rgerganov | hashar, thanks; I am looking for someone that can modify files on the server; specifically https://review-dev.openstack.org/static/hideci.js | 08:09 |
| hashar | SergeyLukjanov: I guess hpcloud would no more respond to nodepool requests for a little bit. No clue about the impact :D | 08:10 |
| SergeyLukjanov | hashar, I think that it shouldn't break anything | 08:10 |
| hashar | rgerganov: apparently not in puppet. So I have no clue :/ | 08:10 |
| *** dims__ has quit IRC | 08:11 | |
| SergeyLukjanov | we've implemented some mechanisms for non-responding clouds before enabling tripleo cloud | 08:11 |
| *** IvanBerezovskiy has joined #openstack-infra | 08:11 | |
| hashar | sounds good :-] | 08:12 |
| *** yfried_ has quit IRC | 08:12 | |
| *** yfried_ has joined #openstack-infra | 08:13 | |
| *** yfried_ has joined #openstack-infra | 08:13 | |
| *** amitpp has quit IRC | 08:14 | |
| *** oomichi has quit IRC | 08:14 | |
| *** amitpp has joined #openstack-infra | 08:14 | |
| *** fifieldt_ has joined #openstack-infra | 08:16 | |
| *** flaper87|afk is now known as flaper87 | 08:17 | |
| *** yfried_ has quit IRC | 08:18 | |
| *** yfried has joined #openstack-infra | 08:18 | |
| *** fifieldt has quit IRC | 08:18 | |
| *** ihrachyshka has joined #openstack-infra | 08:19 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/infra-specs: StoryBoard search spec https://review.openstack.org/97254 | 08:19 |
| *** guangjun has joined #openstack-infra | 08:20 | |
| *** lcheng has quit IRC | 08:21 | |
| *** basha has quit IRC | 08:26 | |
| *** Hal_ has quit IRC | 08:26 | |
| *** jp_at_hp has joined #openstack-infra | 08:26 | |
| *** lcheng has joined #openstack-infra | 08:26 | |
| *** Hal_ has joined #openstack-infra | 08:26 | |
| *** gokrokve has joined #openstack-infra | 08:30 | |
| *** fifieldt_ is now known as fifieldt | 08:31 | |
| *** otherwiseguy has joined #openstack-infra | 08:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adds a docs-specs repository https://review.openstack.org/103115 | 08:33 |
| *** gokrokve has quit IRC | 08:34 | |
| *** Hal_ has quit IRC | 08:35 | |
| *** otherwiseguy has quit IRC | 08:36 | |
| *** derekh_ has joined #openstack-infra | 08:41 | |
| *** yfried has quit IRC | 08:42 | |
| *** dizquierdo has joined #openstack-infra | 08:43 | |
| *** MIDENN_ has quit IRC | 08:47 | |
| *** mdenny has joined #openstack-infra | 08:47 | |
| *** jcoufal has quit IRC | 08:48 | |
| *** jgallard has joined #openstack-infra | 08:48 | |
| *** Hal_ has joined #openstack-infra | 08:50 | |
| *** mrmartin has joined #openstack-infra | 08:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Index the os-collect-config logs in tripleo jobs https://review.openstack.org/99950 | 08:55 |
| *** pelix has joined #openstack-infra | 08:56 | |
| *** gema has quit IRC | 09:00 | |
| *** e0ne has joined #openstack-infra | 09:00 | |
| *** penguinRaider has left #openstack-infra | 09:02 | |
| *** gema has joined #openstack-infra | 09:02 | |
| *** gokrokve has joined #openstack-infra | 09:02 | |
| *** Longgeek has joined #openstack-infra | 09:03 | |
| *** NikitaKo_ has joined #openstack-infra | 09:03 | |
| *** Longgeek has quit IRC | 09:03 | |
| *** gokrokve has quit IRC | 09:03 | |
| *** gokrokve has joined #openstack-infra | 09:04 | |
| *** Longgeek has joined #openstack-infra | 09:04 | |
| *** dims__ has joined #openstack-infra | 09:07 | |
| *** gokrokve has quit IRC | 09:09 | |
| *** dims__ has quit IRC | 09:12 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:17 | |
| *** tcammann_ has joined #openstack-infra | 09:18 | |
| tcammann_ | Hey the gate-oslo-incubator-python33 job has been failing consistently for the last 8 hours or so. e.g. https://review.openstack.org/#/c/99695/ Any ideas? | 09:19 |
| openstackgerrit | Christian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger https://review.openstack.org/104120 | 09:20 |
| *** andreaf has joined #openstack-infra | 09:24 | |
| markmc | tcammann_, looks like pycrypto isn't building | 09:25 |
| *** maxbit has quit IRC | 09:25 | |
| therve | I think it's more failing with the Routes dependency | 09:25 |
| openstackgerrit | Christian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger https://review.openstack.org/104120 | 09:26 |
| pelix | hashar: any idea as to why the bug in mina-sshd wouldn't be ticked if using a host key generated with openssl? | 09:27 |
| hashar | pelix: glad you ask: I have no clue :-( | 09:27 |
| pelix | lol | 09:27 |
| pelix | should we push to have the hash mismatch patch added in the mean time to git-review or wait until a fixed version of gerrit appears? | 09:28 |
| hashar | pelix: I reproduced the issue with some old version of Gerrit I had. I am fighting to upgrade the Gerrit with the version that might fix the issue | 09:29 |
| hashar | how is Gerrit installed in the git-review integration tests ? | 09:29 |
| hashar | Gerrit Code Review 2.8.1-4-ga1048ce ready | 09:29 |
| hashar | \O/ | 09:29 |
| pelix | downloaded from http://gerrit-releases.storage.googleapis.com/ | 09:30 |
| openstackgerrit | Christian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger https://review.openstack.org/104120 | 09:31 |
| hashar | pelix: Christian (our Gerrit/Java guru) cherry picked a few patches and crafted me http://quelltextlich.at/gerrit-2.8.1-4-ga1048ce.war | 09:33 |
| hashar | might want to mirror that to some place and try running tests with it | 09:33 |
| hashar | that fix the hash mismatch issue for me | 09:33 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/devstack-gate: Add option to enable profiling https://review.openstack.org/104123 | 09:33 |
| hashar | might cause over bugs though (we never know) | 09:33 |
| pelix | :D | 09:33 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/devstack-gate: Add option to enable profiling https://review.openstack.org/104123 | 09:35 |
| *** ch has joined #openstack-infra | 09:38 | |
| tcammann_ | markmc: Looks like the wheel package was updated yesterday | 09:38 |
| *** liusheng is now known as liusheng-leave | 09:39 | |
| ch | hi! i've got technical trouble signing the icla on review.o.o -> Server Error/ | 09:39 |
| ch | Cannot store contact information; what am i doing wrong? | 09:40 |
| *** MIDENN_ has joined #openstack-infra | 09:40 | |
| *** mdenny has quit IRC | 09:40 | |
| hashar | pelix: our bug report will be updated with bit more details ( https://bugzilla.wikimedia.org/show_bug.cgi?id=53895 ) | 09:44 |
| pelix | hashar: going to try testing with gerrit-2.8.6 since David Pursehouse did a release this morning | 09:45 |
| pelix | :0 | 09:46 |
| pelix | or :) rather | 09:46 |
| hashar | pelix: might not have the mina sshd patch though | 09:46 |
| *** ch has left #openstack-infra | 09:46 | |
| pelix | it does, but apparently it also has a DB regression, nuts | 09:46 |
| hashar | nice! | 09:47 |
| tcammann_ | I believe the gate-oslo-incubator-python33 is failing due to the wheel releasing v0.23.0 yesterday https://pypi.python.org/pypi/wheel and the Routes is now for some reason invalid. We should change the requirements to use 0.22.0 or below. | 09:47 |
| tcammann_ | Routes uses the .whl format | 09:48 |
| pelix | hashar: hopefully he should have a new release sometime in the next day or two that fixes the DB regression, so for git-review it should be as simple as pointing at the new release. | 09:48 |
| hashar | pelix: awesome! Looking forward to the tests finally passing reliably | 09:51 |
| *** MIDENN_ has quit IRC | 09:51 | |
| derekh_ | In the last hour a commit merged so that tripleo os-collect-config logs are indexed by logstash, but the "message" is being populated with %{logmessage} instead of actual data, has anybody got any idea why ? http://logstash.openstack.org/#eyJzZWFyY2giOiJ0YWdzOm9zLWNvbGxlY3QtY29uZmlnIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI5MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIiwic3RhbXAi | 09:52 |
| derekh_ | OjE0MDQyOTQ1MjA3NTJ9 | 09:52 |
| derekh_ | http://logstash.openstack.org/#eyJzZWFyY2giOiJ0YWdzOm9zLWNvbGxlY3QtY29uZmlnIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI5MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIiwic3RhbXAiOjE0MDQyOTQ1MjA3NTJ9 | 09:52 |
| *** mdenny has joined #openstack-infra | 09:52 | |
| *** mkoderer has quit IRC | 09:52 | |
| tteggel | hpcloud deploy complete "Normality has been restored. Anything you can't cope with is therefore your own problem." :) | 09:53 |
| *** unicell1 has joined #openstack-infra | 09:55 | |
| *** basha has joined #openstack-infra | 09:56 | |
| *** unicell has quit IRC | 09:58 | |
| *** gokrokve has joined #openstack-infra | 10:02 | |
| *** guangjun has quit IRC | 10:02 | |
| *** Longgeek has quit IRC | 10:03 | |
| *** Longgeek has joined #openstack-infra | 10:03 | |
| *** gokrokve has quit IRC | 10:07 | |
| *** skolekonov has joined #openstack-infra | 10:07 | |
| *** Longgeek has quit IRC | 10:07 | |
| *** dims__ has joined #openstack-infra | 10:08 | |
| *** unicell1 has quit IRC | 10:11 | |
| *** hashar has quit IRC | 10:11 | |
| *** hashar has joined #openstack-infra | 10:12 | |
| *** jcoufal has joined #openstack-infra | 10:13 | |
| *** dims__ has quit IRC | 10:13 | |
| *** jgallard has quit IRC | 10:14 | |
| *** Hal_ has quit IRC | 10:15 | |
| *** Longgeek has joined #openstack-infra | 10:21 | |
| *** lcheng has quit IRC | 10:22 | |
| *** NikitaKo_ has quit IRC | 10:24 | |
| *** lcheng has joined #openstack-infra | 10:27 | |
| *** matrohon has quit IRC | 10:32 | |
| *** dims__ has joined #openstack-infra | 10:32 | |
| *** roeyc has joined #openstack-infra | 10:38 | |
| *** fabo has quit IRC | 10:41 | |
| *** fabo has joined #openstack-infra | 10:41 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack-infra/config: Test oslo.db against all supported SA branches https://review.openstack.org/104134 | 10:43 |
| *** matrohon has joined #openstack-infra | 10:45 | |
| sdague | derekh_: I don't think any of the grok rules actually match your logs | 10:46 |
| *** mdenny has quit IRC | 10:49 | |
| sdague | where is your log format definition for these logs in the logstash indexer? | 10:49 |
| *** mdenny has joined #openstack-infra | 10:49 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Restrict 1336172 query on build_name https://review.openstack.org/103933 | 10:54 |
| *** ildikov_ has joined #openstack-infra | 10:57 | |
| *** andreykurilin_ has joined #openstack-infra | 10:57 | |
| derekh_ | sdague: I didn't know there was such a thing, so I guess they need to be added to agent.conf ? | 10:59 |
| *** asselin has quit IRC | 10:59 | |
| *** asselin has joined #openstack-infra | 10:59 | |
| *** ildikov has quit IRC | 11:00 | |
| sdague | ok, how did you expect this to work? :) Maybe we can work backwards from there and figure out what's missing in the config. | 11:02 |
| *** gokrokve has joined #openstack-infra | 11:02 | |
| *** e0ne has quit IRC | 11:03 | |
| *** maxbit has joined #openstack-infra | 11:03 | |
| derekh_ | sdague: I thought that I told logstash the file to index and then it just appeared, anyways will push an update to the config | 11:05 |
| derekh_ | once I figure it out | 11:05 |
| derekh_ | bbiab | 11:06 |
| sdague | so you need to tell it the kind of file it is, as we used that for indexing. | 11:06 |
| sdague | I think the issue is you need to add syslog to the tags | 11:07 |
| sdague | because these look like syslog files | 11:07 |
| derekh_ | sdague: cool, will have an update in a bit, once I'm back, thanks | 11:07 |
| *** gokrokve has quit IRC | 11:08 | |
| *** Longgeek has quit IRC | 11:10 | |
| *** Longgeek has joined #openstack-infra | 11:10 | |
| *** andreykurilin_ has quit IRC | 11:12 | |
| *** otherwiseguy has joined #openstack-infra | 11:15 | |
| *** yaguang has quit IRC | 11:16 | |
| *** otherwiseguy has quit IRC | 11:19 | |
| *** hashar has quit IRC | 11:23 | |
| ttx | notmyname: for swift 2.0.0-rc2 I think the right way to do it at this point would be to cut a temporary proposed/2.0.0 branch and backport to that. | 11:26 |
| ttx | we might need https://review.openstack.org/#/c/102822/ to merge first though (support for proposed/* ACLs) | 11:27 |
| ttx | fungi: when you're up, consider re-approving ^ and modifying https://$GERRIT/#/admin/projects/All-Projects,access too | 11:29 |
| *** yaguang has joined #openstack-infra | 11:30 | |
| *** tkelsey has quit IRC | 11:32 | |
| *** e0ne has joined #openstack-infra | 11:33 | |
| *** bcrochet|afk is now known as bcrochet | 11:33 | |
| *** mkoderer has joined #openstack-infra | 11:40 | |
| *** julim has joined #openstack-infra | 11:45 | |
| *** ArxCruz has joined #openstack-infra | 11:47 | |
| *** amitpp has quit IRC | 11:50 | |
| *** NikitaKo_ has joined #openstack-infra | 11:54 | |
| openstackgerrit | Kashyap Chamarthy proposed a change to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755 https://review.openstack.org/104160 | 11:54 |
| uvirtbot | Launchpad bug 1336755 in ceilometer "telemetry_notification_api.py test times out waiting for Nova notification" [Undecided,New] https://launchpad.net/bugs/1336755 | 11:54 |
| *** saper has joined #openstack-infra | 11:55 | |
| *** NikitaKo_ has quit IRC | 11:55 | |
| *** NikitaKo_ has joined #openstack-infra | 11:56 | |
| openstackgerrit | A change was merged to openstack-infra/config: Change ACLs to apply on proposed/* branches https://review.openstack.org/102822 | 11:58 |
| *** nosnos has quit IRC | 11:59 | |
| *** nosnos has joined #openstack-infra | 11:59 | |
| *** Shrews has quit IRC | 12:00 | |
| *** yaguang has quit IRC | 12:01 | |
| *** rlandy has joined #openstack-infra | 12:02 | |
| *** Shrews has joined #openstack-infra | 12:02 | |
| *** gokrokve has joined #openstack-infra | 12:02 | |
| *** Longgeek_ has joined #openstack-infra | 12:02 | |
| *** mbacchi has joined #openstack-infra | 12:03 | |
| *** nosnos has quit IRC | 12:04 | |
| *** jgallard has joined #openstack-infra | 12:04 | |
| *** Longgeek has quit IRC | 12:06 | |
| openstackgerrit | Jamie Lennox proposed a change to openstack/requirements: Add requests-mock package to global requirements https://review.openstack.org/104067 | 12:06 |
| *** unicell has joined #openstack-infra | 12:06 | |
| *** basha has quit IRC | 12:07 | |
| *** gokrokve has quit IRC | 12:07 | |
| *** Longgeek_ has quit IRC | 12:07 | |
| *** Longgeek has joined #openstack-infra | 12:08 | |
| *** mugsie has joined #openstack-infra | 12:08 | |
| *** otherwiseguy has joined #openstack-infra | 12:14 | |
| *** annegent_ has joined #openstack-infra | 12:15 | |
| *** alexpilotti has joined #openstack-infra | 12:15 | |
| *** rfolco has joined #openstack-infra | 12:16 | |
| *** yaguang has joined #openstack-infra | 12:18 | |
| *** otherwiseguy has quit IRC | 12:18 | |
| *** doug-fish has joined #openstack-infra | 12:19 | |
| *** annegent_ is now known as annegentle_ | 12:21 | |
| *** mdenny has quit IRC | 12:22 | |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/requirements: Bump glanceclient to 0.13.1 https://review.openstack.org/104170 | 12:22 |
| *** mdenny has joined #openstack-infra | 12:22 | |
| openstackgerrit | A change was merged to openstack/requirements: Add python-marconiclient https://review.openstack.org/102806 | 12:23 |
| *** doude has quit IRC | 12:25 | |
| openstackgerrit | A change was merged to openstack/requirements: Update python-cinderclient>=1.0.7 https://review.openstack.org/103056 | 12:25 |
| *** pdmars has joined #openstack-infra | 12:27 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools https://review.openstack.org/104171 | 12:27 |
| *** dstanek_zzz is now known as dstanek | 12:29 | |
| *** doude has joined #openstack-infra | 12:30 | |
| *** pdmars_ has joined #openstack-infra | 12:31 | |
| *** pdmars has quit IRC | 12:31 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Add bug-deferral feature to process_bugs.py https://review.openstack.org/102218 | 12:31 |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Support proposed/* branches in release scripts https://review.openstack.org/102831 | 12:32 |
| *** Hal_ has joined #openstack-infra | 12:34 | |
| *** mestery has quit IRC | 12:35 | |
| *** hashar has joined #openstack-infra | 12:37 | |
| *** talluri has quit IRC | 12:38 | |
| *** talluri has joined #openstack-infra | 12:38 | |
| *** mestery has joined #openstack-infra | 12:39 | |
| *** otherwiseguy has joined #openstack-infra | 12:39 | |
| *** unicell has quit IRC | 12:39 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Support swift and oslo milestone releases https://review.openstack.org/99892 | 12:40 |
| *** BobBall_ has joined #openstack-infra | 12:42 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: The argparse module was added in requirements https://review.openstack.org/103389 | 12:42 |
| *** BobBall has quit IRC | 12:42 | |
| *** talluri has quit IRC | 12:43 | |
| *** dprince has joined #openstack-infra | 12:44 | |
| rgerganov | sdague, hi, do you have credentials for review-dev.openstack.org? I'd like to make a one line change in hideci.js | 12:45 |
| *** adalbas has joined #openstack-infra | 12:45 | |
| sdague | rgerganov: no, sorry. | 12:45 |
| sdague | SergeyLukjanov or fungi probably do | 12:46 |
| rgerganov | sorry for bothering you guys, but there is no easy way to test this locally | 12:46 |
| *** BobBall_ has quit IRC | 12:47 | |
| *** BobBall has joined #openstack-infra | 12:47 | |
| *** BobBall is now known as Guest60921 | 12:47 | |
| *** unicell has joined #openstack-infra | 12:50 | |
| SergeyLukjanov | rgerganov, hey | 12:50 |
| SergeyLukjanov | rgerganov, which kind of access do you need? | 12:51 |
| rgerganov | SergeyLukjanov, I'd like to change ciRegex in https://review-dev.openstack.org/static/hideci.js | 12:51 |
| *** baohua has quit IRC | 12:52 | |
| rgerganov | SergeyLukjanov, it should be / Dev$/ | 12:52 |
| *** baohua has joined #openstack-infra | 12:52 | |
| rgerganov | that is a hack to test my pending patch for hiding CI comments | 12:52 |
| *** dims__ has quit IRC | 12:52 | |
| SergeyLukjanov | Zwkrgybi864957 | 12:52 |
| *** julim has quit IRC | 12:53 | |
| SergeyLukjanov | storing my tab name here ;) | 12:53 |
| *** dims__ has joined #openstack-infra | 12:53 | |
| SergeyLukjanov | rgerganov, it should be updated on the host? | 12:54 |
| rgerganov | SergeyLukjanov, yes | 12:54 |
| *** weshay has joined #openstack-infra | 12:54 | |
| *** julim has joined #openstack-infra | 12:54 | |
| *** radez_g0n3 is now known as radez | 12:55 | |
| fungi | looks like /home/gerrit2/review_site/static/hideci.js on review-dev was last updated three days ago | 12:55 |
| SergeyLukjanov | fungi, morning | 12:55 |
| *** annegentle_ has quit IRC | 12:55 | |
| SergeyLukjanov | fungi, I'm going to update it now, is it ok? | 12:55 |
| rgerganov | fungi, yeah, mordred deployed it | 12:55 |
| fungi | SergeyLukjanov: sure, go ahead. i was hoping we had a small puppet change to deploy it just to review-dev, but apparently not | 12:57 |
| fungi | looks like it was just done manually | 12:57 |
| SergeyLukjanov | fungi, yeah, I've not find it too | 12:57 |
| *** jlibosva has quit IRC | 12:57 | |
| SergeyLukjanov | rgerganov, done | 12:58 |
| *** dkranz has joined #openstack-infra | 12:58 | |
| rgerganov | SergeyLukjanov, thanks! | 12:58 |
| fungi | SergeyLukjanov: looks like it may also require a gerrit restart there (at least i see one in the shell history performed immediately following mordred's edit) | 12:59 |
| *** jlibosva has joined #openstack-infra | 12:59 | |
| *** pblaho has quit IRC | 12:59 | |
| rgerganov | hm, the rest of the changes are gone | 12:59 |
| SergeyLukjanov | fungi, the js script updated already | 12:59 |
| SergeyLukjanov | fungi, https://review-dev.openstack.org/static/hideci.js | 12:59 |
| rgerganov | what about GerritSiteHeader.html? I think it is reverted | 13:00 |
| rgerganov | the "Toggle CI" button doesn't show up | 13:00 |
| *** BobBall_ has joined #openstack-infra | 13:00 | |
| kashyap | Heya folks, Sean Dague suggested an idea on #openstack-qa to host the Fedora image (for DevStack download) to tarballs.openstack.org/fedora since there have been mirroring issues from external mirrors | 13:01 |
| *** mdenny has quit IRC | 13:01 | |
| *** BobBall_ is now known as GnomeKing | 13:01 | |
| *** dangers_away is now known as dangers | 13:02 | |
| *** mfer has joined #openstack-infra | 13:02 | |
| *** gokrokve has joined #openstack-infra | 13:02 | |
| rgerganov | SergeyLukjanov, fungi now my browser requests https://review-dev.openstack.org/static/hideci.js?e=5d13164c690e5a28d09de3e94155baa8 which result in 404 | 13:02 |
| rgerganov | I have no idea where this URL parameter comes from | 13:03 |
| *** chandan_kumar is now known as chandankumar | 13:03 | |
| *** Guest60921 has quit IRC | 13:03 | |
| fungi | rgerganov: is it possible the other files are managed by puppet? if so, it probably overwrote them back to their previous contents configured in the openstack-infra/config project | 13:05 |
| *** tkelsey has joined #openstack-infra | 13:05 | |
| kashyap | I wonder what do admins here think for the above suggestion | 13:05 |
| rgerganov | fungi, this is the puppet change in my patch: https://review.openstack.org/#/c/95743/7/modules/openstack_project/manifests/gerrit.pp | 13:06 |
| *** yaguang has quit IRC | 13:06 | |
| fungi | kashyap: that seems like something we'd only want to turn on in our automated test runs, and not in general for all users of devstack. also having some mechanism to update that cached copy when upstream's image gets changed would probably be warranted | 13:06 |
| kashyap | afazekas, sdague ^ | 13:06 |
| *** gokrokve has quit IRC | 13:07 | |
| kashyap | fungi, Why not for all users of devstack? Just asking out of ignorance | 13:07 |
| fungi | kashyap: we already cache it onto our devstack snapshot images using nodepool, anyway, so it shouldn't be downloaded on every run, but only when we're creating new images | 13:07 |
| fungi | kashyap: we're not the fedora project, so ideally we shouldn't expect devstack users to get fedora images from us | 13:08 |
| fungi | kashyap: just like we don't intend for devstack users to get pypi packages from our pypi mirror, but rather from the real pypi | 13:08 |
| kashyap | fungi, True, this is not a long term solution, | 13:08 |
| *** MarkAtwood has joined #openstack-infra | 13:08 | |
| kashyap | I'm only trying to see if the mirroring issues are for real from Fedora Project | 13:08 |
| kashyap | The admins (Kevin Fenzi) there frequently ask me, if OpenStack upstream faces mirroring issues, and have specific details | 13:09 |
| kashyap | They're ready to jump on it to help out | 13:09 |
| fungi | kashyap: so to take a step back, what problem are you trying to solve? have we been having stale images due to nodepool image update failures from fedora image download problems? | 13:09 |
| fungi | or something else? | 13:09 |
| kashyap | fungi, Let me see if I can summarize - | 13:10 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Search use case 1: Quicknav https://review.openstack.org/102652 | 13:10 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix fingerprint to deal with n-obj hit https://review.openstack.org/104185 | 13:10 |
| fungi | kashyap: because we don't download fedora images for every devstack job we run. we bundle it into the snapshots which we use to spin up devstack test nodes, and that only gets refreshed daily | 13:10 |
| sdague | fungi: right, but if we have a 10% failure rate pulling into nodepool, then how does that impact nodepool builds? | 13:11 |
| kashyap | fungi, Fair enough (I understand the case where it's snapshotted & reused) | 13:11 |
| *** xianghui has quit IRC | 13:12 | |
| fungi | sdague: 1 in 10 nodepool images end up being a day older than the others | 13:12 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: DB api for teams https://review.openstack.org/104186 | 13:12 |
| *** e0ne has quit IRC | 13:12 | |
| * fungi checks to see if it's happening now | 13:12 | |
| sdague | yeh, part of what's hard here is the nodepool logs not being available outside of infra-root | 13:12 |
| *** MarkAtwood has quit IRC | 13:13 | |
| kashyap | Ah, that was what I was about to ask, if the logs are available | 13:13 |
| fungi | sdague: the nodepool image update logs *are* published and readily excessible to everyone | 13:13 |
| fungi | http://nodepool.openstack.org/ | 13:13 |
| *** e0ne has joined #openstack-infra | 13:13 | |
| *** MarkAtwood has joined #openstack-infra | 13:14 | |
| *** GnomeKin1 has joined #openstack-infra | 13:14 | |
| derekh_ | If this file out of date http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/logstash/agent.conf.erb I can't find log-pusher.py anywhere, I'm guessing I should be looking at log-gearman-worker.py | 13:14 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added custom CSS aliases for common storyboard icons. https://review.openstack.org/102963 | 13:14 |
| derekh_ | *Is | 13:14 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Auth error handling https://review.openstack.org/101867 | 13:14 |
| sdague | fungi: oh, | 13:15 |
| *** eharney has joined #openstack-infra | 13:15 | |
| fungi | anyway, a quick survey of our 8 locations where we build devstack-trusty snapshots seem to have successfully updated twice in the past 48 hours... i'll check again around 15:00 when today's runs complete | 13:15 |
| rgerganov | SergeyLukjanov, do you have a minute to check something on the review-dev host? | 13:15 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Notification Priority Constant https://review.openstack.org/103966 | 13:15 |
| fungi | derekh_: maybe what you're looking for is in http://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor | 13:17 |
| *** GnomeKing has quit IRC | 13:17 | |
| sdague | fungi: well I don't see errors in there, but the nodepool logs are pretty interleaved, so it's a little hard to see what's going on :) | 13:18 |
| derekh_ | fungi: cool, thanks | 13:18 |
| fungi | derekh_: i think it got split out into its own top-level puppet module on a refactor | 13:18 |
| fungi | sdague: yeah, it's suboptimal, but the data is there. i end up filtering for the provider+label i'm interested in | 13:19 |
| fungi | sdague: but i'm sure we wouldn't turn down improvements to nodepool's logging which make it easier for others to diagnose failures | 13:19 |
| sdague | fungi: ok, I stick it on my queue.... so sometime around the Q release :) | 13:20 |
| *** MarkAtwood has quit IRC | 13:20 | |
| *** dkliban_afk is now known as dkliban | 13:20 | |
| *** yaguang has joined #openstack-infra | 13:21 | |
| *** mriedem has joined #openstack-infra | 13:22 | |
| fungi | sdague: so soon?!? | 13:22 |
| *** NikitaKo_ has quit IRC | 13:23 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: fix fingerprint to deal with n-obj hit https://review.openstack.org/104185 | 13:23 |
| fungi | anyway, i don't currently see image updates impacted by cache refresh download issues, but if/when it does happen the impact is negligible since we just keep using the previous day's image wherever we failed to update | 13:24 |
| fungi | and we could always stick in some additional retry logic if it does become inconvenient for some reason | 13:24 |
| *** GnomeKin1 is now known as Bobba | 13:24 | |
| sdague | ok | 13:25 |
| fungi | also, if we start building one image and uploading it to all providers via glance, the number of times we download it would be the same whether we're building a new image or updating our network mirror | 13:25 |
| *** rgerganov has quit IRC | 13:25 | |
| sdague | yeh | 13:25 |
| sdague | well, is that true? | 13:26 |
| sdague | because we'd pull it to the mirror once, it would be known good | 13:26 |
| sdague | or are you seeing the image build with yesterday's image? | 13:26 |
| sdague | vs. building from a clean base | 13:26 |
| fungi | would we only pull it to the mirror once? does fedora not release new images? | 13:26 |
| fungi | but also i believe dib does its own local caching, so wouldn't redownload files unnecessarily (others who have worked directly with it should correct me if i've misunderstood) | 13:27 |
| sdague | fungi: these images are very specifically version tagged | 13:28 |
| *** xianghui has joined #openstack-infra | 13:29 | |
| fungi | and we don't update the reference to them in devstack? | 13:29 |
| sdague | not very often | 13:29 |
| sdague | this is the image that's needed for heat testing | 13:30 |
| fungi | i suspect we'd still need to dynamically identify when new versions are in need of being downloaded, whether the trigger is new versions appearing on a remote site or new version references in devstack itself, and automatically act accordingly | 13:30 |
| sdague | afazekas has the 3rd update for that image in 8 months pending | 13:30 |
| *** annegent_ has joined #openstack-infra | 13:31 | |
| fungi | currently nodepool figures out what to download by inspecting devstack source i think, but locally caching that and reusing it across rebuilds if it's still the same shouldn't be too much effort | 13:31 |
| *** julim has quit IRC | 13:34 | |
| *** prad has joined #openstack-infra | 13:35 | |
| *** julim has joined #openstack-infra | 13:35 | |
| sdague | fungi: actually we have an image_list.sh | 13:35 |
| sdague | because the source inspection was really error prone | 13:36 |
| fungi | yeah, that. i remember when it was added now | 13:36 |
| *** lcheng has quit IRC | 13:37 | |
| *** pcrews has joined #openstack-infra | 13:37 | |
| *** jlibosva has quit IRC | 13:38 | |
| *** lcheng has joined #openstack-infra | 13:38 | |
| *** jlibosva has joined #openstack-infra | 13:39 | |
| fungi | hrm, so looking at cache_devstack.py, since it runs on a fresh template we'd need to push the pre-cached files onto it if we wanted to reuse them, so we'd likely need some separate mechanism on the nodepool server to collect, cache and dump them to the templates | 13:40 |
| *** sgordon has joined #openstack-infra | 13:40 | |
| sgordon | anyone looked at ask.o.o today? seems we have a bit of a spam issue... | 13:40 |
| kashyap | Sorry fungi - My internet at my parents is abysmal, and my IRC proxy went bonkers, if you addressed any message for me, likely I haven't got it | 13:40 |
| fungi | but with dib, as i said, i think there's caching already build in | 13:40 |
| fungi | kashyap: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-07-02.log | 13:41 |
| fungi | sgordon: that's awesome | 13:41 |
| kashyap | sgordon, Yeah, I looked at it, the same crap was plagued to ask.fp.o | 13:41 |
| sgordon | fungi, i find it amusing that you only get 5 "report abuse" points per day | 13:42 |
| kashyap | sgordon, I guess I even called that spammer on that number & asked him to refrain (on a request from a ask.fp.o admin) | 13:42 |
| fungi | reed: ^ fyi | 13:42 |
| sgordon | fungi, but it seems you can create around 15 questions in 2 minutes | 13:42 |
| sgordon | before having to create another account (which they did) | 13:42 |
| fungi | sgordon: yeah, amusing indeed | 13:42 |
| *** annegent_ has quit IRC | 13:43 | |
| *** annegent_ has joined #openstack-infra | 13:44 | |
| *** wenlock has joined #openstack-infra | 13:44 | |
| *** jpeeler has quit IRC | 13:45 | |
| *** jpeeler has joined #openstack-infra | 13:45 | |
| *** jpeeler has quit IRC | 13:45 | |
| *** jpeeler has joined #openstack-infra | 13:45 | |
| *** prad has quit IRC | 13:47 | |
| *** homeless has joined #openstack-infra | 13:51 | |
| *** thedodd has joined #openstack-infra | 13:51 | |
| *** unicell has quit IRC | 13:52 | |
| *** lcheng has quit IRC | 13:53 | |
| *** baohua has quit IRC | 13:54 | |
| *** miqui has quit IRC | 13:56 | |
| *** amitpp has joined #openstack-infra | 13:57 | |
| *** NikitaKo_ has joined #openstack-infra | 13:58 | |
| *** mestery_ has joined #openstack-infra | 13:59 | |
| *** manueldesbonnet has joined #openstack-infra | 13:59 | |
| *** mestery has quit IRC | 14:00 | |
| *** mestery_ is now known as mestery | 14:00 | |
| *** amitpp has quit IRC | 14:00 | |
| *** amalagon has joined #openstack-infra | 14:01 | |
| *** lcheng has joined #openstack-infra | 14:01 | |
| *** gokrokve has joined #openstack-infra | 14:02 | |
| *** zz_dimtruck is now known as dimtruck | 14:03 | |
| *** yaguang has quit IRC | 14:03 | |
| *** julim has quit IRC | 14:03 | |
| *** julim has joined #openstack-infra | 14:04 | |
| dhellmann | fungi: thanks for tracking down that wheel issue and getting the fix pushed through | 14:04 |
| * dhellmann makes a note not to take off mondays any more | 14:04 | |
| * fungi makes a note to take mondays off instead | 14:05 | |
| fungi | sgordon: reed: i think we've squashed all the spambot posts for now, but should probably keep an eye out for more | 14:06 |
| sgordon | fungi, it would be preferable to actually delete them | 14:06 |
| *** gokrokve has quit IRC | 14:07 | |
| sgordon | fungi, for that kind of spammer the closed state with all the text still present is good enough for them | 14:07 |
| *** MarkAtwood has joined #openstack-infra | 14:07 | |
| sgordon | so not going to dissuade them in any way | 14:07 |
| fungi | sgordon: agreed. hopefully reed will get in touch with evgeny and get that to happen at some point | 14:07 |
| fungi | sgordon: reporting a bug about it at https://bugs.launchpad.net/openstack-community/+bugs?field.tag=askproject may also help | 14:08 |
| *** MarkAtwood has quit IRC | 14:08 | |
| fungi | sgordon: note that the openstack infrastructure team has no access to that system, so can't really do much about it | 14:08 |
| sgordon | does anybody then? | 14:09 |
| fungi | sgordon: might be that there are ask admins who can delete posts through the webui, though i don't know who they are... hopefully they have some bulk delete/hide feature at their disposal | 14:10 |
| *** rwsu has joined #openstack-infra | 14:10 | |
| *** wenlock has quit IRC | 14:10 | |
| *** jlibosva has quit IRC | 14:11 | |
| fungi | hmm... looks like ask calls them "moderators" | 14:11 |
| *** jlibosva has joined #openstack-infra | 14:11 | |
| fungi | https://wiki.openstack.org/wiki/Community/AskModerators#One_URL.2C_One_Question.2C_One_Answer implies moderators can delete questions | 14:13 |
| manueldesbonnet | Howdy, I'm looking to get https://review.openstack.org/#/c/101475/ approved, if possible (it's a change to pypimirror to allow specifying requirements files on cmdline). | 14:13 |
| manueldesbonnet | Any takers? | 14:13 |
| *** hashar has quit IRC | 14:14 | |
| *** salv-orlando has joined #openstack-infra | 14:14 | |
| fungi | sgordon: oho, looks like i actually have sufficient karma to be a moderator, but someone else has already deleted them now anyway | 14:15 |
| fungi | sgordon: i guess worth keeping in mind for next time | 14:16 |
| sgordon | i dont suppose you know what the magic number is | 14:16 |
| sgordon | i am still on ~1400 | 14:16 |
| fungi | sgordon: the top-right corner of the main interface claims 100 | 14:16 |
| *** yaguang has joined #openstack-infra | 14:17 | |
| *** vhoward- has joined #openstack-infra | 14:17 | |
| fungi | sgordon: when i view a question i see a somewhat discreet "delete" trashcan next to the "close" button | 14:17 |
| sgordon | ah yes | 14:18 |
| *** vhoward- has left #openstack-infra | 14:18 | |
| *** e0ne has quit IRC | 14:19 | |
| *** basha has joined #openstack-infra | 14:19 | |
| *** msabramo has quit IRC | 14:21 | |
| *** afazekas has quit IRC | 14:21 | |
| *** yfried has joined #openstack-infra | 14:22 | |
| *** NikitaKo_ has quit IRC | 14:25 | |
| *** gokrokve has joined #openstack-infra | 14:26 | |
| *** Hal_ has quit IRC | 14:27 | |
| *** gokrokve_ has joined #openstack-infra | 14:27 | |
| *** Hal_ has joined #openstack-infra | 14:30 | |
| yolanda | jeblair, working on changes for nodepool. Looking at your review, looks as if you want to drop the old method of uploading images using snapshots? | 14:30 |
| *** nelsnelson has quit IRC | 14:30 | |
| *** nelsnelson has joined #openstack-infra | 14:30 | |
| *** gokrokve has quit IRC | 14:31 | |
| *** zz_gondoi is now known as gondoi | 14:31 | |
| anteaya | I had a cranky router this morning | 14:32 |
| *** wenlock has joined #openstack-infra | 14:32 | |
| anteaya | ArxCruz: thanks for letting me know | 14:33 |
| ArxCruz | anteaya: np :) | 14:33 |
| ArxCruz | still working, i'll let you know when it's done :) | 14:33 |
| *** msabramo has joined #openstack-infra | 14:33 | |
| *** sabeen2 has joined #openstack-infra | 14:34 | |
| anteaya | krtaylor: yeah, nothing listed though I'm confident the topic will come up in many ways, not the least of which when we discuss vinz and gerrit gui | 14:34 |
| anteaya | ArxCruz: kk, thanks | 14:34 |
| *** jcoufal has quit IRC | 14:34 | |
| anteaya | ArxCruz: is your system disabled then? | 14:34 |
| *** prad has joined #openstack-infra | 14:34 | |
| ArxCruz | anteaya: no, but it's not voting, I need to have enabled to test :/ | 14:35 |
| ArxCruz | sorry | 14:35 |
| anteaya | well as long as it isn't voting | 14:35 |
| anteaya | can you have it enabled on sandbox only and have it test there? | 14:35 |
| anteaya | the community doesn't like to be the testing platform for things | 14:36 |
| anteaya | unless you have conscious consent | 14:36 |
| anteaya | because when you do that | 14:36 |
| anteaya | you train people to ignore your system | 14:36 |
| *** jcoufal has joined #openstack-infra | 14:36 | |
| anteaya | which is a bad choice | 14:37 |
| *** sabeen2 has quit IRC | 14:37 | |
| *** sabeen1 has quit IRC | 14:38 | |
| *** sabeen2 has joined #openstack-infra | 14:38 | |
| krtaylor | anteaya, I was asking because there was a small chance I could go to mid-cycle, hard to justify w/o CI content | 14:38 |
| anteaya | ah okay | 14:39 |
| krtaylor | anteaya, I'll still offer to have a mid-cycle for 3rd party systems in Austin :) | 14:39 |
| anteaya | let me help you justify | 14:39 |
| anteaya | right, a bit late to plan that one I think | 14:39 |
| anteaya | but perhaps we can plan if for k release | 14:39 |
| *** signed8bit has joined #openstack-infra | 14:40 | |
| krtaylor | anteaya, and a bit early to have real content, yes perhaps for K | 14:40 |
| anteaya | also the reason we don't explictly state third party business on the qa/infra meetup agenda is we tend to attract people who want tutorials on how to set up their system rather than wanting to help infra and qa work on infra and qa issues | 14:40 |
| *** Bobba is now known as BobBall | 14:40 | |
| anteaya | so I'm willing to write a letter to whomever needs to read the letter to get you to attend germany | 14:41 |
| anteaya | since I want you to be there | 14:41 |
| anteaya | so what do I do to make that happen? | 14:41 |
| *** basha has quit IRC | 14:41 | |
| *** ociuhandu has quit IRC | 14:42 | |
| krtaylor | anteaya, understood, that is a problem | 14:42 |
| krtaylor | anteaya, I don't think a letter is necessary, but you could help me come up with CI discussion justificsation | 14:43 |
| anteaya | yes, and we are trying to make all the people happy, which is hard | 14:43 |
| krtaylor | and justification | 14:43 |
| anteaya | sure | 14:43 |
| *** gokrokve_ has quit IRC | 14:43 | |
| krtaylor | hehheh | 14:43 |
| anteaya | toss some words on an etherpad and we can move them around until you have what you need | 14:43 |
| anteaya | how's that | 14:43 |
| krtaylor | yes, hard, but we could get a lot of doc/definitions work done | 14:44 |
| krtaylor | anteaya, will do | 14:44 |
| anteaya | tons | 14:44 |
| anteaya | hey I could put infra-manual on the agenda | 14:44 |
| anteaya | would that help? | 14:44 |
| anteaya | I plan on working on that anyway | 14:44 |
| *** jgrimm has joined #openstack-infra | 14:44 | |
| tteggel | hpcloud instance boot times high at the moment.. we're working the issue | 14:44 |
| *** otherwiseguy has quit IRC | 14:44 | |
| anteaya | tteggel: thanks for the heads up | 14:44 |
| anteaya | appreciate it | 14:44 |
| krtaylor | anteaya, do we have a useful set of CI folks attending? | 14:45 |
| tteggel | anteaya: you're welcome | 14:45 |
| * krtaylor needs to review attending | 14:45 | |
| anteaya | tteggel: anything we can do to help? | 14:45 |
| *** yaguang has quit IRC | 14:45 | |
| tteggel | anteaya: don't think so. our DB is in meltdown. we have an couple of mysql experts in attendance | 14:45 |
| anteaya | krtaylor: well all of infra is involved with helping third party be as smooth as possible | 14:45 |
| anteaya | I'm just playing point | 14:46 |
| anteaya | tteggel: very good, don't let me interupt | 14:46 |
| anteaya | tteggel: do say if we can help with anything | 14:46 |
| tteggel | anteaya: will do | 14:46 |
| anteaya | tteggel: thanks | 14:46 |
| anteaya | krtaylor: and we will be discussing vinz, which is the hopeful gerrit gui replacement | 14:47 |
| *** gokrokve has joined #openstack-infra | 14:47 | |
| anteaya | so the whole ml discussion about how to represent third party ci will get much air time | 14:47 |
| anteaya | and your input would be very welcome in that discusssion | 14:47 |
| anteaya | krtaylor: what do you need to see to decide if you want to attend? | 14:48 |
| anteaya | because I guarentee there will be value for you there | 14:48 |
| fungi | sdague: your https://review.openstack.org/104009 nova change is hitting an interesting py26 unit test failure in the gate | 14:48 |
| *** ArxCruz has quit IRC | 14:49 | |
| *** ArxCruz_ has joined #openstack-infra | 14:49 | |
| *** ArxCruz_ has quit IRC | 14:50 | |
| *** andreaf has quit IRC | 14:51 | |
| *** devkulkarni has joined #openstack-infra | 14:51 | |
| *** ArxCruz has joined #openstack-infra | 14:51 | |
| krtaylor | anteaya, its not me, its getting approval, need strong case | 14:52 |
| *** mfer has quit IRC | 14:52 | |
| krtaylor | anteaya, I have to drive to office, bbiab and we'll discuss more | 14:52 |
| sdague | fungi: yeh | 14:53 |
| sdague | honestly, I've not landed an openstack without 3 rechecks in a week :) | 14:53 |
| sdague | and they are all different fails | 14:53 |
| sdague | so my energy to dig has been exhausted | 14:54 |
| *** tkelsey has quit IRC | 14:54 | |
| mtreinish | sdague: push the patch to remove py26 testing from nova | 14:54 |
| mtreinish | it seems like a justified response :) | 14:54 |
| *** mfer has joined #openstack-infra | 14:55 | |
| *** dims__ has quit IRC | 14:55 | |
| *** ociuhandu has joined #openstack-infra | 14:55 | |
| anteaya | krtaylor: kk | 14:56 |
| anteaya | drive safe | 14:56 |
| *** vhoward- has joined #openstack-infra | 14:56 | |
| *** yaguang has joined #openstack-infra | 14:58 | |
| *** miqui has joined #openstack-infra | 14:59 | |
| *** e0ne has joined #openstack-infra | 15:02 | |
| *** krtaylor has quit IRC | 15:05 | |
| *** jaypipes-GOOAL has quit IRC | 15:06 | |
| *** tkelsey_ has joined #openstack-infra | 15:07 | |
| *** doug-fish has quit IRC | 15:08 | |
| *** doug-fish has joined #openstack-infra | 15:09 | |
| *** gokrokve has quit IRC | 15:10 | |
| *** mrmartin has quit IRC | 15:11 | |
| *** gokrokve has joined #openstack-infra | 15:11 | |
| *** virmitio has joined #openstack-infra | 15:13 | |
| *** moted has joined #openstack-infra | 15:13 | |
| *** MarkAtwood has joined #openstack-infra | 15:17 | |
| *** doug-fish has quit IRC | 15:17 | |
| *** doug-fish has joined #openstack-infra | 15:18 | |
| *** xianghui has quit IRC | 15:18 | |
| *** matrohon has quit IRC | 15:18 | |
| *** packet has joined #openstack-infra | 15:19 | |
| *** hashar has joined #openstack-infra | 15:23 | |
| *** dims__ has joined #openstack-infra | 15:23 | |
| mordred | mtreinish: :) | 15:24 |
| *** jlibosva has quit IRC | 15:25 | |
| *** gokrokve has quit IRC | 15:25 | |
| *** dims__ has quit IRC | 15:27 | |
| *** dims__ has joined #openstack-infra | 15:27 | |
| *** matrohon has joined #openstack-infra | 15:28 | |
| *** gondoi is now known as zz_gondoi | 15:32 | |
| *** zz_gondoi is now known as gondoi | 15:35 | |
| *** unicell has joined #openstack-infra | 15:36 | |
| *** james_li has joined #openstack-infra | 15:36 | |
| *** mzoeller has quit IRC | 15:36 | |
| zaro | morning | 15:38 |
| *** deny_ has joined #openstack-infra | 15:39 | |
| *** yfried has quit IRC | 15:40 | |
| openstackgerrit | A change was merged to openstack/requirements: Update glanceclient to 0.13.1 https://review.openstack.org/104070 | 15:40 |
| *** misterpink has quit IRC | 15:40 | |
| hashar | good morning! | 15:40 |
| hashar | if you ever heard of Gerrit hash mismatch bug of doom. We apparently got the solution https://bugzilla.wikimedia.org/show_bug.cgi?id=53895#c28 | 15:41 |
| hashar | Apache Mina SSHD needs to be updated. pelix told me he was going to try a new version of Gerrit that got that bugfix | 15:41 |
| zaro | jhesketh, clarkb : i believe nodepool is on jenkins-dev https://review.openstack.org/#/c/88026 | 15:42 |
| *** jcoufal has quit IRC | 15:42 | |
| *** otherwiseguy has joined #openstack-infra | 15:42 | |
| *** yfried has joined #openstack-infra | 15:43 | |
| *** yaguang has quit IRC | 15:44 | |
| *** afazekas has joined #openstack-infra | 15:44 | |
| *** msabramo has quit IRC | 15:44 | |
| *** chandankumar has quit IRC | 15:44 | |
| zaro | hashar: good to know. which version of gerrit was that bug introduced? | 15:44 |
| hashar | zaro: apparently related to Java version being used | 15:46 |
| hashar | zaro: our bug report is quite old. I don't think anyone bothered to bisect the multiple softwares involved | 15:46 |
| *** changbl has quit IRC | 15:47 | |
| *** gangil has joined #openstack-infra | 15:48 | |
| openstackgerrit | Julien Vey proposed a change to openstack-infra/config: Add solum-infra-guestagent repo to stackforge https://review.openstack.org/99705 | 15:48 |
| *** nati_ueno has joined #openstack-infra | 15:50 | |
| *** ihrachyshka has quit IRC | 15:51 | |
| zaro | hashar: we are on java '(IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)' but have not run into the problem yet, or not that anyone has reported. | 15:53 |
| hashar | I have no idea which version of java we are running :/ | 15:53 |
| *** pdmars_ has quit IRC | 15:54 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Move puppet-trove to Stackforge https://review.openstack.org/104244 | 15:55 |
| fungi | zaro: hashar: actually we run into it all the time in our git-review functional testing | 15:55 |
| *** pballand has joined #openstack-infra | 15:55 | |
| *** msabramo has joined #openstack-infra | 15:55 | |
| jeblair | clarkb, jhesketh: i've put jenkins07 in shutdown mode (maybe jobs will be closer to being done when clarkb comes online) | 15:56 |
| fungi | zaro: hashar: the reason we don't see it in public is that if you have bouncy castle loaded the problem method seems to not be in the code path any longer (presumably one of the routines offloaded to bc) | 15:56 |
| pelix | zaro: hashar: the latest gerrit 2.8.6 has the fix, but broke other stuff (namely DB transactions), but it may still work for the unit tests | 15:56 |
| fungi | s/public/production/ | 15:56 |
| *** roeyc has quit IRC | 15:56 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Move puppet-trove to Stackforge https://review.openstack.org/104244 | 15:57 |
| tteggel | hpcloud instance build time issue resolved. should be back to normal. | 15:57 |
| fungi | the details of the error in mina-sshd are amusing too. if the generated key material starts with a zero byte, some functions strip it causing a mismatch 1 in 256 times | 15:57 |
| *** Sukhdev has joined #openstack-infra | 15:58 | |
| hashar | though on the gerrit-review integration test that happen more frequently apparently | 15:58 |
| *** bdpayne has joined #openstack-infra | 15:58 | |
| fungi | tteggel: great--thanks for the update! luckily it didn't seem to impact us heavily (at least not that i noticed in the graphs) | 15:58 |
| hashar | we have a patch that haven't yet landed after 6 rechecks or so | 15:58 |
| fungi | hashar: i don't think it really happened more often... i think that we actually make roughly that many ssh connections over the course of one test run | 15:58 |
| *** IvanBerezovskiy has left #openstack-infra | 15:58 | |
| *** otherwiseguy has quit IRC | 15:58 | |
| hashar | fungi: oh of course! | 15:58 |
| fungi | so the chances that one gets fouled up by the 1:256 chance is extremely high | 15:59 |
| *** skolekonov has quit IRC | 15:59 | |
| *** pballand has quit IRC | 16:00 | |
| ttx | jeblair: ping re: using proposed/* now for swift 2.0.0-rc2 | 16:00 |
| fungi | if 2.8.6 incorporates an updated mina-sshd with the fix, i'll try a patch switching to that in the tox env | 16:00 |
| *** signed8b_ has joined #openstack-infra | 16:00 | |
| jeblair | ttx: i suspect we may have some gerrit acl work to do; will check | 16:00 |
| *** otherwiseguy has joined #openstack-infra | 16:00 | |
| ttx | jeblair: context: swift 2.0.0 needs a few commits backported, RC1 was tagged on master | 16:01 |
| fungi | jeblair: i updated the all-projects acl this morning when i reapproved ttx's rebased acl patch | 16:01 |
| ttx | jeblair: my proposal would be to create a temporary proposed/2.0.0 branch and backport to it, tag on it | 16:01 |
| jeblair | ttx: i'm guessing fungi already did that :) | 16:01 |
| jeblair | fungi: yay | 16:01 |
| *** signed8bit has quit IRC | 16:01 | |
| zaro | fungi: maybe you should try on 2.9, it's about to be released any day now. | 16:01 |
| ttx | jeblair: afaik we have the tooling aligned now, merge_tags was also pushed | 16:02 |
| *** anode has joined #openstack-infra | 16:02 | |
| *** mestery has quit IRC | 16:02 | |
| ttx | jeblair: if you're right that the magic should just happen with proposed/foo, then I think that will work | 16:02 |
| fungi | zaro: there's a balance between testing git-review with a gerrit version people are likely to already have running in public, and a version of gerrit which doesn't cause tests to arbitrarily fail. but yes i expect 2.9 will start getting tested for git-review changes soon | 16:02 |
| jeblair | ttx: ok, i can't think of anything that would block; i can stand by to assist if there are problems | 16:02 |
| ttx | cool, let me try to manually create the branch from the RC1 tag | 16:03 |
| *** gyee has joined #openstack-infra | 16:03 | |
| zaro | fungi: 2.8.x doesn't have the CHANGE_OWNER acl, 2.9 does. | 16:03 |
| * ttx engages manual mode | 16:03 | |
| *** pballand has joined #openstack-infra | 16:03 | |
| fungi | zaro: awesome--i can't wait for us to upgrade so we have access to start using that | 16:03 |
| tteggel | fungi: welcome. i want to see the error rate from hpcloud drop too | 16:04 |
| fungi | zaro: did they keep the old change screen around in 2.9 after all? | 16:04 |
| ttx | jeblair: the idea would be to merge tags and then delete that proposed/2.0.0 branch | 16:04 |
| jeblair | ttx: sounds good | 16:05 |
| *** bdpayne has quit IRC | 16:05 | |
| zaro | fungi: hmm, i believe so. might default to new change screen but default can be set in gerrit.config | 16:06 |
| *** gokrokve has joined #openstack-infra | 16:06 | |
| ttx | jeblair: ok, proposed/2.0.0 branch craeted, post job making a tarball from it | 16:08 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/git-review: Update tests to use Gerrit 2.8.6 https://review.openstack.org/104249 | 16:08 |
| *** arnaud has joined #openstack-infra | 16:08 | |
| *** ildikov has joined #openstack-infra | 16:09 | |
| jeblair | 2014-07-02 16:08:20.365 | [SCP] uploading file: '/srv/static/tarballs/swift/swift-proposed-2.0.0.tar.gz' | 16:09 |
| zaro | fungi: speaking of.. would it be possible to move this along? https://review.openstack.org/#/c/94675 | 16:10 |
| ttx | jeblair: sounds legit | 16:10 |
| zaro | fungi: it's been in the que for a while and i believe rcarrillocruz is waiting for it to lang. | 16:10 |
| zaro | *lang/land | 16:10 |
| *** lcheng has quit IRC | 16:10 | |
| fungi | zaro: does that appear in 2.8.5? | 16:11 |
| *** ildikov|2 has joined #openstack-infra | 16:11 | |
| *** ildikov has quit IRC | 16:11 | |
| zaro | fungi: no, it's not back ported. | 16:12 |
| fungi | hrm... looks like it's not included in any tags | 16:12 |
| fungi | okay, so this is a backport to 2.8 of something earmarked for 2.9 or 3.0? | 16:12 |
| *** ildikov_ has quit IRC | 16:12 | |
| zaro | yes | 16:12 |
| mordred | jeblair: neat! | 16:13 |
| *** paul-- has quit IRC | 16:14 | |
| fungi | zaro: if we approve 94675 we still need to update the config repo to refer to the new warfile url, right? | 16:15 |
| jeblair | yesterday, i learned that sheng kee makes hong kong style pineapple buns! i'm so excited because i've been jonesing and that was looking to be an expensive breakfast. | 16:15 |
| fungi | zaro: also, do you know whether any of our current hook scripts will break when gerrit starts applying the new command-line option? | 16:16 |
| *** otherwiseguy has quit IRC | 16:16 | |
| ttx | jeblair: notmyname should push a few backports to that branch soon. Could you keep an eye on the tests and make sure they do what they are supposed to do ? | 16:16 |
| jeblair | ttx, notmyname: let me know when | 16:17 |
| ttx | I think they will, the use case is not that weird (shall use master everywhere else) | 16:17 |
| *** rgerganov has joined #openstack-infra | 16:17 | |
| fungi | hashar: i'm exercising 2.8.6 in https://review.openstack.org/104249 to confirm the bug is gone... first pass was all green! | 16:18 |
| *** gangil has left #openstack-infra | 16:18 | |
| *** paul-- has joined #openstack-infra | 16:19 | |
| zaro | fungi: yes, an update to load new ware would be required as well. | 16:19 |
| reed | sgordon, fungi: evgeny is working on a fix for the bot issue, ETA a couple of days... hopefully | 16:19 |
| *** mestery has joined #openstack-infra | 16:19 | |
| fungi | reed: awesome--thanks for the heads up | 16:19 |
| *** atiwari has joined #openstack-infra | 16:19 | |
| *** mriedem has quit IRC | 16:19 | |
| *** jcoufal has joined #openstack-infra | 16:20 | |
| *** doude has quit IRC | 16:20 | |
| pleia2 | good morning | 16:20 |
| zaro | fungi: i haven't tested but should be completely backwards compatible. IIRC the scripts in jeepyb retreives the info by name. rcarrillcruz wanted to update jeepyb with the additional info. | 16:22 |
| annegent_ | reed: around? | 16:24 |
| fungi | zaro: okay, great. the only hook i specifically remember breaking on unrecognized options was the trivial rebase hook, which we don't use now | 16:24 |
| reed | annegent: yes | 16:25 |
| *** ArxCruz has quit IRC | 16:25 | |
| *** ArxCruz has joined #openstack-infra | 16:26 | |
| hashar | fungi: awesome! | 16:26 |
| hashar | fungi: gotta move out. bbl | 16:26 |
| openstackgerrit | Julien Vey proposed a change to openstack-infra/config: Add solum-infra-guestagent repo to stackforge https://review.openstack.org/99705 | 16:27 |
| *** hashar has quit IRC | 16:27 | |
| *** krtaylor has joined #openstack-infra | 16:30 | |
| *** markmc has quit IRC | 16:31 | |
| zaro | fungi: from looking at the code, it looks like it will just ignore the new arg. http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n328 | 16:31 |
| *** basha has joined #openstack-infra | 16:31 | |
| *** david-lyle has joined #openstack-infra | 16:32 | |
| *** ArxCruz has quit IRC | 16:33 | |
| *** ArxCruz_ has joined #openstack-infra | 16:33 | |
| *** ildikov|2 has quit IRC | 16:33 | |
| fungi | zaro: well, yeah, as long as our other hook scripts are written similarly then they should also be fine (remember that this option is being passed as part of all of those hook calls, so every hook script needs to be written to ignore unrecognized options) | 16:33 |
| *** nati_ueno has quit IRC | 16:33 | |
| *** flaper87 is now known as flaper87|afk | 16:34 | |
| ruhe | mordred: hi! question about recent changes in select-mirror.sh. now it uses http://pypi.openstack.org/simple ; who controls this mirror and how can i push a specific package to that mirror? | 16:35 |
| pelix | hashar: I can confirm we'll have to wait for the gerrit 2.8.7 to test git-review against release as 2.8.6 clearly has some issues... | 16:35 |
| *** bdpayne has joined #openstack-infra | 16:35 | |
| *** basha has quit IRC | 16:35 | |
| *** ArxCruz_ has quit IRC | 16:35 | |
| *** ArxCruz has joined #openstack-infra | 16:36 | |
| fungi | ruhe: that mirror is a direct copy of pypi.python.org | 16:36 |
| ruhe | fungi: hi. do you have any idea why a package (which is present on pypi.python.org) wouldn't be present in this mirror? | 16:37 |
| ruhe | fungi: here is the package https://pypi.python.org/pypi/yaql | 16:37 |
| openstackgerrit | A change was merged to stackforge/python-jenkins: Adds assert_node_exists() https://review.openstack.org/100835 | 16:37 |
| fungi | ruhe: checking | 16:38 |
| zaro | fungi: ohh of course. i'll double check the other ones. | 16:38 |
| *** dizquierdo has quit IRC | 16:38 | |
| ruhe | fungi: thanks | 16:38 |
| *** dkehn__ has joined #openstack-infra | 16:39 | |
| jeblair | yolanda: yes -- my guess is that it would be easiest to remove the support for snapshot-based images in nodepool. ideally, we would keep them both, but i think the concurrency requirements are different enough that it would be compilacetd | 16:40 |
| *** Ajaeger has joined #openstack-infra | 16:40 | |
| jeblair | yolanda: so if you find that you agree and it would be easier to remove it, i think that's fine. | 16:40 |
| fungi | mordred: ruhe has found an interesting gap... https://pypi.python.org/simple/yaql/ exists (and has tarball links) while there is no http://pypi.openstack.org/simple/yaql/ at all | 16:40 |
| Ajaeger | Hi infra team, I have a few docu problems. Has anybody now some time for some help, please? | 16:41 |
| jeblair | fungi, mordred: grep yaql /var/log/bandersnatch/mirror.log produces nothing | 16:41 |
| *** dkehnx has quit IRC | 16:41 | |
| yolanda | jeblair, ok, at the moment i was separating cases, but it will be much simpler to just drop it | 16:42 |
| fungi | jeblair: yep | 16:42 |
| yolanda | i can upload something for tomorrow | 16:42 |
| mordred | jeblair: problem with the cron job perhaps? you can run "bandersnatch mirror" at any time and it should do a mirror run real quick | 16:42 |
| jeblair | mordred: it's running regularly | 16:43 |
| mordred | k. well, that's both good and bad | 16:43 |
| jeblair | mordred: http://paste.openstack.org/show/85336/ there's the latest output | 16:43 |
| mordred | oh- I mis-read the above - I thought you were saying there was no output in the log :) | 16:43 |
| jeblair | also, do whe know why it's having that problem with skll every time? | 16:43 |
| jeblair | (it repats that output for every run) | 16:43 |
| *** marcoemorais has joined #openstack-infra | 16:43 | |
| reed | sgordon, jpich, fungi: ask.openstack.org looks better now, thanks those who helped deleting spam and blocking users | 16:44 |
| jeblair | fungi: i'm assuming i should have cronspam with previous mirror output, yeah? | 16:44 |
| fungi | jeblair: yes | 16:44 |
| jeblair | fungi: i can't find cronspam with yaql in it | 16:44 |
| *** thedodd has quit IRC | 16:44 | |
| jeblair | so there was probably never an error reported, or if there was, it must have been in an inital run or something | 16:45 |
| *** annegent_ has quit IRC | 16:45 | |
| *** marcoemorais has quit IRC | 16:45 | |
| *** ArxCruz_ has joined #openstack-infra | 16:45 | |
| *** marcoemorais has joined #openstack-infra | 16:45 | |
| *** hemna__ is now known as hemna | 16:45 | |
| fungi | agreed. i'm digging through the fs tree currently to see if it's somehow being misplaced or under different capitalization | 16:46 |
| jeblair | Ajaeger: how can i help? | 16:46 |
| *** ArxCruz has quit IRC | 16:46 | |
| *** otherwiseguy has joined #openstack-infra | 16:46 | |
| Ajaeger | First issue is: https://review.openstack.org/#/c/103115 was approved - but I do not see the docs-specs repository | 16:46 |
| Ajaeger | jeblair: thanks ^ | 16:46 |
| *** jpich has quit IRC | 16:47 | |
| Ajaeger | jeblair: are some manual steps necessary? | 16:47 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/infra-specs: Add spec for adding cross-project unit test jobs https://review.openstack.org/95885 | 16:47 |
| jeblair | Ajaeger: no, but i think someone broke puppet yesterday | 16:47 |
| Ajaeger | jeblair: that would explain my second issue as well - https://review.openstack.org/#/c/103578/ was approved but jenkins still has the old slave script | 16:48 |
| *** marcoemorais1 has joined #openstack-infra | 16:48 | |
| dhellmann | jeblair: is it safe for me to proceed with implementing ^^? There hasn't been any feedback on the spec, yet, since I think I filed it when you guys were dealing with outages. | 16:49 |
| dhellmann | jeblair: well, not quite true, lifeless did leave some questions which I've answered | 16:49 |
| Ajaeger | jeblair: and perhaps my 3rd one as well - http://pypi.openstack.org/openstack/openstack-doc-tools/ does not have the new 0.16.1 release... | 16:49 |
| fungi | jeblair: mordred: nothing at all for 'find /srv/static/mirror/ | grep -i yaql' so presumably bandersnatch is just ignoring it outright | 16:50 |
| *** marcoemorais has quit IRC | 16:50 | |
| *** zhiyan is now known as zhiyan_ | 16:50 | |
| *** zhiyan_ is now known as zhiyan | 16:51 | |
| Ajaeger | fungi, see also my issue above - normally pypi.o.o has a new tag within an hour but I'm waiting now for four hours for 0.16.1 | 16:51 |
| jeblair | wow | 16:51 |
| jeblair | apparently no one reviewed the emergency revert of the patcch that broke apache | 16:51 |
| rgerganov | mordred: could you please redeploy https://review.openstack.org/#/c/95743/ on review-dev.openstack.org? I have fixed a problem with ciRegex - it was too slow | 16:52 |
| *** pelix has quit IRC | 16:52 | |
| *** harlowja_away is now known as harlowja | 16:52 | |
| *** tkelsey_ has quit IRC | 16:52 | |
| jeblair | Ajaeger: i've approved the patch that should fix puppet runs, things should happen in a few hours | 16:53 |
| zaro | fungi, rcarrillcruz:: double checked, all those jeepyb scripts use the same pattern. will just ignore new params. | 16:53 |
| jeblair | Ajaeger: i don't know why that would not have the latest release | 16:54 |
| jeblair | Ajaeger: we'll need to check the post job | 16:54 |
| Ajaeger | jeblair: ok, will check back for the first two issues later/tomorrow. Thanks! | 16:54 |
| jeblair | Ajaeger: i'm looking up the post job for the third one | 16:55 |
| fungi | http://logs.openstack.org/f8/f8d93fcd5fc4b57b474a5ff7ecbf8d8a5e4e4173/release/openstack-doc-tools-pypi-both-upload/1faf6fc/console.html seems to have uploaded | 16:55 |
| Ajaeger | jeblair: https://jenkins.openstack.org/job/openstack-doc-tools-pypi-both-upload/2/ | 16:55 |
| Ajaeger | fungi: you're too fast for me ;) | 16:55 |
| jeblair | Ajaeger: oh, it's on pypi.python.org: https://pypi.python.org/pypi/openstack-doc-tools/0.16.1 | 16:56 |
| Ajaeger | jeblair, fungi: It's available indeed at https://pypi.python.org/pypi/openstack-doc-tools/ | 16:56 |
| fungi | https://pypi.python.org/pypi/openstack-doc-tools/ has it | 16:56 |
| jeblair | it's just not in the mirror | 16:56 |
| jeblair | hah | 16:56 |
| Ajaeger | jeblair: yeah, it's not in the mirrors | 16:57 |
| Ajaeger | and what's even strangeer is that 0.16 is in the mirror but building uses 0.15 | 16:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Revert "Downgrade puppetlabs-apache to version 0.0.4."" https://review.openstack.org/104038 | 16:57 |
| jeblair | so this could be another manifestation of the bandersnatch problem fungi and hopefully mordred are working on | 16:57 |
| *** dkehn__ is now known as dkehnx | 16:57 | |
| jeblair | that change ^^ should get puppet running again, so maybe in 10 mins or so we'll have new projects | 16:57 |
| fungi | yeo, not at http://pypi.openstack.org/simple/openstack-doc-tools/ | 16:57 |
| mordred | jeblair, fungi: dstufft said something about bandersnatch behavior when it sees a stale thing | 16:57 |
| mordred | dstufft: you aroudn? what was the remediation for that again? | 16:58 |
| Ajaeger | http://logs.openstack.org/19/104219/1/check/gate-security-doc-tox-checklang/3b1f930/console.html - shows that 0.15 is ued | 16:58 |
| Ajaeger | s/ued/used/ | 16:58 |
| *** whoops has joined #openstack-infra | 16:58 | |
| Ajaeger | jeblair: if you think I should come check back later, no problem... | 16:58 |
| jeblair | mordred: i think we have 3 problems: 1) yaql is not in the mirror at all, 2) openstack-doc-tools has an old version in the mirror, and 3) the mirror emits error messages about skll: http://paste.openstack.org/show/85336/ | 16:59 |
| fungi | Ajaeger: yep, it seems that it ended up on upstream pypi but out mirror routine hasn't pulled a copy from there | 16:59 |
| jeblair | dstufft: ^ see 3 problems | 16:59 |
| *** jgallard has quit IRC | 16:59 | |
| mordred | jeblair: I'm operating under the hunch that they're all related - based on something dstufft said a while back that I may have mis-interpreted | 17:00 |
| mordred | dstufft: namely something about noticing when a package has stale entries in the CDN | 17:00 |
| Ajaeger | fungi, jeblair, mordred : Thanks for looking at this! I'll keep silent watching you debug it ;) | 17:01 |
| mordred | I was thinking it meant that there was a possibility that we'd end up with a package stale for a little bit - but it seems that maybe it means that bandersnatch goes bonkers and there is a deeper (and crappier) problem we have to fix | 17:01 |
| *** jamielennox is now known as jamielennox|away | 17:02 | |
| mordred | jeblair: but I think dstufft can clarify and help know where we should look | 17:02 |
| *** Longgeek has quit IRC | 17:02 | |
| fungi | mordred: does bandersnatch have a debug option, so we could tell it for example to update its mirror for just that package and then inspect its decision making? | 17:02 |
| *** otherwiseguy has quit IRC | 17:03 | |
| fungi | the manpage on static.o.o for bandersnatch is... let's just say unenlightening. i'll go hunting actual documentation | 17:04 |
| jeblair | dhellmann: spec lgtm | 17:04 |
| dhellmann | jeblair: cool, thanks | 17:04 |
| *** otherwiseguy has joined #openstack-infra | 17:05 | |
| mordred | fungi: it has something to do with pypi serial numbers | 17:05 |
| jeblair | dhellmann: for the record, i don't think i ignored that the whole time; i think i read it and saw lifeless's suggestions and thought they were good and waited for a response. so i've only been ignoring it for a a day or so! :) | 17:05 |
| dhellmann | jeblair: yeah, that's sort of what I figured happened :-) | 17:06 |
| *** mmaglana has joined #openstack-infra | 17:06 | |
| jeblair | dhellmann: i'm really glad we have all those options documented. i'm sure we're going to be reading them in 6 months :) | 17:07 |
| ruhe | fungi: jeblair: i've just compared contents of pypi.python.org/simple and pypi.openstack.org/simple . the latter one is missing 1081 packages (45588 compared to 44507) | 17:07 |
| dhellmann | jeblair: it's like you can see into the future | 17:07 |
| jeblair | dstufft, mordred: so we're up to 1081 problems! :) | 17:07 |
| fungi | mordred: yeah, i'm guessing this is related to https://bitbucket.org/ctheune/bandersnatch/issue/41/hundreds-of-packages-fail-to-sync ? | 17:07 |
| *** gondoi is now known as zz_gondoi | 17:08 | |
| *** zhiyan is now known as zhiyan_ | 17:08 | |
| *** SumitNaiksatam has quit IRC | 17:09 | |
| *** ArxCruz has joined #openstack-infra | 17:09 | |
| *** derekh_ has quit IRC | 17:11 | |
| *** zz_gondoi is now known as gondoi | 17:11 | |
| *** ArxCruz_ has quit IRC | 17:11 | |
| openstackgerrit | Khai Do proposed a change to stackforge/python-jenkins: Add a gitreview file https://review.openstack.org/104261 | 17:14 |
| *** praneshp has joined #openstack-infra | 17:14 | |
| fungi | mordred: i'm not finding much at all in the way of debugging options for bandersnatch (unless we want to run it under pdb or something) | 17:15 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:15 | |
| ruhe | fungi: as a temporary fix (in case if this problem will not be resolved soon), would it make sense to add "extra-index-url = http://pypi.python.org/simple/" to the select-mirror.sh? | 17:16 |
| *** devkulkarni has quit IRC | 17:16 | |
| *** gondoi is now known as zz_gondoi | 17:16 | |
| Ajaeger | jeblair: docs-specs exists - so the puppet patch fixed it! | 17:16 |
| *** Ryan_Lane has joined #openstack-infra | 17:16 | |
| *** melwitt has joined #openstack-infra | 17:18 | |
| Ajaeger | jeblair: still an empty repo, so puppet is not finished with doing its magic... | 17:19 |
| *** dkehn__ has joined #openstack-infra | 17:21 | |
| *** otherwiseguy has quit IRC | 17:21 | |
| *** amcrn has joined #openstack-infra | 17:23 | |
| *** dkehn__ has quit IRC | 17:23 | |
| *** dkehn__ has joined #openstack-infra | 17:24 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:24 | |
| *** dkehn_ has quit IRC | 17:24 | |
| *** dkehnx has quit IRC | 17:25 | |
| *** crc32 has joined #openstack-infra | 17:25 | |
| *** mriedem has joined #openstack-infra | 17:25 | |
| *** dkehn__ is now known as dkehnx | 17:26 | |
| *** dkehn has joined #openstack-infra | 17:26 | |
| *** sparkycollier has joined #openstack-infra | 17:29 | |
| jeblair | Ajaeger: there's a further puppet problem that prevented manage-projects from running on review.o.o; looking | 17:31 |
| *** mbacchi has quit IRC | 17:33 | |
| mgagne | Does it bother anyone that program names aren't consistent across programs? "Queue service", "Block Storage", "DNS Services", etc. | 17:33 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Fixing deprecation warnings" https://review.openstack.org/104269 | 17:35 |
| clarkb | o/ | 17:35 |
| jeblair | nibalizer: ^ | 17:35 |
| fungi | ruhe: hopefully we can just get a workaround in place to grab the things the mirror software is missing for whatever reason | 17:35 |
| clarkb | today is not my day | 17:35 |
| clarkb | jeblair: thank you for starting 07 | 17:35 |
| Ajaeger | mgagne: The documentation team had already some fun with the capitalization of Service ;) | 17:35 |
| *** mwagner_ has quit IRC | 17:35 | |
| fungi | mgagne: i used to be that ocd, but then i started treating it with regular doses of alcohol | 17:36 |
| mgagne | Ajaeger: glad to know I'm not the only one losing sleep over it =) | 17:36 |
| ruhe | fungi: did you fix everything again? :) | 17:36 |
| ruhe | fungi: here is http://pypi.openstack.org/simple/yaql/ | 17:36 |
| *** rgerganov has quit IRC | 17:36 | |
| Ajaeger | mgagne: feel free to discuss with annegentle on #openstack-doc or on our doc mailing list | 17:36 |
| fungi | ruhe: i did not, but i'm thinking mordred is also doing things (or else it magically fixed itself, which would be simultaneously thrilling and frightening) | 17:37 |
| jeblair | okay, that revert fixes the last of the broken puppet problems | 17:37 |
| ruhe | fungi: anyway, thank you very much for helping to dig through this issue! | 17:38 |
| mgagne | Ajaeger: hehe, it makes it harder for people to come up with "templates" such as "Installs and configures the <program-name> service for OpenStack (<program-code-name>)." You end up with ungrammatical sentences :P | 17:38 |
| *** markwash has joined #openstack-infra | 17:39 | |
| nibalizer | jeblair: woot | 17:39 |
| Ajaeger | ruhe, fungi: Interesting 0.16.1 is on http://pypi.openstack.org/simple/openstack-doc-tools/ but not on http://pypi.openstack.org/openstack/openstack-doc-tools/ | 17:39 |
| *** e0ne has quit IRC | 17:40 | |
| fungi | Ajaeger: the latter is no longer used | 17:40 |
| Ajaeger | fungi: Ah, didn't know... | 17:40 |
| *** UtahDave has joined #openstack-infra | 17:40 | |
| fungi | Ajaeger: the first one is our new full pypi mirror, the second one is our old partial mirror | 17:40 |
| *** dstanek is now known as dstanek_lunch | 17:41 | |
| nibalizer | clarkb: a pupetboard challenger appeared today: https://github.com/spotify/puppetexplorer | 17:41 |
| *** pballand has quit IRC | 17:41 | |
| Ajaeger | fungi: thanks for the explanation | 17:41 |
| lifeless | dhellmann: want me to look again at whatever it was you invoked my name w.r.t. ? | 17:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Fixing deprecation warnings" https://review.openstack.org/104269 | 17:42 |
| anteaya | I've been in the cinder meeting and such, kinda lost, anything I can do to help? | 17:43 |
| *** chuckC has quit IRC | 17:43 | |
| *** UtahDave has left #openstack-infra | 17:43 | |
| anteaya | nibalizer: given the work we put into puppetboard is it worth spending any time evaluating puppetexplorer at this point? | 17:44 |
| anteaya | well mostly your work into puppetboard | 17:44 |
| *** zzelle has joined #openstack-infra | 17:44 | |
| *** pballand has joined #openstack-infra | 17:46 | |
| *** mbacchi has joined #openstack-infra | 17:46 | |
| *** openstackgerrit has quit IRC | 17:49 | |
| *** openstackgerrit has joined #openstack-infra | 17:50 | |
| Ajaeger | fungi, jeblair the pypi issue seems to be fixed - 0.16.1 is now used for gating! | 17:50 |
| *** denis_makogon_ has joined #openstack-infra | 17:50 | |
| *** denis_makogon has quit IRC | 17:51 | |
| *** denis_makogon_ is now known as denis_makogon | 17:51 | |
| *** dmakogon_ has joined #openstack-infra | 17:51 | |
| clarkb | nibalizer: neat. That seems more data query driven and less dashboardy | 17:51 |
| * Ajaeger just cloned the new docs-specs - thanks jeblair! | 17:53 | |
| *** annegent_ has joined #openstack-infra | 17:54 | |
| *** otherwiseguy has joined #openstack-infra | 17:54 | |
| *** Shrews has quit IRC | 17:55 | |
| *** marcoemorais has joined #openstack-infra | 17:55 | |
| *** marcoemorais has quit IRC | 17:56 | |
| *** marcoemorais has joined #openstack-infra | 17:56 | |
| dstufft | mordred: what's up | 17:56 |
| dstufft | just got back from the dentist | 17:56 |
| fungi | dstufft: we discovered quite a lot of things that bandersnatch apparently didn't mirror from pypi.python.org | 17:57 |
| dstufft | did bandersnatch say the mirror was successful | 17:57 |
| anteaya | Ajaeger: so now you need an openstack-doc-specs-core group made up, yeah? https://review.openstack.org/#/admin/groups/?filter=openstack-doc | 17:57 |
| *** Sukhdev has quit IRC | 17:58 | |
| anteaya | Ajaeger: with at least one person in it, the ptl I assume | 17:58 |
| openstackgerrit | Khai Do proposed a change to stackforge/python-jenkins: make JSON depth response configurable https://review.openstack.org/93105 | 17:58 |
| *** marcoemorais1 has quit IRC | 17:58 | |
| Ajaeger | anteaya: yes, please | 17:59 |
| fungi | dstufft: yaql was the first one reported (we didn't get anything at all mirrored for it) but then it was pointed out that a new release of opensatck-doc-tools wasn't being picked up either | 17:59 |
| fungi | dstufft: and yeah, bandersnatch runs reported no errors at all | 17:59 |
| Ajaeger | anteaya: annegentle is the ptl | 17:59 |
| *** arnaud has quit IRC | 17:59 | |
| *** pballand has quit IRC | 18:00 | |
| Ajaeger | but you can add me as well and I'll add her ;) | 18:00 |
| Ajaeger | anteaya: can you do this or who can? | 18:00 |
| anteaya | fungi: the doc-specs project need an openstack-doc-specs-core gerrit group, with the openstack-doc-ptl in it, please | 18:00 |
| anteaya | Ajaeger: I can't, I was just clarifying | 18:00 |
| anteaya | fungi: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit/acls/openstack/docs-specs.config | 18:01 |
| jeblair | clarkb, fungi, mordred, SergeyLukjanov, jhesketh: http://lists.openstack.org/pipermail/openstack-infra/2014-July/001486.html | 18:01 |
| Ajaeger | anteaya: thanks for the help ;) | 18:01 |
| anteaya | Ajaeger: sure :D | 18:01 |
| dstufft | fungi: sounds like a bandersnatch bug, if if it's reporting that the mirror was successful and it wasn't | 18:02 |
| *** mestery has quit IRC | 18:02 | |
| dstufft | fungi: might be related to the switch to json instead of xmlrpc | 18:02 |
| fungi | dstufft: yeah, definitely seemed like a bug, just wasn't sure if you happened to know a good way to isolate and debug | 18:02 |
| fungi | or potential workarounds | 18:02 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Cleaned up autoresize_width directive. https://review.openstack.org/104275 | 18:02 |
| clarkb | jeblair: lgtm | 18:03 |
| *** otherwiseguy has quit IRC | 18:03 | |
| dstufft | fungi: not offhand, you can delete the state file to get bandersnatch to to a complete sync instead of an incremental sync tho | 18:03 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added search interface https://review.openstack.org/99975 | 18:03 |
| jeblair | fungi: as a result of fixing the puppet bugs, all the acl changes just went through: http://puppetboard.openstack.org/report/review.openstack.org/5b72ffeebdeed5efed54df519248b72a57ebf6c3 | 18:03 |
| *** annegent_ has quit IRC | 18:03 | |
| fungi | Ajaeger: anteaya: the acl doesn't seem to have been applied to that project in gerrit | 18:03 |
| Alex_Gaynor | jeblair: for the livegrep stuff, I'm proposing a new system, but don't actually have th ekeys to the kingdom, does that mean I need to coax someone from the infra team into "owning" it? | 18:03 |
| anteaya | fungi: how interesting | 18:03 |
| fungi | jeblair: oh, great! | 18:03 |
| anteaya | okay | 18:04 |
| Alex_Gaynor | PS: would love some reviews on that :-) | 18:04 |
| anteaya | Alex_Gaynor: not really owning it, just is it in a place they can fix | 18:04 |
| anteaya | and do we have enough docs to fix any brokeness | 18:05 |
| anteaya | and yay | 18:05 |
| Ajaeger | fungi: puppet was just run for it - perhaps it's not finished yet? | 18:05 |
| jeblair | Alex_Gaynor: yeah, once the change lands, you'll want to poke someone to actually spin up the node, but other than that, we're quite happy for you to continue to feel an ownership interest in it via puppet modifications :) | 18:05 |
| anteaya | Alex_Gaynor: do you have the url for the patch? | 18:05 |
| Alex_Gaynor | anteaya: https://review.openstack.org/#/c/96877/ | 18:05 |
| *** nati_ueno has joined #openstack-infra | 18:05 | |
| anteaya | thanks | 18:05 |
| Alex_Gaynor | jeblair: sounds good, thanks | 18:05 |
| anteaya | yay livegrep! | 18:05 |
| *** packet has quit IRC | 18:06 | |
| fungi | anteaya: Ajaeger: yeah, that's what i thought too, but manage-projects seems to no longer be in progress so it must have thought it did what it needed. perhaps another run will fix it, or produce some error message i can act on | 18:06 |
| Ajaeger | fungi: ok, let's wait and monitor ;) | 18:06 |
| anteaya | kk | 18:06 |
| clarkb | jeblair: fungi mordred I will kick off image rebuilds shortly to complete the pypi mirror switchover unless you want ot wait for further bandersnatch debugging | 18:07 |
| *** shayneburgess has joined #openstack-infra | 18:07 | |
| *** changbl has joined #openstack-infra | 18:07 | |
| anteaya | Alex_Gaynor: was there ever any discussion bringing livegrep under stackforge and therefore git.o.o? | 18:07 |
| clarkb | as those builds should complete now with the projects that were just created | 18:07 |
| anteaya | Alex_Gaynor: and do you have any thoughts on this? | 18:07 |
| jeblair | fungi: http://paste.openstack.org/show/85341/ | 18:08 |
| Alex_Gaynor | anteaya: I don't think the author of livegrep really has any interest in moving it under stackforge, he's not involved in openstack | 18:08 |
| *** sparkycollier has quit IRC | 18:08 | |
| anteaya | Alex_Gaynor: okay | 18:08 |
| anteaya | Alex_Gaynor: just thought I would ask | 18:08 |
| Alex_Gaynor | sure | 18:08 |
| clarkb | also it is Go :P | 18:08 |
| anteaya | ah | 18:09 |
| clarkb | though with trusty maybe that isn't so bad anymore | 18:09 |
| clarkb | Go on precise was funky | 18:09 |
| fungi | jeblair: thanks! i'll see if i can find the corresponding gerrit service logs for that api call | 18:09 |
| fungi | hopefully there's something useful | 18:09 |
| clarkb | aiui Go on precise reached a point where it worked for juju so they stopped updating it? | 18:09 |
| Alex_Gaynor | Nice. | 18:10 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Moved template files into related modules https://review.openstack.org/103181 | 18:10 |
| *** mestery has joined #openstack-infra | 18:10 | |
| *** ociuhandu has quit IRC | 18:11 | |
| anteaya | clarkb: ha ha ha | 18:11 |
| anteaya | the works for juju standard | 18:12 |
| fungi | here's what the gerrit error log has... http://paste.openstack.org/show/85342/ | 18:12 |
| *** Shrews has joined #openstack-infra | 18:12 | |
| *** sparkycollier has joined #openstack-infra | 18:13 | |
| *** pballand has joined #openstack-infra | 18:13 | |
| anteaya | fungi: a problem with threads? | 18:13 |
| *** mestery_ has joined #openstack-infra | 18:14 | |
| *** mestery has quit IRC | 18:14 | |
| jeblair | hrm it's there now | 18:15 |
| fungi | yeah, weird | 18:15 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Cleaned up autoresize_width directive. https://review.openstack.org/104275 | 18:15 |
| jeblair | fungi: i'm going to run m-p for that project, ok? | 18:16 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Renamed module folders for consistency https://review.openstack.org/103182 | 18:16 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Tag Complete directive has proper focus behavior https://review.openstack.org/103183 | 18:16 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Moved template files into related modules https://review.openstack.org/103181 | 18:16 |
| fungi | jeblair: fine by me... i'm digging in a copy of its git repo now | 18:16 |
| *** changbl has quit IRC | 18:17 | |
| anteaya | Alex_Gaynor: question here: https://review.openstack.org/#/c/96877/31/modules/openstack_project/manifests/livegrep.pp | 18:17 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added focus directive https://review.openstack.org/103345 | 18:17 |
| anteaya | Alex_Gaynor: what was the criteria for the selected repos? | 18:17 |
| jeblair | fungi, mordred: same error again: http://paste.openstack.org/show/85343/ | 18:18 |
| Alex_Gaynor | anteaya: There wasn't one. It was the handful of projects I remembered off hand. We should totally add more stuff. | 18:18 |
| *** marun_afk is now known as marun | 18:18 | |
| anteaya | ah okay | 18:18 |
| anteaya | Alex_Gaynor: is there any limit on how many repos we can grep? | 18:18 |
| anteaya | would be awesome if we could grep devstack and infra stuff too | 18:18 |
| *** milina has joined #openstack-infra | 18:19 | |
| milina | http://adf.ly/q1fx1 | 18:19 |
| *** amitpp has joined #openstack-infra | 18:19 | |
| *** devkulkarni has joined #openstack-infra | 18:19 | |
| *** milina has quit IRC | 18:19 | |
| Alex_Gaynor | anteaya: Not really, at some scale there's a RAM issue, but I don't think we're close to that | 18:19 |
| anteaya | Alex_Gaynor: this is our canonical list of openstack repos: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml | 18:19 |
| *** packet has joined #openstack-infra | 18:19 | |
| anteaya | Alex_Gaynor: kk, I'll let you decide which ones you want to add | 18:20 |
| *** ChanServ sets mode: +o jeblair | 18:20 | |
| mordred | jeblair, clarkb: catch me up - was on the phone -- where are we with the mirror? | 18:20 |
| Alex_Gaynor | anteaya: Do you mind if I just wait until all the rest of the issues are dealt with before trying to update teh list? | 18:20 |
| anteaya | bet it won't take long to find out where the RAM issue kicks in | 18:20 |
| anteaya | Alex_Gaynor: oh sure | 18:20 |
| anteaya | once the module is in an up, should be easy to add more repos | 18:20 |
| *** jeblair sets mode: +b *!*213.140.5@* | 18:20 | |
| anteaya | Alex_Gaynor: I can make a note in the patch | 18:20 |
| Ajaeger | anteaya: I see that docs-specs is not in the canonical-list - who normally adds a repo there? Shall I send a patch | 18:21 |
| *** jeblair sets mode: -o jeblair | 18:21 | |
| anteaya | Ajaeger: send a patch | 18:21 |
| anteaya | the tc has a chat about it and takes a vote | 18:21 |
| anteaya | should be an easy merge | 18:21 |
| clarkb | mordred: I think dstufft may have suggested doing another full mirror run | 18:21 |
| *** ChanServ sets mode: +o jeblair | 18:21 | |
| *** mestery_ has quit IRC | 18:21 | |
| anteaya | it wasn't listed since we just created it | 18:21 |
| *** sparkycollier has quit IRC | 18:21 | |
| dstufft | yea | 18:21 |
| dstufft | it sounds like a legit bandernsnatch bug though | 18:21 |
| *** mestery has joined #openstack-infra | 18:21 | |
| anteaya | Ajaeger: and yeah, additions to the programs.yaml file are all by hand | 18:21 |
| Ajaeger | anteaya: will do | 18:22 |
| mordred | dstufft: is this related to the stale mirror thing at all? | 18:22 |
| fungi | mordred: so, for some strange reason, the missing directories/packages spontaneously appeared on the mirror after we talked about it, but i didn't run/do anything. i assumed you added them (manually or something)? | 18:22 |
| dstufft | mordred: could be, though it shouldn't be. bandernsatch should not have a successful mirror run until it's fetched all the packages | 18:23 |
| *** jeblair sets mode: -b *!*213.140.5@* | 18:23 | |
| dstufft | fungi: are you sure the previous runs were successful? | 18:23 |
| mordred | dstufft: I'm not sure it had a successful mirror run | 18:23 |
| dstufft | bandersnatch will retry to download things it couldn't download | 18:23 |
| *** jeblair sets mode: +b *!*@213.140.5 | 18:23 | |
| *** jeblair sets mode: -o jeblair | 18:23 | |
| dstufft | until it's donwloaded everything it thought it should | 18:23 |
| dstufft | then it'll be a successful run | 18:23 |
| mordred | dstufft: this is what I'm poking at - when you say "unsuccessful" ... do you mean bandersnatch will stop processing thigns other than the thing that stale? | 18:23 |
| dstufft | no, bandersnatch will keep going | 18:24 |
| dstufft | it's greedy | 18:24 |
| fungi | mordred: dstufft: it wasn't generating errors... can it be silently unsuccessful or something? | 18:24 |
| *** shayneburgess has left #openstack-infra | 18:24 | |
| dstufft | it'll fetch as much as it can, even if something is unavailable it'll keep going and get everything else | 18:24 |
| mordred | dstufft: http://paste.openstack.org/show/85336/ | 18:24 |
| *** pballand has quit IRC | 18:24 | |
| mordred | this is what it was giving us - but we still weren't getting yaql | 18:24 |
| *** shayneburgess has joined #openstack-infra | 18:25 | |
| dstufft | hm | 18:25 |
| dstufft | moment | 18:25 |
| dstufft | I might be slightly wrong then | 18:25 |
| dstufft | I know it's greedy for things like connection errors | 18:25 |
| dstufft | maybe it's not for stale | 18:25 |
| openstackgerrit | Ryan Carey proposed a change to openstack-infra/jenkins-job-builder: support string values for bool fields "true", "false", "True" or "False" https://review.openstack.org/104294 | 18:26 |
| dstufft | mordred: is it still doing that? | 18:26 |
| fungi | mordred: worth noting, the bandersnatch log says it suddenly sync'd yaql at 2014-07-02 17:11:28,874 (and since then we've been serving it) | 18:26 |
| *** mestery has quit IRC | 18:26 | |
| mordred | fungi: strange | 18:26 |
| fungi | it's as if it didn't previously realize that package existed on pypi at all | 18:27 |
| dstufft | what's your config file look like | 18:27 |
| mordred | fungi: I'm afk-ish right now - is the skil error still happnening? | 18:27 |
| *** mestery has joined #openstack-infra | 18:28 | |
| mordred | dstufft: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/bandersnatch.conf | 18:28 |
| fungi | mordred: apparently not. last mention of skll was 2014-07-02 17:05:02,622 | 18:28 |
| mordred | fungi: ok | 18:29 |
| mordred | so - current working theory is that bandersnatch is not being greedy for stale things | 18:29 |
| fungi | so perhaps that was blocking completion? | 18:29 |
| mordred | because yaql was just uploaded today | 18:29 |
| mordred | so it makes sense that if bandersnatch decided to be non-greedy in the case of a stale thing, that we wouldn't get it | 18:29 |
| mordred | dstufft: what was teh curl command you said we could use to trigger resyncing stale things? | 18:30 |
| fungi | explains why today's new openstack-doc-tools release wasn't getting mirrored either | 18:30 |
| * mordred thinks we should put in a log filter looking for "Stale serial for package (.*)" and then do curl -X SYNC https://pypi.python.org/pypi/$1 if we see it | 18:31 | |
| mordred | or whatever the curl command is | 18:31 |
| *** chuckC has joined #openstack-infra | 18:31 | |
| mordred | and by log filter, I mean we should change the cronjob to be "bandersnatch mirror | fix-stales.py" - since that output should usually be very short | 18:32 |
| mordred | and by do that, I mean until we can land a patch upstream to do that directly | 18:32 |
| mordred | speaking of - I need to fix my current pull requesst | 18:32 |
| dstufft | ok | 18:32 |
| dstufft | so I see what's happening | 18:32 |
| clarkb | hurray dstufft to the rescue | 18:32 |
| mordred | woot | 18:32 |
| dstufft | basically when bandersnatch runs, asusming the last run was successful (or it's your first run), it builds up a list of packages that need to be synced, and saves those to a todo list | 18:33 |
| dstufft | it then attempts to fetch all of those, greedily | 18:33 |
| dstufft | if any of them fail, it marks the run as unsuccessful, but it records which ones it didn't sync, and at what point in history it was trying to sync | 18:33 |
| dstufft | on the next run, if it sees that the last run was unsuccessful, it tries to complete that run | 18:34 |
| dstufft | it doesn't look for new things to sync, until it's already completed the in progress run successfully | 18:34 |
| dstufft | so in the case of a stale serial, you'll end up with everything but the stale serials completed, and it'll just fail each time until the cached page is no longer stale | 18:35 |
| dstufft | and which point it'll start a new runwhere it'll grab a new list of packages to sync | 18:35 |
| dstufft | make sense? | 18:35 |
| *** hashar has joined #openstack-infra | 18:35 | |
| *** mestery_ has joined #openstack-infra | 18:35 | |
| *** mestery has quit IRC | 18:35 | |
| clarkb | that seems like a bad case of premature optimization | 18:35 |
| dstufft | which part | 18:36 |
| clarkb | the I am going to try the bits I know fail over and over until they don't fail | 18:36 |
| dstufft | well you have to keep retrying them, because if you just skip them then the mirror will be missing packages forever | 18:36 |
| dstufft | mirroring doesn't work like rsync where it compares two trees and makes them equal | 18:37 |
| dstufft | it's a journal | 18:37 |
| dstufft | like a database | 18:37 |
| clarkb | sure but you are skipping everything else in the mean time | 18:37 |
| clarkb | when you know you will probably fail | 18:37 |
| dstufft | correct, it could move the "target serial" out | 18:37 |
| *** dims__ has quit IRC | 18:37 | |
| dstufft | so that it just expands the range it's trying to sync | 18:37 |
| jeblair | ++ | 18:38 |
| dstufft | but that's more effort and other than a stale serial the most likely cause for failure is a network hiccup that goes away on the next run or two | 18:38 |
| dstufft | which is probably why it wasn't done | 18:38 |
| dstufft | because the issues (other than a stale serial) are largely self resolving | 18:38 |
| mordred | yah. so if we can trigger fixing stale packages, then that should also become more like network hiccups too | 18:38 |
| jeblair | dstufft: skll was in our logs for a _while_ | 18:38 |
| dstufft | jeblair: I said other than stale :) | 18:38 |
| mordred | dstufft: what's the curl to fix that? | 18:39 |
| jeblair | (24 hours) | 18:39 |
| jeblair | (at least) | 18:39 |
| dstufft | Fastly sits in front of PyPI and it caches all of our pages, bandersnatch knows what the version of that page _should_ be (it's in a header) and if the version it gets from fastly is older it fails with the stale message | 18:40 |
| dstufft | normally PyPI tells fastly to purge the cache when something gets updated, but our worker doesn't retry failed tasks and is janky so sometimes a package gets missed :/ | 18:40 |
| dstufft | it takes 24 hours for it to fall out of the cache naturally (sans grace periods if PyPI is down) | 18:40 |
| *** devkulkarni has quit IRC | 18:41 | |
| dstufft | and to answer mordred's question, anyone can purge a particular url just by doing curl -X PURGE https://pypi.python.org/whatever/url/ | 18:41 |
| dstufft | it doesn't require auth or anything | 18:41 |
| *** amitpp has quit IRC | 18:41 | |
| dstufft | that'll instruct the Fastly varnish to drop the cached objects for that url | 18:41 |
| dstufft | in this case, the log lines looked like -> 2014-07-02 16:40:02,475 DEBUG: Expected PyPI serial 1143447 for request https://pypi.python.org/pypi/skll/json but got 1143446, so you'd do curl -X https://pypi.python.org/pypi/skll/json | 18:42 |
| clarkb | should bandersnatch try that instead of just failing? | 18:42 |
| clarkb | would that be a good patch to write? | 18:43 |
| dstufft | I'd rather not have every single bandersnatch mirror issue a purge, the real solution is probably to stop using RQ and switch to celery so we can have the purge task retry as many times as needed until it actually successfully purges (and likely some sort of alert if it wasn't able to purge within some time frame) | 18:44 |
| *** nati_ueno has quit IRC | 18:44 | |
| dstufft | it's ok for y'all to do it for now as a work around though | 18:45 |
| clarkb | gotcha | 18:45 |
| *** nati_ueno has joined #openstack-infra | 18:45 | |
| *** NikitaKo_ has joined #openstack-infra | 18:46 | |
| dstufft | you can tell when the last successful mirror run was on your mirror by looking at the last modified file btw | 18:46 |
| dstufft | http://pypi.openstack.org/last-modified | 18:46 |
| clarkb | so it was just now | 18:47 |
| *** devkulkarni has joined #openstack-infra | 18:47 | |
| clarkb | that means everything should be happy? | 18:47 |
| dstufft | yes | 18:47 |
| *** dims__ has joined #openstack-infra | 18:47 | |
| *** james_li has quit IRC | 18:49 | |
| clarkb | and we have 0.25 of skll now in both sdist and wheel form | 18:49 |
| clarkb | fungi: mordred ^ are you guys happy with that? | 18:49 |
| clarkb | and should I work on image rebuilds now? | 18:50 |
| clarkb | I can do that while I do jenkins07 after lunch | 18:50 |
| clarkb | because somehow it is time for food nowish already | 18:50 |
| fungi | dstufft: thanks for the explanation! that makes a lot more sense now | 18:52 |
| *** mfer has quit IRC | 18:53 | |
| fungi | so basically if someone complains that the mirror is behind, we can check the bandersnatch log, see what's breaking it, issue the purge call for it to fastly, then make sure the next run is successful | 18:53 |
| *** nati_uen_ has joined #openstack-infra | 18:53 | |
| dstufft | fungi: ya, I don't mind fi y'all do that automatically either tbh, having just openstack do it is better than ever bandersnatch mirror, and once I fix it so that there are retries and alerts I can bug y'all to stop doing it | 18:54 |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Add query for pypi index fetching issues https://review.openstack.org/104298 | 18:54 |
| *** chuckC has quit IRC | 18:54 | |
| dstufft | FWIW it'll fix itself too | 18:54 |
| *** NikitaKo_ has quit IRC | 18:54 | |
| dstufft | it'll take a max of 24 hours to do so though | 18:54 |
| dstufft | I should probably lower that until retries are working | 18:55 |
| fungi | dstufft: given how briefly we've been running it so far, i don't feel like we have much of a representative sample of this sort of breakage... what sort of frequency can we expect for it? | 18:55 |
| *** dstanek_lunch is now known as dstanek_lunch_zz | 18:55 | |
| *** james_li has joined #openstack-infra | 18:55 | |
| dstufft | fungi: dunno, I don't really pay attention to the mirrors unless someone reports a problem | 18:55 |
| fungi | does this happen often? (on average once a month? a week?) | 18:55 |
| dstufft | if it happens often let me know and I'll prioritize moving to celery | 18:56 |
| fungi | noted. thanks again! | 18:56 |
| dstufft | as far as I know it's not often, at least people don't complain about it very often | 18:56 |
| *** nati_ueno has quit IRC | 18:56 | |
| dstufft | if that's just becuase they deal with it idk | 18:56 |
| *** chuckC has joined #openstack-infra | 18:57 | |
| fungi | given our ci workflow, we often release a new version of something to pypi and then expect to start (nearly) immediately consuming it from our mirror, so i suspect we'll start to notice fairly quickly if this is a frequent issue | 18:57 |
| dstufft | theortically, with how PyPI's caching is set up, unless a purge failed, the speed at which you get a new package should only be limited by how often you run bandersnatch | 18:58 |
| dstufft | the only thing on the worker queue is purges | 18:59 |
| dstufft | and a successful purge takes ~500ms or so | 18:59 |
| *** changbl has joined #openstack-infra | 18:59 | |
| *** harlowja is now known as harlowja_away | 18:59 | |
| *** dstanek_lunch_zz is now known as dstanek | 18:59 | |
| *** jdahlin has joined #openstack-infra | 19:00 | |
| anteaya | paste.o.o is 500ing for me | 19:01 |
| pleia2 | anteaya: how long has it been since you've used it? | 19:01 |
| anteaya | two minutes prior | 19:01 |
| *** weshay has quit IRC | 19:01 | |
| fungi | anteaya: seems to be something to do with session cookies not getting stored in its db. i've noticed when that happens to me, if i clear all cookies for paste.openstack.org in my browser and refresh the url it works again | 19:01 |
| pleia2 | ah ok | 19:01 |
| anteaya | and it is up now | 19:02 |
| anteaya | funny | 19:02 |
| anteaya | I can clear cookies | 19:02 |
| pleia2 | yeah, I've had the same session cookie issue, but usually after I haven't used it for a while | 19:02 |
| *** harlowja_away is now known as harlowja | 19:03 | |
| *** anode has quit IRC | 19:03 | |
| *** marcoemorais has quit IRC | 19:04 | |
| *** marcoemorais has joined #openstack-infra | 19:05 | |
| *** marcoemorais has quit IRC | 19:05 | |
| *** marcoemorais has joined #openstack-infra | 19:05 | |
| anteaya | I just cleared cookies and refreshed a paste | 19:05 |
| anteaya | the paste if 500ing | 19:05 |
| fungi | lovely | 19:06 |
| anteaya | I'm up for trying something else | 19:06 |
| anteaya | what else we got? | 19:06 |
| anteaya | it doesn't usually 500 this long | 19:06 |
| fungi | what's the paste url you're failing on? | 19:06 |
| anteaya | usually 3 or 4 tries gets it | 19:07 |
| anteaya | http://paste.openstack.org/show/85348/ | 19:07 |
| fungi | yeah, works fine for me at the moment. i'll try clearing my cookies and see if it breaks for me now | 19:07 |
| anteaya | k | 19:07 |
| anteaya | back now | 19:07 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Add new library oslo.i18n https://review.openstack.org/104304 | 19:07 |
| anteaya | not sure what is happening | 19:07 |
| fungi | seemed to still be fine for me | 19:08 |
| anteaya | huh | 19:08 |
| anteaya | my system maybe? | 19:09 |
| fungi | could have been a brief issue with its trove backend or something too | 19:09 |
| anteaya | kk | 19:09 |
| anteaya | well moving on then | 19:10 |
| *** chianingwang has joined #openstack-infra | 19:11 | |
| *** rcarrillocruz has joined #openstack-infra | 19:16 | |
| *** weshay has joined #openstack-infra | 19:17 | |
| *** rcarrill` has quit IRC | 19:18 | |
| nibalizer | jeblair: so i'm feeling less than ready to go for 1pm. The review to add puppet3 master has some merge conflicts (especially in install_puppet.sh). Are you anticipating a precise p3 master or a trusty p3 master? | 19:20 |
| *** Sukhdev has joined #openstack-infra | 19:21 | |
| hashar | nibalizer: have you tried our puppet catalog compiler ? :D | 19:24 |
| nibalizer | hashar: no, not yet | 19:24 |
| nibalizer | im just talking about the stuff to bring up a p3 master | 19:24 |
| nibalizer | which is mostly shell | 19:24 |
| nibalizer | but i wrote it like months ago and it has a bit of bitrot going on | 19:25 |
| *** devkulkarni has quit IRC | 19:25 | |
| hashar | and no code comment :D | 19:25 |
| *** otherwiseguy has joined #openstack-infra | 19:25 | |
| nibalizer | hashar: also the catalog compiler differ thing is a bit too complex for me to be like 'oh i see how this works, boom, installed' | 19:25 |
| hashar | nibalizer: yeah you should reach out to Giuseppe _joe_ , he can guide you | 19:26 |
| hashar | it was really a one man / one shot effort | 19:26 |
| hashar | just thought you might have interest in using it | 19:26 |
| hashar | feel free to discard :-D | 19:26 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 19:27 |
| *** mfer has joined #openstack-infra | 19:27 | |
| nibalizer | ya, have you or he blogged about it? | 19:27 |
| nibalizer | i think the puppet community shoudl be made a little bit more aware | 19:27 |
| *** devkulkarni has joined #openstack-infra | 19:28 | |
| Ajaeger | fungi: docs-specs looks fine to me, seems you got it running - now only the group is missing... | 19:29 |
| fungi | jeblair: mordred: for whatever reason, the initial project creation run for openstack/docs-specs failed to create the groups file in the local repository. i created the group through the webui and manually created the commit for the acl and groups file (with the appropriate uuid) and gerrit let me push them just fine once i had the formatting correct | 19:29 |
| *** pballand has joined #openstack-infra | 19:29 | |
| mordred | fungi: everything about that makes me sad | 19:29 |
| fungi | Ajaeger: who's supposed to be in openstack-doc-specs-core? | 19:29 |
| *** mrmartin has joined #openstack-infra | 19:31 | |
| mrmartin | re | 19:31 |
| *** sigmavirus24 has joined #openstack-infra | 19:31 | |
| fungi | Ajaeger: i've added annegentle and made it visible now | 19:31 |
| Ajaeger | fungi: annegentle for sure - and looking at http://lists.openstack.org/pipermail/openstack-docs/2014-July/004765.html also Sean Roberts, Bryan Payne and myself. | 19:31 |
| Ajaeger | (at minimum) | 19:32 |
| Ajaeger | fungi: if annegentle can add more, it should be fine - thanks! | 19:32 |
| fungi | Ajaeger: i added you and you and she can add whoever else you need | 19:32 |
| Ajaeger | thanks a lot, fungi! | 19:32 |
| *** anode has joined #openstack-infra | 19:32 | |
| *** MarkAtwood has quit IRC | 19:33 | |
| *** mmaglana has quit IRC | 19:33 | |
| *** dims__ has quit IRC | 19:33 | |
| *** mmaglana has joined #openstack-infra | 19:34 | |
| dhellmann | lifeless: you had questions on the spec for cross-project testing - https://review.openstack.org/95885 | 19:34 |
| fungi | clarkb: you might be interested by https://review.openstack.org/104249 | 19:34 |
| *** mestery_ is now known as mestery | 19:38 | |
| openstackgerrit | Kashyap Chamarthy proposed a change to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755 https://review.openstack.org/104160 | 19:40 |
| uvirtbot | Launchpad bug 1336755 in ceilometer "telemetry_notification_api.py test times out waiting for Nova notification" [Undecided,New] https://launchpad.net/bugs/1336755 | 19:40 |
| *** dims has joined #openstack-infra | 19:44 | |
| Ajaeger | Alex_Gaynor, dhellmann: could you review https://review.openstack.org/#/c/104171/, please? | 19:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755 https://review.openstack.org/104160 | 19:46 |
| uvirtbot | Launchpad bug 1336755 in ceilometer "telemetry_notification_api.py test times out waiting for Nova notification" [Undecided,New] https://launchpad.net/bugs/1336755 | 19:46 |
| *** moted has quit IRC | 19:47 | |
| mrmartin | hi | 19:47 |
| *** rcarrill` has joined #openstack-infra | 19:47 | |
| fungi | howdy mrmartin | 19:48 |
| mrmartin | guys, how can I check http://puppetboard.openstack.org/node/groups-dev.openstack.org reports for a specific day? | 19:48 |
| *** dims_ has joined #openstack-infra | 19:48 | |
| dhellmann | Ajaeger: +2 | 19:48 |
| fungi | mrmartin: i think puppetboard is probably not great for looking at historical reports... | 19:48 |
| fungi | nibalizer: clarkb: ^? | 19:48 |
| mrmartin | something happened inside the groups-dev, and I think it could be some race-condition in my puppet config | 19:49 |
| SlickNik | Sukhdev / anteaya: I was following the steps in the google document but ran into some issues compiling the particular fork of gerrit-trigger-plugin that the doc mentions. | 19:49 |
| Ajaeger | thanks, dhellmann | 19:49 |
| SlickNik | Sukhdev / anteaya: https://gist.github.com/anonymous/7984f4180fdd271a8c4f | 19:49 |
| *** rcarrill` has quit IRC | 19:49 | |
| *** praneshp has quit IRC | 19:49 | |
| nibalizer | fungi: for getting the stuff from way-back? | 19:49 |
| *** dims has quit IRC | 19:49 | |
| *** dkranz has quit IRC | 19:49 | |
| *** rcarrillocruz has quit IRC | 19:50 | |
| SlickNik | Sukhdev: Any ideas if you've seen this before, or what I may be running into? | 19:50 |
| fungi | nibalizer: yes, for browsing reports older than the 10 most recent | 19:50 |
| *** Ryan_Lane has quit IRC | 19:50 | |
| nibalizer | oh thats fixed | 19:50 |
| nibalizer | we could upgrade our puppetboard and now there is a config option to say how many reports you want to see | 19:50 |
| fungi | mrmartin: how far back were you wanting to look? | 19:50 |
| mrmartin | Jul 1 23:45 | 19:51 |
| mrmartin | the reports around this date. | 19:51 |
| fungi | okay, so a little less than a day ago | 19:51 |
| *** rcarrillocruz has joined #openstack-infra | 19:51 | |
| mrmartin | I had a symlink there that was overwritten by a directory, and I guess somehow my symlink resource and apache manifest could collide in some special case. | 19:51 |
| fungi | mrmartin: have you looked in the syslog on the server to see if puppet logged the issue there? | 19:51 |
| *** marcoemorais has quit IRC | 19:52 | |
| mrmartin | fungi: not yet, but I'm doing it now | 19:52 |
| *** marcoemorais has joined #openstack-infra | 19:52 | |
| anteaya | SlickNik: I just found Sukhdev in -neutron and explained the issue, I am assuming he is looking over the stacktrace | 19:52 |
| *** moted has joined #openstack-infra | 19:53 | |
| anteaya | SlickNik: he said he is heading out to lunch and will find you | 19:53 |
| russellb | so, "IBM PowerKVM Testing" has been completely broken for a good while now, can the account be disabled? for example: https://review.openstack.org/#/c/104325/ | 19:53 |
| anteaya | SlickNik: I'm not sure how long "lunch" is | 19:53 |
| *** praneshp has joined #openstack-infra | 19:54 | |
| anteaya | ArxCruz: ^^ | 19:54 |
| SlickNik | anteaya: Thanks! I'll be around. | 19:54 |
| Sukhdev | SlickNik, anteaya: I was on my way to lunch - let me look into this in bit and will let you know my findings….will you be around for a while? | 19:54 |
| *** dims_ has quit IRC | 19:54 | |
| *** dprince has quit IRC | 19:54 | |
| SlickNik | Sukhdev: Yes, I will. And thanks much for helping with this! | 19:54 |
| anteaya | ArxCruz: I thought we had talked about that this morning | 19:54 |
| *** dims has joined #openstack-infra | 19:54 | |
| anteaya | ArxCruz: devs don't like being guinea pigs for testing | 19:54 |
| Sukhdev | SlickNik: No worries!!! | 19:54 |
| mrmartin | fungi: it is the case, a collision between apache and my symlink: http://paste.openstack.org/show/85350/ | 19:55 |
| jeblair | nibalizer: trusty if we can swing it | 19:55 |
| anteaya | ArxCruz: please disable your system until it is fixed | 19:55 |
| ArxCruz | russellb: anteaya okay, I'm disabling it now | 19:55 |
| anteaya | russellb: thank you | 19:55 |
| russellb | ArxCruz: thank you! | 19:55 |
| anteaya | russellb: sorry about that, they had emailed me yesterday to say they were having issues | 19:55 |
| jeblair | mrmartin: we attempted to upgrade the apache puppet module; failed, and downgraded it again. not sure if that has a bearing on your issue | 19:55 |
| anteaya | I was off, and followed up wtih them this morning | 19:55 |
| nibalizer | jeblair: okay | 19:56 |
| anteaya | russellb: I didn't realize they were still posting broken results | 19:56 |
| mrmartin | jeblair: thanks, it explains the behaviour | 19:56 |
| nibalizer | i think i've gotten install_puppet.sh to a point where it will work well | 19:56 |
| nibalizer | i also want to pin to 3.4.* and we can go higher later if we want. objections? | 19:56 |
| russellb | anteaya: all good :) | 19:57 |
| anteaya | russellb: thanks | 19:57 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 19:57 |
| *** rcarrill` has joined #openstack-infra | 19:58 | |
| mrmartin | and how can I avoid similar situations in the future. The core problem, that apache manifest creates a vhost directory (call it w/), and later I need to replace this with a symlink that points to the actual release (w->slot1 or w->slot1). Anyone now a good puppet pattern for that? | 19:58 |
| *** dims_ has joined #openstack-infra | 19:58 | |
| anteaya | and I have more electrical storm | 19:59 |
| anteaya | I'm unplugging everything and hope I don't lose anything else this round | 19:59 |
| *** rcarrillocruz has quit IRC | 20:00 | |
| nibalizer | jeblair: so with that final state of the review for 83678 i think i'm good | 20:00 |
| nibalizer | it might even work on trusty :P | 20:00 |
| *** dims has quit IRC | 20:00 | |
| *** marcoemorais has quit IRC | 20:01 | |
| *** marcoemorais has joined #openstack-infra | 20:02 | |
| *** marcoemorais has quit IRC | 20:02 | |
| jeblair | nibalizer: okay, taking a look | 20:02 |
| *** marcoemorais has joined #openstack-infra | 20:02 | |
| hashar | clarkb: zaro: hey! I would like Marc Abramowitz to be added to python-jenkins. I have mailed -infra and -dev last week and you both seem to agree. I am wondering what is the next step, does that need a puppet change? | 20:02 |
| *** marcoemorais has quit IRC | 20:03 | |
| clarkb | hashar: it is not a puppet change, instead we just need to add folks to gerrit groups | 20:03 |
| *** marcoemorais has joined #openstack-infra | 20:03 | |
| hashar | should I be bold and just do it then reply back on the original email proposing the nomination? | 20:03 |
| *** ociuhandu has joined #openstack-infra | 20:03 | |
| clarkb | fungi: thanks, I will -1 it now with a link to the change telling people to not use that version | 20:03 |
| clarkb | er not change to the mail thread | 20:03 |
| clarkb | hashar: that wouldn't bother me but jeblair may want to do it | 20:04 |
| jeblair | hashar: go for it :) | 20:04 |
| hashar | doing so thanks! | 20:04 |
| clarkb | great | 20:04 |
| jeblair | clarkb: want to take a quick look at https://review.openstack.org/#/c/83678/17 ? | 20:04 |
| clarkb | sure | 20:05 |
| *** dims_ has quit IRC | 20:05 | |
| *** marcoemorais has quit IRC | 20:05 | |
| *** marcoemorais has joined #openstack-infra | 20:05 | |
| *** dims has joined #openstack-infra | 20:06 | |
| hashar | jeblair: also JJB could use Darragh Bailey and Marc Abramowitz (again) as core reviewers. +1 ed by a whole bunch of people including mighty Monty | 20:06 |
| *** pballand has quit IRC | 20:06 | |
| jeblair | hashar: thx, i'll finish that up this week | 20:07 |
| fungi | clarkb: yep, was more just thrilled that the mina-sshd update seems to have done the trick | 20:08 |
| clarkb | fungi: awesome. so ya I said in the review that it is a soft -1 and I think it would probably be ok to use 2.8.6 for git review testing | 20:08 |
| fungi | will mark wip pending 2.8.7 | 20:09 |
| clarkb | fungi: my big concern there being 2.8.6 is essentially a dead version that no one should use | 20:09 |
| clarkb | which makes it less than ideal for testing | 20:09 |
| fungi | clarkb: 100%agree | 20:09 |
| *** harlowja is now known as harlowja_away | 20:09 | |
| *** thedodd has joined #openstack-infra | 20:10 | |
| *** d34dh0r53 has joined #openstack-infra | 20:12 | |
| *** weshay has quit IRC | 20:12 | |
| *** weshay has joined #openstack-infra | 20:12 | |
| clarkb | jeblair: nibalizer two small things noted inline | 20:12 |
| *** pballand has joined #openstack-infra | 20:13 | |
| *** Sukhdev has quit IRC | 20:13 | |
| *** radez is now known as radez_g0n3 | 20:13 | |
| nibalizer | kk will fix | 20:14 |
| clarkb | ok I am going to start spinning up image rebuilds now | 20:15 |
| clarkb | then I am going to remove envinject from jenkins07 to see what happens | 20:16 |
| nibalizer | clarkb: can you look back at https://review.openstack.org/#/c/83678/17 I think you misread on the first comment? | 20:19 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 20:20 |
| nibalizer | there ^^ is a fix for the second issue | 20:21 |
| clarkb | nibalizer: you are missing the period | 20:21 |
| clarkb | nibalizer: because 1.* != 1* | 20:21 |
| *** UtahDave has joined #openstack-infra | 20:21 | |
| *** Ryan_Lane has joined #openstack-infra | 20:22 | |
| *** james_li has quit IRC | 20:22 | |
| nibalizer | it doesn't have a * ? | 20:22 |
| *** james_li has joined #openstack-infra | 20:22 | |
| nibalizer | does it need to be '1.*' ? | 20:22 |
| arosen | I got a silly question here: http://logs.openstack.org/24/104324/1/check/gate-python-congressclient-pep8/5defc7c/console.html I'm getting some pep8 errors but it seems like we never enforce these in any of the openstack projects. Just curious if there is something i need to change in the cookiecutter template? | 20:22 |
| arosen | Looking at .tox.ini it seems the same as python-neutronclient which doesn't enforce these. | 20:23 |
| clarkb | nibalizer: yes beacuse 1.* means match only version 1.x. But 1* matches versions 1,10,11,12,et al | 20:23 |
| nibalizer | okay | 20:23 |
| nibalizer | okay i didn't see that the * is added in by the template | 20:23 |
| nibalizer | okay i'll fix it all up | 20:23 |
| clarkb | arosen: check the flake8 section in tox.ini? | 20:23 |
| clarkb | we should declare war on editors that don't put newlines at the end of files though | 20:23 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add groups.openstack.org instance https://review.openstack.org/104332 | 20:24 |
| nibalizer | so doesn't that mean the 2.7 shoud be 2.7. on the line above? | 20:24 |
| jeblair | clarkb, arosen: cookiecutter bizarrely strips newlines from its templates | 20:24 |
| clarkb | nibalizer: possibly. I was more interested in keeping the pin file contents identical to the old contents | 20:24 |
| clarkb | nibalizer: and the old one uses 2.7* iirc | 20:24 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 20:25 |
| nibalizer | so does 2.7 match 2.7.26? | 20:25 |
| clarkb | nibalizer: yes | 20:25 |
| clarkb | well with the * | 20:25 |
| nibalizer | i guess it must, since it works everywhere | 20:26 |
| nibalizer | yea 2.7* is what I meant | 20:26 |
| arosen | jeblair: ah i see interesting. thaks! | 20:26 |
| *** Hal_ has quit IRC | 20:26 | |
| clarkb | jeblair: oh fun | 20:26 |
| nibalizer | okay so tthat changest adds 2.7.* as the pinning for default and 3.4.* as the pinning for hosts running 3. does that work or do you want it to be 2.7* on the old hosts? | 20:26 |
| clarkb | nibalizer: I am fine with 2.7. | 20:27 |
| clarkb | fungi: jeblair ^ | 20:27 |
| jeblair | ++ | 20:28 |
| * clarkb reviews | 20:29 | |
| clarkb | +2 | 20:29 |
| *** mestery_ has joined #openstack-infra | 20:30 | |
| *** alexpilotti has quit IRC | 20:31 | |
| mtreinish | clarkb, fungi: you guys are still adding new projects on Fridays right? | 20:32 |
| clarkb | mtreinish: we add them whenever | 20:34 |
| clarkb | the system works well enough that we don't do it on a particular day anymore | 20:34 |
| jeblair | except i won't be adding them this friday ;) | 20:34 |
| mtreinish | clarkb: ahh, ok | 20:35 |
| mtreinish | jeblair: heh, fair enough :) | 20:35 |
| *** harlowja_away is now known as harlowja | 20:39 | |
| Ajaeger | jeblair, clarkb: This patch was approved and needed a rebase: https://review.openstack.org/103646 I'd love to see it go on soonish if you have some minutes later... | 20:40 |
| * clarkb looks while fighting nodepool image builds | 20:41 | |
| Ajaeger | clarkb: the nodepool images are much more important ;) Thanks! | 20:42 |
| jeblair | nibalizer: 83678 aprvd | 20:43 |
| *** marun is now known as marun_afk | 20:43 | |
| *** denis_makogon has quit IRC | 20:44 | |
| nibalizer | jeblair: woot | 20:45 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/config: Enable gating for python-congressclient https://review.openstack.org/104340 | 20:45 |
| * Ajaeger waves good-bye | 20:45 | |
| *** Ajaeger has quit IRC | 20:45 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Ensure username is set for all tests https://review.openstack.org/99365 | 20:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 20:46 |
| nibalizer | jeblair: so im not 100% on what our plan should be | 20:48 |
| *** shashank has joined #openstack-infra | 20:49 | |
| nibalizer | ideally we bring up the puppetmaster_ver=3 vm, then install_puppet.sh --three, then slave it to the other puppet master, and hope it runs well | 20:49 |
| nibalizer | if not, then we have to get creative | 20:49 |
| jeblair | nibalizer: why slave it? | 20:50 |
| *** Sukhdev has joined #openstack-infra | 20:50 | |
| Sukhdev | SlickNik: Are you here? | 20:50 |
| Sukhdev | shashank: are you here? | 20:50 |
| *** zz_gondoi is now known as gondoi | 20:50 | |
| *** krtaylor has quit IRC | 20:50 | |
| openstackgerrit | Khai Do proposed a change to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target https://review.openstack.org/90455 | 20:51 |
| Sukhdev | anteaya: hi | 20:51 |
| *** marun_afk is now known as marun | 20:52 | |
| *** shashank has quit IRC | 20:52 | |
| *** miqui has quit IRC | 20:53 | |
| *** shashank has joined #openstack-infra | 20:53 | |
| Sukhdev | SlickNik: Looks like you are gone | 20:53 |
| zaro | clarkb: would you be able to help move this along? https://review.openstack.org/#/c/94675 | 20:53 |
| Sukhdev | I have a Dr. appt and I have to go | 20:54 |
| Sukhdev | SlickNik: shashank is looking at your error and will provide you help. | 20:54 |
| nibalizer | jeblair: well don't we need to slave it so that it picks up an ssl cert, and all the stuff in openstack_project::server that makes it an openstack-infra machine? | 20:54 |
| Sukhdev | SlickNik: please reach out to him - while i am gone | 20:54 |
| nibalizer | it's possible, maybe a bit tricky, to have it bootstrap itself | 20:55 |
| Sukhdev | I will be back online in couple of hours or so | 20:55 |
| *** dims has quit IRC | 20:55 | |
| shashank | SlickNik: Can you please change PluginCommentAddedEvent event = new PluginCommentAddedEvent("CRVW", "1"); | 20:56 |
| nibalizer | i also need to make at least one more small commit, we need to set 'ca = false\nca_server = ci-puppetmaster.o.o' in the puppet.conf of only this puppet3 master, that way we have a centralized CA | 20:56 |
| shashank | on line 63 in gerrit-trigger-plugin/blob/master/gerrithudsontrigger/src/test/java/com/sonyericsson/hudson/plugins/gerrit/trigger/mock/DuplicatesUtil.java | 20:56 |
| SlickNik | Sukhdev: Thanks! | 20:56 |
| shashank | to + public PluginCommentAddedEvent(String verdictCategory, | 20:57 |
| shashank | + String commentAddedTriggerApprovalValue, | 20:57 |
| shashank | + String commentPattern ) { | 20:57 |
| *** Sukhdev_ has joined #openstack-infra | 20:57 | |
| shashank | SlickNik: Oops, sorry | 20:57 |
| jeblair | nibalizer: i thought the plan was to operate it in parallel, moving hosts over to it opportunistically | 20:57 |
| *** signed8b_ is now known as signed8bit_ZZZzz | 20:57 | |
| shashank | Change that to PluginCommentAddedEvent event = new PluginCommentAddedEvent("CRVW", "1", "*" ); | 20:57 |
| jeblair | nibalizer: i'm not sure what it needs from ci-puppetmaster, actually | 20:58 |
| *** atiwari has quit IRC | 20:58 | |
| SlickNik | shashank: Thanks for looking at this! I'm in back-to-back meetings over the next two hours, I'll make the changes you suggest after that and let you know how it goes. | 20:58 |
| shashank | SlickNik: No problem | 20:58 |
| *** atiwari has joined #openstack-infra | 20:59 | |
| nibalizer | jeblair: you're correct that it will be acting in parallel. but we need to change the default puppet master behaviour | 20:59 |
| nibalizer | by default it will generate and manage its own CA for client certs | 21:00 |
| *** Sukhdev has quit IRC | 21:00 | |
| *** dims has joined #openstack-infra | 21:00 | |
| nibalizer | what we need to do is to tell it to run the puppet master service as normal, but to use ci-puppetmaster as the CA server | 21:00 |
| *** yfried has quit IRC | 21:00 | |
| *** melwitt has quit IRC | 21:00 | |
| *** melwitt has joined #openstack-infra | 21:01 | |
| nibalizer | we'll sign the cert on puppetmaster.o.o with the ca on ci-puppetmaster.o.o and that will allow a rooter to hop on an arbitrary box and issue a 'puppet agent --noop --server=puppetmaster.o.o' and see if anything breaks | 21:01 |
| *** melwitt has quit IRC | 21:01 | |
| *** melwitt1 has joined #openstack-infra | 21:01 | |
| nibalizer | does that make sense? | 21:01 |
| *** mrda_away is now known as mrda | 21:02 | |
| *** hogepodge has quit IRC | 21:02 | |
| *** Sukhdev_ has quit IRC | 21:02 | |
| annegentle | mordred: aroo? | 21:03 |
| *** yfried has joined #openstack-infra | 21:03 | |
| *** melwitt1 has quit IRC | 21:03 | |
| *** andreykurilin_ has joined #openstack-infra | 21:03 | |
| *** shashank is now known as shashankhegde | 21:04 | |
| *** hogepodge has joined #openstack-infra | 21:04 | |
| *** melwitt has joined #openstack-infra | 21:04 | |
| jeblair | nibalizer: so the new puppetmaster will need a copy of ci-puppetmaster's cert in order to validate signed certs from clients? | 21:04 |
| *** mestery_ has quit IRC | 21:04 | |
| *** ArxCruz has quit IRC | 21:04 | |
| *** rfolco has quit IRC | 21:05 | |
| jeblair | nibalizer: or does "ca_server =" cause it to do that validation over the network? (and if so, how?) | 21:05 |
| *** adalbas has quit IRC | 21:06 | |
| *** atiwari has quit IRC | 21:06 | |
| clarkb | ok I am removing envinject plugin from jenkins07 now | 21:06 |
| *** markwash has quit IRC | 21:06 | |
| *** gokrokve has quit IRC | 21:06 | |
| nibalizer | jeblair: so the new puppetmaster will need its own cert, we can't re-use the cert from ci-puppetmaster | 21:07 |
| *** mfer has quit IRC | 21:07 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: New Story Autofocus https://review.openstack.org/104342 | 21:07 |
| jeblair | nibalizer: yeah, that's not what i was trying to ask; i'm curious about how the new puppetmaster will validate the signatures of client certs presented to it | 21:07 |
| nibalizer | ca = false tells the puppet daemon not to respond to certificate signing requests, if it gets one it just errors and closes the connection, thats how we prevent puppetmaster from running its own ca | 21:08 |
| nibalizer | ca_server = ci-pupputmaster will tell puppetmaster that it needs to check in to that host to get a certificate | 21:08 |
| jeblair | nibalizer: oh, we should do that on ci-puppetmaster too -- we never accept signing requests | 21:08 |
| *** gokrokve has joined #openstack-infra | 21:08 | |
| nibalizer | no it does more than that, it turns off all certificate authority actions too | 21:08 |
| nibalizer | how do you put certs out there | 21:08 |
| nibalizer | ? | 21:08 |
| nibalizer | do you puppet cert generate then copy? | 21:08 |
| jeblair | nibalizer: yes, because we launch them from ci-puppetmaster, so we have a trusted path for the copy | 21:09 |
| nibalizer | puppetmaster.o.o will validate clients by checking to see if their cert has been signed by the ca.pem that puppetmaster.o.o got from puppetmaster-ci.o.o | 21:09 |
| nibalizer | clients perform a reciprocal validation | 21:09 |
| jeblair | nibalizer: okay, so checking the ca.pem from ci-puppetmaster is a side effect of setting ca_server=, right? | 21:10 |
| nibalizer | well, no | 21:10 |
| nibalizer | if we're doing cert generate, copy, then there is no need for ca_server to be set | 21:10 |
| nibalizer | its only important if using puppet agent, then puppet cert sign to do the initial key setup | 21:11 |
| nibalizer | key/cert setup, i should say | 21:11 |
| nibalizer | actually puppet doesn't /do/ any certificate validation, it hands all that off to apache | 21:11 |
| jeblair | nibalizer: ok, so what will cause puppetmaster to validate client certs are signed by ci-puppetmaster's ca.pem? | 21:11 |
| jeblair | or apache even :) | 21:11 |
| nibalizer | so its our job to put ci-puppetmaster.o.o's ca.pem file in the right place then point at it in apache and then we're golden | 21:12 |
| nibalizer | jeblair: apache configuration dirctives set that up | 21:12 |
| jeblair | ./sites-available/puppetmaster: SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem | 21:12 |
| openstackgerrit | Khai Do proposed a change to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target https://review.openstack.org/90455 | 21:12 |
| jeblair | that one i guess :) | 21:12 |
| nibalizer | jeblair: so if you look in that file there is a section that allows unauthenticated clients to submit certificate signing requests | 21:13 |
| *** dims_ has joined #openstack-infra | 21:13 | |
| nibalizer | we could presumably remove that to get the behaviour you spoke of a bit ago: rejecting certificate requests | 21:13 |
| clarkb | shutting down jenkins07 now | 21:14 |
| jeblair | nibalizer: i don't see that option immediately, but that might be nice, yeah. but we can worry about it later | 21:14 |
| *** mmaglana has quit IRC | 21:14 | |
| jeblair | nibalizer: so should we just copy the ci-puppetmaster ca files (cert and key) over to the new puppetmaster? | 21:14 |
| *** dims has quit IRC | 21:15 | |
| clarkb | and starting now | 21:15 |
| *** julim has quit IRC | 21:15 | |
| nibalizer | jeblair: no | 21:16 |
| nibalizer | we should puppet cert generate puppetmaster.o.o , then copy those files over, as well as the ca.pem from the CA and the CRL | 21:16 |
| nibalizer | also... crud, so trusty has apache 2.4 | 21:16 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 21:17 | |
| nibalizer | so we'll have to add logic to detect and handle that in our puppetmaster vhost template | 21:17 |
| jeblair | nibalizer: sorry, i was only asking about the ca. so yes, agreed that the machine cert itself should be unique for the new server, but as for the CA -- copy it over completely? | 21:17 |
| clarkb | https://jenkins07.openstack.org/job/gate-horizon-selenium/922/parameters/? jobs are running and we still list parameters | 21:17 |
| clarkb | so hopefully this really does fix it :) | 21:17 |
| *** gyee has quit IRC | 21:18 | |
| *** annegent_ has joined #openstack-infra | 21:18 | |
| nibalizer | jeblair: no no, we can't copy it over | 21:18 |
| nibalizer | it needs to be unique | 21:18 |
| *** dims_ has quit IRC | 21:18 | |
| nibalizer | because every time you sign a cert with a ca it gets a serial number and those numbers are what is used to revoke it in the crl | 21:19 |
| nibalizer | so best pratice is to only have one if you can get away with it | 21:19 |
| nibalizer | second best practice if you really need two machines, is to keep it on nfs or something so the files can never disagree | 21:19 |
| *** james_li has quit IRC | 21:20 | |
| nibalizer | but we don't need that CA directory on puppetmaster.o.o it can live on ci-puppetmaster.o.o for the forseeable future | 21:20 |
| jeblair | nibalizer: okay, so just copy the ca cert for now, but not the key? i mean, we'll have a ca on that host, it just won't be usuable in that state. | 21:20 |
| *** mmaglana has joined #openstack-infra | 21:21 | |
| nibalizer | ca = false on puppetmaster.o.o tells pm.o.o to be a 'dumb' puppet master, just compiling catalogs and such, leaving cert signing to a different puppetmaster | 21:21 |
| nibalizer | jeblair: it won't have a ca at all, the only file you need to copy over that has ca in it is /var/lib/puppet/ssl/certs/ca.pem | 21:21 |
| nibalizer | but thats just the root ca certificate, same file as is in /etc/pki and /etc/ssl and in your firefox | 21:22 |
| nibalizer | its only purpose is to validate that a certificate presented to you has been signde by that certificate authority | 21:22 |
| nibalizer | and its only purpose for pm.o.o is to make sure clients have really been signed by the CA on ci-puppetmaster.o.o | 21:22 |
| nibalizer | also, is the puppetmaster apache vhost managed in puppet? I don't see it in infra/config anywhere | 21:23 |
| jeblair | nibalizer: i'm assuming something will create those files and set up a valid ca. so at that point, i think it's likely we will have a broken ca because the key doesn't match the cert. but we don't care because we don't plan to use it. | 21:23 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Use tripleo-core for dib-utils https://review.openstack.org/104349 | 21:23 |
| jeblair | nibalizer: i expect that it comes from the .deb | 21:23 |
| *** signed8bit_ZZZzz is now known as signed8b_ | 21:23 | |
| jeblair | clarkb: erm, that screen was supposed to go away | 21:23 |
| nibalizer | oh the puppetmaster-passenger package, that makes sense | 21:23 |
| clarkb | jeblair: yes I think jenkins provides that screen | 21:24 |
| nibalizer | jeblair: yes, probably the post-install(and start) script of the puppemaster-passenger package will generate a ca | 21:24 |
| clarkb | jeblair: I don't think the plugin does | 21:24 |
| jeblair | clarkb: jenkins-dev does not have it | 21:24 |
| nibalizer | but a combination of putting ca=false in the [master] section of puppet.conf and rm -fr /var/lib/puppet/ssl will take care of that | 21:25 |
| clarkb | jeblair: oh | 21:25 |
| nibalizer | then you can copy over the four files you need from ci-puppetmaster.o.o | 21:25 |
| clarkb | jeblair: the plugin is definitely uninstalled | 21:25 |
| *** abhishek__ has joined #openstack-infra | 21:25 | |
| abhishek__ | hi all | 21:26 |
| abhishek__ | can any one review this patch? | 21:26 |
| abhishek__ | https://review.openstack.org/#/c/102215/ | 21:26 |
| abhishek__ | thank you | 21:26 |
| *** mmaglana has quit IRC | 21:26 | |
| jeblair | clarkb: i wonder if 'rebuild plugin' is responsible? | 21:26 |
| *** jp_at_hp has quit IRC | 21:27 | |
| clarkb | jeblair: /me looks | 21:27 |
| *** sgordon has quit IRC | 21:27 | |
| jeblair | oh, hrm. some jobs do have it... | 21:28 |
| jeblair | on jenkins-dev | 21:28 |
| fungi | i've not been reviewing the puppetmaster^3 changes closely enough... how do we address hiera sync from the old master? entry at a time? wholesale copying of the hiera file? | 21:28 |
| clarkb | oh weird | 21:28 |
| clarkb | jeblair: I think removal of that screen is less pressing | 21:29 |
| clarkb | it would have been nice to have but the more important thing is getting the env vars in without modifying them | 21:29 |
| pleia2 | fungi: I had assumed wholesale copying | 21:29 |
| jeblair | clarkb: yeah | 21:29 |
| pleia2 | shouldn't change that much | 21:30 |
| jeblair | pleia2, fungi: yeah, i think we probably just double our work there for a while :) | 21:30 |
| pleia2 | some of our dependencies are a bit tangled up (referenceing gerrit pubkey on a few servers, for instance) so I think chopping it up for the move is more work | 21:30 |
| jeblair | nibalizer: ok so i think we know what we're doing now; do you want to write that second patch before i launch the new host? | 21:30 |
| jeblair | nibalizer: or go aheand and launch it, and fix it up with the update later? | 21:31 |
| *** dims has joined #openstack-infra | 21:31 | |
| *** SumitNaiksatam has quit IRC | 21:32 | |
| clarkb | arg I think centos6 builds are broken again due to facter | 21:32 |
| clarkb | fix incoming | 21:35 |
| *** mrmartin has quit IRC | 21:36 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Readd facter-2* exclude on centos6. https://review.openstack.org/104351 | 21:36 |
| nibalizer | jeblair: as you pointed out | 21:36 |
| clarkb | jeblair: nibalizer ^ | 21:36 |
| nibalizer | whatever i do with ca = false is moot because the pacakge is gonna go do things anyways cus its 'helping' | 21:37 |
| nibalizer | im about to submit the patch | 21:37 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf https://review.openstack.org/104352 | 21:37 |
| *** rcarrillocruz has joined #openstack-infra | 21:38 | |
| openstackgerrit | David Lyle proposed a change to openstack-infra/reviewstats: Updating Horizon core with recent changes https://review.openstack.org/104353 | 21:38 |
| nibalizer | also i had some good early results from running beaker on localhost, nothing to share yet really, but hopefully by the end of the week something you all can see/poke at | 21:38 |
| reed | fungi, jeblair: when you have time, this is good news finally https://review.openstack.org/#/c/104332/ | 21:38 |
| jeblair | reed: cool | 21:39 |
| jeblair | nibalizer: lgtm | 21:39 |
| *** rcarrill` has quit IRC | 21:40 | |
| jeblair | clarkb: lgtm | 21:40 |
| jeblair | clarkb: and i'll trade you https://review.openstack.org/#/c/104352/ :) | 21:40 |
| *** signed8b_ is now known as signed8bit_ZZZzz | 21:41 | |
| openstackgerrit | A change was merged to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target https://review.openstack.org/90455 | 21:41 |
| *** ociuhandu has quit IRC | 21:42 | |
| clarkb | jeblair: nibalizer that change failed testing | 21:44 |
| *** signed8bit_ZZZzz is now known as signed8b_ | 21:45 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf https://review.openstack.org/104352 | 21:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Readd facter-2* exclude on centos6. https://review.openstack.org/104351 | 21:46 |
| nibalizer | clarkb: ya, just posted a fix | 21:46 |
| nibalizer | sorry i needed to verify that undef works how I think it does | 21:46 |
| *** sweston has joined #openstack-infra | 21:52 | |
| *** chianingwang has quit IRC | 21:53 | |
| *** jamielennox|away is now known as jamielennox | 21:55 | |
| *** james_li has joined #openstack-infra | 21:55 | |
| *** abhishek__ has quit IRC | 21:57 | |
| zaro | msabramo: noticed you approved your change own change. Sorry I didn't make you aware, but I think we kinda have a policy against that. | 21:57 |
| *** melwitt has quit IRC | 21:57 | |
| *** dangers is now known as dangers_away | 21:58 | |
| *** MarkAtwood has joined #openstack-infra | 21:59 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf https://review.openstack.org/104352 | 22:01 |
| *** mbacchi has quit IRC | 22:04 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Adding default value for hiera lookup https://review.openstack.org/104358 | 22:05 |
| msabramo | zaro: Oops. sorry about that | 22:05 |
| *** mmaglana has joined #openstack-infra | 22:06 | |
| msabramo | oh I see you pushed up patch sets but probably just to rebase it; it was originally me | 22:07 |
| msabramo | duh | 22:07 |
| msabramo | sorry bout that | 22:07 |
| *** rcarrillocruz has quit IRC | 22:08 | |
| *** eharney has quit IRC | 22:08 | |
| *** rcarrillocruz has joined #openstack-infra | 22:08 | |
| mattoliverau | Guten morgen | 22:08 |
| *** penguinRaider has joined #openstack-infra | 22:09 | |
| *** doug-fish has left #openstack-infra | 22:10 | |
| hashar | msabramo: hi! I have added you as a core reviewer to python-jenkins \O/ | 22:11 |
| msabramo | hashar: Yes, I saw and I am honored by that :-) | 22:12 |
| hashar | zaro: my bad, should have talked about the double CR+2 with Mark before adding him :/ | 22:12 |
| msabramo | though apparently I immediately abused my power :-) | 22:12 |
| msabramo | what happens now? military tribunal? :-) | 22:12 |
| fungi | guten abend mattoliverau | 22:13 |
| hashar | the idea is to seek review from two other core contributors, one of them (or even a third one) would then formally approve | 22:13 |
| *** andreykurilin_ has quit IRC | 22:13 | |
| hashar | so you have some kind of a double check. | 22:13 |
| hashar | zaro: I can't afford anytime this week to review stuff onJJB. Will definitely allocate half a day next week though, then I am on vacations | 22:14 |
| mattoliverau | fungi: I see you've been practicing too :P | 22:14 |
| *** nelsnelson has quit IRC | 22:15 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 22:15 | |
| *** d34dh0r53 is now known as mostly_d34dh0r53 | 22:16 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 22:18 |
| hashar | zaro: maybe python-jenkins could use a minor release just before that pbr patches that landed in | 22:19 |
| hashar | zaro: then bump version to 0.3 ? | 22:19 |
| jeblair | probably a good idea | 22:19 |
| zaro | msabramo: tribunal would be interesting, but i think newbie pass is ok this time :) | 22:20 |
| zaro | hashar: ++, will do tomorrow maybe. | 22:21 |
| hashar | zaro: left a note on the pbr change . Thanks! | 22:21 |
| nibalizer | jeblair: clarkb want to review https://review.openstack.org/#/c/104358/ ? | 22:21 |
| hashar | crashing to bed. Have fun folks | 22:21 |
| *** otherwiseguy has quit IRC | 22:21 | |
| nibalizer | turns out you HAVE to use the <% -%> syntax if you dont' provide any string output | 22:22 |
| zaro | hashar: although it probably should be major release since last release was a year ago and there seems to be lots of changes? | 22:22 |
| jeblair | nibalizer: is https://review.openstack.org/#/c/104352/ ready? | 22:22 |
| nibalizer | jeblair: yes | 22:23 |
| *** gondoi is now known as zz_gondoi | 22:24 | |
| zaro | jeblair: have an opinion? should release be 0.3 or 0.2.2? | 22:24 |
| *** melwitt has joined #openstack-infra | 22:24 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/release-tools: Use python interpreter in the $PATH https://review.openstack.org/104365 | 22:24 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/release-tools: Add requests requirement needed by spec2bp https://review.openstack.org/104366 | 22:24 |
| jeblair | zaro: i'd do 0.3.0 | 22:24 |
| *** melwitt1 has joined #openstack-infra | 22:24 | |
| *** maxbit has quit IRC | 22:24 | |
| zaro | hashar, msabramo : ^ i agree with jeblair | 22:25 |
| hashar | 0.3.0 without pbr so ? | 22:25 |
| zaro | hashar: yes. then bump next one to 0.4 | 22:25 |
| hashar | sounds good to me | 22:26 |
| zaro | hashar: goodnight | 22:27 |
| hashar | thank you! | 22:28 |
| *** melwitt has quit IRC | 22:28 | |
| *** shashankhegde has quit IRC | 22:30 | |
| msabramo | yeah I agree with 0.3.0 for next version of python-jenkins without pbr and then 0.4.0 for pbr | 22:31 |
| clarkb | err: /Stage[main]/Openstack_project::Thick_slave/Package[python-requests]/ensure: change from 1.1.0-4.el6 to absent failed: Execution of '/bin/rpm -e python-requests-1.1.0-4.el6.noarch' returned 1: error: Failed dependencies: | 22:32 |
| msabramo | even without pbr, it changed quite a lot and changed maintainership and everything | 22:32 |
| clarkb | python-requests is needed by (installed) cloud-init-0.7.4-2.el6.noarch | 22:32 |
| clarkb | I think yum is trying to tell me it couldn't upgrade python-requests because the version of cloud init that is installed dpeends on the version it already has? | 22:32 |
| ianw | jeblair: https://review.openstack.org/#/c/101110/ acquired a few more reviews; any more issues? | 22:32 |
| anteaya | so more rain, right now we are at 6.5 inches in 48 hours, not sure but it might break a record | 22:32 |
| jeblair | ianw: aprvd | 22:33 |
| clarkb | anteaya: are you commuting by boat yet? | 22:33 |
| clarkb | jeblair: so should we move forward on envinject or do a bit more sluething to remove the parameters page first? | 22:33 |
| clarkb | jeblair: I guess we should also run a job on 07 to check that it really does fix the parameters | 22:33 |
| ianw | jeblair: thanks! | 22:34 |
| jeblair | clarkb: i'd move forward on that to solve the error and regroup on the parameters page since it's less critical | 22:34 |
| clarkb | ok I am going to put 06 and 05 in shutdown mode now | 22:34 |
| anteaya | clarkb: no commute | 22:34 |
| *** annegent_ has quit IRC | 22:34 | |
| anteaya | but I am sizing up the available timber for the ark | 22:35 |
| anteaya | I should be in good shape | 22:35 |
| anteaya | the deck posts will be my beams | 22:35 |
| clarkb | and done put in shutdown mode | 22:35 |
| *** hashar has quit IRC | 22:36 | |
| *** otherwiseguy has joined #openstack-infra | 22:36 | |
| *** jgrimm has quit IRC | 22:37 | |
| *** zzelle has quit IRC | 22:37 | |
| *** melwitt1 has quit IRC | 22:38 | |
| jeblair | clarkb: quick question on your zuul poll patch... | 22:39 |
| jeblair | clarkb: would it be possible to get the test cases from the other class? it looks like just code duplication there... | 22:39 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 22:40 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 22:41 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Track last allocations to ensure forward-progress https://review.openstack.org/101110 | 22:41 |
| clarkb | jeblair: maybe? | 22:42 |
| clarkb | it has been a while I would haveto go reread | 22:42 |
| clarkb | I think the reason for that was we didn't want to run all of the tests from the other class | 22:42 |
| clarkb | jhesketh: jeblair https://jenkins07.openstack.org/job/experimental-swift-logs-config-pep8/9/console | 22:43 |
| *** anode has quit IRC | 22:43 | |
| clarkb | that still isn't quite working right and it appears that maybe jenkins without envinject truncates the variable? | 22:43 |
| clarkb | so much :( | 22:43 |
| *** anode has joined #openstack-infra | 22:44 | |
| *** pballand has quit IRC | 22:44 | |
| *** shashankhegde has joined #openstack-infra | 22:45 | |
| clarkb | jeblair: I remember now. The sleeps are what is important | 22:45 |
| jeblair | clarkb: oh, ok. thx. | 22:45 |
| clarkb | jeblair: those tests are different and have artificial waits that force the polling to occur | 22:45 |
| clarkb | similar but differnet | 22:45 |
| jeblair | clarkb: er, any chance we can make those a bit faster? :) | 22:46 |
| clarkb | we have 1 second precision, so 2 seconds maybe? | 22:46 |
| jeblair | clarkb: for the timer tests i do tenths of a second | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Move oslo default test node to trusty. https://review.openstack.org/103660 | 22:46 |
| clarkb | jeblair: I think that may end up being shorter than gits fetch time | 22:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Move trove default test node to trusty. https://review.openstack.org/103678 | 22:47 |
| jeblair | clarkb: if we depend on something happening in a sleep, then it's a racy test anyway | 22:47 |
| clarkb | jeblair: it shouldn't be racy because we open the flood gates after the wait | 22:48 |
| clarkb | the self.update_repo() calls make it sanish | 22:48 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/infra-specs: StoryBoard Permissions Specifications https://review.openstack.org/103667 | 22:49 |
| sweston | clarkb: fungi: hello. Would either of you have a few minutes to provide some direction on how to get console logs out of Jenkins? | 22:50 |
| *** maxbit has joined #openstack-infra | 22:50 | |
| clarkb | sweston: can you be a little bit more specific? what are you trying to do? | 22:50 |
| sweston | clarkb: sure. | 22:50 |
| anteaya | he is trying to get utc timestamps into logs for third party ci | 22:51 |
| sweston | clarkb: trying to get the output for the dsvm-tempest-full job | 22:51 |
| clarkb | we use a plugin to do that | 22:51 |
| sweston | is it the html publisher? | 22:51 |
| clarkb | no | 22:52 |
| clarkb | one sec | 22:52 |
| sweston | kk | 22:52 |
| fungi | anteaya: you have my sympathies. next island south of us is currently under a mandatory evacuation order and this is only the first cyclone of the season (hopefully we get a pass on this one) | 22:52 |
| clarkb | sweston: https://wiki.jenkins-ci.org/display/JENKINS/Timestamper is what we use at version 1.3.1 | 22:52 |
| *** david-lyle has quit IRC | 22:53 | |
| jeblair | clarkb: point being, even with sleep(3) we can't be certain it's actually polled and failed; what would be nice is looping with a small sleep until at least one poll failure has happened. that probably means adding some instrumentation; possibly monkeypatching. but then we can be assured the test dtrt, and can speed it up. | 22:53 |
| *** devkulkarni has quit IRC | 22:54 | |
| anteaya | fungi: :( | 22:54 |
| *** david-lyle has joined #openstack-infra | 22:54 | |
| anteaya | fungi: where do they go? and how secure are their houses when they evacuate? | 22:54 |
| clarkb | jeblair: sure. I think jhesketh asked for something ismilar but I don't think we do any of that in zuul's test frame work today | 22:54 |
| anteaya | fungi: where would you go if you had to? you just arrived | 22:54 |
| clarkb | jeblair: its very functional testing and not mock testing | 22:54 |
| jeblair | clarkb: yeah, i don't want to change that | 22:55 |
| *** dimtruck is now known as zz_dimtruck | 22:55 | |
| jeblair | clarkb: however, we go to great lengths to functional test as quickly as possible | 22:55 |
| fungi | anteaya: we "go west" | 22:55 |
| *** krtaylor has joined #openstack-infra | 22:55 | |
| jeblair | fungi: go far enough and you hit a desert :) | 22:56 |
| clarkb | jeblair: do you have suggestions on how you would do mock like functional testing here? monkeypatch in the poll functions with one that sets a flag? | 22:56 |
| fungi | jeblair: indeed | 22:56 |
| anteaya | fungi: got that wagon packed, just in case? | 22:56 |
| *** msabramo has quit IRC | 22:56 | |
| anteaya | krtaylor: hey there | 22:56 |
| anteaya | krtaylor: so can you come to germany yet? | 22:56 |
| fungi | anteaya: we just go visit family in the mountains for a few days if it comes to it. not really a huge hassle | 22:56 |
| anteaya | fungi: kk, glad you have a plan if need be | 22:57 |
| *** camunoz_gone is now known as camunoz | 22:57 | |
| sweston | clarkb: ok. right now I am using version 1.5.12, and it is configured from the scp publisher defined in macros.yaml file here: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/macros.yaml | 22:57 |
| anteaya | I'm on top of a hill so pretty safe | 22:57 |
| anteaya | I'm in the place everyone else comes to in times of bad things happening | 22:57 |
| clarkb | sweston: I think that newer versions may not work because they split the data out or something silly | 22:58 |
| clarkb | sweston: also you haev to set the format in your global config | 22:58 |
| *** virmitio has quit IRC | 22:58 | |
| sweston | clarkb: ah, ok. I will look into that. | 22:58 |
| anteaya | sweston: and keep notes please so you can share at a future meeting | 22:58 |
| anteaya | and so we can add to docs | 22:59 |
| *** devkulkarni has joined #openstack-infra | 22:59 | |
| *** david-lyle has quit IRC | 22:59 | |
| clarkb | ok all of the rax images that matter have been updated | 22:59 |
| clarkb | hpcloud is in progress | 22:59 |
| sweston | clarkb: anteaya: yes, absolutely. at some point I'd like to have a discussion with infra about how to adapt the infra's puppet manifests in the config repo to run third party infrastructure. this is what I've been working on for the last several weeks. | 23:00 |
| jeblair | sweston: are you attending the qa/infra meetup in germany? | 23:01 |
| *** ociuhandu has joined #openstack-infra | 23:01 | |
| *** blamar_ has joined #openstack-infra | 23:01 | |
| *** blamar has quit IRC | 23:01 | |
| *** blamar_ is now known as blamar | 23:01 | |
| sweston | jeblair: no, unfortunately not. may I somehow contribute to the discussion/meetup online? | 23:02 |
| jeblair | sweston: well, we'll probably have some specs come out of it; https://review.openstack.org/#/c/99990/ is a related first step | 23:02 |
| jeblair | sweston: but several folks are interested in that area, your help will be appreciated :) | 23:03 |
| jhesketh | Morning | 23:03 |
| sweston | jeblair: sure, I would be happy to share what I have so far | 23:03 |
| anteaya | morning jhesketh | 23:04 |
| sweston | jeblair: I have all of my changes in github, when you are ready I will be happy to send the link | 23:04 |
| jeblair | sweston: great, best way to share is to submit patches to gerrit, and write/review specs | 23:04 |
| *** amcrn has quit IRC | 23:04 | |
| sweston | jeblair: ok. | 23:04 |
| *** blamar has quit IRC | 23:04 | |
| sweston | jeblair: suggestions on where/how to get started? | 23:05 |
| clarkb | jhesketh: hello | 23:05 |
| clarkb | jhesketh: please see https://jenkins07.openstack.org/job/experimental-swift-logs-config-pep8/9/console | 23:05 |
| clarkb | that ran without envinject | 23:05 |
| *** amcrn has joined #openstack-infra | 23:05 | |
| *** otherwiseguy has quit IRC | 23:05 | |
| anteaya | sweston: have you reviewed all open patches with topic:third-party ? https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:third-party,n,z | 23:06 |
| sweston | anteaya: no, first time I've seen these | 23:06 |
| jhesketh | clarkb: yeah just saw that | 23:06 |
| jhesketh | that's kind of disappointing | 23:06 |
| anteaya | sweston: great, let's start there | 23:07 |
| jhesketh | clarkb: it looks like it's truncated at the \n | 23:07 |
| sweston | anteaya: happy to. I will start on the patch review this evening. | 23:07 |
| *** sigmavirus24 has quit IRC | 23:08 | |
| jhesketh | clarkb, jeblair: Maybe for a work around for being unable to send multiple line variables to Jenkins I could patch zuul to also send "SWIFT_logs_HMAC_BODY_PART{1,4}" | 23:08 |
| jhesketh | what do you think? | 23:08 |
| anteaya | sweston: thanks | 23:08 |
| anteaya | I just checked the annual rainfall stats, we doubled the usual monthy rainfall for june in two days | 23:09 |
| sweston | anteaya: yw. clarkb:, jeblair: thanks for the guidance. looking forward to working with you all some more :-) | 23:09 |
| anteaya | usual is 86 mm, or 3.25 inches | 23:09 |
| jeblair | jhesketh, clarkb: the "ZUUL_CHANGES" parameter that zuul sends separates them with '^' for that reason; back when gerrit-git-prep used it (a loooong time ago), it did some sed magic or something to split it on that char | 23:10 |
| *** msabramo has joined #openstack-infra | 23:10 | |
| anteaya | for a month | 23:10 |
| jhesketh | jeblair: right, except the hmac body is in the format that the signature used | 23:10 |
| jhesketh | and the format that swift expects | 23:11 |
| jhesketh | so I'd like to keep sending the hmac_body as it is but then also provide a workaround for Jenkins | 23:11 |
| jeblair | jhesketh: yup, just adding information :) | 23:11 |
| jhesketh | cool :-) | 23:11 |
| *** annegent_ has joined #openstack-infra | 23:12 | |
| clarkb | I would be on board with either approach | 23:12 |
| clarkb | I have 05 and 06 in shutdown mode still | 23:12 |
| clarkb | I don't see any reason to leave envinject in place as we wanted it gone | 23:13 |
| clarkb | so I will probably continue on my path of removing it | 23:13 |
| jeblair | clarkb, jhesketh: is upgrading envinject to the new version a solution? | 23:13 |
| *** maxbit has quit IRC | 23:13 | |
| clarkb | jeblair: yes I think that was an alternative | 23:14 |
| *** whoops has quit IRC | 23:14 | |
| jhesketh | :-( that was what I wanted to do last week | 23:14 |
| jhesketh | we've done well to remove the envinject stuff, I think adding it back in would be a step backwards | 23:14 |
| clarkb | ya | 23:14 |
| jeblair | jhesketh: well, that was based on the understanding that the problem was that envinject was corrupting data; that seems to be falso | 23:15 |
| clarkb | jeblair: it was corrupting the data | 23:15 |
| jeblair | we definitely don't want to use it in jobs | 23:16 |
| jhesketh | I'm actually unsure why we don't want it to be used? | 23:16 |
| jeblair | however, if it helps the transition to not-jenkins, i'm fine keeping it around | 23:16 |
| jhesketh | or is it simply that we want to reduce our module consumption to tie us to jenkins less | 23:16 |
| jeblair | jhesketh: because it's a jenkins-specific way of describing a job | 23:16 |
| jeblair | jhesketh: yes | 23:16 |
| jhesketh | I'm more inclined to have the workaround in zuul, but that would add 4 more variables to the gearman work data that may have performance effects | 23:17 |
| phschwartz | :( my brain hurts. This is the worst day I have had coding in about 12 years. I made the stupid decision to move monday from Vi due to non-stop plugin issue and sick of my 3k line config to emacs after not using it for almost 10 years. Then I had yesterday off for jury duty. Talking about making your life hell. | 23:19 |
| clarkb | phschwartz: the answer is to just stick with vim | 23:20 |
| clarkb | I had similar problems with vim on saucy crashing on yaml files due to segfault in regex engine | 23:20 |
| SlickNik | phschwartz: evil mode :) | 23:20 |
| clarkb | I tried evil but it is no where close enough | 23:20 |
| jeblair | jhesketh, clarkb: i think the issue may be in gearman-plugin | 23:20 |
| clarkb | too many things break | 23:20 |
| clarkb | jeblair: oooh | 23:20 |
| clarkb | interesting | 23:20 |
| jeblair | i'll have more in a sec | 23:21 |
| phschwartz | clarkb: I have been having a mix of that and YCM dying non-stop on my 14.04 installs. It has been driving me insane. | 23:21 |
| phschwartz | clarkb: Which spawned the change. | 23:21 |
| clarkb | trusty has been fine with me but I avoid plugins for hte most part | 23:21 |
| clarkb | too many habits from when I had to have working vim on all the platforms | 23:22 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/StartJobWorker.java#n144 | 23:22 |
| jeblair | https://github.com/jenkinsci/parameterized-trigger-plugin/pull/36 | 23:22 |
| *** Sukhdev has joined #openstack-infra | 23:22 | |
| clarkb | jeblair: interesting | 23:22 |
| clarkb | but also a simplish fix | 23:22 |
| jeblair | http://javadoc.jenkins-ci.org/hudson/model/TextParameterValue.html | 23:22 |
| clarkb | I should probably continue to remove envinject then? | 23:23 |
| clarkb | 05 and 06 are close to being done and I can do whatever | 23:23 |
| *** alexandra has joined #openstack-infra | 23:24 | |
| *** rlandy has quit IRC | 23:25 | |
| *** alexandra is now known as asettle | 23:25 | |
| Sukhdev | SlickNik: ping | 23:25 |
| jeblair | clarkb, jhesketh: it's not yet clear to me if that will change behavior in our case | 23:25 |
| clarkb | looking at the logstash worker queue graph for today I want to declare success on that front and I should probably move on to do ssds on all the nodes | 23:25 |
| clarkb | jeblair: roger | 23:25 |
| *** r1chardj0n3s has left #openstack-infra | 23:25 | |
| SlickNik | Sukhdev: Spoke to shashank, and he offerred a workaround. | 23:25 |
| SlickNik | Sukhdev: I haven't had a chance to try it out yet. | 23:26 |
| *** annegent_ has quit IRC | 23:26 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move troveclient default test node to trusty. https://review.openstack.org/103679 | 23:26 |
| SlickNik | Sukhdev: Will do so this evening, and let you know how it goes. | 23:26 |
| *** sabeen2 has quit IRC | 23:26 | |
| SlickNik | Sukhdev: Thanks so much for your help with this! | 23:26 |
| Sukhdev | SlickNik: good - I discussed it with him before I left - he is very familiar with the code | 23:26 |
| *** mugsie has quit IRC | 23:26 | |
| anteaya | SlickNik Sukhdev shashankhegde thanks to all of you for working on this | 23:27 |
| anteaya | I really appreciate it | 23:27 |
| Sukhdev | SlickNik: no worries -- let me know it goes and then we will take it from there | 23:27 |
| *** james_li has quit IRC | 23:27 | |
| anteaya | please keep notes to so we can use them for getting this patch upstreamed please | 23:27 |
| anteaya | I'd like that to happen if possible | 23:27 |
| jhesketh | jeblair: hmm, it seems like a good lead | 23:27 |
| jeblair | jhesketh, clarkb: i should be able to test it in a few mins | 23:28 |
| Sukhdev | anteaya: once SlickNik confirms the fix, I'll update the document so that others can benefit from it | 23:28 |
| jhesketh | jeblair: you're patching it up then ? :-) | 23:28 |
| shashankhegde | anteaya: It still requires a bit of work before I can push it upstream | 23:28 |
| anteaya | Sukhdev: thank you | 23:28 |
| anteaya | shashankhegde: understood, and thank you | 23:29 |
| shashankhegde | I'll try to complete it and push it so that everyone can take advantage of it | 23:29 |
| *** devkulkarni has quit IRC | 23:29 | |
| anteaya | shashankhegde: thank you | 23:29 |
| *** signed8bit_ZZZzz is now known as signed8b_ | 23:29 | |
| anteaya | shashankhegde: do you work on jenkins plugins? | 23:30 |
| *** signed8b_ has quit IRC | 23:30 | |
| *** signed8bit has joined #openstack-infra | 23:30 | |
| *** thedodd has quit IRC | 23:31 | |
| Sukhdev | anteaya: shashankhegde and I both work for Arista. He and I put together the whole Arista CI solution - | 23:31 |
| anteaya | ah | 23:31 |
| anteaya | your silent partner | 23:31 |
| anteaya | you brought him out of silence | 23:31 |
| anteaya | :D | 23:32 |
| Sukhdev | anteaya: yes, silent, but, very powerful partner :-) | 23:32 |
| anteaya | the best kind to have | 23:32 |
| anteaya | :D | 23:32 |
| Sukhdev | anteaya: no kidding!!! | 23:32 |
| anteaya | okay I have to get a walk in when it isn't raining and the sun is still out | 23:33 |
| *** gyee has joined #openstack-infra | 23:33 | |
| anteaya | back in a while | 23:33 |
| *** signed8bit has quit IRC | 23:34 | |
| *** UtahDave has quit IRC | 23:35 | |
| jeblair | jhesketh, clarkb: i see the problem | 23:39 |
| jeblair | clarkb: yes, please proceed with removing envinject | 23:39 |
| clarkb | ok will do | 23:41 |
| *** atiwari has joined #openstack-infra | 23:42 | |
| clarkb | 06 is ready so I am going ahead with it. 05 has one job left | 23:42 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix env var interpretation in zuul swift upload https://review.openstack.org/104376 | 23:42 |
| jeblair | clarkb, jhesketh: ^ | 23:42 |
| clarkb | jeblair: will that actually fix it though? on the parameters page it looks like the value is truncated? | 23:43 |
| clarkb | jeblair: I agree that is a bug | 23:43 |
| jeblair | clarkb: why do you think it's truncated? | 23:44 |
| jeblair | i mean, it looks like it has multiple values strung together... | 23:44 |
| clarkb | jeblair: because it doesn't show the parameter on multiple lines | 23:44 |
| clarkb | and I think when I tested it on jenkins-dev it showed me multiple lines | 23:44 |
| clarkb | I would haveto double check that though | 23:45 |
| jeblair | clarkb: the newlines don't seem to be displayed, but my local testing suggests they are there. | 23:45 |
| *** oomichi has joined #openstack-infra | 23:45 | |
| clarkb | jeblair: ah ok | 23:45 |
| clarkb | anyways +2 on the fix | 23:45 |
| jeblair | clarkb: it's possible the string -> text change may change the display of that on the parameter listing page | 23:45 |
| clarkb | ohhhh ya good point | 23:45 |
| clarkb | a rendering vs content thing | 23:46 |
| jeblair | clarkb: however, i haven't done that yet -- i just set up the base test case locally and noticed it worked :) | 23:46 |
| jhesketh | jeblair: +A since your testing works.. however I'm wondering how it got there in the first place | 23:46 |
| jhesketh | ie I tested that script :-s | 23:46 |
| jeblair | jhesketh: i actually have not tested that script, i just tested the gearman->jenkins path to make sure the env variable looks correct; so i'm pretty sure that's the problem | 23:47 |
| jhesketh | maybe I fat-fingered it after the test | 23:48 |
| *** jcoufal has quit IRC | 23:48 | |
| clarkb | 06 is done | 23:48 |
| *** devkulkarni has joined #openstack-infra | 23:48 | |
| *** nati_uen_ has quit IRC | 23:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix env var interpretation in zuul swift upload https://review.openstack.org/104376 | 23:50 |
| jeblair | jhesketh, clarkb: yes, changing to TextParameter causes jenkins to display it in a multiline field | 23:50 |
| jeblair | (well, causes everything to display in a multiline field, which is a little awkward) | 23:50 |
| jeblair | this may be a good general improvement to gearman-plugin, for folks who would want to see such info, or possibly when interacting with other plugins (like rebuild, which uses saved data) | 23:50 |
| jeblair | maybe it should scan the value for '\n' and chose string or text appropriately | 23:51 |
| *** IAX has joined #openstack-infra | 23:51 | |
| jeblair | (or just accept the wasted realestate and use text everywhere) | 23:51 |
| openstackgerrit | A change was merged to openstack/requirements: Update openstack-doc-tools https://review.openstack.org/104171 | 23:52 |
| *** shayneburgess has quit IRC | 23:53 | |
| *** unicell has quit IRC | 23:53 | |
| clarkb | ok 05 is ready now | 23:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gearman-plugin: Use TextParameterValue instead of String https://review.openstack.org/104386 | 23:54 |
| jeblair | in case we decide we want that^ | 23:54 |
| jeblair | but yeah, doing a "python -c" command on a parameter with a newline works either way | 23:55 |
| *** shashankhegde has quit IRC | 23:55 | |
| jhesketh | jeblair: I don't mind the scan option... it's probably a good idea since it'll allow people to see all the hmac lines if they need | 23:55 |
| clarkb | 05 restarted | 23:55 |
| clarkb | putting 04 and 03 in shutdown mode now then probably walking home and picking up from there | 23:56 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Export prepare_node variables rather than pass them around https://review.openstack.org/104387 | 23:56 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Add path to installing puppet 3 on nodepool nodes https://review.openstack.org/104388 | 23:56 |
| clarkb | ok thats done | 23:57 |
| *** prad has quit IRC | 23:59 | |
| *** wenlock has quit IRC | 23:59 | |
| *** msabramo has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!