| sdague | jar | 00:00 |
|---|---|---|
| *** sarob has quit IRC | 00:00 | |
| anteaya | happy utc valentine's day everybody | 00:00 |
| jeblair | jog0: since this bug is a behavior change, that suggests: (a) we are doing way more lookups than before (b) they have changed something about their servers (rate limiting/reliability/etc) | 00:00 |
| jog0 | fungi: http://status.openstack.org/elastic-recheck/gate.html | 00:00 |
| *** bhuvan has joined #openstack-infra | 00:00 | |
| openstackgerrit | John Dickinson proposed a change to openstack-infra/config: clean up swiftclient jobs https://review.openstack.org/73447 | 00:01 |
| jeblair | jog0: suggestion: determine whether (a) is happening, and if so, is it necessary, or did we do something silly that's causing extra lookups that we should revert | 00:01 |
| mtreinish | sdague: I guess I just don't ever run java if I can help it I didn't realize the syntax was the same | 00:01 |
| sdague | yeh, that's why I use it | 00:01 |
| jog0 | jeblair: ahh | 00:01 |
| fungi | jeblair: or c) the recent rise in conficker traffic has started beating their nameservers very, very hard | 00:01 |
| jog0 | jeblair: not sure how to figure out if we are doing extra lookups | 00:02 |
| *** oanufriev has quit IRC | 00:02 | |
| jog0 | fungi: that would be pretty wild of conflicker is hurting our gate | 00:02 |
| *** oubiwann_ has joined #openstack-infra | 00:02 | |
| jeblair | jog0: and if it we really are doing the correct number of lookups (and it's huge), then i think we need to do what fungi suggests and add a caching resolver to devstack | 00:02 |
| mordred | prolly not a terrible idea | 00:02 |
| jeblair | jog0: (i think it should go in devstack since it would, apparently, be needed to run tests in some environments) | 00:03 |
| jeblair | fungi: i think i need more time than eod-now to find/fix the gate/re-enqueue bug. i think we should revert todays layout and logging changes now. | 00:04 |
| fungi | jeblair: i confer. we have debug data we can pour over | 00:05 |
| fungi | er, concur | 00:05 |
| fungi | want to propose the reverts or shall i? | 00:06 |
| jeblair | fungi: i'm on it | 00:06 |
| jog0 | jeblair: sounds reasonable should i add devstack to that bug? | 00:07 |
| jeblair | jog0: yeah | 00:07 |
| *** flaper87 is now known as flaper87|afk | 00:07 | |
| *** sarob_ has quit IRC | 00:08 | |
| *** virmitio has quit IRC | 00:08 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert requiring a recent positive check result https://review.openstack.org/73449 | 00:08 |
| jeblair | fungi: +3 ^ ? | 00:08 |
| fungi | yuppp | 00:09 |
| jog0 | clarkb: btw I just saw some 502s from elasticSearch | 00:10 |
| fungi | waiting for check to report before i approve, since, well, you know why | 00:10 |
| clarkb | jog0: may be related to creating the new index for the day as the 0000 hour rolled over | 00:13 |
| sdague | sorry, catching up, what do we think is impacting us from devstack? | 00:14 |
| *** e0ne has joined #openstack-infra | 00:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert requiring a recent positive check result https://review.openstack.org/73449 | 00:14 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Remove taskflow sqlalchemy 0.9 testing https://review.openstack.org/73445 | 00:14 |
| jog0 | clarkb: cool just wanted to let you know | 00:15 |
| fungi | sdague: devstack-gate/devstack/tempest/grenade might be making lots of dns lookups | 00:15 |
| sdague | hmmm... could be | 00:15 |
| sdague | well we are installing dnsmasq already for nova, might as well configure it for upstream cache | 00:16 |
| fungi | sdague: we got a ticket from rackspace about one single-use slave doing so many queries that they configured their resolvers to throttle it | 00:16 |
| sdague | that's weird | 00:16 |
| sdague | the pypi lookups? | 00:16 |
| sdague | I guess I could believe that, pip doesn't have a real offline mode | 00:16 |
| fungi | sdague: and we have an open bug which is apparently the current highest e-r match about name lookup timeouts | 00:17 |
| *** bhuvan has quit IRC | 00:17 | |
| sdague | anyway, I think it's dinner + beer time here. I'll chat with folks in the morning | 00:17 |
| *** bhuvan has joined #openstack-infra | 00:18 | |
| fungi | awesome--enjoy | 00:18 |
| *** senthil has quit IRC | 00:18 | |
| *** e0ne has quit IRC | 00:19 | |
| *** david-lyle has quit IRC | 00:19 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Remove taskflow sqlalchemy 0.9 testing https://review.openstack.org/73445 | 00:22 |
| *** gokrokve has quit IRC | 00:24 | |
| jog0 | sdague: o/ | 00:24 |
| *** cadenzajon has quit IRC | 00:24 | |
| *** gokrokve has joined #openstack-infra | 00:24 | |
| *** wenlock_ has quit IRC | 00:25 | |
| openstackgerrit | A change was merged to openstack/requirements: Update to new hp3parclient https://review.openstack.org/72776 | 00:26 |
| openstackgerrit | Jenkins proposed a change to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 00:26 |
| *** senthil has joined #openstack-infra | 00:27 | |
| *** wenlock_ has joined #openstack-infra | 00:27 | |
| *** gokrokve has quit IRC | 00:29 | |
| *** yamahata has quit IRC | 00:29 | |
| harlowja_ | hi guys, could we get a quick look over https://review.openstack.org/#/c/73445/ (to unblock taskflow reviews) | 00:30 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Remove taskflow sqlalchemy 0.9 testing https://review.openstack.org/73445 | 00:32 |
| *** chris_johnson has joined #openstack-infra | 00:34 | |
| *** chris_johnson has quit IRC | 00:35 | |
| openstackgerrit | A change was merged to openstack/requirements: Add cmd2 for cliff https://review.openstack.org/73412 | 00:41 |
| *** CaptTofu has quit IRC | 00:43 | |
| *** prad has quit IRC | 00:47 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add grenade-dsvm-rolling test https://review.openstack.org/64200 | 00:48 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Support grenade-svm-rolling rename to *-partial-ncpu https://review.openstack.org/73460 | 00:48 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Set up opportunistic bare-metal postgres db https://review.openstack.org/73461 | 00:50 |
| *** lcheng_ has quit IRC | 00:56 | |
| *** prad has joined #openstack-infra | 00:56 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: eliminate false positives for bug 1253482 https://review.openstack.org/69997 | 00:58 |
| *** mrodden1 has quit IRC | 00:59 | |
| *** jooools has quit IRC | 01:02 | |
| harlowja_ | thx fungi | 01:03 |
| *** sarob has joined #openstack-infra | 01:09 | |
| *** atiwari has quit IRC | 01:12 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Mark resolved queries https://review.openstack.org/73466 | 01:12 |
| *** e0ne has joined #openstack-infra | 01:13 | |
| *** sarob has quit IRC | 01:15 | |
| *** yamahata has joined #openstack-infra | 01:15 | |
| *** yaguang has joined #openstack-infra | 01:18 | |
| *** e0ne has quit IRC | 01:18 | |
| *** melwitt has quit IRC | 01:19 | |
| *** sarob has joined #openstack-infra | 01:19 | |
| *** VijayT has quit IRC | 01:24 | |
| *** sarob has quit IRC | 01:24 | |
| *** mrodden has joined #openstack-infra | 01:25 | |
| *** gokrokve has joined #openstack-infra | 01:35 | |
| *** wenlock_ has quit IRC | 01:35 | |
| *** wenlock has quit IRC | 01:35 | |
| *** gokrokve has quit IRC | 01:39 | |
| *** gokrokve has joined #openstack-infra | 01:39 | |
| *** nosnos has joined #openstack-infra | 01:39 | |
| *** gokrokve has quit IRC | 01:40 | |
| *** gokrokve has joined #openstack-infra | 01:40 | |
| *** gokrokve_ has joined #openstack-infra | 01:41 | |
| *** jergerber has quit IRC | 01:42 | |
| *** gokrokve has quit IRC | 01:45 | |
| *** senthil has quit IRC | 01:52 | |
| *** Hefeweizen has quit IRC | 01:54 | |
| *** mrodden1 has joined #openstack-infra | 01:54 | |
| *** mrodden has quit IRC | 01:55 | |
| *** banix has quit IRC | 01:56 | |
| *** david-lyle has joined #openstack-infra | 01:59 | |
| *** prad has quit IRC | 02:02 | |
| *** derekh has quit IRC | 02:06 | |
| *** sarob has joined #openstack-infra | 02:07 | |
| *** thomasem has joined #openstack-infra | 02:08 | |
| *** jhesketh has quit IRC | 02:09 | |
| *** zul has quit IRC | 02:12 | |
| *** e0ne has joined #openstack-infra | 02:13 | |
| *** dprince has quit IRC | 02:15 | |
| *** dprince has joined #openstack-infra | 02:17 | |
| *** e0ne has quit IRC | 02:18 | |
| *** CaptTofu has joined #openstack-infra | 02:20 | |
| *** thomasem has quit IRC | 02:22 | |
| *** sarob has quit IRC | 02:23 | |
| *** sarob has joined #openstack-infra | 02:23 | |
| *** morganfainberg is now known as morganfainberg_Z | 02:24 | |
| lifeless | testenvs are back online | 02:27 |
| *** mrodden has joined #openstack-infra | 02:28 | |
| *** sarob has quit IRC | 02:28 | |
| *** mrodden1 has quit IRC | 02:29 | |
| *** wenlock has joined #openstack-infra | 02:31 | |
| *** amrith has joined #openstack-infra | 02:31 | |
| amrith | when I try to "git review" I get an error: | 02:32 |
| *** dprince has quit IRC | 02:32 | |
| amrith | amrith@amrith-vbox:~/stackalytics$ git review | 02:32 |
| amrith | Enter passphrase for key '/home/amrith/.ssh/id_rsa': | 02:32 |
| amrith | Enter passphrase for key '/home/amrith/.ssh/id_rsa': | 02:32 |
| amrith | fatal: ICLA contributor agreement requires current contact information. | 02:32 |
| amrith | Please review your contact information: | 02:32 |
| amrith | https://review.openstack.org/#/settings/contact | 02:32 |
| amrith | fatal: The remote end hung up unexpectedly | 02:32 |
| amrith | amrith@amrith-vbox:~/stackalytics$ | 02:32 |
| amrith | I checked my profile | 02:32 |
| amrith | I have provided valid contact information | 02:32 |
| amrith | where else should I be looking for possible issues? | 02:33 |
| *** oubiwann is now known as oubiwann-home | 02:34 | |
| clarkb | amrith: doe the username shown by `git remote -v` in the gerrit remote match the username in the webui? | 02:36 |
| *** sarob has joined #openstack-infra | 02:39 | |
| *** dstanek has joined #openstack-infra | 02:39 | |
| lifeless | does python-gear automatically reconnect to the server? | 02:40 |
| clarkb | lifeless: yes | 02:40 |
| * lifeless bounces geard | 02:42 | |
| lifeless | I'm trying to figure out why we have: 2014-02-14 02:18:35.848 | + ./testenv-client -b 192.168.1.48:4730 -- ./toci_devtest.sh | 02:42 |
| lifeless | with no further output | 02:42 |
| lifeless | we have 31 test environments AFAICT | 02:43 |
| lifeless | enough for 10 patches | 02:43 |
| lifeless | there are only 9 in zuul | 02:43 |
| lifeless | and all are showing that hang | 02:43 |
| lifeless | that is the broker IP addr | 02:44 |
| lifeless | clarkb: is there a manhole or some way to get a trace of activity ? | 02:44 |
| *** dstanek has quit IRC | 02:45 | |
| clarkb | lifeless: 'status' against port 4730 returns the list of jobs, the number of queued instances of each job, the number of jobs being run and the number of workers available for that job | 02:45 |
| lifeless | are jobs the same as endpoints ? | 02:46 |
| lifeless | I mean | 02:46 |
| lifeless | ia job is a named function one can call ? | 02:46 |
| clarkb | yeah | 02:46 |
| lifeless | ok | 02:46 |
| clarkb | sorry I am overloading jenkins terms here | 02:46 |
| lifeless | so I have all the callbacks | 02:46 |
| lifeless | but not the te function | 02:47 |
| *** sarob has quit IRC | 02:47 | |
| lifeless | clarkb: can you see attached clients ? | 02:47 |
| *** sarob has joined #openstack-infra | 02:47 | |
| clarkb | lifeless: no idea can you netstat for that? | 02:48 |
| lifeless | lockenv is what I'm looking for | 02:48 |
| lifeless | clarkb: not really - I can see connections yes but | 02:48 |
| lifeless | self.worker = gear.Worker('testenv-client-%s' % self.name) | 02:48 |
| lifeless | that diagnostic | 02:48 |
| lifeless | would be super useful | 02:48 |
| clarkb | lifeless: sorry I am not on machine with keys atm can switch when I finish debugging amrith's issue | 02:49 |
| lifeless | no worries, I'm logged into the broker geard | 02:50 |
| lifeless | just trying to see whats wrong | 02:50 |
| lifeless | I suspect remaining testenv bug | 02:50 |
| *** sarob has quit IRC | 02:52 | |
| lifeless | I can at least ssh into the macines | 02:52 |
| lifeless | hah, that ssh key is clearly wrong :) | 02:53 |
| lifeless | '{"remote-operations":"1", "host-ip":"192.168.1.162", "seed-ip":"192.168.1.163", "node-macs":"52:54:00:31:4c:6e 52:54:00:69:07:a0 52:54:00:d9:ef:c1 52:54:00:7b:19:f2", "ssh-key":"-----BEGIN RSA PRIVATE' | 02:53 |
| lifeless | oh, :200 stripped - fine | 02:53 |
| *** jhesketh_ has joined #openstack-infra | 02:55 | |
| lifeless | hahahahaha bad broker address | 02:56 |
| lifeless | ok this I can fix | 02:56 |
| *** mgagne has quit IRC | 03:02 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow workers to send back metadata https://review.openstack.org/66173 | 03:05 |
| *** dcramer_ has joined #openstack-infra | 03:06 | |
| *** ryanpetrello has joined #openstack-infra | 03:09 | |
| *** zhiyan_ is now known as zhiyan | 03:10 | |
| *** wenlock has quit IRC | 03:13 | |
| *** e0ne has joined #openstack-infra | 03:13 | |
| *** whayutin_ has quit IRC | 03:13 | |
| fungi | amrith: there's some additional detail on troubleshooting cla-related errors like that one in our faq https://wiki.openstack.org/wiki/CLA-FAQ so i'd suggest reading through that first if you haven't already, and then asking any questions it's not answering for you | 03:15 |
| *** pcrews has quit IRC | 03:16 | |
| clarkb | fungi: sorry we mostly sorted ti over there -> | 03:16 |
| clarkb | fungi: is a case of multiple accounts causing confusion | 03:16 |
| fungi | ahh. ignore me then! | 03:16 |
| clarkb | there should be a bug with details from amrith indicating which account should go away | 03:16 |
| fungi | yet another multi-account issue. yay | 03:16 |
| fungi | i'll check if you haven't already fixed it | 03:16 |
| clarkb | 9664 and 10396 are the ids | 03:17 |
| clarkb | I think amrith is trying to sort out which to keep | 03:17 |
| fungi | okay | 03:18 |
| lifeless | ok, we have a lockenv up | 03:18 |
| lifeless | clarkb: one thing gear could do better | 03:18 |
| *** e0ne has quit IRC | 03:18 | |
| lifeless | clarkb: status should show functions for which there are no workers, only requests. | 03:18 |
| lifeless | clarkb: e.g. lockenv 23 0 0 | 03:19 |
| fungi | my plan for catching up on e-mail real quick and then knocking around for a quiet evening of relaxation has turned into spending all evening not quite being able to believe just how far behind on e-mail i was | 03:19 |
| clarkb | I think that would be an extension to the protocol , maybe a status-extra or just ignore what the protocol says | 03:19 |
| *** oubiwann_ has quit IRC | 03:19 | |
| clarkb | fungi: you are more caught up than me, new device fun is slowing me down | 03:19 |
| lifeless | clarkb: huh, seemed like a bug in the server to me :) | 03:19 |
| clarkb | lifeless: that behavior is part of the gearman protocol | 03:20 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow workers to send back metadata https://review.openstack.org/66173 | 03:20 |
| *** oubiwann has joined #openstack-infra | 03:20 | |
| clarkb | I think (its possible geard just does it wrong) | 03:20 |
| *** jcooley_ has joined #openstack-infra | 03:26 | |
| *** gyee has quit IRC | 03:26 | |
| *** ryanpetrello has quit IRC | 03:26 | |
| markmcclain | hmmm…is anyone working on the fact the gate can't merge anything? | 03:36 |
| amrith | clarkb/fungi, I will enter the bug and request multi-account merge. thanks! | 03:37 |
| amrith | there are two more ids ;) | 03:38 |
| *** ryanpetrello has joined #openstack-infra | 03:42 | |
| *** ryanpetrello has quit IRC | 03:43 | |
| *** jhesketh_ has quit IRC | 03:45 | |
| markmcclain | sdague, jog0: http://logstash.openstack.org/#eyJzZWFyY2giOiJcInFlbXU6IGxpbnV4IGtlcm5lbCB0b28gb2xkIHRvIGxvYWQgYSByYW0gZGlza1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNDQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTIzNDk1MjA3MDh9 | 03:46 |
| markmcclain | 70k hits < 4hrs | 03:46 |
| *** thomasem has joined #openstack-infra | 03:47 | |
| *** mriedem has quit IRC | 03:48 | |
| *** flaper87|afk has quit IRC | 03:48 | |
| *** sarob has joined #openstack-infra | 03:48 | |
| *** mriedem has joined #openstack-infra | 03:48 | |
| jeblair | markmcclain: it looks like the first occurance is about 3 hours ago. the images we're running tests on are 0.8 hours old, so this was happening on both the old and new images. | 03:50 |
| *** flaper87|afk has joined #openstack-infra | 03:50 | |
| *** thomasem has quit IRC | 03:51 | |
| jeblair | markmcclain: it seems to affect all devstack jobs and all providers | 03:51 |
| *** flaper87|afk is now known as flaper87 | 03:51 | |
| markmcclain | jeblair: yeah was surprised at how widespread it was | 03:52 |
| *** oubiwann has quit IRC | 03:52 | |
| *** sarob has quit IRC | 03:53 | |
| *** talluri has joined #openstack-infra | 03:54 | |
| jeblair | cloud-init facter and grub-legacy-ec2 are the only debs that are different | 03:57 |
| jeblair | i wonder if the grub-legacy-ec2 change is related | 04:01 |
| markmcclain | I was wondering the same thing | 04:02 |
| jeblair | markmcclain: i have to go now, but i think this is looking to be a non-infra bug so it should be diagnosable and correctible without my help; hopefully someone waking up now can pitch in on it. :) | 04:07 |
| markmcclain | jeblair: yeah…I have to head out too.. thanks for digging on it | 04:08 |
| *** amcrn_ has quit IRC | 04:08 | |
| *** mriedem has quit IRC | 04:10 | |
| *** vogxn has joined #openstack-infra | 04:10 | |
| *** chandan_kumar has joined #openstack-infra | 04:11 | |
| *** amcrn has joined #openstack-infra | 04:12 | |
| *** e0ne has joined #openstack-infra | 04:13 | |
| *** wenlock has joined #openstack-infra | 04:15 | |
| *** CaptTofu has quit IRC | 04:16 | |
| *** ryanpetrello has joined #openstack-infra | 04:16 | |
| *** e0ne has quit IRC | 04:18 | |
| StevenK | jeblair, clarkb, fungi: So grub-legacy-ec2 and cloud-init are both built from the cloud-init source package, but the 0.6.3-0ubuntu1.10 changes only look to touch smartos | 04:18 |
| fungi | 70 thousand hits in less than 4 hours? that doesn't even seem possible | 04:20 |
| clarkb | its not 70k jobs just matches | 04:21 |
| fungi | oh, many recurrent lines in each job | 04:21 |
| fungi | yeah, i'm curious to see who fixes what to resolve it, but i doubt there's much we can do... i suppose devstack itself could pin a specific package version | 04:22 |
| *** ryanpetrello has quit IRC | 04:23 | |
| *** UtahDave has joined #openstack-infra | 04:23 | |
| fungi | as long as the old version is still retrievable from the apt repos | 04:23 |
| StevenK | fungi: On archive.ubuntu.com? | 04:23 |
| fungi | StevenK: i believe we hit apt mirrors within our cloud providers | 04:24 |
| fungi | so would probably need to be there | 04:24 |
| StevenK | 0.6.3-0ubuntu1.9 is already gone from primary | 04:24 |
| *** jcooley_ has quit IRC | 04:24 | |
| StevenK | It's still fetchable from Launchpad | 04:25 |
| fungi | no idea how quickly the cloud providers expire previoys package versions from their mirrors | 04:25 |
| StevenK | fungi: It's usually not up to them | 04:25 |
| *** jp_at_hp has quit IRC | 04:25 | |
| StevenK | If they just mirror what is on their upstream mirror | 04:25 |
| StevenK | And I'd be surprised if mirror operators choose to keep expired binaries around, given current mirror churn | 04:26 |
| StevenK | fungi: You can't just dpkg -i it and then hold it? | 04:27 |
| fungi | StevenK: devstack probably could | 04:28 |
| StevenK | That seems like the wrong layer, to my mind | 04:29 |
| fungi | devstack installs those packages. why would it be the wrong layer? | 04:29 |
| StevenK | Ah | 04:30 |
| fungi | at least i assume they're in the apts list in devstack, but i'll confirm that | 04:30 |
| StevenK | I thought they were installed into the images | 04:30 |
| fungi | no, because different branches of devstack can want different packages installed | 04:31 |
| *** harlowja_ is now known as harlowja_away | 04:31 | |
| fungi | depending on whether it's grizzly, havana or icehouse devstack | 04:32 |
| *** stevemar has joined #openstack-infra | 04:33 | |
| StevenK | fungi: I can not find any mention of grub-legacy-ec2 or an install of cloud-init in devstack | 04:33 |
| fungi | yeah, i'm looking as well | 04:34 |
| fungi | unless they're dependencies of something else in the apts lists | 04:34 |
| fungi | checking a setup log to see | 04:34 |
| *** wenlock has quit IRC | 04:35 | |
| StevenK | On my saucy system, grub-legacy-ec2 has no rdeps | 04:35 |
| *** vogxn has quit IRC | 04:35 | |
| stevemar | anyone know whats going on with the gate failures | 04:36 |
| StevenK | And the only rdep of cloud-init is ec2-init | 04:36 |
| *** lcheng_ has joined #openstack-infra | 04:36 | |
| fungi | no references to either of those packages in openstack-infra/config or openstack-infra/devstack-gate either | 04:37 |
| fungi | stevemar: not yet, but we welcome anyone figuring it out | 04:37 |
| *** amcrn has quit IRC | 04:37 | |
| StevenK | fungi: I think they're both on the cloud image base | 04:37 |
| StevenK | I think they come with cloud-init at least | 04:37 |
| StevenK | But it's been years since I poked at the cloud image build machinery | 04:38 |
| fungi | seems to just be tempest jobs and not tempest large-ops | 04:38 |
| *** david_lyle_ has joined #openstack-infra | 04:38 | |
| *** amcrn has joined #openstack-infra | 04:38 | |
| StevenK | And I don't think I have a local copy of the precise seeds ... | 04:38 |
| *** saju_m has joined #openstack-infra | 04:39 | |
| fungi | StevenK: i can't imagine we'd be doing anything to interface with either of those packages if we're not declaring that they should be installed... if devstack/tempest really do anything with them then they can't just assume anyone running those tools will already have either of those packages | 04:39 |
| *** david-lyle has quit IRC | 04:39 | |
| fungi | so it could be a red herring | 04:39 |
| StevenK | fungi: Sure, I'm investigating since it was pointed it out that was the only source package that changed | 04:40 |
| StevenK | And when it was accepted into -updates fits the timeline | 04:40 |
| *** Sukhdev has joined #openstack-infra | 04:42 | |
| StevenK | fungi: Right, cloud-init and grub-legacy-ec2 are in the cloud image itself -- http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ubuntu.precise/view/head:/cloud-image | 04:45 |
| fungi | yeah, both those files were present on a bare-precise node i grabbed | 04:45 |
| fungi | so nothing devstack pulls in would have even ended up on that node type | 04:45 |
| fungi | more important, i think, is to get nova and/or qa devs to work out why it would be causing breakage | 04:46 |
| fungi | we don't want to prematurely "fix" this from the infra side of things and reduce the urgency of someone looking into the underlying cause | 04:47 |
| fungi | because whatever this is, it probably just broke for everyone running a devstack with ubuntu unattended updates | 04:47 |
| fungi | or worse, everyone running an openstack with ubuntu unattended updates | 04:47 |
| *** sarob has joined #openstack-infra | 04:48 | |
| fungi | having the gate humming along quietly while the rest of the world burns isn't a great place to focus efforts | 04:49 |
| clarkb | ++ | 04:49 |
| *** fandi has joined #openstack-infra | 04:49 | |
| fungi | given the error pattern in the console log for failing jobs, unless i'm looking at the wrong failures it seems like it's tempest tests which rely on launching new nodes and nova not reporting those nodes as launched successfully (over and over and over) | 04:51 |
| fungi | so i concur that if the host copy of cloud-init or grub-legacy-ec2 is being embedded into the cirros images it's launching, that could maybe explain it | 04:52 |
| *** chandan_kumar has quit IRC | 04:52 | |
| *** bodepd has quit IRC | 04:52 | |
| *** bodepd has joined #openstack-infra | 04:53 | |
| lifeless | fungi: cloud provider ubuntu mirrors? | 04:53 |
| *** sarob has quit IRC | 04:53 | |
| fungi | or, rather, inserted into those vms as they're being created | 04:53 |
| lifeless | fungi: therein lies a tae :) | 04:53 |
| lifeless | s/tae/tail/ | 04:53 |
| *** sarob has joined #openstack-infra | 04:53 | |
| lifeless | fungi: Canonical tries to arrange to run the Ubuntu mirror themselves, as part of the source-ip-mapped cloud-archive.ubuntu.com | 04:53 |
| lifeless | so you get a geographically local mirror in any public cloud | 04:54 |
| lifeless | fungi: but. politics. | 04:54 |
| fungi | lifeless: oh, so http://mirror.rackspace.com/ubuntu/ is actually just mapping to an ubuntu-managed pool somewhere? | 04:54 |
| SergeyLukjanov | morning! | 04:55 |
| fungi | (using http rewrites i guess, since there's probably also non-ubuntu stuff being hosted from that same domain name) | 04:55 |
| lifeless | fungi: no; there's a special dns name for it (I forget what it is) and generally ubuntu cloud images have that | 04:55 |
| lifeless | fungi: and ubuntu have a dns server with public range views on it | 04:56 |
| fungi | lifeless: oh, we're not running ubuntu cloud images, necessarily. we're running rackspace (or whatever) images of ubuntu | 04:56 |
| lifeless | it might even be on archive.u.c now, | 04:56 |
| * lifeless shrugs | 04:56 | |
| lifeless | fungi: yes, running the reference image in e.g. aws this happens, AIUI. | 04:56 |
| lifeless | fungi: StevenK: is probably more up to date, or we could ask smoser | 04:57 |
| fungi | in the tripleo-ci cloud we have actual ubuntu cloud images, because you have glance | 04:57 |
| lifeless | fyes :) | 04:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove taskflow sqlalchemy 0.9 testing https://review.openstack.org/73445 | 04:57 |
| lifeless | fungi: erm, yes, but you saw my reference to politics | 04:57 |
| fungi | rackspace seems to doctor their images with inconsistent sets of packages, links to their own local mirror package inside rackspace, et cetera | 04:58 |
| clarkb | have we confirmed the problem is related to those packages? can we throw up a simple devsatck change that downgrades them first? | 05:00 |
| fungi | clarkb: no confirmation has been done. this was just a basic timing correlation i think | 05:00 |
| clarkb | ya seems likely just wondering if we had managed to confirm | 05:01 |
| *** talluri has quit IRC | 05:01 | |
| clarkb | notmyname: https://review.openstack.org/#/c/73447/1 I get removing the py33 job, but isn't swiftclient used in the functional tests? | 05:04 |
| fungi | i would think a simple devstack patch to d/l the previous versions of those from archive.u.c and dpkg -i them into the system before starting services | 05:04 |
| fungi | would be pretty quick to try | 05:04 |
| notmyname | clarkb: because https://github.com/openstack/swift/commit/1f3ae6d8dadef838f1ea7fa571339932074fbe38 | 05:05 |
| notmyname | clarkb: swift does not use python-swiftclient any more, so the swift functional tests don't even exercise any python-swiftclient code | 05:06 |
| clarkb | notmyname: gotcha | 05:07 |
| notmyname | clarkb: and the main reason for pulling swiftclient out of swift as a dependency is so that we can do revs of swiftclient with py3 support without pulling in a bunch of transitive dependencies to swift that swift can't possibly use (because swift doesn't support py3) | 05:07 |
| notmyname | clarkb: so ya, it used to be there. you aren't misremembering. the final "remove swiftclient" patch only landed very recently (~ a month ago) | 05:08 |
| clarkb | ok +2 from me | 05:08 |
| notmyname | clarkb: thanks :-) | 05:09 |
| notmyname | clarkb: we'll add the py33 stuff back whenever we can actually have a hope of it passing :-) | 05:10 |
| notmyname | clarkb: did you see my note about python-swiftclient releases earlier this afternoon? | 05:10 |
| *** jcooley_ has joined #openstack-infra | 05:13 | |
| *** e0ne has joined #openstack-infra | 05:13 | |
| clarkb | I saw them, not sure I processed them | 05:13 |
| clarkb | should I reread them? | 05:13 |
| *** chandan_kumar has joined #openstack-infra | 05:14 | |
| clarkb | looks like it went smoothly \o/ | 05:14 |
| fungi | StevenK: worth noting, the machines i'm looking at, some of them upgraded grub-legacy-ec2 almost 24 hours ago | 05:14 |
| fungi | so in theory this should have been biting us much, much earlier in the day | 05:14 |
| notmyname | clarkb: just that if everything blows up by the am, maybe cert checking in swiftclient broke something unexpected :-) | 05:14 |
| notmyname | clarkb: although I strongly suspect that won't happen ;-) | 05:15 |
| fungi | StevenK: 2014-02-13 06:36:35 upgrade grub-legacy-ec2 0.6.3-0ubuntu1.9 0.6.3-0ubuntu1.10 | 05:15 |
| fungi | StevenK: (that's utc time) | 05:15 |
| notmyname | clarkb: I jsut finished my announce email. I'll send it out tomorrow | 05:15 |
| clarkb | notmyname: sounds good thanks | 05:15 |
| StevenK | fungi: Hm | 05:16 |
| StevenK | So about 23 hours ago | 05:16 |
| StevenK | Why does that now smell of red and herring | 05:17 |
| fungi | StevenK: that was gleaned from the dpkg.log of a random static host, but since devstack does an apt-get upgrade when it starts, it should have been finding it at least that long ago too | 05:17 |
| *** nicedice has quit IRC | 05:17 | |
| *** e0ne has quit IRC | 05:18 | |
| fungi | StevenK: do you know whether anyone's compared the pip freeze between failing and succeeding runs? maybe one of our actual dependencies had a new release a few hours back | 05:21 |
| StevenK | fungi: I don't think so. | 05:22 |
| clarkb | swiftlcient! >_> | 05:22 |
| StevenK | Haha | 05:22 |
| notmyname | wait what? | 05:22 |
| notmyname | :-) | 05:22 |
| notmyname | clarkb: fungi: StevenK: something I need to look at? swiftclient messing up? | 05:25 |
| fungi | notmyname: doubtful | 05:25 |
| notmyname | fungi: FWIW, the swiftclient change happened at Thu Feb 13 2014 16:41:39 GMT-0800 (PST) | 05:26 |
| *** dkliban has joined #openstack-infra | 05:27 | |
| notmyname | so about 5 hours ago | 05:27 |
| fungi | notmyname: too early for what we're looking for anyway, sounds like | 05:27 |
| fungi | oh, pst | 05:27 |
| *** talluri has joined #openstack-infra | 05:27 | |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: Quote group names in case of spaces https://review.openstack.org/72188 | 05:28 |
| fungi | well, maybe. seems unlikely it could be involved unless glance is using swiftclient to get cirros images into and out of a swift backing store and the error is bubbling up as vms failing to launch | 05:28 |
| *** talluri has joined #openstack-infra | 05:29 | |
| *** hdd has quit IRC | 05:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update gerrit project rename instructions https://review.openstack.org/71948 | 05:30 |
| notmyname | fungi: I don't know enough about that process to know if it's unlikely or not | 05:31 |
| fungi | nor do i, i'm afraid | 05:32 |
| notmyname | fungi: eg https://jenkins07.openstack.org/job/gate-devstack-dsvm-cells/491/console ? | 05:32 |
| *** chris_johnson has joined #openstack-infra | 05:32 | |
| notmyname | fungi: does it fail on your local devstack? if so, try manually pinning the python-swiftclient version at <2.0 and see if that fixes it. if it still fails, even once, you know that's not the issue | 05:34 |
| fungi | notmyname: hmm, that one looks different. what i was looking at was http://logs.openstack.org/51/71851/2/check/check-tempest-dsvm-full/6a30552/console.html | 05:34 |
| fungi | but anyway, i should shuffle off to bed and leave this to people who have daylight upon them | 05:35 |
| zaro | fungi: is it possible to tell gerrit gitprep to clone into sub directory? | 05:36 |
| clarkb | zaro: I don't think so, I believe the git clone $URL . is hard coded. You could probably make it a parameter with a default value of . though | 05:37 |
| clarkb | then change it for the gerrit jobs | 05:37 |
| zaro | ok. let me look into that. buck and gerrit dir needs to be at same level. | 05:38 |
| fungi | zaro: well, your job could move the contents of the curdir into a subdir before grabbing buck | 05:38 |
| fungi | as an alternative | 05:39 |
| fungi | i want to say we have at least one job which does that... maybe the jjb xml comparison job? | 05:39 |
| *** sarob has quit IRC | 05:41 | |
| *** vogxn has joined #openstack-infra | 05:41 | |
| *** sarob has joined #openstack-infra | 05:41 | |
| *** sarob has joined #openstack-infra | 05:41 | |
| *** garyk1 has joined #openstack-infra | 05:42 | |
| *** sarob has quit IRC | 05:42 | |
| *** chris_johnson has quit IRC | 05:43 | |
| *** garyk has quit IRC | 05:44 | |
| *** AlexF has joined #openstack-infra | 05:44 | |
| zaro | fungi: hmm that's a good idea but does that work on subsequent job runs? because gerrit git prep will run again right? | 05:44 |
| openstackgerrit | A change was merged to openstack-infra/gear: Solve hacking/pbr requirements conflict https://review.openstack.org/72767 | 05:45 |
| clarkb | fungi: ^ I just approved a couple gaer changes | 05:45 |
| clarkb | pretty sure we consume that via releases so +2 | 05:46 |
| openstackgerrit | A change was merged to openstack-infra/gear: Remove tox locale overrides https://review.openstack.org/72216 | 05:46 |
| fungi | looks like jgriffith opened bug 1280072 against nova for our new gate blocker | 05:48 |
| fungi | clarkb: cool | 05:48 |
| *** HenryG has quit IRC | 05:48 | |
| fungi | zaro: i think gerrit-git-prep will blow away and reclone the workspace contents if it can't do a remote update in them, but it's getting late so i could be wrong | 05:50 |
| clarkb | zaro: fungi: yeah it does a git clean -xfd | 05:51 |
| fungi | clarkb: well, git clean won't buy you much if you also move the .git dir, but maybe skipping dotfiles when moving (mv * newdir/) would solve that too | 05:52 |
| clarkb | oh right, it starts from scratch if there is no .git | 05:52 |
| *** Sukhdev has quit IRC | 05:53 | |
| fungi | though you know what, optimizing for reusing this workspace is kinda pointless. this will run on a single-use node anyway | 05:53 |
| clarkb | oh right | 05:53 |
| *** dstanek has joined #openstack-infra | 05:54 | |
| *** chandankumar_ has joined #openstack-infra | 05:54 | |
| clarkb | zaro: ^ so I woudln't worry about it a whole lot | 05:55 |
| clarkb | single use nodes make it mostly unnecessary to worry about the second time a test runs there | 05:55 |
| fungi | i would have pushed the change today to switch all other non-sensitive jobs to single-use, but ran out of round tuits | 05:55 |
| clarkb | and still making me look bad :P | 05:56 |
| clarkb | tomorrow is going to be a catch up on review day since I think the logstash fires are mostly put out | 05:56 |
| fungi | i would love to catch up on review | 05:56 |
| clarkb | I do need to scale back the number of workers per node though and start puppet again, but that is easy | 05:56 |
| *** orionNebula has quit IRC | 05:56 | |
| fungi | i was staying up to catch up on e-mail so that hopefully tomorrow will be a get-shit-done day not a i-can't-figure-out-what-all-i'm-still-behind-on day | 05:57 |
| fungi | so with that in mind, i'm going to get sleep done, and try to make an early start of it | 05:57 |
| clarkb | good night | 05:58 |
| *** saju_m has quit IRC | 05:58 | |
| fungi | also need to take the gf out to lunch or something, in honor of j.g. hallmark's pocketbook | 05:58 |
| fungi | assuming the roads are clear enough | 05:58 |
| clarkb | fungi: before you go | 05:58 |
| * fungi nods | 05:59 | |
| zaro | clarkb: good night. thanks. | 05:59 |
| clarkb | fungi: is the mysql permissions thing on slaves fixed? and if so can you point me at what the problem is/was | 05:59 |
| zaro | clarkb: i'll bring the dvd for you tomorrow. | 05:59 |
| *** HenryG has joined #openstack-infra | 05:59 | |
| clarkb | zaro: nah its ok I don't need it | 06:00 |
| fungi | clarkb: i think it's fixed now but haven't rechecked. patch was https://review.openstack.org/72799 | 06:00 |
| clarkb | zaro: I got it sorted, yazz helped out | 06:00 |
| clarkb | fungi: danke | 06:00 |
| *** chandankumar_ has quit IRC | 06:00 | |
| *** dstanek has quit IRC | 06:03 | |
| *** dstanek has joined #openstack-infra | 06:09 | |
| *** AlexF has quit IRC | 06:11 | |
| *** AlexF has joined #openstack-infra | 06:13 | |
| *** vogxn has quit IRC | 06:13 | |
| *** e0ne has joined #openstack-infra | 06:13 | |
| *** chandan_kumar has quit IRC | 06:15 | |
| *** CaptTofu has joined #openstack-infra | 06:17 | |
| *** iv_m has quit IRC | 06:18 | |
| *** e0ne has quit IRC | 06:18 | |
| *** vkozhukalov has joined #openstack-infra | 06:21 | |
| *** CaptTofu has quit IRC | 06:21 | |
| *** AlexF has quit IRC | 06:23 | |
| *** jcooley_ has quit IRC | 06:23 | |
| *** AlexF has joined #openstack-infra | 06:24 | |
| *** dkliban has quit IRC | 06:30 | |
| *** chandan_kumar has joined #openstack-infra | 06:32 | |
| *** sarob has joined #openstack-infra | 06:43 | |
| *** yolanda has joined #openstack-infra | 06:44 | |
| *** AlexF has quit IRC | 06:47 | |
| *** sarob has quit IRC | 06:48 | |
| *** mrmartin has joined #openstack-infra | 06:51 | |
| *** AlexF has joined #openstack-infra | 06:54 | |
| *** maurosr has joined #openstack-infra | 07:04 | |
| *** dstanek has quit IRC | 07:08 | |
| *** stevemar has quit IRC | 07:08 | |
| *** chandan_kumar has quit IRC | 07:18 | |
| *** chandan_kumar has joined #openstack-infra | 07:21 | |
| *** ociuhandu has joined #openstack-infra | 07:21 | |
| *** afazekas has quit IRC | 07:21 | |
| *** saju_m has joined #openstack-infra | 07:25 | |
| *** gokrokve_ has quit IRC | 07:31 | |
| *** gokrokve has joined #openstack-infra | 07:31 | |
| *** denis_makogon has joined #openstack-infra | 07:33 | |
| *** vkozhukalov has quit IRC | 07:33 | |
| *** dstanek has joined #openstack-infra | 07:34 | |
| *** gokrokve has quit IRC | 07:36 | |
| *** AlexF has quit IRC | 07:38 | |
| *** yolanda has quit IRC | 07:38 | |
| *** dstanek has quit IRC | 07:38 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix wrong pathname in source/target for ops-guide https://review.openstack.org/73521 | 07:39 |
| *** mrmartin has quit IRC | 07:40 | |
| *** yolanda has joined #openstack-infra | 07:41 | |
| *** markwash has joined #openstack-infra | 07:42 | |
| *** sarob has joined #openstack-infra | 07:43 | |
| *** afazekas has joined #openstack-infra | 07:48 | |
| *** lcheng_ has quit IRC | 07:49 | |
| *** sarob has quit IRC | 07:49 | |
| notmyname | fungi: StevenK: clarkb: full disclosure. we found an issue with python-swiftclient 2.0. it's now been fixed. finishing up last jenkins test now and then I'll tag 2.0.1. see if that clears up any gate issues | 07:57 |
| clarkb | notmyname: https://review.openstack.org/#/c/70378/ btw | 07:59 |
| clarkb | if that does fix the gate then ^ should prevent such fails in the futre | 07:59 |
| notmyname | clarkb: yes, absolutely | 07:59 |
| notmyname | (/me is happy that's not in now to help with getting patches landed to fix stuff) | 07:59 |
| notmyname | clarkb: ok, our patch has landed. tagging now. can you rerun some of the failing gate jobs to see if they clear up? | 08:00 |
| clarkb | notmyname: there seems to be a steady rate of test in the check queue, I think we will know soon enough | 08:00 |
| clarkb | https://jenkins02.openstack.org/job/check-tempest-dsvm-full/9104/ I want to say that test started late enough to pick up the swiftlclient change | 08:01 |
| notmyname | clarkb: I just now pushed the tag | 08:02 |
| notmyname | clarkb: ok, pypi just now got it https://pypi.python.org/pypi/python-swiftclient/2.0.1 | 08:02 |
| *** bhuvan has quit IRC | 08:03 | |
| AJaeger | notmyname: not yet on http://pypi.openstack.org/openstack/python-swiftclient/ | 08:04 |
| *** bhuvan has joined #openstack-infra | 08:04 | |
| notmyname | clarkb: and I'll be honset :-) https://twitter.com/notmyname/status/434218642641649664 | 08:04 |
| notmyname | *honest | 08:05 |
| notmyname | AJaeger: do you know how long that will take? or is it simply lazy-loaded when needed? | 08:06 |
| clarkb | it is probably an hour out | 08:06 |
| clarkb | but integrstion tests are from source so no rush | 08:06 |
| notmyname | does that mean patches won't pick it up until then? | 08:06 |
| clarkb | the test I linked should use it | 08:07 |
| AJaeger | clarkb, is there any chance to get my stupid typo fix reviewed, please? https://review.openstack.org/73521 | 08:07 |
| clarkb | AJaeger: done | 08:09 |
| AJaeger | clarkb, thanks a lot! | 08:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix wrong pathname in source/target for ops-guide https://review.openstack.org/73521 | 08:12 |
| *** bhuvan_ has joined #openstack-infra | 08:12 | |
| *** bhuvan has quit IRC | 08:12 | |
| *** unicell has joined #openstack-infra | 08:14 | |
| *** mrmartin has joined #openstack-infra | 08:14 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Ruby metrics plugin support added https://review.openstack.org/73528 | 08:15 |
| unicell | anyone around? | 08:15 |
| unicell | There're so many gate jobs failed https://bugs.launchpad.net/nova/+bug/1280072 | 08:16 |
| unicell | seems to be caused by VM boot failure with a "qemu: linux kernel too old to load a ram disk“ in n-cpu log | 08:17 |
| *** CaptTofu has joined #openstack-infra | 08:17 | |
| *** flaper87 has quit IRC | 08:18 | |
| *** flaper87 has joined #openstack-infra | 08:18 | |
| unicell | Just wonder if could some infra change causing this issue? | 08:18 |
| clarkb | unicell read the scrollback. initial analysis is no not an infra change | 08:19 |
| clarkb | but most of us are either sleeping or should be sleeping so feel free to investigate | 08:19 |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Ruby metrics plugin support added https://review.openstack.org/73528 | 08:19 |
| unicell | clarkb: will try to dig into it. free free to sleep. :) | 08:20 |
| unicell | s/free free/feel free | 08:20 |
| *** shardy_afk is now known as shardy | 08:21 | |
| *** CaptTofu has quit IRC | 08:22 | |
| *** skraynev_afk is now known as skraynev | 08:25 | |
| notmyname | clarkb: well, it seems that jobs are still failing. ones that should have swiftclient 2.0.1 | 08:26 |
| *** markwash has quit IRC | 08:27 | |
| *** yamahata has quit IRC | 08:27 | |
| unicell | notmyname: anything to do with the swiftclient? | 08:31 |
| *** ociuhandu has quit IRC | 08:32 | |
| notmyname | unicell: I have no hard evidence that any change in python-swiftclient has caused the gate issues. however, there is some suspicious timing with a swiftclient release done several hours ago. a fix was just pushed about an hour ago, though | 08:33 |
| *** ok_delta has joined #openstack-infra | 08:36 | |
| *** dstanek has joined #openstack-infra | 08:37 | |
| unicell | notmyname: this gate failure seems be affecting both nova and neutron side. the earliest failure I can track is about 6 hours ago | 08:37 |
| *** loquacities has quit IRC | 08:40 | |
| *** morganfainberg_Z has quit IRC | 08:42 | |
| *** gokrokve has joined #openstack-infra | 08:42 | |
| *** loquacities has joined #openstack-infra | 08:42 | |
| *** sarob has joined #openstack-infra | 08:44 | |
| *** gokrokve has quit IRC | 08:46 | |
| *** morganfainberg_Z has joined #openstack-infra | 08:47 | |
| *** morganfainberg_Z is now known as morganfainberg | 08:47 | |
| *** sarob has quit IRC | 08:49 | |
| *** rlandy has joined #openstack-infra | 08:50 | |
| *** mkoderer has quit IRC | 08:54 | |
| *** mkoderer has joined #openstack-infra | 08:55 | |
| *** markmc has joined #openstack-infra | 08:56 | |
| *** dstanek has quit IRC | 08:57 | |
| *** hemna has quit IRC | 08:59 | |
| *** jgallard has joined #openstack-infra | 09:01 | |
| *** vkozhukalov has joined #openstack-infra | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** bhuvan_ has quit IRC | 09:11 | |
| *** bhuvan has joined #openstack-infra | 09:12 | |
| *** derekh has joined #openstack-infra | 09:13 | |
| *** yassine has joined #openstack-infra | 09:13 | |
| *** jp_at_hp has joined #openstack-infra | 09:13 | |
| *** hemna has joined #openstack-infra | 09:14 | |
| *** marun has joined #openstack-infra | 09:16 | |
| *** hashar has joined #openstack-infra | 09:18 | |
| *** marun has quit IRC | 09:21 | |
| *** macdomat has joined #openstack-infra | 09:23 | |
| macdomat | Hi all, re gate job, could it be a re-introduction of https://bugs.launchpad.net/nova/+bug/607355 in some form (unless you already have the answer!) | 09:24 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Basic gating and release jobs of openstackid project https://review.openstack.org/73549 | 09:25 |
| *** jroovers|afk has joined #openstack-infra | 09:26 | |
| *** bhuvan has quit IRC | 09:27 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 09:27 | |
| *** gilliard has joined #openstack-infra | 09:28 | |
| *** bhuvan has joined #openstack-infra | 09:28 | |
| *** UtahDave has quit IRC | 09:33 | |
| *** amcrn has quit IRC | 09:34 | |
| *** DinaBelova_ is now known as DinaBelova | 09:35 | |
| *** fbo_away is now known as fbo | 09:38 | |
| *** DeeJay1 has joined #openstack-infra | 09:38 | |
| *** sarob has joined #openstack-infra | 09:44 | |
| *** mrfatstrat has joined #openstack-infra | 09:44 | |
| *** jooools has joined #openstack-infra | 09:46 | |
| *** sarob_ has joined #openstack-infra | 09:46 | |
| *** sarob has quit IRC | 09:49 | |
| *** mrfatstrat has quit IRC | 09:49 | |
| *** mrmartin has quit IRC | 09:50 | |
| *** sarob_ has quit IRC | 09:51 | |
| *** rossella_s has joined #openstack-infra | 09:51 | |
| *** saju_m has quit IRC | 09:51 | |
| *** vkozhukalov has quit IRC | 09:53 | |
| *** macdomat has quit IRC | 09:56 | |
| openstackgerrit | Qiu Yu proposed a change to openstack/requirements: Temporarily cap swiftclient version to fix upload error https://review.openstack.org/73560 | 09:57 |
| openstackgerrit | Qiu Yu proposed a change to openstack/requirements: Temporarily cap swiftclient version to fix upload error https://review.openstack.org/73560 | 09:58 |
| *** zhiyan is now known as zhiyan_ | 09:59 | |
| *** denis_makogon has quit IRC | 10:03 | |
| *** vkozhukalov has joined #openstack-infra | 10:05 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 10:07 |
| *** hemna has quit IRC | 10:08 | |
| *** mrfatstrat has joined #openstack-infra | 10:10 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 10:11 |
| *** b3nt_pin has joined #openstack-infra | 10:16 | |
| *** max_lobur_afk is now known as max_lobur | 10:17 | |
| *** CaptTofu has joined #openstack-infra | 10:18 | |
| openstackgerrit | Qiu Yu proposed a change to openstack/requirements: Temporarily cap swiftclient version to fix glance gate failure https://review.openstack.org/73560 | 10:19 |
| *** mrfatstrat has quit IRC | 10:19 | |
| *** hemna has joined #openstack-infra | 10:21 | |
| *** nosnos has quit IRC | 10:21 | |
| *** CaptTofu has quit IRC | 10:23 | |
| *** hashar has quit IRC | 10:23 | |
| *** hashar has joined #openstack-infra | 10:25 | |
| *** therve has joined #openstack-infra | 10:27 | |
| *** bhuvan has quit IRC | 10:31 | |
| *** bhuvan has joined #openstack-infra | 10:31 | |
| mika | jeblair: regarding the jjb issue I asked, a friend of mine came up with https://github.com/linuxmaniac/jenkins-job-builder/commit/3ca77a761fde77c383625d13d531455bdebf3fb8 - maybe you want to have a look at that and let us know whether we should take care of submitting that for review in upstream? | 10:31 |
| therve | The gate seems to be failing with "qemu: linux kernel too old to load a ram disk". Is it a known problem? | 10:32 |
| unicell | therve: https://bugs.launchpad.net/openstack-ci/+bug/1280072 | 10:33 |
| *** david_lyle_ has quit IRC | 10:33 | |
| *** dstanek has joined #openstack-infra | 10:34 | |
| therve | unicell, So it's a swift problem which corrupts the ram disk? | 10:35 |
| unicell | therve: I believe it is the new swiftclient version breaks the glance image upload | 10:37 |
| unicell | thus vm failed to boot with corrupted image | 10:37 |
| therve | Nice :/ | 10:37 |
| *** dstanek has quit IRC | 10:39 | |
| *** boris-42_ has quit IRC | 10:43 | |
| *** yamahata has joined #openstack-infra | 10:43 | |
| *** sarob has joined #openstack-infra | 10:44 | |
| *** sarob has quit IRC | 10:48 | |
| *** yaguang has quit IRC | 10:52 | |
| *** jatt has joined #openstack-infra | 10:54 | |
| tristanC | unicell: swiftclient is now using python-requests, this got merged 10 hours ago. I'm trying to understand what's going on... | 10:56 |
| *** ociuhandu has joined #openstack-infra | 10:57 | |
| tristanC | just did a swift upload of a 40Mo file with the same requests version tempest is using (2.1.0) and it worked | 11:07 |
| tristanC | now trying to do that with glance | 11:07 |
| openstackgerrit | Steven Hardy proposed a change to openstack-infra/reviewstats: Add jasondunsmore to heat-core https://review.openstack.org/73570 | 11:09 |
| *** ociuhandu has quit IRC | 11:12 | |
| *** ok_delta has quit IRC | 11:12 | |
| *** jgallard has quit IRC | 11:13 | |
| tristanC | alright, I can reproduce the corruption bug on glance download | 11:16 |
| *** CaptTofu has joined #openstack-infra | 11:17 | |
| *** mrfatstrat has joined #openstack-infra | 11:18 | |
| *** CaptTofu has quit IRC | 11:18 | |
| *** CaptTofu has joined #openstack-infra | 11:19 | |
| *** leifmadsen has quit IRC | 11:19 | |
| *** CaptTofu has quit IRC | 11:20 | |
| *** CaptTofu has joined #openstack-infra | 11:20 | |
| *** CaptTofu has quit IRC | 11:21 | |
| *** CaptTofu has joined #openstack-infra | 11:21 | |
| *** jatt has quit IRC | 11:21 | |
| *** jroovers|afk has quit IRC | 11:23 | |
| *** jroovers has joined #openstack-infra | 11:23 | |
| *** CaptTofu has quit IRC | 11:24 | |
| *** leifmadsen has joined #openstack-infra | 11:25 | |
| *** jhesketh__ has quit IRC | 11:27 | |
| *** CaptTofu has joined #openstack-infra | 11:27 | |
| *** jang has joined #openstack-infra | 11:29 | |
| *** CaptTofu has quit IRC | 11:32 | |
| *** rfolco has joined #openstack-infra | 11:32 | |
| *** CaptTofu has joined #openstack-infra | 11:32 | |
| *** ArxCruz has joined #openstack-infra | 11:34 | |
| *** yamahata has quit IRC | 11:37 | |
| *** fandi has quit IRC | 11:41 | |
| *** alexpilotti has joined #openstack-infra | 11:41 | |
| *** dims_ has quit IRC | 11:42 | |
| *** sarob has joined #openstack-infra | 11:44 | |
| *** jroovers|afk has joined #openstack-infra | 11:46 | |
| *** mrfatstrat has quit IRC | 11:47 | |
| *** jroovers has quit IRC | 11:48 | |
| *** sarob has quit IRC | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:49 | |
| *** apevec has joined #openstack-infra | 11:51 | |
| apevec | afazekas, ping - any ideas about http://lists.openstack.org/pipermail/openstack-stable-maint/2014-February/002154.html | 11:51 |
| apevec | as ihrachys found libvirtd has issues, but it's not libvirt, it's the same version as day before | 11:52 |
| apevec | how can I see versions of all packages on the test node? | 11:52 |
| apevec | I want to compare Feb 13 (known good) vs Feb 14 (bad) | 11:52 |
| apevec | e.g. http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-havana/411ac46/ (Feb13) | 11:53 |
| apevec | vs http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-havana/3cd8ddc/ (failure today) | 11:54 |
| *** dims_ has joined #openstack-infra | 11:54 | |
| afazekas | apavec: from the console log: http://paste.openstack.org/show/65401/ | 11:54 |
| apevec | ok found it, dpkg-l.txt | 11:54 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Refactor REST API to work on top of DB API https://review.openstack.org/72357 | 11:54 |
| *** boris-42_ has joined #openstack-infra | 11:56 | |
| apevec | afazekas, no changes in any package version | 11:56 |
| afazekas | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcInZpckV4ZWNXaXRoSG9va1wiIGFuZCBtZXNzYWdlOiBcInBtLWlzLXN1cHBvcnRlZFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJhbGwiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkyMzc4OTM0MzUzfQ== | 11:56 |
| afazekas | this message can been found in earlier logs | 11:57 |
| apevec | afazekas, ok, that's updated package, but still end dpkg -l shows no diffs between good and bad run | 11:57 |
| *** chandan_kumar has quit IRC | 11:58 | |
| apevec | so it's not power mgmt | 11:58 |
| apevec | so it's qemu: linux kernel too old to load a ram disk | 11:58 |
| *** talluri has quit IRC | 11:59 | |
| *** talluri has joined #openstack-infra | 12:00 | |
| * afazekas searching for SUCCESSFUL job with this message | 12:00 | |
| apevec | afazekas, there are two linux-image installed in both runs, now which one was running in each case? | 12:01 |
| apevec | is uname -r logged anywhere in devstack? | 12:02 |
| *** talluri has quit IRC | 12:04 | |
| apevec | afazekas, one diff I noticed: Feb13 node was devstack-precise-hpcloud-az2-1367798 | 12:09 |
| apevec | Feb14 devstack-precise-1391998468 | 12:09 |
| *** andre__ has joined #openstack-infra | 12:09 | |
| apevec | is that from different cloud? | 12:09 |
| apevec | SergeyLukjanov, ^^^ ? | 12:10 |
| SergeyLukjanov | apevec, reading scrollback | 12:10 |
| apevec | tl;dr - all tempest periodic jobs failed this morning, trying to figure out why | 12:11 |
| apevec | from libvirtd.logo looks like different kernel | 12:11 |
| SergeyLukjanov | apevec, ok, got it | 12:12 |
| apevec | and now gate is failing too https://review.openstack.org/73402 | 12:12 |
| *** hashar has quit IRC | 12:12 | |
| SergeyLukjanov | apevec, anyway thd one more diff is jenkins07 / jenkins02 | 12:13 |
| *** banix has joined #openstack-infra | 12:13 | |
| *** dpyzhov has joined #openstack-infra | 12:13 | |
| apevec | what is difference among all our jenkins' ? | 12:14 |
| SergeyLukjanov | apevec, it should be no difference ;) | 12:14 |
| SergeyLukjanov | apevec, except slave pools | 12:14 |
| apevec | right, slave where worked yesterday was hpcloud | 12:14 |
| ihrachys | libvirt machine settings that failed: http://logs.openstack.org/02/73402/1/gate/gate-tempest-dsvm-full/ba93148/logs/screen-n-cpu.txt.gz?level=WARNING Anything suspicious? | 12:14 |
| apevec | not sure which one is today? | 12:15 |
| markmc | apevec, fwiw, it'd be good to file a bug and track your findings there | 12:15 |
| SergeyLukjanov | apevec, due to the | 12:15 |
| SergeyLukjanov | 15:53:41] <apevec> e.g. http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-havana/411ac46/ (Feb13) | 12:15 |
| SergeyLukjanov | [15:54:05] <apevec> vs http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-havana/3cd8ddc/ (failure today) | 12:15 |
| markmc | apevec, even for 'reverify bug ###' later | 12:15 |
| SergeyLukjanov | both slaves are hpcloud | 12:16 |
| afazekas | 'qemu: linux kernel too old to load a ram disk' | 12:16 |
| apevec | ihrachys, please file LP for this, you noticed it first | 12:16 |
| apevec | ihrachys, so you get all the glory :) | 12:16 |
| apevec | afazekas, right, that's what kills qemu process | 12:17 |
| apevec | afazekas, but I cannot tell which kernel was actually running | 12:17 |
| ihrachys | afazekas: this message looks like a generic one. same as for pm-is-supported | 12:18 |
| apevec | maybe I'm blind, but don't see kernel version in kern_log.txt oro syslog | 12:18 |
| afazekas | apavec: if it is not updated: http://logs.openstack.org/02/73402/1/gate/gate-tempest-dsvm-full/ba93148/logs/dpkg-l.txt.gz | 12:18 |
| apevec | afazekas, there are two linux-image packages installed | 12:18 |
| apevec | how do you tell which one was running? | 12:18 |
| ihrachys | afazekas: ha, not really: http://logs.openstack.org/02/73402/1/gate/gate-tempest-dsvm-full/ba93148/logs/screen-n-cpu.txt.gz#_2014-02-14_10_08_47_046 | 12:19 |
| ihrachys | see Traceback | 12:19 |
| apevec | ihrachys, that's obviously fatal qemu error, process exits | 12:20 |
| *** ociuhandu has quit IRC | 12:21 | |
| ihrachys | apevec: the question is whether 'qemu: linux kernel too old' part is the reason, or just part of the stdout of the failed qemu | 12:25 |
| apevec | yeah, we can't tell, qemu.log is not captured | 12:27 |
| therve | ihrachys, https://bugs.launchpad.net/openstack-ci/+bug/1280072 | 12:28 |
| therve | It's swiftclient bug | 12:28 |
| apevec | wow | 12:28 |
| tristanC | therve: and the fix is waiting here https://review.openstack.org/#/c/73585/ | 12:29 |
| therve | ah cool | 12:30 |
| ihrachys | therve: tristanC: great! | 12:30 |
| therve | tristanC, Should we land a requirements change in the mean time, or do you thing the release can happen quickly enough? | 12:31 |
| therve | s/thing/think | 12:32 |
| tristanC | tristanC: what do you mean by a requirements change ? I think this can be merge quickly, though no swiftcore are available right now | 12:32 |
| therve | tristanC, I mean change global requirements.txt to add swiftclient < 2.0 | 12:33 |
| tristanC | oups, tab fail, therve: ^ | 12:33 |
| therve | To unblock the gate | 12:33 |
| apevec | therve, from pip-freeze I see tempest is taking swiftclient master | 12:34 |
| apevec | so we need to land the fix | 12:34 |
| *** weshay has joined #openstack-infra | 12:34 | |
| therve | apevec, That sounds surprising | 12:35 |
| apevec | http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-havana/3cd8ddc/logs/pip-freeze.txt.gz | 12:35 |
| tristanC | hum, is swiftclient the reason of the 'linux kernel too old' ? | 12:35 |
| *** yamahata has joined #openstack-infra | 12:35 | |
| *** dstanek has joined #openstack-infra | 12:35 | |
| apevec | I guess that qemu message is just side-effect if image is corrupted | 12:36 |
| therve | tristanC, Yeah | 12:36 |
| afazekas | ihrachys, apevec: a corrupted guest kernel image can cause similar issues | 12:36 |
| therve | That's as bad as an error message as you can get :) | 12:36 |
| *** mflobo has joined #openstack-infra | 12:37 | |
| *** senthil has joined #openstack-infra | 12:38 | |
| tristanC | we may need to push that requirement change as this is preventing the swiftclient fix to psas | 12:38 |
| *** dstanek has quit IRC | 12:40 | |
| *** yamahata has quit IRC | 12:40 | |
| therve | Well if it uses master we're screwed | 12:41 |
| therve | We may need to force the change in | 12:41 |
| apevec | it used swiftclient branch of the review http://logs.openstack.org/85/73585/1/check/check-swift-dsvm-functional/d15276b/logs/pip-freeze.txt.gz | 12:44 |
| apevec | so it's failing for some other reason | 12:44 |
| *** sarob has joined #openstack-infra | 12:44 | |
| apevec | ERROR: Invalid OpenStack Nova credentials. ? | 12:45 |
| therve | http://logs.openstack.org/85/73585/1/check/check-swift-dsvm-functional/d15276b/logs/screen-s-object.txt.gz I think | 12:46 |
| *** sarob has quit IRC | 12:49 | |
| *** smarcet has joined #openstack-infra | 12:52 | |
| *** AJaeger has quit IRC | 12:53 | |
| *** hashar has joined #openstack-infra | 12:55 | |
| tristanC | therve: I don't that new error. Seems related to chunk encoding | 12:55 |
| tristanC | don't understand* | 12:56 |
| *** yamahata has joined #openstack-infra | 12:56 | |
| *** lcostantino has joined #openstack-infra | 12:58 | |
| openstackgerrit | Florent Flament proposed a change to openstack-infra/config: Add ospurge project https://review.openstack.org/73308 | 12:59 |
| *** dkranz has quit IRC | 12:59 | |
| *** banix has quit IRC | 13:01 | |
| *** mwagner_lap has joined #openstack-infra | 13:03 | |
| *** senthil has quit IRC | 13:03 | |
| *** balar has quit IRC | 13:06 | |
| *** gmoro has joined #openstack-infra | 13:06 | |
| *** balar has joined #openstack-infra | 13:06 | |
| *** b3nt_pin is now known as beagles | 13:06 | |
| *** dcramer_ has quit IRC | 13:06 | |
| *** yamahata has quit IRC | 13:07 | |
| *** esker has quit IRC | 13:07 | |
| *** esker has joined #openstack-infra | 13:08 | |
| *** esker has quit IRC | 13:08 | |
| *** esker has joined #openstack-infra | 13:09 | |
| *** oubiwann has joined #openstack-infra | 13:10 | |
| *** dpyzhov has quit IRC | 13:12 | |
| *** esker has quit IRC | 13:13 | |
| *** dkranz has joined #openstack-infra | 13:15 | |
| *** dpyzhov has joined #openstack-infra | 13:15 | |
| *** pdmars has joined #openstack-infra | 13:16 | |
| *** dhellmann_ is now known as dhellmann | 13:18 | |
| *** dhellmann is now known as dhellmann_ | 13:18 | |
| *** dhellmann_ is now known as dhellmann | 13:18 | |
| *** jgallard has joined #openstack-infra | 13:19 | |
| *** dpyzhov has quit IRC | 13:22 | |
| openstackgerrit | A change was merged to openstack-infra/config: Replace install-deploy-guide-$LANG jobs https://review.openstack.org/73397 | 13:22 |
| *** dims_ has quit IRC | 13:24 | |
| *** e0ne has quit IRC | 13:24 | |
| *** e0ne has joined #openstack-infra | 13:25 | |
| *** markmcclain has quit IRC | 13:25 | |
| *** dims_ has joined #openstack-infra | 13:26 | |
| *** dpyzhov has joined #openstack-infra | 13:27 | |
| *** thomasem has joined #openstack-infra | 13:27 | |
| *** ociuhandu has joined #openstack-infra | 13:27 | |
| *** annegentle has quit IRC | 13:32 | |
| *** vkozhukalov has quit IRC | 13:34 | |
| *** senthil has joined #openstack-infra | 13:37 | |
| *** yamahata has joined #openstack-infra | 13:38 | |
| *** e0ne has quit IRC | 13:38 | |
| *** dstanek has joined #openstack-infra | 13:38 | |
| *** andre__ has quit IRC | 13:44 | |
| *** sarob has joined #openstack-infra | 13:44 | |
| *** vkozhukalov has joined #openstack-infra | 13:47 | |
| *** bookwar has quit IRC | 13:48 | |
| *** dstanek has quit IRC | 13:50 | |
| *** dkliban has joined #openstack-infra | 13:51 | |
| *** leifmadsen has quit IRC | 13:56 | |
| *** leifmadsen has joined #openstack-infra | 13:56 | |
| *** andre__ has joined #openstack-infra | 13:57 | |
| *** dstanek has joined #openstack-infra | 13:59 | |
| *** bookwar has joined #openstack-infra | 14:01 | |
| *** stevemar has joined #openstack-infra | 14:01 | |
| *** dpyzhov has quit IRC | 14:03 | |
| *** afazekas has quit IRC | 14:04 | |
| *** dpyzhov has joined #openstack-infra | 14:04 | |
| *** senthil has quit IRC | 14:04 | |
| SergeyLukjanov | AJaeger, ping | 14:05 |
| *** oubiwann has quit IRC | 14:09 | |
| *** rlandy has quit IRC | 14:10 | |
| *** mfer has joined #openstack-infra | 14:10 | |
| *** yamahata has quit IRC | 14:12 | |
| *** dstanek has quit IRC | 14:13 | |
| *** mriedem has joined #openstack-infra | 14:14 | |
| *** annegentle has joined #openstack-infra | 14:14 | |
| *** yamahata has joined #openstack-infra | 14:15 | |
| *** afazekas has joined #openstack-infra | 14:16 | |
| *** prad has joined #openstack-infra | 14:17 | |
| *** sarob has quit IRC | 14:17 | |
| *** esker has joined #openstack-infra | 14:18 | |
| *** dstanek has joined #openstack-infra | 14:19 | |
| *** resker has joined #openstack-infra | 14:20 | |
| *** zhiyan_ is now known as zhiyan | 14:21 | |
| *** wchrisj has joined #openstack-infra | 14:21 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 14:21 |
| *** esker has quit IRC | 14:23 | |
| *** banix has joined #openstack-infra | 14:25 | |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: make gate-pycadf-python33 non-voting https://review.openstack.org/73603 | 14:25 |
| *** ociuhandu has quit IRC | 14:26 | |
| * ttx wonders if there is a simple way to see the gerrit username of someone | 14:26 | |
| *** senthil has joined #openstack-infra | 14:32 | |
| *** DinaBelova is now known as DinaBelova_ | 14:34 | |
| *** saper_ is now known as saper | 14:34 | |
| *** dprince has joined #openstack-infra | 14:35 | |
| *** DinaBelova_ is now known as DinaBelova | 14:35 | |
| *** Ajaeger has joined #openstack-infra | 14:38 | |
| *** e0ne has joined #openstack-infra | 14:38 | |
| *** yamahata has quit IRC | 14:41 | |
| *** yamahata has joined #openstack-infra | 14:43 | |
| *** pafuent has joined #openstack-infra | 14:43 | |
| *** sarob has joined #openstack-infra | 14:44 | |
| SergeyLukjanov | ttx, I don't know how do it | 14:47 |
| SergeyLukjanov | ttx, I'm always looking on reviewstats and trying to map folk to names specified there | 14:48 |
| *** sarob has quit IRC | 14:49 | |
| *** jnoller has joined #openstack-infra | 14:50 | |
| *** jroovers|afk has quit IRC | 14:50 | |
| *** oubiwann has joined #openstack-infra | 14:52 | |
| *** oubiwann has quit IRC | 14:52 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/reviewstats: Support reviewers stats for stable branches https://review.openstack.org/73613 | 14:52 |
| jaypipes | SergeyLukjanov: good evening! :) When you set up your CI slave nodes for savannah, did you use the jenkins::slave puppet module from openstack-infra/config? | 14:52 |
| ttx | SergeyLukjanov: haha | 14:53 |
| *** oubiwann has joined #openstack-infra | 14:53 | |
| ttx | apevec, adam_g: see https://review.openstack.org/73613 for a version of reviewstats that lets us generate review stats for stable/* | 14:53 |
| ruhe | jaypipes: yes we did | 14:54 |
| ruhe | jaypipes: here is a link to our ci config https://github.com/savanna-project/savanna-ci/tree/master/config/deploy-ci | 14:54 |
| *** oubiwann has quit IRC | 14:54 | |
| *** oubiwann_ has joined #openstack-infra | 14:54 | |
| SergeyLukjanov | jaypipes, morning (?) | 14:55 |
| jaypipes | SergeyLukjanov: yep, morning :) | 14:55 |
| *** hashar has quit IRC | 14:57 | |
| jaypipes | ruhe: thx, I did not know about that script. Interesting to see similarities between it and http://github.com/jaypipes/os-ext-testing | 14:57 |
| jaypipes | ruhe: I think this might be the piece I am missing: https://github.com/savanna-project/savanna-ci/blob/master/config/deploy-ci/deploy.sh#L256 | 14:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable py26 tests for storyboard https://review.openstack.org/70675 | 14:59 |
| jaypipes | ruhe: I am getting this on the jenkins master when adding a new node with SSH key containing master private key: http://paste.openstack.org/show/65453/ | 14:59 |
| *** hdd has joined #openstack-infra | 15:00 | |
| jaypipes | ruhe: and the slave has the private key in /home/jenkins/.ssh/id_rsa | 15:00 |
| *** ryanpetrello has joined #openstack-infra | 15:00 | |
| jaypipes | ruhe: so I'm wondering if it's a known_hosts issue or something (but I installed the no-host-key-checking jenkinsuser.pp module on the slave, so that should not matter, right? | 15:00 |
| ruhe | jaypipes: yes, you need to override jenkins public keys in puppet scripts. otherwise the one from jenkins.openstack.org will be used | 15:03 |
| jaypipes | ruhe: doh. that's probably it :) ty sir. | 15:04 |
| *** mgagne has joined #openstack-infra | 15:04 | |
| ruhe | jaypipes: also you need to use credentials-id in nodepool config to pass key for slave nodes | 15:04 |
| jaypipes | ruhe: I'm not using nodepool (yet). Just creating the node manually in jenkins for now. | 15:05 |
| ruhe | jaypipes: which means you need to manually create credentials in jenkins ui and copy credentials ID from url | 15:05 |
| jaypipes | yup, I did that. | 15:05 |
| ruhe | jaypipes: oh, ok | 15:05 |
| ruhe | jaypipes: you should use it, it's a really good software :) | 15:06 |
| jaypipes | ruhe: yes, I know. I just taking things step by step for a tutorial... I'll add nodepool in a bit. | 15:06 |
| fungi | ttx: i can query it in the database... the new gerrit may have a feature to let you do it in the gui/api | 15:07 |
| fungi | also, glad people figured out what was causing the tempest failures. it was well past my bedtime when they came to my attention | 15:08 |
| boris-42_ | sdague pls merge it https://review.openstack.org/#/c/73025/ already 2x+2 and 1x+1 =) | 15:08 |
| ruhe | jaypipes: i hope eventually we will not need something like http://github.com/jaypipes/os-ext-testing or https://github.com/savanna-project/savanna-ci/tree/master/config/deploy-ci ; we only need to make puppet modules modular and configurable enough to allow 3rd parties to use them without modification | 15:08 |
| boris-42_ | sdague it is blocking work in Rally =( | 15:08 |
| jaypipes | ruhe: agreed. | 15:08 |
| *** bhuvan has quit IRC | 15:08 | |
| *** hashar has joined #openstack-infra | 15:10 | |
| *** bhuvan has joined #openstack-infra | 15:10 | |
| *** bhuvan has quit IRC | 15:10 | |
| *** bhuvan has joined #openstack-infra | 15:10 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Extracted DB API layer https://review.openstack.org/72356 | 15:13 |
| jaypipes | ruhe: Slave successfully connected and online ... w00t. :) | 15:16 |
| *** krtaylor has quit IRC | 15:16 | |
| fungi | tristanC: notmyname: we should probably go ahead and merge mtreinish's https://review.openstack.org/70378 too once the swiftclient fix gets in | 15:17 |
| *** boris-42_ has quit IRC | 15:17 | |
| fungi | i think it would have prevented this | 15:17 |
| *** krtaylor has joined #openstack-infra | 15:17 | |
| notmyname | fungi: likely so. can you join -swift where we're talking about the swiftclient stuff? | 15:17 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: dequeue abandoned changes https://review.openstack.org/65467 | 15:17 |
| fungi | notmyname: you bet | 15:17 |
| notmyname | fungi: you coming? | 15:22 |
| fungi | notmyname: oh... i joined #swift, no wonder it was so quiet :/ | 15:22 |
| *** dcramer_ has joined #openstack-infra | 15:23 | |
| notmyname | students with an interest in the future of technology | 15:23 |
| ruhe | jaypipes: congrats :) | 15:23 |
| *** jroovers has joined #openstack-infra | 15:33 | |
| *** david-lyle has joined #openstack-infra | 15:33 | |
| *** hashar has quit IRC | 15:34 | |
| *** hashar has joined #openstack-infra | 15:34 | |
| hashar | holy hell | 15:36 |
| hashar | I managed to write a test for Zuul \O/ | 15:36 |
| mordred | hashar: w00t! | 15:36 |
| hashar | the first version was horrible James commented on it and took me a good half hour to figure out what he was explaining to me :D | 15:37 |
| fungi | hashar: congrats! i haven't even done that yet | 15:37 |
| hashar | fungi: I should write some documentation maybe :] | 15:37 |
| fungi | hashar: i should read some documentation maybe | 15:38 |
| hashar | Zuul test writing 101 | 15:38 |
| hashar | fungi: there is none afaik. At least not a high level doc, you get to read the code | 15:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Make python-swiftclient gate symmetrical https://review.openstack.org/70378 | 15:39 |
| fungi | code is documentation. c'mon man, this is sparta^H^H^H^H^H^Hfree software! | 15:39 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix typo in openstack-install-deploy-guide-{lang} job https://review.openstack.org/73624 | 15:40 |
| *** DeeJay1 has quit IRC | 15:40 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 15:43 | |
| *** katyafervent is now known as katyafervent_awa | 15:43 | |
| jeblair | fungi: good morning | 15:44 |
| *** matrohon has quit IRC | 15:44 | |
| jeblair | hashar: hi there! | 15:44 |
| mordred | look, it's jeblair | 15:44 |
| *** sarob has joined #openstack-infra | 15:44 | |
| SergeyLukjanov | jeblair, morning | 15:44 |
| jeblair | fungi: what's the latest on the qemu thing? | 15:45 |
| jeblair | SergeyLukjanov, mordred: good morning to you too! | 15:45 |
| Ajaeger | Morning, here's my weekend wish from the documentation site - a few fixes for our gates: https://review.openstack.org/#/c/73331/ , https://review.openstack.org/#/c/73185/, https://review.openstack.org/#/c/73624/ Thanks! | 15:46 |
| jeblair | ttx: i found a username recently by finding a change they had reviewed then ran the ssh command "gerrit query ####" which outputs the usernames of people who commented | 15:47 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 15:48 | |
| hashar | jeblair: good morning :] | 15:48 |
| fungi | jeblair: fix is being retested here with tempest jobs added to swiftclient gating: https://review.openstack.org/73585 | 15:48 |
| *** ociuhandu has joined #openstack-infra | 15:49 | |
| jeblair | fungi: yay we found a hole in the gate! | 15:49 |
| *** sarob has quit IRC | 15:49 | |
| fungi | jeblair: to be fair, we knew about the hole in the gate, and mtreinish had 70378 up for review for weeks | 15:49 |
| hashar | jeblair: I finished tests for Zuul abandoned changes :-D https://review.openstack.org/#/c/65467/ Nothing urgent. | 15:50 |
| fungi | proof that if it's not tested it's broken | 15:50 |
| jeblair | fungi: ++ | 15:50 |
| mtreinish | fungi: I've been fighting the "I told you so" urge all morning :) | 15:51 |
| jeblair | mtreinish: i'm sorry i didn't get to that :( i'm personally quite behind on reviews due to firefighting. you certainly never had to convince me of the value of it. | 15:51 |
| *** markwash has joined #openstack-infra | 15:52 | |
| mtreinish | jeblair: no it's not that I'm not upset it already had 3 +2s | 15:52 |
| mtreinish | I'm behind on reviews myself too | 15:52 |
| fungi | i'm partly at fault for wanting to wait for the ptl to acknowledge his awareness of that change being approved | 15:53 |
| fungi | but, um, it's been acknowledged now ;) | 15:53 |
| jeblair | fungi: /win 12 | 15:53 |
| *** rkukura has joined #openstack-infra | 15:54 | |
| jeblair | wow. i'm going to eat breakfast and try this "morning" thing again. | 15:54 |
| fungi | that's #-qa for me | 15:54 |
| fungi | ;) | 15:54 |
| *** gokrokve has joined #openstack-infra | 15:54 | |
| *** markmcclain has joined #openstack-infra | 15:54 | |
| *** markmcclain has quit IRC | 15:54 | |
| *** markmcclain has joined #openstack-infra | 15:54 | |
| notmyname | fungi: seems that all the "problems" here are because of PTLs. go us! | 15:54 |
| fungi | notmyname: i should be an honorary ptl in that case. i cause all sorts of problems | 15:55 |
| notmyname | lol | 15:55 |
| *** wenlock has joined #openstack-infra | 15:56 | |
| *** mflobo has quit IRC | 15:56 | |
| *** thedodd has joined #openstack-infra | 15:56 | |
| * jeblair votes for fungi as openstack ptl | 15:57 | |
| fungi | platform: i'll technical all the openstacks | 15:58 |
| *** e0ne_ has joined #openstack-infra | 16:02 | |
| mordred | window 12 for me is Shrews | 16:03 |
| *** wchrisj is now known as wchrisj|away | 16:03 | |
| Shrews | yay for knowing order of importance! | 16:03 |
| *** markwash has quit IRC | 16:03 | |
| Shrews | my life will be complete if i make top 10 | 16:04 |
| *** wchrisj|away is now known as wchrisj | 16:05 | |
| mordred | Shrews: too bad for you I value knowing your number more than I value maintaining a strict value ordering | 16:05 |
| fungi | Shrews: i'm afraid to admit that your window is somewhere in the 40s for me. probably because i lurk in way too many channels | 16:05 |
| *** markwash has joined #openstack-infra | 16:06 | |
| *** AlexF has joined #openstack-infra | 16:06 | |
| *** e0ne has quit IRC | 16:06 | |
| Shrews | fungi: well i was going to buy you a beer after the meetup next week, but now i must reconsider | 16:07 |
| fungi | Shrews: however, if you can overlook that, we should totally grab drinks after the user group meeting next week | 16:07 |
| *** beagles is now known as seagulls | 16:07 | |
| fungi | bwahahaha | 16:07 |
| fungi | has olaph recovered from the plague finally? he needs to get out more | 16:08 |
| *** jgrimm has joined #openstack-infra | 16:08 | |
| Shrews | i think he may have succumbed to snowmaggedon | 16:08 |
| fungi | i just acted like a shut-in for several days. finally shoveled some of the ice crust off my walkway so i can go check the mailbox | 16:09 |
| jeblair | fungi: any mail? | 16:09 |
| fungi | jeblair: i haven't actually checked yet, just shoveled | 16:10 |
| fungi | at the insistence of the gf. i would have just walked on it, but whatever | 16:10 |
| jeblair | fungi: i think it's safest to just go ahead and sit down and slide. | 16:11 |
| fungi | i could... we're on a pretty steep hill... but the mailbox is in the other direction, against the flow of gravity | 16:12 |
| fungi | well, against the pull of gravity or against the flow of spacetime | 16:12 |
| *** gokrokve has quit IRC | 16:14 | |
| *** ryanpetrello has quit IRC | 16:14 | |
| *** gokrokve has joined #openstack-infra | 16:15 | |
| * mordred got a picture of olaph's cow yesterday | 16:16 | |
| anteaya | ha ha ha | 16:16 |
| *** russellb is now known as rustlebee | 16:16 | |
| anteaya | favourtism | 16:16 |
| mordred | that's right | 16:16 |
| *** alaski is now known as lascii | 16:17 | |
| fungi | was the photo from before or after it was butchered and cooked up all tasty-like? | 16:17 |
| fungi | or was it the milk cow? | 16:18 |
| anteaya | fungi is 20, jeblair was in the mid-twenties but is now in the 90's due to nick changes and mordred is 29 | 16:18 |
| anteaya | fungi was just the first one I needed to pm | 16:18 |
| *** pcrews has joined #openstack-infra | 16:19 | |
| fungi | glad to know there's an alternate universe where i'm still 20 rather than pushing 40 | 16:19 |
| *** gokrokve has quit IRC | 16:19 | |
| anteaya | I'm all about alternatives | 16:20 |
| *** hashar has quit IRC | 16:20 | |
| *** afazekas has quit IRC | 16:21 | |
| olaph | the milk cow. she sends you all her love | 16:21 |
| anteaya | ha ha ha | 16:21 |
| *** e0ne_ has quit IRC | 16:22 | |
| *** AlexF has quit IRC | 16:24 | |
| *** e0ne has joined #openstack-infra | 16:25 | |
| *** vkozhukalov has quit IRC | 16:26 | |
| *** svarnau has joined #openstack-infra | 16:27 | |
| *** AlexF has joined #openstack-infra | 16:27 | |
| *** rcleere has joined #openstack-infra | 16:28 | |
| *** AlexF has quit IRC | 16:32 | |
| *** bnemec is now known as beekneemech | 16:32 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:35 | |
| *** mrodden has quit IRC | 16:37 | |
| *** AlexF has joined #openstack-infra | 16:39 | |
| *** mriedem is now known as mcriedemannoid | 16:41 | |
| *** e0ne has quit IRC | 16:42 | |
| *** talluri has joined #openstack-infra | 16:43 | |
| *** atiwari has joined #openstack-infra | 16:43 | |
| *** e0ne has joined #openstack-infra | 16:44 | |
| *** AlexF has quit IRC | 16:44 | |
| *** sarob has joined #openstack-infra | 16:44 | |
| *** senthil has quit IRC | 16:45 | |
| fungi | https://jenkins.openstack.org/computer/mirror26/ says "Offline due to Gearman request" | 16:45 |
| fungi | i'm going to reboot it and then relaunch the slave agent on it | 16:46 |
| *** gokrokve has joined #openstack-infra | 16:46 | |
| fungi | we have changes hanging in the release pipeline waiting for py26 mirror runs | 16:46 |
| jeblair | fungi: oops, we may have messed up the offline regex in layout.yaml | 16:46 |
| fungi | oh | 16:46 |
| jaypipes | feedback welcomed from you guys :) http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ | 16:47 |
| fungi | did it somehow match on periodic-mirror-python26? | 16:47 |
| fungi | i bet it did... checking | 16:47 |
| *** markmcclain has quit IRC | 16:48 | |
| *** sarob has quit IRC | 16:49 | |
| fungi | yep, working on the trivial negative match for it now, and i've onlined it again | 16:50 |
| *** leifmadsen is now known as blitzrage | 16:50 | |
| fungi | good news is, it won't get offlined until the periodic jobs run again tonight | 16:50 |
| *** mrodden has joined #openstack-infra | 16:50 | |
| fungi | so i can fix this after lunch | 16:50 |
| jeblair | fungi: cool. in other news, i found the problem with the gate-enqueue-after-check thing yesterday. | 16:50 |
| clarkb | :) have fun /me is almost really here | 16:51 |
| fungi | the swift fix is confirmed and at the front of the gate. got to take the gf out to lunch in celebration of st. hallmark's day | 16:51 |
| clarkb | jeblair and wrote a patch I remember reviewing | 16:51 |
| * fungi will bbiaw | 16:51 | |
| openstackgerrit | Steven Hardy proposed a change to openstack-infra/reviewstats: Add jasondunsmore to heat-core https://review.openstack.org/73570 | 16:51 |
| jeblair | clarkb: that's a different thing. :) this is where it was supposed to enqueue into gate on a +1 from jenkins if there was an approved vote, but it wasn't. | 16:52 |
| fungi | jeblair: i will be thrilled to review the fix upon my return, if there are no new fires | 16:52 |
| clarkb | oh | 16:52 |
| * fungi goes up, up and away! | 16:52 | |
| jeblair | clarkb: the thing you reviewed was about what happens when there's a -2 verified | 16:52 |
| clarkb | yup, I didnt realize there were two things | 16:53 |
| *** stevemar has quit IRC | 16:53 | |
| clarkb | anyways afk for a short time to un afk | 16:53 |
| *** saju_m has joined #openstack-infra | 16:54 | |
| *** DinaBelova is now known as DinaBelova_ | 16:54 | |
| *** stevemar has joined #openstack-infra | 16:54 | |
| *** senthil has joined #openstack-infra | 16:54 | |
| *** jooools has quit IRC | 16:55 | |
| *** ryanpetrello has joined #openstack-infra | 16:56 | |
| *** vkozhukalov has joined #openstack-infra | 16:56 | |
| *** e0ne has quit IRC | 17:00 | |
| *** jgallard has quit IRC | 17:03 | |
| *** boris-42_ has joined #openstack-infra | 17:08 | |
| *** yamahata has quit IRC | 17:10 | |
| *** garyk1 has quit IRC | 17:12 | |
| *** markmc has quit IRC | 17:15 | |
| *** yamahata has joined #openstack-infra | 17:15 | |
| *** senthil has quit IRC | 17:17 | |
| *** vkozhukalov has quit IRC | 17:17 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Re-enable requiring a recent check result for gate https://review.openstack.org/73418 | 17:18 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori - Configuration Discovery https://review.openstack.org/73667 | 17:19 |
| jeblair | clarkb, fungi, sdague, SergeyLukjanov: https://review.openstack.org/73418 should get us where we thought we were yesterday. | 17:20 |
| *** pblaho has joined #openstack-infra | 17:20 | |
| *** lcheng has quit IRC | 17:20 | |
| *** yolanda has quit IRC | 17:21 | |
| *** ryanpetrello has quit IRC | 17:23 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Re-enable requiring a recent check result for gate https://review.openstack.org/73418 | 17:24 |
| *** yolanda has joined #openstack-infra | 17:25 | |
| *** reed has joined #openstack-infra | 17:25 | |
| *** reed has quit IRC | 17:25 | |
| *** reed has joined #openstack-infra | 17:25 | |
| *** dpyzhov has quit IRC | 17:26 | |
| *** VijayT has joined #openstack-infra | 17:26 | |
| *** wchrisj has quit IRC | 17:27 | |
| jeblair | fungi, clarkb: i'm going to do some interactive debugging on nodepool that will involve some stop/starts and running it foreground inside of screen. i'm having trouble reproducing the memory leak locally | 17:28 |
| *** hogepodge has joined #openstack-infra | 17:31 | |
| *** shardy is now known as shardy_afk | 17:32 | |
| *** dpyzhov has joined #openstack-infra | 17:32 | |
| clarkb | wfm | 17:37 |
| *** hogepodge has quit IRC | 17:37 | |
| clarkb | the reason that layout change works is zuul checks submittability correct? | 17:37 |
| jeblair | clarkb: yeah, for everything involving the gate pipeline, mentally add "and can be submitted if jenkins votes +2" to all the trigger conditions | 17:38 |
| *** hogepodge has joined #openstack-infra | 17:40 | |
| *** markmcclain has joined #openstack-infra | 17:40 | |
| *** rossella_s has quit IRC | 17:41 | |
| reed | hello folks | 17:43 |
| *** afazekas has joined #openstack-infra | 17:43 | |
| *** afazekas has quit IRC | 17:43 | |
| reed | I'd be glad to help you investigate errors like this one for gerrit "fatal: ICLA contributor agreement requires current contact information." | 17:43 |
| reed | a user reported it to me but I'm not sure I can help him | 17:44 |
| *** sarob has joined #openstack-infra | 17:44 | |
| clarkb | reed was the user amrith? | 17:45 |
| reed | clarkb, yes :) | 17:45 |
| clarkb | reed: I think we got amrith sorted last night. Last step is amrith should submit a bug sorting out which of the multiple accounts needs to die | 17:45 |
| reed | got it | 17:45 |
| reed | damn multiple accounts | 17:45 |
| *** stevemar has quit IRC | 17:46 | |
| *** stevemar2 has joined #openstack-infra | 17:46 | |
| *** stevemar2 is now known as stevemar | 17:46 | |
| *** fbo is now known as fbo_away | 17:48 | |
| *** sarob has quit IRC | 17:49 | |
| *** svarnau has left #openstack-infra | 17:51 | |
| *** dpyzhov has quit IRC | 17:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 17:52 |
| *** yolanda has quit IRC | 17:52 | |
| *** afazekas has joined #openstack-infra | 17:53 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack-dev/hacking: Switch over to oslosphinx https://review.openstack.org/73676 | 17:53 |
| *** talluri has quit IRC | 17:56 | |
| *** yolanda has joined #openstack-infra | 17:56 | |
| *** nicedice has joined #openstack-infra | 17:57 | |
| *** DinaBelova_ is now known as DinaBelova | 17:57 | |
| *** DinaBelova is now known as DinaBelova_ | 17:58 | |
| *** AlexF has joined #openstack-infra | 17:59 | |
| *** sabari has joined #openstack-infra | 18:00 | |
| *** cadenzajon has joined #openstack-infra | 18:01 | |
| clarkb | jeblair: going to push https://review.openstack.org/#/c/72805/ through and get puppet running everywhere | 18:02 |
| *** sabari has quit IRC | 18:02 | |
| *** dangers_away is now known as dangers | 18:02 | |
| *** sabari has joined #openstack-infra | 18:02 | |
| jeblair | clarkb: +2 | 18:03 |
| *** derekh has quit IRC | 18:04 | |
| clarkb | then finish up laptop transition and then code review | 18:04 |
| *** pblaho has quit IRC | 18:05 | |
| mordred | clarkb: btw - I have a script for turning off the trackpad should that be a thing you want to do | 18:06 |
| *** seagulls is now known as beagles | 18:06 | |
| *** enikanorov_ has joined #openstack-infra | 18:06 | |
| clarkb | mordred: I turned it off via xfce settings | 18:06 |
| clarkb | was simple | 18:07 |
| clarkb | mordred: also ubuntu will use systemd | 18:07 |
| *** tomhe has joined #openstack-infra | 18:07 | |
| notmyname | python-swiftclient 2.0.2 tag pushed to gerrit. should be in pypi momentarily | 18:08 |
| mordred | clarkb: oh wow, really? | 18:08 |
| mordred | SpamapS: ^^ | 18:08 |
| *** beagles is now known as seagulls | 18:08 | |
| mordred | clarkb: I actually never thought about looking in xfce settings :) | 18:08 |
| fungi | we have a gate full of mostly green stuff once more | 18:09 |
| clarkb | hurray | 18:09 |
| *** w_ has joined #openstack-infra | 18:09 | |
| *** hemna_ has joined #openstack-infra | 18:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9 https://review.openstack.org/72805 | 18:11 |
| *** olaph has quit IRC | 18:11 | |
| tomhe | Hello, I have a Zuul question: I run tests on embedded target hardware in my check and gate pipelines, but the hardware resources are a bottleneck so I would like to model my projects in layout.yaml so that the hardware verification job only runs if all the host verification jobs pass. Something like "run job c, but only if jobs a and b pass". Is it possible to model such a dependency in layout.yaml? | 18:12 |
| fungi | reed: right, in short what happens is that the user ends up with one account tied to their ssh username and another tied to their web session... then signs the cla and submits contact info as the second while their patch submissions get rejected by the first | 18:12 |
| reed | fungi, why does that happen? | 18:12 |
| fungi | tomhe: you can run job b if a passes, and then run job c if b passes | 18:12 |
| fungi | tomhe: but success criteria can't be grouped for job dependencies easily | 18:13 |
| jeblair | tomhe: see http://ci.openstack.org/zuul/zuul.html#projects | 18:13 |
| fungi | reed: usually the cause is that they sign into gerrit from lp, which creates an account associated with their openid url and pulls in their ssh username based on their lp username, then later they do something which causes launchpad to start using a different openid url (switch e-mail addresses, combine lp accounts, or just plan have more than one lp account) | 18:14 |
| tomhe | Thanks, that's what i thought. | 18:14 |
| *** zhiyan is now known as zhiyan_ | 18:15 | |
| fungi | jeblair: my gut says that the nodepool memory leak is from persisting novaclient sessions to providers, but i have no evidence whatsoever to back that up | 18:15 |
| jeblair | fungi: that would explain my inability to reproduce it with local testing | 18:16 |
| jeblair | it's a _fast_ leak too. | 18:16 |
| fungi | it could just as easily be paramiko not garbage-collection or something like that | 18:16 |
| *** garyk has joined #openstack-infra | 18:16 | |
| fungi | s/collection/collecting/ | 18:16 |
| reed | fungi, thanks | 18:17 |
| reed | fungi, is there something we can do with the new openid provider to solve this particular issue? | 18:18 |
| clarkb | reed: partially, if merging accounts don't change openids | 18:18 |
| fungi | reed: yes, make sure that people don't get more than one openid url, or add an ldap backend and point gerrit at that or something similar | 18:18 |
| fungi | s/backend/frontend/ | 18:19 |
| fungi | but mainly just keep their openid url stable and all shouldl be well in gerritland | 18:19 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori - Configuration Discovery https://review.openstack.org/73667 | 18:19 |
| fungi | gerrit likes to autocreate an account whenever it sees a new openid | 18:19 |
| clarkb | fungi: reed: we will still have trouble if people actually create multiple accuonts | 18:20 |
| fungi | and doesn't give the user any way to merge them. it lets you add multiple openids to your account, but you have to know your additional openid urls prior to signing into gerrit wit them, which is sort of cart-before-the-horse | 18:20 |
| reed | fungi, clarkb: we'll have to file a feature request or something against that project to remember that | 18:20 |
| *** max_lobur is now known as max_lobur_afk | 18:20 | |
| *** mcriedemannoid has quit IRC | 18:21 | |
| *** mriedem has joined #openstack-infra | 18:21 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori - Configuration Discovery https://review.openstack.org/73667 | 18:21 |
| fungi | the launchpad openid provider implementers/maintainers claim (and i can see their point) that the openid itself shouldn't be the unique key for an account. the provider should have a key like username or something, and the consumer application should tell you when you're authenticating with a previously-unseen openid and walk you through creating or combining accounts | 18:22 |
| *** mrodden has quit IRC | 18:23 | |
| fungi | but regardless, that's not how gerrit implemented their consumer layer, and lp likes to spontaneously grow new openids for existing accounts, so they're sort of incompatible implementations | 18:23 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Change description of oldest reviews https://review.openstack.org/73686 | 18:25 |
| *** sarob has joined #openstack-infra | 18:26 | |
| *** enikanorov has quit IRC | 18:26 | |
| *** jnoller has quit IRC | 18:27 | |
| *** wchrisj has joined #openstack-infra | 18:27 | |
| jeblair | i forgot to set the statsd vars before starting nodepool in screen, so the graph is wrong; i'll fix that on the next restart | 18:28 |
| *** jgrimm has quit IRC | 18:28 | |
| fungi | are there good python libraries you can easily import to do memory profiling without heavily overhauling the original source? | 18:29 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add gerrit-git-prep option to clone into a sub directory https://review.openstack.org/73687 | 18:29 |
| clarkb | fungi: python profiling tools don't like threads | 18:29 |
| *** starmer has joined #openstack-infra | 18:29 | |
| fungi | grr | 18:30 |
| *** yassine has quit IRC | 18:31 | |
| *** ociuhandu has quit IRC | 18:31 | |
| SpamapS | mordred: yeah, systemd. What are ya gonna do right? :-P | 18:32 |
| *** mrodden has joined #openstack-infra | 18:32 | |
| fungi | SpamapS: start a kilothread on debian-devel called fsck-lennart? | 18:32 |
| *** mrodden has quit IRC | 18:32 | |
| *** mrodden has joined #openstack-infra | 18:33 | |
| zaro | jeblair: wanted to get your opinion on change https://review.openstack.org/#/c/73687 | 18:33 |
| SpamapS | fungi: well I was going to threaten physical violence too, but somebody stole that idea. | 18:33 |
| zaro | jeblair: was wondering if that would be bad? | 18:33 |
| fungi | SpamapS: that idea's older than our species. no use hoping to get the first post there | 18:34 |
| *** wchrisj has quit IRC | 18:34 | |
| clarkb | all elasticsearch machines are now back to normal puppeting, goign t odo the same for logstash workers and drop the worker count to 2 per box | 18:35 |
| SpamapS | fungi: there are no new ideas, just new applications of the same ideas. :) | 18:36 |
| *** jungleboyj has joined #openstack-infra | 18:36 | |
| *** zhiyan_ is now known as zhiyan | 18:38 | |
| fungi | (mostly new and interesting ways of applying threats of physical violence from a hushmail account) | 18:38 |
| pleia2 | hah | 18:39 |
| *** bknudson has quit IRC | 18:39 | |
| fungi | also, kilothread is one of my new favoritest words. there are not many mailing lists out there which can rival the gigantic alt.flame and alt.cascade threads of old | 18:40 |
| fungi | debian-devel should be forklifted back into usenet | 18:41 |
| fungi | (and i don't just mean through a m2n gateway) | 18:41 |
| jeblair | since i bet russ already does that | 18:42 |
| mordred | fungi: we should move all of our mailman traffic to usenet | 18:45 |
| fungi | jeblair: i'm sure it's served via afs too | 18:46 |
| *** thedodd has quit IRC | 18:46 | |
| fungi | mordred: we could certainly refine the oldschoolness of our community that way | 18:46 |
| *** jroovers has quit IRC | 18:47 | |
| fungi | we can offer to uucp posts in addition to straight nntp | 18:47 |
| *** gyee has joined #openstack-infra | 18:48 | |
| fungi | and offer uucp routing addresses for the mail-to-news gateway | 18:48 |
| * fungi has sufficiently beaten this dead horse | 18:48 | |
| *** rfolco has quit IRC | 18:50 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 18:52 |
| *** andre__ has quit IRC | 18:55 | |
| *** bknudson has joined #openstack-infra | 18:56 | |
| *** salv-orlando_ has joined #openstack-infra | 18:56 | |
| *** salv-orlando has quit IRC | 18:56 | |
| *** salv-orlando_ is now known as salv-orlando | 18:56 | |
| *** garyk has quit IRC | 18:57 | |
| *** bookwar has quit IRC | 19:02 | |
| *** andre__ has joined #openstack-infra | 19:03 | |
| fungi | i hate to say it, but i feel like 73418 needs a flowchart for the sanity of all involved | 19:05 |
| * fungi normally hates flowcharts | 19:06 | |
| *** arborism has joined #openstack-infra | 19:06 | |
| *** e0ne has joined #openstack-infra | 19:06 | |
| clarkb | fungi: is that why you have -1'd? your comments are descriptive not prescribing any changes (just making sure i understand your concner) | 19:07 |
| fungi | oh, i forgot to click save when i added to the last comment. fixing | 19:07 |
| fungi | no, nevermind, i did | 19:08 |
| fungi | the comment at line 63 | 19:08 |
| jeblair | fungi: responded | 19:08 |
| * anteaya is not a fan of being eaten by a grue | 19:08 | |
| *** e0ne has quit IRC | 19:10 | |
| *** ociuhandu has joined #openstack-infra | 19:10 | |
| clarkb | logstash workers should be happy now | 19:10 |
| fungi | jeblair: okay, that seems fine then. thanks for the schooling | 19:10 |
| *** saju_m has quit IRC | 19:11 | |
| SergeyLukjanov | jeblair, fungi, clarkb, sorry, was afk | 19:12 |
| SergeyLukjanov | looking on 73418 | 19:12 |
| clarkb | jeblair: fungi: if yo uguys are ready to keep an eye on zuul I will approve | 19:14 |
| clarkb | I guess I will let SergeyLukjanov have a look too | 19:14 |
| jeblair | fungi, clarkb: thanks to objgraph i have determined that the Config objects are not being gc'd in nodepool; it generates a new one of those each time through the loop. they have no backreferences. | 19:14 |
| fungi | clarkb: i am | 19:14 |
| *** andre__ has quit IRC | 19:14 | |
| clarkb | jeblair: ouch | 19:15 |
| *** andre__ has joined #openstack-infra | 19:15 | |
| fungi | jeblair: yeeowch | 19:15 |
| jeblair | it's a start, at least. | 19:15 |
| *** alexpilotti has quit IRC | 19:15 | |
| SergeyLukjanov | clarkb, 2 mins please ;) | 19:15 |
| clarkb | SergeyLukjanov: no rush | 19:15 |
| *** lcostantino has quit IRC | 19:16 | |
| notmyname | mordred: jeblair: jog0: fungi: http://lists.openstack.org/pipermail/openstack-dev/2014-February/027172.html | 19:16 |
| fungi | notmyname: thanks@ | 19:16 |
| fungi | ! | 19:16 |
| SergeyLukjanov | jeblair, clarkb, fungi, small question - will the check rules be applied before the gate ones? | 19:17 |
| jeblair | notmyname: thank you | 19:17 |
| *** ociuhandu has quit IRC | 19:17 | |
| clarkb | SergeyLukjanov: in this case yes, order is preserved but it depends on your layout list | 19:18 |
| jeblair | SergeyLukjanov: technically i believe so, but it considers all pipelines for all events and will happily enqueue a change into multiple pipelines if it matches. so those _should_ be designed to only match either one or the other. | 19:18 |
| *** amcrn_ has joined #openstack-infra | 19:18 | |
| *** amcrn_ is now known as amcrn | 19:18 | |
| *** mrodden has quit IRC | 19:18 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 19:19 |
| *** arborism has quit IRC | 19:19 | |
| *** e0ne has joined #openstack-infra | 19:19 | |
| SergeyLukjanov | jeblair, clarkb, got it, thx | 19:20 |
| SergeyLukjanov | +2 from me too | 19:20 |
| SergeyLukjanov | I hope I understand now how it works | 19:20 |
| *** zul has joined #openstack-infra | 19:20 | |
| *** zul has quit IRC | 19:22 | |
| *** alexpilotti has joined #openstack-infra | 19:23 | |
| *** gokrokve has quit IRC | 19:23 | |
| zaro | https://review.openstack.org/#/c/73687 | 19:23 |
| *** gokrokve has joined #openstack-infra | 19:24 | |
| zaro | jeblair: was just wondering whether this is a bad idea? https://review.openstack.org/#/c/73687 | 19:24 |
| *** rfolco has joined #openstack-infra | 19:26 | |
| jeblair | zaro: or instead you could cd into the directory before calling it; see the check-dg-tempest-dsvm-full job for an example | 19:27 |
| jeblair | zaro: it uses the experimental-devstack-checkout builder macro | 19:27 |
| *** starmer has quit IRC | 19:27 | |
| openstackgerrit | A change was merged to openstack-infra/config: Re-enable requiring a recent check result for gate https://review.openstack.org/73418 | 19:27 |
| zaro | jeblair: you mean call bash script correct? | 19:28 |
| *** gokrokve has quit IRC | 19:28 | |
| jeblair | zaro: yep | 19:29 |
| jeblair | zaro: just make a shell builder that does 'mkdir; cd; ggp' | 19:29 |
| zaro | jeblair: ok. that seems like a good alternative. thanks. | 19:29 |
| jeblair | zaro: we still might want to do your idea with ggp; it's a good one. but this will keep you moving without us having to worry about changing ggp right now. | 19:30 |
| jog0 | notmyname: thanks | 19:31 |
| zaro | jeblair: cool. i used the copy to subdir after clone workaround in the job, but i like your idea better. | 19:31 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Change description of oldest reviews https://review.openstack.org/73686 | 19:31 |
| *** thedodd has joined #openstack-infra | 19:32 | |
| *** CaptTofu_ has joined #openstack-infra | 19:33 | |
| *** CaptTofu has quit IRC | 19:34 | |
| fungi | what was the gotcha we determined with using owncloud's "swift" support? there was some significant but non-obvious limitation with it right? | 19:39 |
| * anteaya tries to remember | 19:39 | |
| * anteaya reads http://doc.owncloud.org/server/5.0/admin_manual/configuration/custom_mount_config.html#openstack-swift again | 19:41 | |
| anteaya | in my notes I have that I had to open the gui and add swift from there | 19:42 |
| *** smarcet has left #openstack-infra | 19:42 | |
| anteaya | and at the time i tried that, I was running a version of owncloud that was too old to have that access to the gui | 19:43 |
| anteaya | I have not attempted anything with owncloud lately | 19:43 |
| anteaya | it appears owncloud 6 is available now http://owncloud.org/six/ | 19:44 |
| anteaya | which was the release that was going to make auto config available | 19:44 |
| *** mrodden has joined #openstack-infra | 19:45 | |
| *** _david_ has joined #openstack-infra | 19:45 | |
| *** zhiyan is now known as zhiyan_ | 19:45 | |
| *** markwash_ has joined #openstack-infra | 19:45 | |
| anteaya | looking promising: http://doc.owncloud.org/server/6.0/admin_manual/configuration/configuration_automation.html | 19:45 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Don't offline after periodic-mirror-python26 https://review.openstack.org/73701 | 19:46 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 19:46 |
| *** markwash has quit IRC | 19:47 | |
| *** markwash_ is now known as markwash | 19:47 | |
| fungi | anteaya: right! we determined that installing and configuring owncloud via puppet wasn't actually doable at the time | 19:47 |
| fungi | now i remember | 19:47 |
| *** banix has quit IRC | 19:47 | |
| anteaya | yes | 19:48 |
| jeblair | configuring gerrit via puppet isn't doable. | 19:48 |
| anteaya | but now it looks like it is worth another try | 19:48 |
| anteaya | looks like i have to config a mysql first then the owncloud then enable swift from the owncloud gui | 19:49 |
| fungi | jeblair: good point | 19:49 |
| fungi | ...jenkins... | 19:49 |
| jeblair | especially if owncloud's config information is stored in a (trove) database, that's fine if some of the bootstrap stuff doesn't live in puppet | 19:50 |
| anteaya | okay | 19:50 |
| *** denis_makogon_ has joined #openstack-infra | 19:50 | |
| fungi | right, presumably that would at least allow us to rebuild it and reconnect it to the old database | 19:50 |
| anteaya | I will see about standing one up and see how much trove/swift we can point at | 19:50 |
| anteaya | I think I got it working with trove last time | 19:51 |
| anteaya | yes, looks like the msql is required to fire it up, then we can modify | 19:51 |
| * anteaya goes back to trying to figure out how to stand up a gerrit with puppet for the purposes of testing manage-projects | 19:52 | |
| *** denis_makogon_ is now known as denis_makogon | 19:52 | |
| *** DinaBelova_ is now known as DinaBelova | 19:53 | |
| *** Hefeweizen has joined #openstack-infra | 19:54 | |
| *** seagulls is now known as beagles | 19:56 | |
| *** rfolco has quit IRC | 19:56 | |
| *** vkozhukalov has joined #openstack-infra | 20:02 | |
| *** AaronGr is now known as AaronGr_Zzz | 20:05 | |
| *** mriedem1 has joined #openstack-infra | 20:06 | |
| *** AaronGr_Zzz is now known as AaronGr | 20:07 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 20:08 |
| *** mriedem has quit IRC | 20:08 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Switch over to oslosphinx https://review.openstack.org/73676 | 20:10 |
| *** mgagne has quit IRC | 20:14 | |
| *** amcrn has quit IRC | 20:15 | |
| *** apevec has quit IRC | 20:15 | |
| *** alexpilotti_ has joined #openstack-infra | 20:16 | |
| *** amcrn has joined #openstack-infra | 20:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Don't offline after periodic-mirror-python26 https://review.openstack.org/73701 | 20:16 |
| *** alexpilotti has quit IRC | 20:18 | |
| *** hogepodge has quit IRC | 20:20 | |
| *** alexpilotti_ has quit IRC | 20:23 | |
| *** senthil_ has joined #openstack-infra | 20:23 | |
| *** gokrokve has joined #openstack-infra | 20:24 | |
| *** cadenzajon has quit IRC | 20:29 | |
| *** gokrokve has quit IRC | 20:31 | |
| *** gokrokve has joined #openstack-infra | 20:31 | |
| mordred | anteaya, fungi: wasnt' there something about the swift support in owncloud not actually being usable for storing the content? | 20:32 |
| *** yolanda has quit IRC | 20:32 | |
| *** gokrokve_ has joined #openstack-infra | 20:32 | |
| sdague | jeblair: where do we stand on freshness check? | 20:33 |
| fungi | sdague: https://review.openstack.org/73418 merged | 20:33 |
| fungi | sdague: i'm exercising it at https://review.openstack.org/62206 now | 20:33 |
| fungi | reverify caused it to restart into the check pipeline | 20:34 |
| anteaya | mordred: I don't think I got far enough to find out | 20:34 |
| fungi | we'll see if it ends up in the gate if/when that succeeds | 20:34 |
| mordred | k | 20:34 |
| *** AlexF has quit IRC | 20:34 | |
| *** e0ne has quit IRC | 20:35 | |
| anteaya | mordred: I was instructed to use the gui but couldn't get the text box or button to show up so I could add swift | 20:35 |
| mordred | nice | 20:35 |
| *** gokrokve has quit IRC | 20:35 | |
| sdague | fungi: ok, cool, was wondering about being about to lean on +A on old results and it would trigger all fine | 20:35 |
| AaronGr | is there any chance some of the people who reviewed https://review.openstack.org/#/c/66072/ previously could re-check it and maybe merge it? | 20:37 |
| *** gokrokve_ has quit IRC | 20:37 | |
| fungi | sdague: should be fine to do so, assuming it works as we intend | 20:38 |
| sdague | coolio | 20:39 |
| fungi | AaronGr: i would calculate that chance as being a non-zero value, at least | 20:39 |
| AaronGr | fungi: as would i, given an unknowable future. | 20:40 |
| fungi | AaronGr: i'm hedging my bed slightly, by actually reviewing it | 20:41 |
| fungi | s/bed/bet/ | 20:41 |
| * anteaya wonders how fungi would hedge a bed | 20:41 | |
| *** sarob has quit IRC | 20:41 | |
| fungi | anteaya: with very short shrubberies | 20:42 |
| anteaya | and a herring? | 20:42 |
| *** sarob has joined #openstack-infra | 20:42 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: install gerrit core plugins https://review.openstack.org/65368 | 20:42 |
| AaronGr | fungi: you're my new favorite person. | 20:43 |
| fungi | AaronGr: i'll try to fix that misperception soon, don't worry | 20:43 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/73709 | 20:44 |
| * anteaya hads AaronGr the 'u' he forgot | 20:46 | |
| anteaya | I know thou art a canuck | 20:46 |
| *** sarob has quit IRC | 20:46 | |
| fungi | AaronGr: edumacate me on teh rubies. what does the @ in @certname mean there (in modules/openstack_project/templates/puppet.conf.erb)? | 20:47 |
| *** sarob has joined #openstack-infra | 20:47 | |
| mordred | wow. I missed something | 20:47 |
| mordred | dhellmann: we renamed oslosphinx to oslosphinx? | 20:47 |
| mordred | gah | 20:47 |
| dhellmann | mordred: yes | 20:48 |
| fungi | mordred: yes, for great justice | 20:48 |
| mordred | dhellmann: what's the tl;dr ? | 20:48 |
| dhellmann | mordred: we didn't have enough going on | 20:48 |
| mordred | ok | 20:48 |
| AaronGr | fungi: best practice is to @variable vs @variable, to prevent pulling in a global in place of a local afaik | 20:48 |
| mordred | great | 20:48 |
| fungi | namespaces, namespaces, namespaces | 20:48 |
| mordred | Im glad we foudn somethign to make us busy | 20:48 |
| dhellmann | mordred: namespace packages and in editable and production mode do not mix | 20:48 |
| dhellmann | sorry, that first response wasn't actually in response to your question :-) | 20:48 |
| fungi | AaronGr: ahh, so @ signifies a global? | 20:48 |
| mordred | dhellmann: are you just saing "python packaging is broken" using more words? | 20:49 |
| dhellmann | yes, I am being specific about the way its brokenness hit us this time | 20:49 |
| mordred | awesome. I'm so excited about taht | 20:49 |
| AaronGr | fungi: turns out i was wrong: http://docs.puppetlabs.com/guides/templating.html | 20:50 |
| mordred | btw - I believe we should just start using npm and bower to package our python code | 20:50 |
| AaronGr | see under referencing variables fungi: t's to prevent collision with ruby method names. | 20:50 |
| *** oubiwann_ has quit IRC | 20:50 | |
| fungi | AaronGr: ah! we should be doing this in many more places, i guess | 20:51 |
| AaronGr | fungi: yep, and there will even be warnings in the syslog on the puppetmaster about it | 20:51 |
| fungi | mordred: can't we just put our packages on cpan? | 20:51 |
| AaronGr | fungi: for now, just adjusting what i'm working on, to keep the patch small | 20:52 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Make main savanna dg job 'full' https://review.openstack.org/73712 | 20:52 |
| clarkb | fungi: pleia2 has a change up with that addition of @s for fedora puppet | 20:52 |
| fungi | AaronGr: that's fine, just making a mental note to use safer variable reference syntax in the future | 20:52 |
| clarkb | I need to test that somewhere and make sure it doesn't break firewall rules | 20:52 |
| *** alexpilotti has joined #openstack-infra | 20:53 | |
| *** mburned has quit IRC | 20:53 | |
| fungi | clarkb: AaronGr: i'm going to noop test 66072 on the master just make sure puppet doesn't see problems i missed, but otherwise lgtm | 20:53 |
| AaronGr | fungi: thanks! | 20:54 |
| *** mwagner_lap has quit IRC | 20:54 | |
| *** lcestari has quit IRC | 20:55 | |
| clarkb | fungi: ty | 20:55 |
| *** lcestari has joined #openstack-infra | 20:55 | |
| clarkb | I think I have the most important new machine bits sorted, I am going to dive into review now | 20:55 |
| *** lcestari has quit IRC | 20:55 | |
| dhellmann | I just had a docs job fail because it couldn't install sphinx -- is that something we're seeing a lot of, or is this new? | 20:56 |
| dhellmann | https://jenkins01.openstack.org/job/gate-oslo.config-docs/386/console | 20:56 |
| dhellmann | hmm, I wonder if that version # is right | 20:56 |
| *** _david_ has quit IRC | 20:58 | |
| clarkb | fungi: is there a change to have git-review tests reduce concurrency? if not oen of us should write it and have the other review and approve | 20:58 |
| *** melwitt has joined #openstack-infra | 20:58 | |
| clarkb | fungi: I think the tsets are more reliable due to some of the other changes that happened so reducing concurrency should get us out of the woods | 20:58 |
| fungi | dhellmann: should find 1.1.3 (i see it on the mirror) | 20:58 |
| anteaya | AaronGr: I always believed @variable was an instance variable, am I wrong here | 20:58 |
| fungi | dhellmann: though "Cannot fetch index base URL http://pypi.openstack.org/openstack/" | 20:58 |
| fungi | dhellmann: could be broken network, or static.o.o didn't respond | 20:59 |
| dhellmann | fungi: didn't I hear something about sporadic dins issues? | 20:59 |
| dhellmann | fungi: the patch just after it in the gate passed ok | 20:59 |
| dhellmann | I'll resubmit it in a while | 20:59 |
| fungi | dhellmann: we have been seeing a few of those. quite a few apparently. seems to only affect slaves in rackspace too | 20:59 |
| dhellmann | hmm, when I run that command locally I get a weird error about "source_dir must be a directory name" and then it reports the name of a thing that is a directory | 21:00 |
| fungi | dns timeouts i mean. in this case not sure if the failure is dns related (pip may be hiding that from us) | 21:00 |
| *** lcestari has joined #openstack-infra | 21:00 | |
| dhellmann | yeah, that seems likely | 21:00 |
| fungi | you know, for kids! | 21:01 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 21:01 |
| *** gokrokve has joined #openstack-infra | 21:02 | |
| *** dprince has quit IRC | 21:03 | |
| *** shardy_afk is now known as shardy | 21:03 | |
| dhellmann | the sphinx error I get locally looks like it is http://bugs.python.org/issue19570 | 21:03 |
| *** gokrokve_ has joined #openstack-infra | 21:04 | |
| *** gyee has quit IRC | 21:04 | |
| fungi | dhellmann: ahh, yeah, that looks familiar. i forgot how/whether we work around it | 21:04 |
| fungi | oh, we pin <1.2 | 21:05 |
| fungi | and use a new enough pip so as to not automagically install pre-release versions | 21:05 |
| *** mburned has joined #openstack-infra | 21:05 | |
| *** gokrokv__ has joined #openstack-infra | 21:05 | |
| dhellmann | ah, I ended up with 1.2b3 in this environment | 21:05 |
| *** Ajaeger has quit IRC | 21:06 | |
| fungi | computers are conspiring against you | 21:06 |
| dhellmann | ok, well, I just wanted to verify the docs did build so I ran sphinx-build directly and all is well | 21:06 |
| *** gokrokve has quit IRC | 21:06 | |
| *** lcestari has quit IRC | 21:07 | |
| clarkb | fungi: want me to push the git-review test reduction change? if I do that can you push it through? | 21:07 |
| *** krtaylor has quit IRC | 21:08 | |
| *** lcestari has joined #openstack-infra | 21:08 | |
| fungi | clarkb: sure | 21:08 |
| *** gokrokve_ has quit IRC | 21:08 | |
| *** lcestari has quit IRC | 21:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: do not gate on unit tests, docs, or pep8 https://review.openstack.org/73714 | 21:09 |
| sdague | so.... can folks look at that? | 21:09 |
| sdague | because I'm curious if it will have other fall out | 21:10 |
| *** gokrokv__ has quit IRC | 21:10 | |
| *** pafuent has left #openstack-infra | 21:11 | |
| *** cadenzajon has joined #openstack-infra | 21:11 | |
| anteaya | sdague: I too am curious, as I stated in my comment | 21:12 |
| *** DinaBelova is now known as DinaBelova_ | 21:13 | |
| *** _david_ has joined #openstack-infra | 21:13 | |
| *** denis_makogon has quit IRC | 21:13 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/git-review: Reduce testr concurrnecy to 2. https://review.openstack.org/73715 | 21:15 |
| clarkb | fungi: ^ | 21:15 |
| clarkb | sdague: looking | 21:15 |
| clarkb | sdague: I think that is all that is necessary less areas where the template isn't used but all of openstack* should use the template at this point | 21:15 |
| sdague | yeh, spot check seemed sane on that | 21:16 |
| *** alexpilotti has quit IRC | 21:16 | |
| sdague | oh, not quite | 21:17 |
| sdague | how do I convert something to template? | 21:17 |
| fungi | sdague: you want to create a new template, or add an existing template to another project? | 21:17 |
| clarkb | sdague: if you look at say openstack/nova you will see template application | 21:17 |
| clarkb | iirc it is 'templates:\n - template-name\n' | 21:18 |
| *** mfisch has joined #openstack-infra | 21:18 | |
| fungi | however this can also be implemented piecemeal if necessary. if we miss jobs for a project or two, we can spot those after it's applied and take care of them in a follow-on change | 21:19 |
| mfisch | is zuul backed up? I have a commit with a +2 that's just been sitting there since last night | 21:19 |
| fungi | mfisch: got a link to the review/> | 21:19 |
| fungi | ? | 21:19 |
| clarkb | mfisch: not that backed, up can you link to the change? | 21:19 |
| mfisch | fungi: https://review.openstack.org/#/c/70837/ | 21:19 |
| clarkb | fungi: :P | 21:19 |
| clarkb | mfisch: jenkins has voted, what were you expecting to happen instead? | 21:20 |
| fungi | mfisch: it failed to merge, then you ran recheck instead of reverify | 21:20 |
| fungi | oh, it's not approved anyway | 21:20 |
| fungi | right, what clarkb said then | 21:20 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: do not gate on unit tests, docs, or pep8 https://review.openstack.org/73714 | 21:20 |
| sdague | there are a few other projects which are weird, like heat | 21:20 |
| mfisch | I thought +2 was approved, missed that | 21:21 |
| fungi | mfisch: so you have a successful check verification from jenkins and positive core review votes from dolphm and bknudson, but no approval | 21:22 |
| fungi | mfisch: once a core reviewer for keystone approves that change, then it will be tested again in the gate and (hopefully, if all goes well) merge | 21:23 |
| mfisch | fungi: yep, I misconstrued +2 as == approval | 21:23 |
| mfisch | thanks and sorry for the noise | 21:23 |
| fungi | mfisch: no worries | 21:23 |
| *** changbl has quit IRC | 21:24 | |
| *** changbl has joined #openstack-infra | 21:24 | |
| *** hogepodge has joined #openstack-infra | 21:26 | |
| clarkb | jeblair: fungi: jenkins slave sudoer rights, is that something that should wait for post weekend due to image build delays? | 21:26 |
| clarkb | if we are comfortable pushing that through today I can approve things as I review them. Otherwise will mark votes as appropriate less approvals | 21:27 |
| fungi | clarkb: i'll be around all weekend if we need to undo it | 21:27 |
| clarkb | in that case I will use the go button | 21:27 |
| jeblair | clarkb: we should probably be around for the image build on that; i'd say trigger one now, but i keep restarting nodepool, so lets wait till next week | 21:27 |
| *** Ace_Rimmer has joined #openstack-infra | 21:27 | |
| jeblair | oh | 21:27 |
| clarkb | jeblair: unless we want to give fungi a break | 21:27 |
| clarkb | which is a very valid thing to do :) | 21:27 |
| fungi | next week is fine by me | 21:28 |
| jeblair | cool | 21:28 |
| clarkb | ok, I have starred the beginning of the stack to remind me where to look next week | 21:28 |
| clarkb | will review now | 21:28 |
| *** Ace_Rimmer has left #openstack-infra | 21:29 | |
| fungi | clarkb: i also have a couple changes to git cloning for image builds hanging out there which might want to wait for us to test image building again | 21:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: do not gate on unit tests, docs, or pep8 https://review.openstack.org/73714 | 21:29 |
| clarkb | fungi: roger | 21:29 |
| clarkb | I think that stuff and zaro's gerrit changes are high on my list | 21:29 |
| clarkb | then swing around and do chronological review | 21:30 |
| fungi | i'm still trying to figure out why puppet agent --test --noop --environment=fungi isn't showing any delta at all on the puppet master (i have 66072 cherry-picked into my dev env) | 21:30 |
| clarkb | fungi: pretty sure it shouldn't right? | 21:31 |
| clarkb | its an in place cleanup so no delta is good | 21:31 |
| fungi | ahh, yep. if we *did* have a delta, that would indicate a problem | 21:31 |
| clarkb | ya | 21:31 |
| fungi | totally agree | 21:31 |
| fungi | clearly friday afternoon brain cache miss | 21:31 |
| anteaya | another reason you should take the weekend off | 21:32 |
| anteaya | clear the cache | 21:32 |
| fungi | i normally clean my cache with an all-natural grain alcohol | 21:32 |
| jeblair | sdague: then next time you respin that, can you change the commit message? it asserts some things that i don't think we actually believe (eg, gating on unit tests doesn't make sense). basically i don't think it's contextualized right. i'm fine with it being shorter or longer; mostly just different. | 21:33 |
| anteaya | fungi: stored in a mason jar, I presume? | 21:33 |
| fungi | anteaya: no, but that would be amusing | 21:33 |
| *** oubiwann has joined #openstack-infra | 21:33 | |
| anteaya | something to shoot for | 21:34 |
| *** sabari has quit IRC | 21:37 | |
| *** sabari has joined #openstack-infra | 21:37 | |
| *** banix has joined #openstack-infra | 21:39 | |
| anteaya | jaypipes: I love that your workbox is called cranky | 21:39 |
| jaypipes | anteaya: heh | 21:41 |
| jaypipes | man... partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk is a beast to download... took almost half an hour :( | 21:41 |
| clarkb | jeblair: please see comments on https://review.openstack.org/#/c/72779/ | 21:42 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add grenade-dsvm-rolling test https://review.openstack.org/64200 | 21:43 |
| *** dstanek has quit IRC | 21:44 | |
| fungi | jaypipes: maybe you want https://packages.debian.org/search?keywords=openstack-debian-images | 21:45 |
| jeblair | clarkb: i didn't put revoke-sudo on jobs on long-running jenkins slaves which don't have sudo enabled; however, it does seem like it would be unlikely to hurt, so i could go ahead and add them just for more better future proofing. | 21:45 |
| jaypipes | fungi: nah, this is something in prepare_devstack.sh nodepool script... I think. | 21:45 |
| jeblair | so on the nodepool memleak front, i was wrong (user error with objgraph); the problem is not related to config reloads | 21:46 |
| clarkb | jeblair: yeah I was thinking it would reduce the amount of thought cycles needed t odetermine if it is necessary | 21:46 |
| fungi | jaypipes: oh | 21:46 |
| jaypipes | fungi: just noticed it was taking a while when executing that script on slave nodes.. | 21:46 |
| jeblair | jaypipes: yep. and we don't use it either. | 21:46 |
| jaypipes | LOL :) | 21:46 |
| mordred | maybe we should delete the reference to it | 21:46 |
| jaypipes | well, luckily, it only gets pulled once. | 21:46 |
| jaypipes | and cached... | 21:46 |
| jeblair | mordred: devstack _can_ use it; however, i think since devstack has grown plugin mechanisms, perhaps including that in the default config could be revisited | 21:47 |
| anteaya | installing it never is faster than installing it once | 21:47 |
| mordred | anteaya: ++ | 21:47 |
| fungi | jaypipes: also, 73073 and 73076 will probably speed up image builds a good bit too | 21:47 |
| mordred | jeblair: nod | 21:47 |
| *** mriedem1 is now known as mriedem | 21:48 | |
| mordred | fungi: any reason to not +A those? | 21:48 |
| jeblair | mordred: we're giving fungi the weekend off | 21:48 |
| *** jeckersb has joined #openstack-infra | 21:48 | |
| *** dpyzhov has joined #openstack-infra | 21:48 | |
| fungi | mordred: jeblair is troubleshooting nodepool memory leaks, and we'd probably want to babysit the image builds | 21:48 |
| mordred | jeblair: great reason | 21:49 |
| fungi | we apparently don't keep enough bare-centos6 nodes on hand... 73715,1 has been waiting on a worker for gate-git-review-python26 for 35 minutes and counting | 21:50 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1280072 https://review.openstack.org/73719 | 21:51 |
| fungi | shall i bump the minimum reserve? | 21:51 |
| jeblair | fungi: just a sec | 21:51 |
| fungi | jeblair: no hurry, just making a mental note that we ought to tune it up | 21:51 |
| jeblair | fungi: well, now isn't the best time to evaluate node starvation because of my debugging | 21:52 |
| fungi | good point | 21:52 |
| *** sarob has quit IRC | 21:58 | |
| jaypipes | fungi, mordred, jeblair: Here is some output from devstack-based test runs on a slave: http://paste.openstack.org/show/65580/ . Curious about the sudo issue there... I'm executing devstack-checkout macros and then trying to ./safe-devstack-gate-wrap.sh in the job. How do you solve sudo problems on the Jenkins slaves upstream? | 21:58 |
| *** sarob has joined #openstack-infra | 21:59 | |
| jaypipes | I've installed the jenkins::jenkinsuser manifest which installs the jenkins-sudoers-grep sudoers config, but anything else I'm missing you can think od? | 21:59 |
| jaypipes | of.. | 21:59 |
| jeblair | jaypipes: devstack-precise nodes are contstructed by nodepool with the puppet param to allow sudo | 21:59 |
| *** masayukig has joined #openstack-infra | 21:59 | |
| jeblair | jaypipes: unit test slaves do not allow sudo | 21:59 |
| mordred | jaypipes: what jeblair said | 22:00 |
| jaypipes | jeblair: aha... got it. let me pass sudo => true. | 22:00 |
| jaypipes | cheers | 22:00 |
| jeblair | jaypipes: and everything about that is about to change (see https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:sudoers,n,z ) | 22:00 |
| jaypipes | crap :) | 22:00 |
| jaypipes | thx for the heads up! | 22:00 |
| jeblair | well, almost everything | 22:00 |
| jaypipes | jeblair: well, looks like those patches actually clean all that up substantially, which is great. when they merge, they merge into my solution, too... so I should be ok. | 22:01 |
| jaypipes | (since i'm using as close to upstream as I could) | 22:02 |
| *** dpyzhov has quit IRC | 22:02 | |
| jeblair | yeah, the end goal is all slaves start by allowing sudo and then jobs revoke it themselves when not needed | 22:02 |
| jeblair | (this only works for single use slaves; but all our slaves are going to be single use) | 22:03 |
| *** mfer has quit IRC | 22:03 | |
| jaypipes | got it. makes sense now that nodepool is responsible for spinning up and maintaining the pool. | 22:03 |
| *** sarob has quit IRC | 22:03 | |
| *** bhuvan has quit IRC | 22:04 | |
| *** gokrokve has joined #openstack-infra | 22:04 | |
| *** bhuvan has joined #openstack-infra | 22:05 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1280464 https://review.openstack.org/73726 | 22:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Mark resolved queries https://review.openstack.org/73466 | 22:08 |
| *** david-lyle has quit IRC | 22:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: disable the jeepyb trivial rebase hook on review-dev.o.o https://review.openstack.org/69800 | 22:10 |
| clarkb | zaro: ^ | 22:10 |
| *** krtaylor has joined #openstack-infra | 22:14 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1280072 https://review.openstack.org/73719 | 22:14 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1280464 https://review.openstack.org/73726 | 22:14 |
| *** masayukig has quit IRC | 22:14 | |
| *** dpyzhov has joined #openstack-infra | 22:14 | |
| *** thomasem has quit IRC | 22:15 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed a change to openstack/requirements: Update hp3parclient low version number https://review.openstack.org/73727 | 22:16 |
| *** khyati_ has joined #openstack-infra | 22:17 | |
| *** dpyzhov has quit IRC | 22:17 | |
| *** _david_ has quit IRC | 22:18 | |
| *** senthil_ has quit IRC | 22:19 | |
| *** andre__ has quit IRC | 22:19 | |
| *** banix has quit IRC | 22:21 | |
| anteaya | hemna: ah you are walter the fourth | 22:24 |
| hemna_ | anteaya, yah | 22:24 |
| anteaya | a buddy of mine is george the fourth, he doesn't go by it though | 22:24 |
| hemna_ | King George! :) | 22:25 |
| anteaya | hemna_: cool, nice to place a number to the nick | 22:25 |
| anteaya | ah, not so much | 22:25 |
| anteaya | just unimaginative family naming | 22:25 |
| hemna_ | soren, question regarding my patch actually | 22:25 |
| hemna_ | my cinder driver unit tests are still failing because it's using 2.9.2 on the jenkins box | 22:25 |
| hemna_ | even though my last patch to update the requirement to <4.0.0 landed | 22:26 |
| hemna_ | soren, my latest patch is to update the low number to >=3.0.0 from >=2.0.0 | 22:26 |
| hemna_ | pypi.openstack.org/openstack/hp3parclient has 3.0.0, so I figured it was safe to post the new patch requiring a minimum of 3.0.0 | 22:26 |
| hemna_ | s/soren/so | 22:27 |
| *** prad has quit IRC | 22:29 | |
| mordred | hrm. interesting that it's using 2.9.2 if the irror has 3.0.0 | 22:30 |
| *** dpyzhov has joined #openstack-infra | 22:30 | |
| *** CaptTofu_ has quit IRC | 22:31 | |
| *** banix has joined #openstack-infra | 22:31 | |
| BobBall | Just noticed the spike in test nodes... Dunno if it's a problem if if you guys are already aware :) | 22:32 |
| hemna_ | mordred, the mirror has both 2.9.2 and 3.0.0 | 22:32 |
| *** bhuvan has quit IRC | 22:33 | |
| *** hogepodge has quit IRC | 22:33 | |
| jeblair | BobBall: i'm causing it. :) friday afternoon interactive debugging. | 22:33 |
| *** bhuvan has joined #openstack-infra | 22:33 | |
| *** resker has quit IRC | 22:33 | |
| BobBall | haha | 22:34 |
| *** rwsu has quit IRC | 22:34 | |
| *** esker has joined #openstack-infra | 22:34 | |
| BobBall | That's the second best sort of debugging. The best being four-pint debugging of course. | 22:34 |
| jgriffith | hemna_: I still don't see any value in those external libs at all | 22:35 |
| mordred | BobBall: ++ | 22:36 |
| jgriffith | hemna_: I thought you were working on changing that? | 22:36 |
| hemna_ | we'll look at removing the unit test dependence on the external lib. | 22:37 |
| hemna_ | but probably not until juno | 22:37 |
| *** esker has quit IRC | 22:39 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Run all non-sensitive jobs on single-use workers https://review.openstack.org/73732 | 22:40 |
| fungi | that's a thing. probably an ugly implementation, but we can refine it | 22:40 |
| jeblair | woo | 22:41 |
| fungi | clarkb: the current node starvation in the check pipeline is delaying results for the git-review change, so we can't approve it just yet anyway... i need to pop out and grab some dinner but i can check back in on it later | 22:41 |
| clarkb | fungi: ya no rush | 22:41 |
| clarkb | it is getting late before the weekend after all | 22:42 |
| *** rcleere has quit IRC | 22:42 | |
| fungi | oh, 73732 has a typo :( | 22:42 |
| *** VijayT has quit IRC | 22:42 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Run all non-sensitive jobs on single-use workers https://review.openstack.org/73732 | 22:43 |
| fungi | all better | 22:43 |
| *** dpyzhov has quit IRC | 22:44 | |
| *** mriedem has quit IRC | 22:44 | |
| *** gokrokve_ has joined #openstack-infra | 22:45 | |
| clarkb | fungi: left a comment with a question | 22:46 |
| *** dpyzhov has joined #openstack-infra | 22:47 | |
| fungi | clarkb: replied. i can look into it after dinner | 22:48 |
| * fungi disappears | 22:48 | |
| *** gokrokve has quit IRC | 22:48 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 22:48 |
| *** pdmars has quit IRC | 22:50 | |
| *** hogepodge has joined #openstack-infra | 22:50 | |
| *** dpyzhov has quit IRC | 22:50 | |
| jeblair | clarkb, fungi: jobs /win 22 | 22:52 |
| *** dcramer_ has quit IRC | 22:52 | |
| jeblair | jeez | 22:52 |
| jeblair | clarkb, fungi: replied in review | 22:52 |
| *** hogepodge has quit IRC | 22:53 | |
| *** mrodden has quit IRC | 22:57 | |
| sdague | jeblair: sorry, was away for a bit. Because we use a single template, is there any good way to pull those jobs on just integrated projects? Should we make a second template? | 22:58 |
| jeblair | sdague: i think probably a second template; openstack_python_jobs or something | 22:58 |
| *** sarob has joined #openstack-infra | 22:59 | |
| sdague | jeblair: or, what if there is a gate-on-python-jobs template? | 23:00 |
| sdague | then just add that to the projects that want to do that? | 23:00 |
| sdague | hmmm... actually that's probably more error prone | 23:00 |
| jeblair | sdague: yeah, the other way is following the conventions in jjb too | 23:00 |
| *** hogepodge has joined #openstack-infra | 23:00 | |
| clarkb | AaronGr: fungi: https://review.openstack.org/#/c/66072/ lgtm, did not approve as I am hoping fungi our fearless tester may be willing to babysit | 23:01 |
| clarkb | if he has ENOTIME let me know and I can babysit | 23:01 |
| jeblair | clarkb, fungi: actually, i want AaronGr to babysit | 23:01 |
| jeblair | clarkb, fungi: which means that we need a puppetboard up first | 23:01 |
| clarkb | I see | 23:01 |
| jeblair | clarkb, fungi: i discussed this with AaronGr recently, it shouldn't be a surprise... | 23:02 |
| clarkb | jeblair: ok | 23:02 |
| clarkb | will keep that in mind as I review, thanks | 23:02 |
| jeblair | clarkb, fungi: his plan is to make major changes to puppet, which is great, but it's not efficient for us to just tail logs all day seeing if something broke somewhere | 23:02 |
| *** rwsu has joined #openstack-infra | 23:04 | |
| *** vkozhukalov has quit IRC | 23:04 | |
| *** sarob has quit IRC | 23:05 | |
| AaronGr | jeblair: sorry, i hadn't realized you meant all changes. I'll hold off on any other patches/requests. | 23:06 |
| *** banix has quit IRC | 23:06 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: do not gate on unit tests, docs, or pep8 https://review.openstack.org/73714 | 23:07 |
| ewindisch | I'm seeing automatic rechecks happening after people review patchsets | 23:07 |
| ewindisch | I presume that shouldn't be happening? | 23:07 |
| jeblair | ewindisch: depends on how old the last recheck was | 23:08 |
| ewindisch | jeblair: ah. okay. yeah, the review sat for 3 days from the last check, then sdague gave it a +2 and it rechecked | 23:08 |
| jeblair | ewindisch: i think we're currently rechecking things at 72 hours, and then if something is approved, we recheck if its older than 24h | 23:09 |
| ewindisch | acknowledged- thanks | 23:09 |
| *** sarob has joined #openstack-infra | 23:09 | |
| sdague | ewindisch: right, this is working as designed | 23:10 |
| clarkb | the goal being that reviewers always have relatively up to date results so that out of date code that can't get through the gate doesn't end up in the gate | 23:10 |
| *** thedodd has quit IRC | 23:12 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerritlib: remove extra item from listGroups and listProjects methods https://review.openstack.org/70301 | 23:13 |
| anteaya | AaronGr: what do you need to get a puppetboard up? | 23:14 |
| *** sarob has quit IRC | 23:14 | |
| anteaya | my understanding is puppetdb, is that correct? | 23:14 |
| anteaya | and nibalizer was working on that and then hit puppetdb firewall not mixing with our iptables solution | 23:15 |
| AaronGr | yes | 23:15 |
| anteaya | is there something new in that | 23:15 |
| anteaya | or that is where we stand | 23:15 |
| AaronGr | that's current state afaik | 23:15 |
| jog0 | jeblair: when did the 24h rule land? | 23:15 |
| anteaya | so we need an item on next week's agenda to hash out puppetdb firewall meshing with our iptables, correct? | 23:16 |
| clarkb | is puppet db running its own firewall? | 23:16 |
| anteaya | yes | 23:16 |
| clarkb | why? | 23:16 |
| anteaya | no idea | 23:16 |
| * clarkb thinks puppetdb should be a db not a firewall | 23:16 | |
| anteaya | this is what I understand from nibalizer | 23:16 |
| clarkb | nibalizer: ^ poke | 23:16 |
| anteaya | I agree with you clarkb | 23:16 |
| sdague | jog0: earlier today | 23:16 |
| sdague | top of gate is currently reset on a py27 fail in neutron | 23:17 |
| anteaya | yes | 23:17 |
| anteaya | just talked to markmcclain about it | 23:17 |
| sdague | the bulk of failures I saw earlier today where in unit tests and pep8 (just spot looking) | 23:17 |
| *** eharney has quit IRC | 23:17 | |
| sdague | anteaya: yeh, this is mostly in response to jog0's review comments | 23:17 |
| *** sarob has joined #openstack-infra | 23:17 | |
| anteaya | he will talk to nicira vshield people about fixing their races in their unit tests | 23:17 |
| mordred | puppetdb is running a firewall? | 23:18 |
| anteaya | sdague: very good | 23:18 |
| jog0 | sdague: were those before or after the 24h patch landed? | 23:18 |
| anteaya | mordred: yes, this is nibalizer obstacle de jour | 23:18 |
| sdague | jog0: after | 23:18 |
| anteaya | or of the week | 23:18 |
| jog0 | clarkb: http://status.openstack.org/elastic-recheck/data/uncategorized.html hasn't updated since yesterday | 23:18 |
| sdague | the py27 is | 23:18 |
| jog0 | see timestamp | 23:18 |
| clarkb | mordred: anteaya: fungi https://review.openstack.org/#/c/66521/3 are we still running manage-projects by hand for those? | 23:18 |
| anteaya | jog0: I saw all tests pass on that patch this afternoon | 23:19 |
| anteaya | it would have passed check | 23:19 |
| sdague | the others were before, but they weren't stale failures, they were races, either inherent or network based | 23:19 |
| jog0 | anteaya: which patch? | 23:19 |
| anteaya | clarkb: I think that automatically is back up, though it might fall over | 23:19 |
| anteaya | jog0: https://review.openstack.org/#/c/65838/ | 23:19 |
| anteaya | jog0: I watched in the gate this afternoon | 23:20 |
| clarkb | anteaya: thanks | 23:20 |
| anteaya | all green at least once | 23:20 |
| clarkb | jog0: looks like the lock is in place and the crons haven't udpated, did I merge that change? | 23:20 |
| anteaya | clarkb: np | 23:20 |
| sdague | jog0: also - https://review.openstack.org/#/c/67395/ nova failing on py26, on what looks like a node setup issue | 23:20 |
| clarkb | no I didn't merge that change so cron thing isn't a problem | 23:20 |
| sdague | caused gate reset | 23:20 |
| clarkb | jog0: but yeah something died and didn't clean up the lock best I can tell | 23:21 |
| clarkb | will clean it up | 23:21 |
| sdague | right now | 23:21 |
| jog0 | clarkb: thanks | 23:21 |
| anteaya | clarkb: https://review.openstack.org/#/c/72178/ | 23:21 |
| anteaya | and has a dependency, the logging patch | 23:22 |
| *** rwsu has quit IRC | 23:22 | |
| clarkb | hrm maybe it wasn't locked, don't know what is going on | 23:23 |
| *** krtaylor has quit IRC | 23:24 | |
| jog0 | sdague: so that looks like a new race caused by taht patch | 23:24 |
| jog0 | sdague: https://review.openstack.org/#/c/65838/ that is | 23:24 |
| jog0 | anteaya: ^ | 23:24 |
| jog0 | only one hit for that bug in the check queue and its for that patch | 23:24 |
| jog0 | if we didn't have that py27 this would have landed and broke neutron | 23:25 |
| *** dims_ has quit IRC | 23:25 | |
| jog0 | erp wait | 23:25 |
| jog0 | ignore that, I clicked the wrong button | 23:25 |
| *** sarob has quit IRC | 23:25 | |
| jog0 | sdague: this is the type of race condition that you don't like, you win this one! | 23:26 |
| clarkb | jog0: not sure what is going on, do they work for you locally? | 23:26 |
| * jog0 files a neutron bug | 23:26 | |
| *** sarob has joined #openstack-infra | 23:26 | |
| anteaya | nibalizer AaronGr Hunner: new item on next week's agenda to discuss puppetdb firewall and iptables conflict | 23:26 |
| *** mriedem has joined #openstack-infra | 23:27 | |
| jeblair | anteaya: why do we need to talk about that at the meeting? | 23:27 |
| clarkb | anteaya: is that conflict described anywere? | 23:27 |
| anteaya | jeblair: well discusion seemed to disolve the other day when nibalizer brought it up | 23:27 |
| sdague | jog0: I will be very surprised if you look at the fails over python27 in the gate, and > 10% of them were a real issue. | 23:27 |
| anteaya | clarkb: not that I know of, just going by what nibalizer said | 23:27 |
| clarkb | anteaya: was that logged in channel? | 23:27 |
| sdague | real issue == caused by that patch | 23:28 |
| * clarkb googles | 23:28 | |
| anteaya | if we can discuss it earlier I am all for it | 23:28 |
| sdague | this is why I'm promoting this POV | 23:28 |
| anteaya | just don't want it to not get discussed | 23:28 |
| clarkb | google fails me | 23:28 |
| jog0 | sdague: I assume you mean starting since the 24 hour rule landed? | 23:29 |
| jeblair | anteaya: what needs discussing? | 23:29 |
| mordred | clarkb: I should just ignore that puppetdb requires postgres shoudln't i? | 23:29 |
| clarkb | mordred: yes | 23:29 |
| anteaya | jeblair: puppetdb has a firewall that interferes with our iptables | 23:29 |
| sdague | jog0: ever. It's not just a freshness issue | 23:30 |
| jeblair | anteaya: yep, someone needs to solve that. | 23:30 |
| clarkb | jog0: so theory time, elasticsearch upgrade tickles something in e-r? | 23:30 |
| anteaya | yes | 23:30 |
| clarkb | jeblair: the all and gate json files are old too | 23:30 |
| clarkb | er jog0 ^ | 23:30 |
| anteaya | so it felt like to me that nibalizer needed some additional direction about how to proceed | 23:30 |
| anteaya | because after he announced that I haven't seen him since | 23:30 |
| jog0 | clarkb: it failed for me locally | 23:30 |
| *** banix has joined #openstack-infra | 23:30 | |
| *** sarob has quit IRC | 23:31 | |
| jog0 | ValueError: time data u'2014-02-14T16:07:47.718+00:00' does not match format '%Y-%m-%dT%H:%M:%S.%fZ' | 23:31 |
| jog0 | ahh I know the issue | 23:31 |
| *** lascii is now known as alaski | 23:31 | |
| openstackgerrit | Kenny Jones proposed a change to openstack-dev/pbr: Updated parse_requirements to align to PIP https://review.openstack.org/73737 | 23:31 |
| jog0 | I guess not | 23:32 |
| anteaya | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-02-12.log timestamp 2014-02-12T00:15:37 | 23:32 |
| sdague | what' failing? | 23:32 |
| jeblair | anteaya: i think he has a pretty good grasp of the issues, but if he has something further to ask, i'm sure any of us would be happy to help. but i think this is a rather vague topic for the meeting. | 23:32 |
| sdague | on e-r | 23:32 |
| anteaya | I am fine with that characterisation | 23:32 |
| jog0 | sdague: timestamp = time.strptime(result.timestamp, "%Y-%m-%dT%H:%M:%S.%fZ") | 23:32 |
| anteaya | but not having puppetboard is blocking more and more work | 23:32 |
| jog0 | looks like a different timestamp format in ne ES | 23:33 |
| clarkb | jog0: try replace fZ with Z? | 23:33 |
| jog0 | new | 23:33 |
| anteaya | and i have no idea how to address this myself | 23:33 |
| anteaya | so since it is expanding its importance, I feel it needs some airtime | 23:33 |
| jog0 | clarkb: there is the new +00:00 | 23:33 |
| jeblair | anteaya: then people need to shift priorities. | 23:33 |
| anteaya | which people? | 23:33 |
| clarkb | jeblair: yeah it gives you timezone | 23:33 |
| clarkb | gah jog0^ | 23:34 |
| anteaya | jeblair: who and what priorities would you like them to have? | 23:34 |
| jeblair | anteaya: people for whom the lack of a puppetboard is blocking work should probably start working on puppetboard. | 23:35 |
| anteaya | okay | 23:35 |
| jeblair | anteaya: i don't get to set priorities for people though. i'm not their manager. | 23:35 |
| anteaya | well puppetboard isn't blocking any of my work | 23:35 |
| anteaya | I foolishly figured that since I know ruby and rails (or did at one point) that puppet stuff would be something I could understand | 23:36 |
| anteaya | but I don't | 23:36 |
| anteaya | it is not a well written tool | 23:36 |
| sdague | clarkb: is that going to remain constant? | 23:36 |
| anteaya | and doesn't fit any rails conventions | 23:36 |
| clarkb | sdague: it should | 23:36 |
| sdague | because +%z isn't valid in strptime | 23:36 |
| anteaya | so I am happy to not be involved | 23:36 |
| clarkb | sdague: ? /me looks harder | 23:36 |
| anteaya | and can easily remove the item from the meeting agenda | 23:37 |
| sdague | clarkb: ValueError: 'z' is a bad directive in format '%Y-%m-%dT:%M:%S.%f%z' | 23:37 |
| sdague | %z is what it should be | 23:37 |
| anteaya | done | 23:37 |
| clarkb | right its int te table | 23:37 |
| clarkb | sdague: I guess I don't understand why %z isn't valid | 23:37 |
| sdague | I don't either | 23:38 |
| jog0 | me neither | 23:38 |
| *** banix has quit IRC | 23:40 | |
| jog0 | clarkb: is it safe to say that the timezone will always be UTC or rather +00:00? | 23:40 |
| clarkb | http://bugs.python.org/issue6641 | 23:40 |
| clarkb | jog0: yes | 23:40 |
| jog0 | clarkb: good | 23:40 |
| *** dims_ has joined #openstack-infra | 23:40 | |
| sdague | http://stackoverflow.com/questions/14762518/python-datetime-strptime-and-strftime-how-to-preserve-the-timezone-informat | 23:41 |
| sdague | it's a platform level issue | 23:41 |
| clarkb | man strftime(3) on status.o.o shows %z | 23:42 |
| sdague | jog0: the general suggestion is to use python dateutil | 23:42 |
| jog0 | I have an ugly hack | 23:42 |
| jog0 | patch coming soon | 23:42 |
| jog0 | we always now its +00:00 so just hard code | 23:42 |
| clarkb | jog0: that should be fine | 23:44 |
| sdague | honestly, why not actually bring in the dateutil parse | 23:44 |
| sdague | to future proof us here | 23:44 |
| clarkb | wfm | 23:44 |
| *** blitzrage has quit IRC | 23:44 | |
| jog0 | sdague: I am fine with that, but don't exepct the timezone to change away from +00:00 | 23:46 |
| sdague | until the next upgrade, when it goes back to UTC, or something :) | 23:46 |
| clarkb | jog0: it might be different on other folks' ES | 23:47 |
| clarkb | or that ^ | 23:47 |
| sdague | I have a fix, let me run local unit tests first | 23:47 |
| sdague | then I'm calling it a night | 23:48 |
| *** sarob has joined #openstack-infra | 23:48 | |
| jog0 | sdague: cool, I have the alternate version | 23:49 |
| jog0 | I like how such a simple thing is so hard in python | 23:52 |
| nibalizer | anteaya: hi | 23:52 |
| nibalizer | i haven't made the openstack PR yet, but i've sortof resloved the problem | 23:53 |
| nibalizer | once https://github.com/puppetlabs/puppetlabs-puppetdb/pull/119 is merged, we can pull puppetdb module in from install_modules.sh | 23:54 |
| sdague | jog0: you have clearly not spent enough time with calendars, otherwise you would realize it is always hard | 23:54 |
| nibalizer | and i've been able to rectify that with openstack_project::base | 23:54 |
| nibalizer | though not openstack_project::server | 23:54 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: parse both new and old timestamp formats https://review.openstack.org/73740 | 23:54 |
| sdague | so I'm hitting an issue where time delta subtraction is failing, so I give up for now | 23:55 |
| jog0 | sdague: pushing that up just in case, I prefer your solution | 23:55 |
| * fungi readz all teh scrollbacks | 23:55 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wip: partial solution to real date parsing https://review.openstack.org/73741 | 23:55 |
| sdague | jog0: so the top level logic is right | 23:56 |
| sdague | the issue is in the facet calculation | 23:56 |
| sdague | and the math to build histograms | 23:56 |
| sdague | and, with that, I'm done, time to make dinner | 23:56 |
| sdague | good weekend all | 23:56 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Parse both new and old ES timestamp formats https://review.openstack.org/73740 | 23:56 |
| jog0 | sdague: o/ | 23:57 |
| jog0 | sdague: whats missing from your patch? | 23:59 |
| *** dangers is now known as dangers_away | 23:59 | |
| *** melwitt has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!