| sdague | jeblair: yeh, there were just plenty of other things to do on a Sat :) | 00:01 |
|---|---|---|
| *** erfanian has joined #openstack-infra | 00:01 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: provide time in queue in zuul ui https://review.openstack.org/65993 | 00:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: clean up possible js incompatibilities https://review.openstack.org/66057 | 00:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make merge conflict changes black https://review.openstack.org/66056 | 00:01 |
| sdague | that's the confirmed fix | 00:02 |
| *** dcramer_ has quit IRC | 00:02 | |
| *** dcramer_ has joined #openstack-infra | 00:14 | |
| *** erfanian has quit IRC | 00:16 | |
| *** erfanian has joined #openstack-infra | 00:26 | |
| *** CaptTofu has quit IRC | 00:26 | |
| *** CaptTofu has joined #openstack-infra | 00:27 | |
| *** rnirmal has joined #openstack-infra | 00:28 | |
| *** CaptTofu has quit IRC | 00:31 | |
| *** ryanpetrello has quit IRC | 00:40 | |
| openstackgerrit | Sean Dague proposed a change to openstack-dev/pbr: whitelist netifaces in integration tests https://review.openstack.org/66146 | 00:41 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Allow for sqla 0.8... finally https://review.openstack.org/64831 | 00:45 |
| *** sdake has joined #openstack-infra | 00:46 | |
| *** sdake has quit IRC | 00:49 | |
| sdague | jeblair: did something happen with nodepool? We seem to be shrinking on available nodes | 00:52 |
| jeblair | sdague: where are you seeing that? | 01:02 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch cinder to use bare-precise nodes https://review.openstack.org/65732 | 01:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch glance to use bare-precise nodes https://review.openstack.org/65733 | 01:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch keystone to use bare-precise nodes https://review.openstack.org/65734 | 01:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch heat to use bare-precise nodes https://review.openstack.org/65735 | 01:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch horizon to use bare-precise nodes https://review.openstack.org/65736 | 01:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch ceilometer to use bare-precise nodes https://review.openstack.org/65737 | 01:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch swift to use bare-precise nodes https://review.openstack.org/65738 | 01:11 |
| *** DennyZhang has joined #openstack-infra | 01:14 | |
| *** niska has quit IRC | 01:17 | |
| *** niska has joined #openstack-infra | 01:18 | |
| sdague | jeblair: on the status page | 01:18 |
| jeblair | sdague: i need more details. it looks normal to me. what looks amiss to you? | 01:24 |
| sdague | I guess the fall off of nodes, but I supose we're actually getting through the queue now | 01:24 |
| sdague | so it's all good | 01:25 |
| jeblair | sdague: yeah, the top of the green line is i think about where it's supposed to be | 01:25 |
| sdague | jeblair: I left a comment in the zuul review, I'm not really sure how to get access to the queue with items in it in the tests, so hints at that, would be appreciated | 01:25 |
| jeblair | sdague: a bunch of nodes just got used, so it's replacing them (yellow) | 01:25 |
| jeblair | sdague: note that we currently have 3 node types -- devstack-precise, devstack-precise-check, and bare-precise; so it's normal to have some ready nodes at all times, even when we're at capacity | 01:26 |
| jeblair | (they would be the ready nodes of the wrong type for whatever crunch we have) | 01:27 |
| lifeless | jeblair: (also tripleo-gate :P) | 01:27 |
| jeblair | lifeless: true, there's one of those. :) | 01:28 |
| jeblair | sdague: ack, i'll take a look at the zuul change in a few minutes after i translate my physical coordinates | 01:28 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add match for bad neutron dictionary compare https://review.openstack.org/66150 | 01:33 |
| *** sarob has joined #openstack-infra | 01:33 | |
| *** dstanek has joined #openstack-infra | 01:37 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add match for bad neutron dictionary compare https://review.openstack.org/66150 | 01:40 |
| *** sarob has quit IRC | 01:42 | |
| *** sarob has joined #openstack-infra | 01:42 | |
| *** sarob has quit IRC | 01:47 | |
| *** DennyZhang has quit IRC | 01:50 | |
| *** DennyZhang has joined #openstack-infra | 01:50 | |
| * fungi returns | 01:51 | |
| *** loq_mac has joined #openstack-infra | 01:53 | |
| *** sarob has joined #openstack-infra | 01:53 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: added ER query for swift not starting in grenade https://review.openstack.org/66151 | 01:53 |
| fungi | gate now @17 ans droppig another 6-8 in the next 15 minutes sans additional errors. i should leave for dinner more often | 01:54 |
| sdague | heh | 01:55 |
| sdague | the reset rate is pretty high still | 01:55 |
| fungi | yes, i've been noticing that all day. need more people fixing broken openstacks | 01:55 |
| StevenK | fungi: O hai, can you peer at https://review.openstack.org/#/c/65872/ ? | 01:56 |
| fungi | i'm mostly amused that with present tuning, the test nodes graph gains sanity ~queue depth of 30 | 01:56 |
| *** mkoderer has quit IRC | 01:56 | |
| *** sarob has quit IRC | 01:57 | |
| *** sarob has joined #openstack-infra | 01:58 | |
| *** mkoderer_ has joined #openstack-infra | 01:59 | |
| jeblair | fungi: i'm sure you noticed i merged all the bare-precise changes; i don't think they have taken effect for any running jobs | 02:01 |
| StevenK | fungi: Thanks for the review. | 02:01 |
| fungi | jeblair: hmmm... that's been ~50 minutes too | 02:02 |
| *** sarob has quit IRC | 02:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add a Loading paragraph onto zuul's status page https://review.openstack.org/65872 | 02:02 |
| fungi | jeblair: they did | 02:03 |
| fungi | jeblair: https://review.openstack.org/#/c/65872/ | 02:03 |
| fungi | "Took 2 min 12 sec on bare-precise-rax-iad-1064881" | 02:03 |
| fungi | er, wrong link... https://jenkins04.openstack.org/job/gate-glance-pep8/1/ | 02:04 |
| jeblair | fungi: oh i wasn't expecting them to kick in until about now | 02:04 |
| fungi | k | 02:04 |
| fungi | then, yes, seems to be working as intended ;) | 02:04 |
| jeblair | with propogation delay, etc | 02:04 |
| *** krtaylor has joined #openstack-infra | 02:07 | |
| fungi | got it | 02:09 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: added ER query for swift not starting in grenade https://review.openstack.org/66151 | 02:10 |
| fungi | jeblair: i'm assuming you saw in scrollback, but i ended up quiescing and restarting jenkins02 earlier today. it was exhibiting lots of webui timeouts and proxy errors, and runaway cpu utilization right around 20bg memory utilized. after restarting it's <10gb | 02:13 |
| *** wenlock has joined #openstack-infra | 02:14 | |
| fungi | though the load average graph from around the restart is pretty insane... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=822&rra_id=all | 02:14 |
| *** blamar has quit IRC | 02:15 | |
| fungi | arount 17:40 utc, that spike to around 375, i think it's about an order of magnitude lower than what top was reporting | 02:16 |
| jeblair | fungi: i did see the bit about the restart. the loadavg is news to me. hrm. | 02:16 |
| fungi | honestly, no guess what was going on there | 02:17 |
| fungi | the asymptotic shape almost seems like a counter inversion of some kind | 02:17 |
| *** wenlock has quit IRC | 02:19 | |
| fungi | when i spotted it, top was reporting a 10-minute load average of over 2000 | 02:19 |
| anteaya | fungi: at some point, which means when you feel like it, can you create a gerrit group for 3rd party test accounts that do not have voting enabled? | 02:19 |
| fungi | anteaya: that would be... just not adding them to a group | 02:20 |
| anteaya | well I would like so way I could navigate to a url within gerrit that would list them | 02:20 |
| anteaya | s/so way/a way so | 02:21 |
| *** sdake has joined #openstack-infra | 02:21 | |
| *** sdake has quit IRC | 02:21 | |
| *** sdake has joined #openstack-infra | 02:21 | |
| fungi | the external testing tools group has acl privs to allow verify voting in the -1 to +2 range, but any account can add comments | 02:21 |
| anteaya | true | 02:22 |
| jeblair | anteaya, fungi: i think what would be most useful is a group that has all accounts, and then a group that allows them to vote (a subset of the first) | 02:22 |
| jeblair | (the first being for administrative purposes, the second for actual permissions) | 02:22 |
| fungi | er, -1 to +1 range (typo) | 02:22 |
| *** DennyZhang has quit IRC | 02:22 | |
| fungi | i can duplicate the current entries from the external testing tools group into a probationary tools group or something like that, and add the system accounts we added which have been retroactively removed from ett | 02:24 |
| *** vipuls has quit IRC | 02:24 | |
| anteaya | ett? | 02:24 |
| fungi | external testing tools | 02:24 |
| anteaya | thanks, fry brain here | 02:24 |
| fungi | it would be double accounting, to some extent, but i'm not too bothered by that | 02:24 |
| fungi | anteaya: no worries. my brain is always fried | 02:25 |
| anteaya | :D | 02:25 |
| anteaya | let's try that | 02:25 |
| fungi | have we had lots of rogue -1 behavior? | 02:26 |
| anteaya | the word probationary which would have made me cringe 2 weeks ago has a lovely ring to it today | 02:26 |
| anteaya | enough that we need to take action | 02:26 |
| fungi | the last broken one brought to my attention wasn't even voting, just leaving 20+ comments on each review | 02:26 |
| *** vipul has joined #openstack-infra | 02:26 | |
| anteaya | and today I noticed one account posting 5 responses per event | 02:26 |
| fungi | sounds remarkably thorough ;) | 02:27 |
| anteaya | yup, comment spam is the next issue | 02:27 |
| anteaya | sigh | 02:27 |
| jeblair | fungi: you want double accounting? what's wrong with "external testing tools" and "external testing tools that may vote"? | 02:27 |
| anteaya | shame that zero of the comments contanined useful information | 02:27 |
| jeblair | (i don't like the idea of a group that _can't_ do something, that seems weird.) | 02:27 |
| fungi | well, external testing tools that can't vote is a group that can't do something | 02:28 |
| jeblair | fungi: right, which is why i'm suggesting we not have that group. :) | 02:28 |
| anteaya | I stand corrected, this one actually is offering logs with a clickable link | 02:28 |
| anteaya | the logs look possibly usefuel | 02:28 |
| anteaya | useful too | 02:28 |
| fungi | then i'm misreading your suggestion of "a group that has all accounts, and then a group that allows them to vote" | 02:29 |
| jeblair | fungi: i'm suggesting that we have a group that contains _all_ external testing tools, so that we can identify all of them (there are a lot of accounts, it's nice to know which are external testing tools) | 02:29 |
| jeblair | fungi: that group would have no permissions associated with it. it would only get new tools added when we create accounts for them | 02:29 |
| jeblair | fungi: the second group is used to give permission to vote. we can freely move tools in or out of that group as needed without forgetting that the account is an external testing tool. | 02:30 |
| fungi | ahh, right. but assuming most of then get to the point where they can vote, that's double-accounting. having the group that allows them to vote be included into the group of accounts, and then moving accounts between one group or the other, would not | 02:30 |
| jeblair | fungi: yes, in an ideal world, both groups would be duplicates. | 02:31 |
| fungi | so when you can't vote, you're in the external testing tools group as a direct member, but when you can vote you're in the testing tools that can vote group, which is included into the external testing tools group | 02:31 |
| jeblair | fungi: i would recommend not doing that | 02:31 |
| jeblair | fungi: i don't think any tool should ever be removed from the external testing tools group | 02:31 |
| fungi | even if it's just being moved to a group which is included into the other group? | 02:32 |
| jeblair | fungi: because this depends on humans to remember to do that. we don't actually have a double entry system, so we can't verify that. | 02:32 |
| jeblair | fungi: yes, because there is no move. there is add and please also remember to delete. | 02:32 |
| *** dcramer_ has quit IRC | 02:32 | |
| jeblair | fungi: i believe the value of the list of external testing tools is in never forgetting which of the 20k accounts are external testing tools. | 02:33 |
| fungi | true. there is also, though, the outside chance that an account gets added to the voting group without being added to the larger group then (accident or whatever) | 02:33 |
| jeblair | if we start deleting entries from that group we will forget. | 02:33 |
| jeblair | fungi: i think if we adopt the 'no-voting by default' rule, then there's a nice progression from (a) add to external account tools group and then (b) after trial period, add to voting group | 02:34 |
| fungi | but i agree. given gerrit lacks a "move this account from group a to group b" operation, that's even more error prone | 02:34 |
| fungi | okay, so we create a new "voting external tools" group, or similar, duplicate some subset of the current ett group to it, and change the acl to refer to that | 02:35 |
| fungi | anyway, let's approve the "no voting by default" change to the docs and announce it, if we're ready to do so. should we notify each tool contact address individually or just send a note to the -dev ml about it? | 02:36 |
| fungi | and do we know at this point which tools are going to be grandfathered into the new voting group? or is it all of the current ett members? | 02:37 |
| jeblair | fungi: i think it would be okay to approve that (and all those comments should just be additional patches, we don't have to get everything done in one patch) | 02:38 |
| jeblair | fungi: maybe anteaya can help with the question on which tools since she's in the trenches | 02:39 |
| anteaya | right now the only tools I would consider for grandfathering is smokestack, turbo-hipster and the fuel testing system | 02:40 |
| anteaya | fuel folks seem to be happy and it is only testing the stackforge repos | 02:41 |
| anteaya | dprince is very responsive, I see no reason to change his flow | 02:41 |
| anteaya | mikal and harlowja_away are fine if they are quarentined for a bit | 02:41 |
| anteaya | I personally don't have a reason to do so, but I'm fine if we do since they might be a good guinea pig for the approval process | 02:42 |
| anteaya | jhesketh my mistake | 02:43 |
| fungi | anteaya: remember, there also systems voting exclusively on specific projects, for example the stackforge puppet people have their own ci | 02:46 |
| anteaya | good point | 02:46 |
| anteaya | I haven't looked at their system at all | 02:46 |
| *** oubiwann has joined #openstack-infra | 02:47 | |
| anteaya | I'm biased regarding how we address the non-voting by default desemination | 02:47 |
| anteaya | if would be easiest for me if we just shut them down and then address stability and process for enbiling voting | 02:47 |
| anteaya | but I am tired and frustrated so not feeling very diplomatic right now | 02:48 |
| *** tma996 has joined #openstack-infra | 02:48 | |
| fungi | clarkb had a good suggestion about implementing project-specific acls to limit which project's changes a bot is allowed to vote on, but currently we haven't been asking operators which projects they want to vote on changes for, so we're missing a mapping there | 02:48 |
| *** alexpilotti has joined #openstack-infra | 02:49 | |
| anteaya | my response would be what kind of time estimate would their be to enable that mapping and how much damage and frustration is expected during that window? | 02:49 |
| fungi | there was also a suggestion that the decision on which bots are ready to vote be left up to "the ptl" but then we need a similar mapping to determine which ptl that is | 02:50 |
| anteaya | all good suggestions worthy of consideration | 02:51 |
| anteaya | I guess there is a long term solution and a short term need for action | 02:51 |
| anteaya | I'm thinking short term | 02:51 |
| anteaya | and think a good chat about longer term is in order | 02:51 |
| fungi | with a broad brush, we have ci voting on impact to nova drivers, cinder drivers, neutron drivers, other specific projects (some stackforge-specific things) and cross-project ci | 02:52 |
| *** dstanek has quit IRC | 02:52 | |
| anteaya | I'm open to a better short term solution | 02:53 |
| anteaya | that won't make the current situation worse and gives us some time to collect opinions and enact a better longer term solution | 02:54 |
| *** alexpilotti has quit IRC | 02:54 | |
| fungi | mostly just worried right now that if infra revokes voting rights, the operators will be petitioning the infra team on their relative merits and readiness to vote | 02:54 |
| anteaya | fine | 02:54 |
| anteaya | jeblair had a suggestion to expand some gerrit permissions to me and let those operators petition me as well | 02:55 |
| fungi | i'm not sure i'm in a suitable position to evaluate their voting readiness (though i'm willing to try, if asked) | 02:55 |
| anteaya | but yes, taking action to stem one tide that creates another is not necessarily what we want here | 02:56 |
| anteaya | fungi: fair enough | 02:56 |
| anteaya | personally I would heed the advice of the ptl of the project in questions, look for logs and a stable url and guage responsiveness of teh operators | 02:57 |
| anteaya | while being willing to revoke again if I made a mistake or if there is a significant change on behaviour | 02:57 |
| anteaya | in behaviour | 02:57 |
| fungi | and if you're evaluating their readiness, are you keeping track of bots voting on just neutron, or also nova? cinder? random stackforge projects? we have no enforcement in place at the moment to keep a nova testing bot from going crazy on neutron (or glance, or keystone) changes either, just the current reactionary removal of access | 02:58 |
| anteaya | good questions | 02:58 |
| anteaya | I'm grouping 3rd party testing systems as an entity regardless of what projects they are listning to | 02:59 |
| *** tma996 has quit IRC | 02:59 | |
| anteaya | I think 3rd party testing system should have a standard and they should meet that standard | 02:59 |
| fungi | in which case there is no "the ptl" to consult on readiness | 02:59 |
| *** tma996 has joined #openstack-infra | 02:59 | |
| anteaya | in some cases, that may be true, yes | 02:59 |
| fungi | and particularly for cross-project testing bots | 03:00 |
| anteaya | the standard if they restrict themselves to listening to stackforge only might be a different standard in some respects to if they are listening to openstack projects | 03:00 |
| anteaya | but there is a minimum of similarity | 03:00 |
| fungi | not trying to be obstinate, just pointing out the current things i think may not be getting taken into account | 03:00 |
| anteaya | right | 03:01 |
| anteaya | and I appreciate it | 03:01 |
| anteaya | for the cross project testing bots, then if the meet the standard (to be determined) they are good to go | 03:01 |
| anteaya | for project specific, the ptl can advocate | 03:02 |
| anteaya | for instance the existance and behaviour of recheck syntax | 03:02 |
| anteaya | cross project would adhere to the decision of the group | 03:02 |
| anteaya | project specific would have a ptl override clause for that | 03:02 |
| anteaya | unless that became a problem then we would discuss and adjust | 03:03 |
| fungi | and i think we definitely need to focus on proof by example for graduation to voting status, if we're going to start with a non-voting class. after all, i think if i'd asked all the operators who ended up being disruptive whether they were ready for voting rights in the beginning, most of them would probably have insisted that they were | 03:03 |
| anteaya | agreed | 03:03 |
| anteaya | proof my example | 03:03 |
| anteaya | by | 03:03 |
| anteaya | I can't English today | 03:04 |
| fungi | today isn't a day. it's a weekend | 03:04 |
| anteaya | that's it | 03:04 |
| anteaya | question for you, do you feel comfortable expanding gerrit rights on 3rd party testing accound admin to me? | 03:05 |
| anteaya | I don't want a responsibility that causes stress or concern for the team | 03:06 |
| anteaya | but I do think I can help out here | 03:06 |
| fungi | it's worth discussing. it's easy to do from a gerrit standpoint (put the 3pt evaluation board group as the owner of the voting version of the ett group) | 03:07 |
| anteaya | right | 03:07 |
| fungi | so anyway, if we're going to start bots non-voting and ask them to prove themselves, we need to be able to point them not just to a list of expectations, but specific criteria by which they will be evaluated for readiness, timelines, et cetera | 03:07 |
| anteaya | and if I'm in that group fine and if that group is someone else fine | 03:07 |
| anteaya | agreed | 03:08 |
| anteaya | but we also need to stem the spam in the meantime | 03:08 |
| anteaya | the voting spam | 03:08 |
| *** sarob has joined #openstack-infra | 03:08 | |
| anteaya | I'm not aware of any bots doing vote spam atm but the potential to have another wave is there | 03:09 |
| *** dcramer_ has joined #openstack-infra | 03:09 | |
| *** michchap_ has joined #openstack-infra | 03:09 | |
| *** michchap has quit IRC | 03:10 | |
| fungi | and they've been impacting disjoint groups of developers. mostly either nova or neutron (as far as the ones which have been brought to my attention so far) | 03:10 |
| fungi | so whoever is making decisions on which bots have earned the right to vote, or revoking the same, benefit from having collective insight into the various projects which have third parties providing significant input | 03:12 |
| anteaya | I agree | 03:13 |
| *** wenlock has joined #openstack-infra | 03:13 | |
| *** sarob has quit IRC | 03:13 | |
| fungi | i'm comfortable confirming that a bot is needlessly spamming or failing to adhere to some specific set of criteria and revoking access, but i don't feel like i have sufficient grounding in the day-to-day of the projects where these bots are voting to be able to evaluate their readiness to vote | 03:16 |
| anteaya | I accept that | 03:16 |
| anteaya | I can evaluate a more of what they would need to vote and would take input from others to enact a decision | 03:17 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/zuul: make enqueue_time passable to addChange https://review.openstack.org/66095 | 03:17 |
| sdague | jeblair: ok, should be good now | 03:18 |
| anteaya | for cross project, a sampling of effected projects and feedback on the level of usefulness of comments and logs provided to developers | 03:18 |
| sdague | when my wife insists on watching figure skating... I end up getting a lot of code done... | 03:18 |
| anteaya | sdague: but the costumes, they are so lovely | 03:18 |
| fungi | sdague: i can sympathize | 03:19 |
| anteaya | I can't watch figure skating | 03:19 |
| fungi | it's as pointless to me as sportball | 03:19 |
| anteaya | olympics yes, but not generally | 03:19 |
| anteaya | cricket, now that's a game | 03:19 |
| anteaya | and curling | 03:19 |
| sdague | I love curling | 03:19 |
| sdague | you ever do it? | 03:20 |
| fungi | caber toss, now there's a sport | 03:20 |
| StevenK | Right, my wife and I are happy to watch olympic figure skating | 03:20 |
| anteaya | yes, I curled all through high school, was skip of the team in grade 13 | 03:20 |
| sdague | anteaya: nice | 03:20 |
| anteaya | haven't had time to curl in years | 03:20 |
| sdague | I did a 2 week class after the 2006 games | 03:20 |
| sdague | it was a ton of fund | 03:20 |
| sdague | fun | 03:20 |
| anteaya | but I watch the big events | 03:20 |
| anteaya | nice | 03:20 |
| anteaya | how was your draw? | 03:21 |
| sdague | however, the club is just over an hour drive | 03:21 |
| anteaya | that interferes | 03:21 |
| sdague | anteaya: about what you'd expect after 2 weeks. Not great | 03:21 |
| sdague | but it was still fun | 03:21 |
| anteaya | cool | 03:21 |
| anteaya | as long as you get on and off the ice without falling over, you are good | 03:21 |
| anteaya | and buy a drink for your counterpart if you win | 03:22 |
| sdague | :) | 03:22 |
| anteaya | that part isn't covered in the official rules | 03:22 |
| anteaya | used to curl with my family in bonspiels | 03:23 |
| anteaya | for those not here, Perth is hot | 03:24 |
| anteaya | I don't remember being anywhere this hot before | 03:24 |
| anteaya | so according to the zuul status page, some tests are running over an hour | 03:25 |
| anteaya | is this due to some of the gate blocking bug fixes that rustlebee details in his email? | 03:25 |
| anteaya | I had thought we were down to 40 minutes or so per patch | 03:26 |
| anteaya | http://lists.openstack.org/pipermail/openstack-dev/2014-January/024052.html | 03:27 |
| fungi | anteaya: i think the hpcloud vms are slower for us than they used to be | 03:27 |
| anteaya | interesting | 03:27 |
| anteaya | and that is a big step backwards | 03:27 |
| mordred | hey all | 03:28 |
| fungi | mordred: hi!!! | 03:28 |
| sdague | mordred: aren't you supposed to be sportsball drinking? | 03:28 |
| StevenK | mordred: Hai! | 03:28 |
| mordred | it turns out my hong kong laptop can still tlk to my irc server | 03:28 |
| sdague | nice | 03:28 |
| mordred | so hopefully I'll be a real human again soon | 03:28 |
| anteaya | mordred: yay | 03:29 |
| fungi | mordred: the prc didn't totally cripple it? sounds like the backdoor is working | 03:29 |
| mordred | fungi: I guess I'm not important enough for the prc | 03:29 |
| fungi | me either | 03:29 |
| fungi | (or maybe that's just what the nsa wants us to think) | 03:30 |
| sdague | mordred: well, when you get real humany - https://review.openstack.org/#/c/66146/ (well a working version of it) is all we need to bump sqla to 0.8 | 03:30 |
| mordred | StevenK: howdy | 03:30 |
| sdague | though I'm still confused why the requirements integration tests are in the pbr tree | 03:30 |
| mordred | sdague: hysterical raisins | 03:30 |
| mordred | sdague: the whoel system needs overhaul, tbh | 03:30 |
| sdague | I actually also believe that the only thing between us an sqla 0.9 is a few migrate work arounds | 03:31 |
| sdague | though I could be overly optimistic | 03:31 |
| fungi | sdague: that started as "make sure pbr changes don't break all the things" | 03:31 |
| jeblair | mordred: fyi we like totally revoked all your access, but you can have it back when you're all set up... | 03:32 |
| mordred | jeblair: yes, I saw - and thank you | 03:32 |
| lifeless | fungi: if you have a minute - https://bugs.launchpad.net/openstack-ci/+bug/1268248 | 03:32 |
| uvirtbot | Launchpad bug 1268248 in openstack-ci "derekh has two entries in gerrit, can't add as reviewer" [Undecided,New] | 03:32 |
| mikal | anteaya: do you have a headache? Do you need some painkillers? | 03:32 |
| mordred | btw - don't leave your laptop in your backseat if yuo park anywhere in SF | 03:32 |
| fungi | lifeless: thanks. i'll fix | 03:32 |
| jeblair | mordred: we weren't sure about things like launchpad or pypi creds, so we assumed the worst. | 03:32 |
| fungi | lifeless: or rather, clean up. the fix is no longer using lp for openid, but that's a ways out | 03:32 |
| mordred | jeblair: best choice | 03:33 |
| mordred | jeblair: enjoying oz? | 03:33 |
| anteaya | mikal: yes, headache, pass on painkillers - I don't take medication, sitting in the hall is working fine, thanks | 03:34 |
| * anteaya goes to get her water | 03:34 | |
| StevenK | mordred: I think leaving a laptop visible anywhere in the car when parked anywhere is a terrible idea | 03:34 |
| mikal | anteaya: fair enough | 03:35 |
| mordred | StevenK: well, I had gotten used to New York where leaving a laptop just laying on the side of the road is perfectly safe | 03:35 |
| mikal | anteaya: what about gatorade or something like that? | 03:35 |
| mikal | mordred: not true! The cops would come out and explode it. | 03:35 |
| mordred | but I forgot that I'd left civilization when I moved to california | 03:35 |
| StevenK | mordred: For instance, Sydney would have the same problem as SF | 03:35 |
| jeblair | mordred: yes, lovely; i'm in a room with StevenK, lifeless, mikal, jhesketh, anteaya, and others | 03:35 |
| mordred | jeblair: woot | 03:36 |
| sdague | mordred: you are in the hipster jungle, it's brutal out there | 03:36 |
| jeblair | mordred: also matthewoliver who isn't in the room but is a sysadmin-type in oz who wants to help us. at least, mikal and i keep telling him that. | 03:37 |
| mordred | sdague: no, I'd driven up there - I'm actually down in south bay ... one more reason to avoid the hipsters | 03:37 |
| anteaya | mikal: where can I get a gatorade? I would drink that | 03:37 |
| mordred | jeblair: tell me where I should mail beer for him to | 03:38 |
| sdague | what's mikal's hoodie count up to? | 03:38 |
| mordred | I AM wearing a hoodie though | 03:38 |
| *** dcramer_ has quit IRC | 03:38 | |
| jeblair | mordred: i think "matthew oliver, australia" should get to him. there aren't that many ppl in this country. | 03:38 |
| StevenK | mordred: It is 105degF | 03:38 |
| StevenK | Anti-hoodie weather | 03:38 |
| sdague | my hoodies are all in the laundry. this whole working from home 100% of the time means openstack hoodies are my work uniform | 03:39 |
| sdague | which means I clearly need more | 03:39 |
| jeblair | you have to tie the little strings for the hood before you wash them | 03:39 |
| jeblair | i'm not really familiar with hoodies. i learned that the hard way. | 03:39 |
| mordred | sdague: weirdly, my favorite hoodie has become my hp hoodie, which is weird - but they did a REALLY good job with construction, and the logo is very small | 03:40 |
| StevenK | mordred: When do I get a HP hoodie? | 03:41 |
| mordred | StevenK: it comes as part of the blood transfusion. lifeless told you about that part, right? | 03:41 |
| mordred | jeblair: the WORST part is that I went to the thinkpad site to look for a replacement, and they're all BATSHIT | 03:42 |
| fungi | lifeless: derekh selectable for you now? | 03:42 |
| mordred | as in, I believe I'm going to give up and just use my hp laptop because the new thinkpads are all terrible looking | 03:42 |
| StevenK | mordred: Really? The X240 didn't look too bad | 03:42 |
| mordred | apparently we all want touch screens | 03:43 |
| sdague | blame microsoft | 03:43 |
| *** Loquacity has quit IRC | 03:44 | |
| sdague | so I actually like the old/new style keyboard on the thinkpads. So I picked up an X1 carbon before christmas | 03:44 |
| jeblair | mordred: i think you should get an x240 and tell me if i'm going to hate it. :) | 03:44 |
| sdague | however... the new new keyboard is crazy pants | 03:45 |
| jeblair | mordred: or get hp to make a dev laptop. :) | 03:45 |
| mordred | jeblair: I'm sure you will | 03:45 |
| sdague | https://twitter.com/obra/status/419995228838588417 | 03:45 |
| StevenK | The only thing I don't like about the X240 is the crazy touchpad | 03:45 |
| StevenK | I want buttons! | 03:45 |
| mordred | jeblair: I figured out how to turn the touchpad off on my hp folio - and it' not terrible | 03:45 |
| StevenK | Actual, physical buttons! | 03:46 |
| * fungi blames ibm. i think the last ibm portable pc he used was a 5155 model 68 | 03:46 | |
| sdague | StevenK: yeh, that's why I got the X1 carbon | 03:46 |
| mordred | i mean, it's not a thinkpad, but neither is the x240 | 03:46 |
| anteaya | where do I file a bug for the foundation website, so that a new foundation member gets an indication of success when the form is submitted? | 03:46 |
| fungi | anteaya: which form? the foundation membership registration form? | 03:47 |
| fungi | anteaya: anyway, https://bugs.launchpad.net/openstack-org | 03:47 |
| anteaya | yes the foundation membership registration form | 03:47 |
| fungi | anteaya: also, ask them where you can download the source ;) | 03:48 |
| fungi | (only half kidding) | 03:48 |
| sdague | heh | 03:48 |
| jeblair | mordred: indeed :( | 03:49 |
| anteaya | fungi: I have a big enough can of worms currently | 03:49 |
| StevenK | sdague: The X1 Carbon didn't do it for me either | 03:49 |
| anteaya | not interested in poking that monster | 03:49 |
| StevenK | I have a X201 currently, and I love it to bits, aside from the battery being pretty terrible due to conferences | 03:50 |
| sdague | StevenK: what didn't you like about it? just curious. | 03:51 |
| StevenK | sdague: My main problem is it doesn't look like a Thinkpad | 03:51 |
| mordred | X240 supports "up to 8G memory" | 03:51 |
| sdague | heh | 03:51 |
| StevenK | But the new design language doesn't either | 03:51 |
| fungi | i've travelled with netbooks for so long now that laptops seem more like luggables | 03:51 |
| StevenK | mordred: Yes, it's terrible | 03:51 |
| mordred | what a pile of garbage | 03:52 |
| *** dcramer_ has joined #openstack-infra | 03:52 | |
| mordred | I'm not going to fly in the face of all of the political flak I get for running a lenovo in meetings at hp | 03:52 |
| sdague | mordred: yeh, I don't get why they aren't offering 16 yet in the ultrabooks | 03:52 |
| mordred | just to run a piece of hardware that's substandard | 03:52 |
| mordred | I mean, I actually am not a lenovo fanboi - I ran lenovo because it was the best product available | 03:53 |
| mordred | but if it's not the best product available... sigh | 03:53 |
| fungi | mordred: just wheel around an hp k-series on a hand cart | 03:53 |
| mordred | fungi: yah | 03:53 |
| StevenK | Hahaha | 03:53 |
| sdague | I think someone was looking at making a new run of motherboards for the x220 | 03:54 |
| mordred | honestly - the folio that lifeless and I both have is not terrible, seriously. It's NOT as good as my old lenovo - but it's certainly comparble with the new ones | 03:54 |
| fungi | hp/ux on parisc. you'll be the envy of the conference room | 03:54 |
| jeblair | mordred: http://www.bunniestudios.com/blog/?p=2686 | 03:54 |
| StevenK | Why not ia64? Then you can't run anything, either. | 03:54 |
| jeblair | oh that's old | 03:55 |
| jeblair | mordred: http://www.bunniestudios.com/blog/?p=3597 | 03:55 |
| jeblair | mordred: look, you can use a thinkpad keyboard! :) | 03:55 |
| anteaya | https://bugs.launchpad.net/openstack-org/+bug/1268251 | 03:55 |
| uvirtbot | Launchpad bug 1268251 in openstack-org "submitting registration form should return success indicator" [Undecided,New] | 03:55 |
| mordred | jeblair: nice | 03:55 |
| *** Loquacity has joined #openstack-infra | 03:56 | |
| mordred | sdague: what was that review you were wanting? | 03:56 |
| sdague | mordred: https://review.openstack.org/#/c/66146/ | 03:57 |
| *** vlo1 has joined #openstack-infra | 03:57 | |
| lifeless | mordred: how do we know this is you? | 03:57 |
| sdague | though I don't know why it doesn't pass | 03:57 |
| mordred | lifeless: butttoast | 03:57 |
| StevenK | mordred has been replaced by Watson? | 03:58 |
| mikal | anteaya: the drinks machine near the NOC has gatorade. I'd be happy to go get you one. | 03:58 |
| mikal | (Sorry, I only just noticed this message) | 03:58 |
| lifeless | mordred: the current gen of the folio should be pretty awesome, really. | 03:58 |
| fungi | mordred: if i were the lowlife who swiped your laptop in the california wasteland, i'd say "butttoast too" | 03:58 |
| anteaya | mikal: I accept | 03:59 |
| lifeless | mordred: it might even do 32GB of ram | 03:59 |
| mordred | fungi: then you'll just have to wait to see if I push up 100 buggy patches in the next day | 03:59 |
| sdague | jeblair: my zuul change seems to have failed - with no useful fail output - http://logs.openstack.org/95/66095/3/check/gate-zuul-python27/71c53d0/console.html | 03:59 |
| fungi | mordred: we'll know it's you then, agreed | 03:59 |
| lifeless | not without an ssh key in gerrit.... | 04:00 |
| anteaya | fungi: vlo1 is sitting beside me and is a new contributor | 04:00 |
| sdague | StevenK: well, that would be a test. They had to put a vulgarity filter in front of watson, because apparently one of the test matches was *hilarious* | 04:00 |
| anteaya | gerrit id is 9948 | 04:00 |
| fungi | anteaya: what's the problem? | 04:00 |
| mordred | sdague: that's going to require more work than that patch, sadly | 04:00 |
| sdague | mordred: ok | 04:01 |
| anteaya | fungi: would you be so kind as to check the db for his account and tell me the status of his foundation membership? | 04:01 |
| mordred | sdague: probably needs a patch to run-mirror too | 04:01 |
| *** oubiwann has quit IRC | 04:01 | |
| sdague | mordred: ok, well we can't current change requirements because of that | 04:01 |
| anteaya | fungi: when he tries to submit the cla it says his foundation memebership doesn't exist | 04:01 |
| sdague | so if you know how to fix it, that would be cool | 04:01 |
| jeblair | sdague: yes, i'm extremely frustrated by that. it's a timeout, and i have no idea how to find out what test is failing, or how to get the log output from that test. | 04:01 |
| fungi | anteaya: well, all i can do is confirm whether gerrit was able to get confirmation on a contact info update. but i'll look | 04:01 |
| anteaya | and I just watched him submit his foundation membership registration | 04:01 |
| mordred | sdague: yes. I grok the problem. the solution is probably beyond me given my current computing situation | 04:02 |
| anteaya | fungi: thanks | 04:02 |
| anteaya | fungi: and understood | 04:02 |
| jeblair | sdague: i'm at the mercy of someone who understands how that is supposed to work with testr. | 04:02 |
| lifeless | jeblair: whats up? | 04:02 |
| fungi | anteaya: oh, make sure the primary e-mail address on his foundation profile is the same as his preferred e-mail address in gerrit | 04:02 |
| anteaya | fungi: he only has one email | 04:03 |
| anteaya | it is consisstent across the sign up | 04:03 |
| anteaya | we had discussed that last night | 04:03 |
| mordred | sdague: I will endeavor to get myself into a place where I can help fix that soonish | 04:03 |
| sdague | mordred: cool | 04:03 |
| vlo1 | openstack@systems-networking.de | 04:03 |
| sdague | mordred: who's looking after migrate at this point? | 04:04 |
| mordred | sdague: there is a team - it's in gerrit | 04:04 |
| sdague | ok, mostly distro folks, cool | 04:05 |
| fungi | vlo1: welcome! and apologies for the inconvenience. i'm taking a look now | 04:05 |
| sdague | any idea how responsive the sqla maintainer is? | 04:05 |
| mordred | sdague: very. he's cool | 04:06 |
| sdague | ok, cool, I'll see which of these bugs I can just get fixed up there instead | 04:06 |
| vlo1 | fungi: thanks | 04:06 |
| fungi | vlo1: anteaya: i see the account in gerrit, and gerrit's db confirms that the cla has not been agreed to nor has contact information been filed. checking the logs to see if we're maybe seeing connectivity issues to the foundation site or something | 04:09 |
| fungi | [2014-01-12 03:50:31,146] ERROR com.google.gerrit.server.contact.EncryptedContactStore : Cannot store encrypted contact information | 04:10 |
| fungi | that was ~20 minutes ago | 04:11 |
| fungi | java.io.IOException: Connection failed: 404 | 04:11 |
| fungi | so i'm guessing something is wrong with the foundation site (either the foundation site's contact information submission endpoint is broken or it's having trouble confirming the right profile/address) | 04:12 |
| anteaya | fungi: so our gerrit connection to the foundation site is unhappy | 04:12 |
| anteaya | who do we get for a fix? | 04:12 |
| *** vlo1 has quit IRC | 04:13 | |
| anteaya | vlo1: so you have done everything correctly from what we can assess so far, well done | 04:13 |
| anteaya | he's gone :( | 04:13 |
| fungi | vlo: i only see that once in the log. maybe trying a second time will work? could have just been temporary | 04:13 |
| anteaya | I can walk down the hall and update him | 04:13 |
| anteaya | I'll let him know | 04:14 |
| *** vlo1 has joined #openstack-infra | 04:16 | |
| fungi | i'm passing out shortly. it's getting late | 04:17 |
| jeblair | sdague: lifeless helped me narrow it down to tests.test_scheduler.TestScheduler.test_client_enqueue_negative | 04:17 |
| anteaya | fungi: k, thanks for your help | 04:17 |
| lifeless | jeblair: I think :) | 04:18 |
| jeblair | sdague: i don't have output for why it failed, but hopefully that narrows it down a bit | 04:18 |
| jeblair | sdague: i'd say 'recheck no bug' for your change | 04:18 |
| fungi | lifeless: derekh selectable for you now? | 04:18 |
| lifeless | fungi: I'll check shortly - if he's selectable for you though, I'm sure it will be fine | 04:18 |
| fungi | lifeless: seems fine. should be all set | 04:19 |
| sdague | jeblair: yeh, I did | 04:21 |
| sdague | jeblair: it passed the 2nd time | 04:21 |
| sdague | it would be super awesome to get that and the queue times stuff merged by monday :) | 04:21 |
| sdague | and with that, it's time to go watch some QI and call it a night. | 04:22 |
| vlo1 | fungi: i tried it again - same error | 04:26 |
| *** rakhmerov has quit IRC | 04:26 | |
| anteaya | vlo1: looks like we are in a holding pattern until we get someone who can access the foundation site and assess it from their end | 04:27 |
| anteaya | vlo1: sorry about that, but at leasat you are on irc so you know how to find us going forward | 04:28 |
| anteaya | vlo1: this being a weekend, it might take until Monday to find a fix | 04:28 |
| vlo1 | fungi: OK | 04:29 |
| *** loq_mac has quit IRC | 04:29 | |
| anteaya | mikal: gatorade was it, starting to feel better | 04:30 |
| anteaya | water, who knew? | 04:30 |
| *** loq_mac has joined #openstack-infra | 04:33 | |
| anteaya | https://bugs.launchpad.net/openstack-org/+bug/1268254 | 04:37 |
| uvirtbot | Launchpad bug 1268254 in openstack-org "no communication with gerrit" [Undecided,New] | 04:37 |
| *** senk has joined #openstack-infra | 04:38 | |
| *** sandywalsh has quit IRC | 04:44 | |
| *** mattoliverau has joined #openstack-infra | 04:47 | |
| anteaya | mattoliverau: yay | 04:47 |
| *** loq_mac has quit IRC | 04:51 | |
| *** vlo1 has quit IRC | 04:53 | |
| *** dcramer_ has quit IRC | 04:55 | |
| *** rakhmerov has joined #openstack-infra | 04:57 | |
| *** rakhmerov1 has joined #openstack-infra | 04:59 | |
| *** rakhmerov has quit IRC | 04:59 | |
| *** rakhmerov1 has quit IRC | 05:03 | |
| *** wenlock has quit IRC | 05:16 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Allow concurrency to be tweaked for tempest https://review.openstack.org/65804 | 05:22 |
| *** rnirmal has quit IRC | 05:26 | |
| *** rakhmerov has joined #openstack-infra | 05:29 | |
| *** michchap_ has quit IRC | 05:33 | |
| *** michchap has joined #openstack-infra | 05:34 | |
| portante | jeblair, sdague: perhaps another zuul enhancement could be to indicate if a job in the gate is there because of a reverify or not | 05:50 |
| *** marun has quit IRC | 05:59 | |
| *** senk has quit IRC | 06:23 | |
| *** senk has joined #openstack-infra | 06:23 | |
| *** loq_mac has joined #openstack-infra | 06:26 | |
| jeblair | portante: neat idea; trigger cause could be tracked | 06:28 |
| *** vlo1 has joined #openstack-infra | 06:30 | |
| *** loq_mac has quit IRC | 06:31 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add -U to pip install command in tox.ini https://review.openstack.org/61517 | 06:33 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow workers to send back metadata https://review.openstack.org/66173 | 06:38 |
| *** loq_mac has joined #openstack-infra | 06:38 | |
| *** gyee has quit IRC | 06:40 | |
| *** david-lyle has quit IRC | 06:40 | |
| *** gyee has joined #openstack-infra | 06:40 | |
| *** david-lyle has joined #openstack-infra | 06:40 | |
| clarkb | made it to cottlesloe for breakfast and beaching then did maritime museum in fremantle, now a nap | 06:44 |
| StevenK | \o/ | 06:45 |
| lifeless | a | 06:46 |
| jeblair | anteaya: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/review.pp#n110 | 06:47 |
| *** vlo1 has quit IRC | 06:50 | |
| *** vlo1 has joined #openstack-infra | 06:50 | |
| *** senk has quit IRC | 06:51 | |
| *** vlo1 has quit IRC | 06:52 | |
| *** sdake has quit IRC | 06:55 | |
| anteaya | clarkb: how were the waves today? | 06:58 |
| clarkb | small but the water is too cold for swimming | 07:00 |
| anteaya | nooooo | 07:00 |
| anteaya | what a shame | 07:00 |
| StevenK | Compared to the 6,000degrees the beach sand is? | 07:01 |
| clarkb | StevenK yes I like warm water | 07:01 |
| anteaya | I wonder how the weather could get hotter and the water colder | 07:01 |
| StevenK | anteaya: That's what happens | 07:01 |
| anteaya | was great when pleia2 and I went | 07:01 |
| anteaya | ah | 07:01 |
| StevenK | Hot weather == cold water ; cold weather == warm water | 07:02 |
| anteaya | I probably would have gone swimming anyway | 07:02 |
| *** sdake has joined #openstack-infra | 07:07 | |
| anteaya | but I have gone into glacier feed rivers previously, so that's me | 07:14 |
| StevenK | anteaya: I think Australia is not a conducive environment for glaciers :-) | 07:17 |
| anteaya | I agree with that assessemtn | 07:18 |
| *** salv-orlando has joined #openstack-infra | 07:30 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Adding apache redirect for the cacti url. https://review.openstack.org/66181 | 07:32 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Adding apache redirect for the cacti url. https://review.openstack.org/66181 | 07:40 |
| *** tma996 has quit IRC | 07:40 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/git-review: Correct .Fl typo WRT --compare in the manual page https://review.openstack.org/66182 | 07:42 |
| *** loq_mac has quit IRC | 07:43 | |
| *** loq_mac has joined #openstack-infra | 07:47 | |
| *** sdake has quit IRC | 07:50 | |
| jeblair | fungi, clarkb: i think graphite may have upgraded to a broken version; i'm seeing a traceback when i look at images | 08:00 |
| jeblair | fungi, clarkb: but i'm not in a position to fix now. | 08:00 |
| jeblair | i expect that statsd will still be collecting data though | 08:00 |
| jeblair | so once graphite is fixed, hopefully we won't be mising data | 08:00 |
| *** sdake has joined #openstack-infra | 08:02 | |
| *** loq_mac has quit IRC | 08:04 | |
| *** tma996 has joined #openstack-infra | 08:41 | |
| ttx | jeblair: still around? | 08:52 |
| *** salv-orlando has quit IRC | 08:55 | |
| *** tma996 has quit IRC | 09:11 | |
| *** tma996 has joined #openstack-infra | 09:11 | |
| ttx | jeblair: sent you email instead | 09:15 |
| *** nijaba has quit IRC | 09:24 | |
| *** tma996 has quit IRC | 09:28 | |
| *** yolanda has joined #openstack-infra | 09:33 | |
| *** nijaba has joined #openstack-infra | 09:49 | |
| *** fbo_away is now known as fbo | 09:52 | |
| *** nijaba has quit IRC | 09:58 | |
| *** tma996 has joined #openstack-infra | 09:59 | |
| *** nijaba has joined #openstack-infra | 10:03 | |
| *** nijaba has joined #openstack-infra | 10:03 | |
| *** nicedice has joined #openstack-infra | 10:18 | |
| *** flaper87|afk is now known as flaper87 | 10:28 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 10:32 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags https://review.openstack.org/63580 | 10:32 |
| *** tma996 has quit IRC | 10:33 | |
| *** rcarrillocruz has joined #openstack-infra | 10:43 | |
| *** vogxn has joined #openstack-infra | 10:47 | |
| *** mozawa has joined #openstack-infra | 10:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!