| jeblair | so i think 'internet. lol.' is the best explanation we have for this issue right now. i think sdague's change to bump the connect timeout to 30s is worth a shot, but i think that's experimental -- we need to follow up and see if it changes anything. | 00:00 |
|---|---|---|
| *** markvoelker has quit IRC | 00:00 | |
| jeblair | regardless of whether that improves things, i think the eventual real fix for this may lie in mordred's work for the launcher in zuulv3 | 00:00 |
| *** markvoelker has joined #openstack-infra | 00:01 | |
| mordred | jeblair: ++ | 00:02 |
| mordred | jeblair: I'm +2 on 'internet. lol.' | 00:02 |
| clarkb | what changes in the launcher in v3 to avoid ssh problems? | 00:03 |
| clarkb | also running git-review during db backups is like watching paint dry | 00:04 |
| * clarkb is trying to start back up again on newton-xenial changes | 00:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run multinode jobs on xenial and trusty https://review.openstack.org/364598 | 00:04 |
| clarkb | so that we have some ready to go as soon as dhellmann is done with milestone 3 | 00:04 |
| *** fguillot has quit IRC | 00:04 | |
| clarkb | it turns out I can just troll the zuul stats for getting a good idea of things that need to be converted | 00:04 |
| clarkb | like nova functional tests | 00:04 |
| *** ijw has joined #openstack-infra | 00:05 | |
| jeblair | clarkb: mordred is looking into some fancy pants ansible plugins that will (a) let us do the things we said we were going to do in v3, and (b) give us more control over the part of the process that's handled by the async module now (which has been the source of a lot of our problems, including this one) | 00:07 |
| clarkb | gotcha | 00:07 |
| clarkb | apparently today is the first day of football being back | 00:08 |
| clarkb | I think I need to go plant myself in front of tv now | 00:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560 | 00:08 |
| mordred | clarkb: football! | 00:09 |
| *** ijw has quit IRC | 00:10 | |
| *** vinaypotluri has quit IRC | 00:12 | |
| *** sputnik13_ has quit IRC | 00:13 | |
| *** fguillot has joined #openstack-infra | 00:16 | |
| *** pvaneck has quit IRC | 00:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received https://review.openstack.org/364322 | 00:17 |
| *** Genek has joined #openstack-infra | 00:21 | |
| *** xarses has quit IRC | 00:22 | |
| *** thorst has joined #openstack-infra | 00:24 | |
| *** claudiub has quit IRC | 00:24 | |
| *** Genek has quit IRC | 00:24 | |
| *** SumitNaiksatam has quit IRC | 00:24 | |
| *** ijw has joined #openstack-infra | 00:25 | |
| *** ijw has quit IRC | 00:26 | |
| *** annegentle has quit IRC | 00:31 | |
| *** annegentle has joined #openstack-infra | 00:32 | |
| *** ijw has joined #openstack-infra | 00:35 | |
| *** rfolco has quit IRC | 00:35 | |
| *** markvoelker has quit IRC | 00:35 | |
| *** tonytan4ever has joined #openstack-infra | 00:38 | |
| *** xarses has joined #openstack-infra | 00:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "tripleo-ui: add missing jobs for release management" https://review.openstack.org/364452 | 00:40 |
| *** rfolco has joined #openstack-infra | 00:40 | |
| *** rfolco has quit IRC | 00:40 | |
| *** Julien-zte has joined #openstack-infra | 00:41 | |
| *** fguillot has quit IRC | 00:41 | |
| *** markvoelker has joined #openstack-infra | 00:44 | |
| *** zz_dimtruck is now known as dimtruck | 00:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Add tripleo-ci-centos-7-ovb-ha-ipv6 into tripleo-ci jobs https://review.openstack.org/364324 | 00:45 |
| *** Jeffrey4l has joined #openstack-infra | 00:46 | |
| *** fguillot has joined #openstack-infra | 00:47 | |
| *** Swami__ has joined #openstack-infra | 00:49 | |
| *** shashank_hegde has joined #openstack-infra | 00:49 | |
| *** shashank_hegde has quit IRC | 00:49 | |
| *** shashank_hegde has joined #openstack-infra | 00:50 | |
| *** kaisers_ has joined #openstack-infra | 00:51 | |
| *** Swami has quit IRC | 00:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560 | 00:55 |
| *** kaisers_ has quit IRC | 00:55 | |
| pabelanger | ianw: mind a review on 363639 | 00:57 |
| ianw | pabelanger: ok, didn't we fix that? must have been a similar one | 01:00 |
| pabelanger | ianw: ya, same issue, just different jobs | 01:01 |
| *** ddieterly has joined #openstack-infra | 01:02 | |
| *** ddieterly is now known as ddieterly[away] | 01:02 | |
| openstackgerrit | Merged openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template https://review.openstack.org/364455 | 01:05 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918 | 01:06 |
| timothyb89 | ianw: ah, sorry, I guess I wasn't very clear, it will need 2 fully separate output directories right now | 01:08 |
| ianw | timothyb89: yep, i think https://review.openstack.org/#/c/364560/ latest gets it | 01:10 |
| timothyb89 | ianw: but I could change stackviz-export to write into an existing directory properly | 01:10 |
| *** ijw has quit IRC | 01:10 | |
| ianw | for grenade, i've just made a copy in old & new | 01:11 |
| *** ijw has joined #openstack-infra | 01:11 | |
| ianw | i'm not sure if we really on want results for "new" ... i think that is what is sort of tacitly happening atm | 01:11 |
| *** ijw has quit IRC | 01:11 | |
| ianw | timothyb89: also, what's with the zip options? can we use that to compress everything? | 01:12 |
| ianw | that might be a good follow-on | 01:12 |
| timothyb89 | ianw: right now zipping is broken because the log server sends files with the wrong type | 01:12 |
| openstackgerrit | Merged openstack-infra/project-config: Run ansible-role-ubuntu-trusty jobs on master https://review.openstack.org/363639 | 01:12 |
| timothyb89 | ianw: so we can't decompress them in the browser | 01:12 |
| timothyb89 | ianw: working around that is on the agenda but there's some things that need to happen first | 01:13 |
| ianw | timothyb89: oh, is that something anyone is working on? i guess it's os-loganalyzers fault? | 01:13 |
| ianw | ok | 01:14 |
| timothyb89 | ianw: at this point I'm planning to move all of the decompress+parsing to the client side | 01:14 |
| *** zshuo has joined #openstack-infra | 01:15 | |
| timothyb89 | ianw: we'll have ~500KB of content and it will just directly read from the raw log files, compressed or not | 01:15 |
| *** zhurong has joined #openstack-infra | 01:16 | |
| *** signed8bit_Zzz is now known as signed8bit | 01:17 | |
| pabelanger | clarkb: image build done, starting upload to osic-cloud1 | 01:18 |
| *** salv-orlando has joined #openstack-infra | 01:18 | |
| *** ijw has joined #openstack-infra | 01:20 | |
| *** spzala has joined #openstack-infra | 01:20 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads https://review.openstack.org/364607 | 01:21 |
| *** waht has joined #openstack-infra | 01:21 | |
| *** ddieterly[away] is now known as ddieterly | 01:23 | |
| *** ijw has quit IRC | 01:24 | |
| *** spzala has quit IRC | 01:25 | |
| *** ddieterly has quit IRC | 01:27 | |
| *** salv-orlando has quit IRC | 01:27 | |
| *** markvoelker has quit IRC | 01:28 | |
| *** shashank_hegde has quit IRC | 01:29 | |
| *** hsammic has quit IRC | 01:29 | |
| *** Julien-zte has quit IRC | 01:29 | |
| *** Genek has joined #openstack-infra | 01:31 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 01:31 | |
| *** akshai has joined #openstack-infra | 01:31 | |
| *** Genek has quit IRC | 01:33 | |
| *** markvoelker has joined #openstack-infra | 01:35 | |
| *** ilyashakhat_mobi has quit IRC | 01:35 | |
| *** mtanino has quit IRC | 01:36 | |
| *** annegentle has quit IRC | 01:37 | |
| *** akshai has quit IRC | 01:37 | |
| *** aeng has quit IRC | 01:37 | |
| *** Genek has joined #openstack-infra | 01:38 | |
| *** yamamoto_ has joined #openstack-infra | 01:38 | |
| *** mriedem has quit IRC | 01:38 | |
| *** mlavalle has quit IRC | 01:38 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:39 | |
| *** Julien-zte has joined #openstack-infra | 01:40 | |
| *** mriedem has joined #openstack-infra | 01:40 | |
| mriedem | this is approved but looks stuck https://review.openstack.org/#/c/363937/ | 01:40 |
| *** markvoelker has quit IRC | 01:40 | |
| mriedem | does that need to be re-approved or something? | 01:40 |
| *** gyee has quit IRC | 01:47 | |
| *** kzaitsev_mb has quit IRC | 01:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Run bashate test over devstack-gate too https://review.openstack.org/236819 | 01:48 |
| *** caowei has joined #openstack-infra | 01:48 | |
| *** spzala has joined #openstack-infra | 01:49 | |
| *** signed8bit is now known as signed8bit_Zzz | 01:49 | |
| *** yanyanhu has joined #openstack-infra | 01:50 | |
| *** Thelo_ has joined #openstack-infra | 01:50 | |
| *** mriedem has quit IRC | 01:51 | |
| *** Thelo_ has quit IRC | 01:52 | |
| *** SumitNaiksatam has quit IRC | 01:53 | |
| *** esberglu has joined #openstack-infra | 01:58 | |
| *** zhurong_ has joined #openstack-infra | 01:59 | |
| *** zhurong has quit IRC | 01:59 | |
| *** shashank_hegde has joined #openstack-infra | 02:01 | |
| pabelanger | clarkb: ubuntu-xenial online is osic-cloud1 | 02:02 |
| pabelanger | clarkb: and working properly | 02:03 |
| *** markvoelker has joined #openstack-infra | 02:04 | |
| *** chlong has joined #openstack-infra | 02:06 | |
| *** tphummel has quit IRC | 02:06 | |
| clarkb | pabelanger: cool thanks | 02:07 |
| *** esberglu has quit IRC | 02:08 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 02:15 |
| *** tqtran has quit IRC | 02:15 | |
| *** roxanaghe has joined #openstack-infra | 02:15 | |
| fungi | if mridem comes back, i added a second approval on 363937 | 02:16 |
| fungi | it was initially approved before one of its depends-on merged | 02:16 |
| *** signed8bit_Zzz is now known as signed8bit | 02:18 | |
| *** rhallisey has quit IRC | 02:18 | |
| *** sdake has quit IRC | 02:21 | |
| *** markvoelker has quit IRC | 02:23 | |
| *** Swami has joined #openstack-infra | 02:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Run with cells v2 in placement and neutron grenade jobs https://review.openstack.org/363937 | 02:25 |
| *** tonytan4ever has quit IRC | 02:25 | |
| fungi | and there it is | 02:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 02:27 |
| fungi | also, while i'll be around some tomorrow, my availability for the maintenance later in the day will be determined in part by which direction this hurricane decides to steer itself | 02:27 |
| *** Swami__ has quit IRC | 02:27 | |
| *** roxanaghe has quit IRC | 02:28 | |
| tonyb | what OS is signing01.ci.openstack.org running? Xenial? | 02:28 |
| fungi | trusty | 02:28 |
| tonyb | fungi: Thanks. | 02:28 |
| cloudnull | pabelanger: w00t! | 02:28 |
| fungi | tonyb: spotted an issue with it? | 02:28 |
| *** tonytan4ever has joined #openstack-infra | 02:29 | |
| *** salv-orlando has joined #openstack-infra | 02:29 | |
| tonyb | fungi: I don't have a root cause but the symptom is that the automatic u-c bumps are being generated against master as opposed the stable/$branch | 02:29 |
| *** Genek has quit IRC | 02:29 | |
| tonyb | fungi: see: http://logs.openstack.org/7a/7ad1cd3a0da7a95fb7c14cf9eeb9ae683c247efa/release-post/tag-releases/82a4a38/console.html#_2016-09-01_15_55_56_932171 | 02:29 |
| cloudnull | evenings btw | 02:29 |
| *** spzala has quit IRC | 02:30 | |
| tonyb | fungi: but that commands "work" for me so I was takign a wild swing that it migh have a different version of git | 02:30 |
| *** Apoorva has quit IRC | 02:30 | |
| fungi | tonyb: odds are it doesn't have extra magic to map a git tag back to the correct branch | 02:30 |
| fungi | looking now | 02:30 |
| tonyb | fungi: oh? I look forwatf to seeing that magic | 02:31 |
| fungi | tonyb: oh, this is an older run... | 02:31 |
| fungi | tonyb: i think dhellmann has a change up to fix this | 02:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 02:31 |
| tonyb | Oh I'll go look | 02:31 |
| *** salv-orlando has quit IRC | 02:32 | |
| tonyb | fungi: "older" is in the last 24hours so it's not *that* old :) | 02:32 |
| fungi | i'm hunting too, because i think i got busy fixing something else and neglected to review it yet | 02:32 |
| fungi | tonyb: yeah, just a few hours ago | 02:32 |
| *** yamahata has quit IRC | 02:32 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 02:33 | |
| * tonyb sees the fix | 02:33 | |
| fungi | tonyb: https://review.openstack.org/364497 i think? | 02:33 |
| tonyb | fungi: thanks | 02:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0 https://review.openstack.org/236815 | 02:33 |
| fungi | tonyb: i've approved it just now--looked correct to me though i didn't run it to confirm | 02:34 |
| tonyb | fungi: Thanks. | 02:35 |
| fungi | we'll just see on the next tag whether it proposes the constraints change against the correct reqs branch | 02:35 |
| *** Swami__ has joined #openstack-infra | 02:35 | |
| ianw | timothyb89: ok, https://review.openstack.org/#/c/364560/ is it's final form if you wouldn't mind checking the results wrt stackviz. i'm working on the bashate failures ... it's a silly thing and i've got an upgrade patch in the works | 02:36 |
| *** Swami has quit IRC | 02:38 | |
| *** Swami_ has quit IRC | 02:38 | |
| *** ilyashakhat_mobi has quit IRC | 02:38 | |
| *** Swami has joined #openstack-infra | 02:38 | |
| tonyb | fungi: Do you have a few mins to talk about project renames? | 02:38 |
| *** kaisers_ has joined #openstack-infra | 02:40 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:44 | |
| *** kaisers_ has quit IRC | 02:45 | |
| *** thorst has quit IRC | 02:45 | |
| *** thorst has joined #openstack-infra | 02:46 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add diskimage-builder meeting agenda https://review.openstack.org/364519 | 02:48 |
| openstackgerrit | Merged openstack-infra/project-config: fix branch handling in clone_repo https://review.openstack.org/364497 | 02:48 |
| *** piet has joined #openstack-infra | 02:51 | |
| *** thorst has quit IRC | 02:54 | |
| *** rlandy|bbl is now known as rlandy | 02:57 | |
| *** dimtruck is now known as zz_dimtruck | 02:58 | |
| *** asettle has joined #openstack-infra | 03:05 | |
| *** fguillot is now known as fguillot_afk | 03:05 | |
| *** signed8b_ has joined #openstack-infra | 03:06 | |
| *** Genek has joined #openstack-infra | 03:06 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: NPM DSVM jobs are now voting. https://review.openstack.org/363200 | 03:07 |
| *** fguillot_afk is now known as fguillot | 03:07 | |
| *** asettle has quit IRC | 03:10 | |
| *** signed8bit has quit IRC | 03:10 | |
| *** sflanigan is now known as sflanigan_afk | 03:10 | |
| *** cody-somerville has joined #openstack-infra | 03:11 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads https://review.openstack.org/364607 | 03:13 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 03:13 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread https://review.openstack.org/363217 | 03:13 |
| *** aeng has joined #openstack-infra | 03:15 | |
| *** mtanino has joined #openstack-infra | 03:16 | |
| *** piet has quit IRC | 03:17 | |
| *** kzaitsev_mb has quit IRC | 03:19 | |
| *** armax has quit IRC | 03:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/release-tools: Change to checked-out directory https://review.openstack.org/364644 | 03:21 |
| *** shashank_hegde has quit IRC | 03:21 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 03:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads https://review.openstack.org/364607 | 03:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 03:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic worker threads https://review.openstack.org/363217 | 03:29 |
| *** signed8b_ has quit IRC | 03:32 | |
| *** vinaypotluri has joined #openstack-infra | 03:33 | |
| *** shashank_hegde has joined #openstack-infra | 03:37 | |
| *** zz_dimtruck is now known as dimtruck | 03:37 | |
| *** salv-orlando has joined #openstack-infra | 03:39 | |
| *** sflanigan_afk is now known as sflanigan | 03:41 | |
| *** cody-somerville has quit IRC | 03:41 | |
| *** coolsvap has joined #openstack-infra | 03:42 | |
| *** gouthamr_ has quit IRC | 03:45 | |
| *** ianw has quit IRC | 03:46 | |
| *** links has joined #openstack-infra | 03:46 | |
| *** tonytan_brb has joined #openstack-infra | 03:47 | |
| *** salv-orlando has quit IRC | 03:47 | |
| *** ijw has joined #openstack-infra | 03:47 | |
| *** Genek has quit IRC | 03:48 | |
| *** rfolco has joined #openstack-infra | 03:49 | |
| *** tonytan4ever has quit IRC | 03:49 | |
| *** ijw has quit IRC | 03:52 | |
| *** thorst has joined #openstack-infra | 03:52 | |
| *** kushal has joined #openstack-infra | 03:53 | |
| *** yfried has joined #openstack-infra | 03:54 | |
| *** fguillot has quit IRC | 03:57 | |
| *** thorst has quit IRC | 03:59 | |
| *** chlong has quit IRC | 04:01 | |
| *** dimtruck is now known as zz_dimtruck | 04:02 | |
| *** yfried has quit IRC | 04:03 | |
| *** Julien-zte has quit IRC | 04:05 | |
| *** cody-somerville has joined #openstack-infra | 04:05 | |
| *** senk has joined #openstack-infra | 04:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:15 | |
| *** chlong has joined #openstack-infra | 04:18 | |
| *** kzaitsev_mb has quit IRC | 04:20 | |
| *** ianw has joined #openstack-infra | 04:21 | |
| *** cody-somerville has quit IRC | 04:21 | |
| *** spzala has joined #openstack-infra | 04:30 | |
| *** psachin has joined #openstack-infra | 04:32 | |
| stevemar | boom goes the dynamite! | 04:34 |
| stevemar | post_failures all over the gate | 04:34 |
| *** spzala has quit IRC | 04:35 | |
| *** csmart has quit IRC | 04:35 | |
| *** Swami__ has quit IRC | 04:35 | |
| *** Swami has quit IRC | 04:35 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 04:35 | |
| stevemar | every job is failing, on the last day of newton-3, damn you murphy's law | 04:35 |
| *** d34dh0r53 has quit IRC | 04:36 | |
| *** cloudnull has quit IRC | 04:36 | |
| *** rlandy has quit IRC | 04:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:36 | |
| *** sigmavirus has quit IRC | 04:37 | |
| *** bhagyashri has quit IRC | 04:38 | |
| *** ilyashakhat_mobi has quit IRC | 04:39 | |
| *** rfolco has quit IRC | 04:41 | |
| *** tonytan_brb has quit IRC | 04:44 | |
| *** _sigmavirus24 has joined #openstack-infra | 04:44 | |
| *** mtanino has quit IRC | 04:45 | |
| *** cloudnull has joined #openstack-infra | 04:45 | |
| *** dkehn_ has quit IRC | 04:47 | |
| *** dkehn has quit IRC | 04:47 | |
| *** d34dh0r53 has joined #openstack-infra | 04:47 | |
| *** yfried has joined #openstack-infra | 04:47 | |
| *** salv-orlando has joined #openstack-infra | 04:50 | |
| *** gongysh has quit IRC | 04:54 | |
| *** caowei has quit IRC | 04:54 | |
| *** yfried has quit IRC | 04:54 | |
| *** nstolyarenko has joined #openstack-infra | 04:55 | |
| *** jerryz has joined #openstack-infra | 04:56 | |
| *** thorst has joined #openstack-infra | 04:56 | |
| *** nstolyarenko has quit IRC | 04:57 | |
| *** senk has quit IRC | 04:58 | |
| *** yamamoto_ has quit IRC | 04:58 | |
| *** salv-orlando has quit IRC | 04:59 | |
| AJaeger | infra-root ^ | 05:00 |
| AJaeger | tonyb: we do the rename tonight - anything I can help with? | 05:01 |
| jhesketh | hmm, that's not good | 05:01 |
| * jhesketh looks | 05:01 | |
| jhesketh | log server is full | 05:03 |
| *** kzaitsev_mb has quit IRC | 05:03 | |
| *** thorst has quit IRC | 05:04 | |
| *** rajinir has quit IRC | 05:05 | |
| *** caowei has joined #openstack-infra | 05:06 | |
| jhesketh | #status alert The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified. | 05:06 |
| openstackstatus | jhesketh: sending alert | 05:06 |
| *** dolphm has quit IRC | 05:07 | |
| *** netsin has quit IRC | 05:08 | |
| *** fungi has quit IRC | 05:08 | |
| jhesketh | ianw: are you around? | 05:08 |
| -openstackstatus- NOTICE: The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified. | 05:09 | |
| *** lbragstad has quit IRC | 05:09 | |
| *** ChanServ changes topic to "The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified." | 05:09 | |
| AJaeger | thanks, jhesketh - and thanks stevemar for notifying | 05:09 |
| *** gongysh has joined #openstack-infra | 05:10 | |
| *** claudiub has joined #openstack-infra | 05:11 | |
| openstackstatus | jhesketh: finished sending alert | 05:12 |
| *** dolphm has joined #openstack-infra | 05:13 | |
| *** timsim has quit IRC | 05:14 | |
| *** jtomasek|afk is now known as jtomasek | 05:15 | |
| *** yfried has joined #openstack-infra | 05:16 | |
| *** lbragstad has joined #openstack-infra | 05:18 | |
| *** waht has quit IRC | 05:20 | |
| *** coolsvap is now known as coolsvap_ | 05:21 | |
| *** dkehn has joined #openstack-infra | 05:21 | |
| *** fungi has joined #openstack-infra | 05:22 | |
| fungi | what an ideal time for my irc client to lose access to freenode | 05:23 |
| AJaeger | morning, fungi! | 05:24 |
| fungi | jhesketh: so https://review.openstack.org/360551 merged a week ago but i've just noticed it hasn't taken effect | 05:24 |
| fungi | because that's not the file we're using any longer. looks like it's been superseded by one in the puppet-openstackci repo but never cleaned up in system-config afterward? | 05:24 |
| AJaeger | thanks a lot for your deep digging into the pip/virtualenv etc. situation! | 05:24 |
| jhesketh | fungi: that's the run running according to crontab | 05:25 |
| *** dkehn_ has joined #openstack-infra | 05:25 | |
| fungi | jhesketh: right, but you want a lower retention | 05:25 |
| *** jerryz has quit IRC | 05:25 | |
| fungi | jhesketh: anyway, my _strong_ recommendation is to run a copy of the log rotation script leaving out the compression bit so it gets to deleting stuff sooner, and dial down the retention | 05:26 |
| jhesketh | fungi: ah right, I see what you mean | 05:26 |
| jhesketh | fungi: I was just running it, but I didn't take out the compression... I'll do that now | 05:26 |
| fungi | jhesketh: there's one in my homedir already set to -mtime 60 | 05:26 |
| jhesketh | oh even easier | 05:27 |
| *** chlong has quit IRC | 05:27 | |
| fungi | i discovered you were already running the script because when i tried to run it the flock immediately bailed on me | 05:27 |
| jhesketh | fungi: I've restarted with your script | 05:27 |
| *** adriant has quit IRC | 05:27 | |
| fungi | then some minutes after that my irc client told me it lost contact with freenode, so the ~10 minutes i had been talking to you in here about it, i was really just talking to myself | 05:27 |
| AJaeger | what fun, fungi ;) | 05:28 |
| jhesketh | fungi: I was also attaching a new volume to grow it, but that'll require taking the partition offline and rsyncing the mess created while it reszes | 05:28 |
| jhesketh | fungi: woops :-s | 05:28 |
| fungi | jhesketh: i'd be curious how you expect to attach a new volume to it | 05:28 |
| jhesketh | fungi: following http://docs.openstack.org/infra/system-config/sysadmin.html#expanding-an-existing-logical-volume | 05:29 |
| fungi | afaik 16 xvd devices is the max for the version of xen rackspace is using | 05:29 |
| *** yfried has quit IRC | 05:29 | |
| jhesketh | fungi: ah, I was wondering if I was running into a limit because it hasn't attached yet | 05:29 |
| fungi | we have one for the rootfs, one for the ephemeral disk, and 14x 1tb (the maximum size rackspace will let us create them) cinder devices attached | 05:29 |
| jhesketh | (and had vague memories of that being why we didn't have more alrady) | 05:29 |
| fungi | yep | 05:29 |
| fungi | there is no way i know of to add more local storage to that server | 05:30 |
| jhesketh | oh well, the lower retention is working | 05:31 |
| fungi | entirely separate from this, ipv6 in rax-iad seems to have gone sideways (which is why i dropped off irc) | 05:31 |
| fungi | though i think that won't impact our ci for a couple of reasons, one of which being that we still have iad completely disabled in nodepool | 05:32 |
| *** adriant has joined #openstack-infra | 05:32 | |
| jhesketh | #status ok Space has been freed up on the log server. If you have POST_FAILURE results it is now safe to issue a 'recheck' | 05:32 |
| openstackstatus | jhesketh: sending ok | 05:32 |
| *** Julien-zte has joined #openstack-infra | 05:32 | |
| jhesketh | fungi: there's been some funky networking issues in rax the last few days it would seem | 05:33 |
| *** javeriak has joined #openstack-infra | 05:33 | |
| jhesketh | if only we stored our logs in some kind of cloud storage solution... ;-p | 05:33 |
| fungi | jhesketh: also, i should have mentioned it, but you probably want to run that log deletion script under screen... took >24 hours to complete the last few times i ran it manually | 05:33 |
| jhesketh | fungi: yep, it's on a screen under my user if you wanted to join | 05:34 |
| fungi | oh, cool | 05:34 |
| fungi | anyway, i'm going to bed. i'm glad i peeked at irc one last time first :/ | 05:34 |
| fungi | thanks for taking care of this! | 05:34 |
| jhesketh | it has all the gzip junk from earlier, but other than that it's working | 05:34 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 05:35 | |
| jhesketh | fungi: thanks for your help, sorry to have disturbed your night | 05:35 |
| -openstackstatus- NOTICE: Space has been freed up on the log server. If you have POST_FAILURE results it is now safe to issue a 'recheck' | 05:35 | |
| fungi | jhesketh: you disturbed nothing--i looked at the computer by choice! ;) | 05:35 |
| fungi | night all | 05:35 |
| jhesketh | thanks nonetheless :-) | 05:36 |
| jhesketh | night! | 05:36 |
| *** javeriak_ has joined #openstack-infra | 05:37 | |
| *** senk has joined #openstack-infra | 05:37 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 05:37 | |
| *** javeriak has quit IRC | 05:38 | |
| openstackstatus | jhesketh: finished sending ok | 05:38 |
| *** timsim has joined #openstack-infra | 05:43 | |
| *** pgadiya has joined #openstack-infra | 05:46 | |
| *** chlong has joined #openstack-infra | 05:48 | |
| *** _nadya_ has joined #openstack-infra | 05:50 | |
| *** _nadya_ has quit IRC | 05:50 | |
| tonyb | AJaeger: I was just wondering what I need to do in the requirements repo for projects that have been renamed? I auumes that for a period of time we need to list both old and new packages or the proposal-bot will be confused | 05:51 |
| *** yamamoto_ has joined #openstack-infra | 05:55 | |
| *** nstolyarenko has joined #openstack-infra | 05:57 | |
| AJaeger | For projects.txt: rename them directly | 05:59 |
| AJaeger | for global-requirements: As soon as a new release is out: ADd a new entry | 06:00 |
| AJaeger | Once all projects have imported the sync, remove the old one. | 06:00 |
| AJaeger | That's what I would do - would that work? | 06:00 |
| *** csmart_ has joined #openstack-infra | 06:00 | |
| AJaeger | remember to backport the projects.txt change. | 06:01 |
| *** martinkopec has joined #openstack-infra | 06:01 | |
| AJaeger | I would ignore global-requiremens in stable branches for this | 06:01 |
| *** nstolyarenko has quit IRC | 06:02 | |
| *** csmart_ is now known as csmart | 06:02 | |
| *** thorst has joined #openstack-infra | 06:02 | |
| *** bhagyashri has joined #openstack-infra | 06:02 | |
| *** hichihara has joined #openstack-infra | 06:05 | |
| *** aeng has quit IRC | 06:06 | |
| *** hashar has joined #openstack-infra | 06:07 | |
| *** thorst has quit IRC | 06:09 | |
| *** netsin has joined #openstack-infra | 06:12 | |
| *** salv-orlando has joined #openstack-infra | 06:16 | |
| *** kaisers_ has joined #openstack-infra | 06:17 | |
| *** adriant has quit IRC | 06:19 | |
| *** Thelo_ has joined #openstack-infra | 06:20 | |
| *** Thelo_ has quit IRC | 06:22 | |
| *** kaisers_ has quit IRC | 06:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:29 | |
| *** woodster_ has quit IRC | 06:29 | |
| *** Julien-zte has quit IRC | 06:30 | |
| *** Julien-zte has joined #openstack-infra | 06:36 | |
| *** thorst has joined #openstack-infra | 06:36 | |
| *** ilyashakhat_mobi has quit IRC | 06:37 | |
| *** rcernin has joined #openstack-infra | 06:40 | |
| *** shashank_hegde has quit IRC | 06:41 | |
| *** Ravikiran_K has joined #openstack-infra | 06:41 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 06:42 |
| *** thorst has quit IRC | 06:44 | |
| *** andreas_s has joined #openstack-infra | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:48 | |
| *** chlong has quit IRC | 06:49 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 06:50 | |
| *** berendt has joined #openstack-infra | 06:50 | |
| *** gongysh has quit IRC | 06:51 | |
| *** florianf has joined #openstack-infra | 06:51 | |
| *** florianf has quit IRC | 06:52 | |
| *** kzaitsev_mb has quit IRC | 06:53 | |
| *** Julien-zte has quit IRC | 06:53 | |
| *** drifterza has joined #openstack-infra | 06:55 | |
| *** ilyashakhat_mobi has quit IRC | 06:56 | |
| *** shashank_hegde has joined #openstack-infra | 06:57 | |
| *** maishsk_ has joined #openstack-infra | 07:01 | |
| *** tesseract- has joined #openstack-infra | 07:02 | |
| *** signed8bit has joined #openstack-infra | 07:02 | |
| *** maishsk has quit IRC | 07:03 | |
| *** maishsk_ is now known as maishsk | 07:03 | |
| *** florianf has joined #openstack-infra | 07:03 | |
| *** javeriak_ has quit IRC | 07:05 | |
| *** javeriak has joined #openstack-infra | 07:05 | |
| *** signed8bit has quit IRC | 07:06 | |
| *** shashank_hegde has quit IRC | 07:07 | |
| *** kushal has quit IRC | 07:08 | |
| *** HeOS has joined #openstack-infra | 07:09 | |
| *** Thelo_ has joined #openstack-infra | 07:11 | |
| *** sflanigan has quit IRC | 07:14 | |
| *** salv-orlando has quit IRC | 07:14 | |
| *** kaisers_ has joined #openstack-infra | 07:15 | |
| *** mpaolino has joined #openstack-infra | 07:16 | |
| *** javeriak has quit IRC | 07:19 | |
| *** jpich has joined #openstack-infra | 07:19 | |
| *** mikelk has joined #openstack-infra | 07:20 | |
| *** Hal1 has joined #openstack-infra | 07:21 | |
| *** shashank_hegde has joined #openstack-infra | 07:22 | |
| *** shardy_afk has joined #openstack-infra | 07:22 | |
| *** askb has joined #openstack-infra | 07:23 | |
| *** Thelo_ has quit IRC | 07:26 | |
| *** javeriak has joined #openstack-infra | 07:26 | |
| *** spzala has joined #openstack-infra | 07:30 | |
| *** Na3iL has joined #openstack-infra | 07:30 | |
| *** javeriak has quit IRC | 07:31 | |
| AJaeger | tonyb: still around? | 07:32 |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 07:34 |
| *** Thelo_ has joined #openstack-infra | 07:34 | |
| *** kaisers_ has quit IRC | 07:34 | |
| *** esikachev has joined #openstack-infra | 07:34 | |
| *** spzala has quit IRC | 07:35 | |
| tonyb | AJaeger: On and off ... trying to make dinner but also trying to make sure I don't cause probelms for people with -2's | 07:35 |
| *** ifarkas_afk is now known as ifarkas | 07:36 | |
| AJaeger | ;) Enjoy dinner! | 07:36 |
| tonyb | :) | 07:37 |
| ttx | AJaeger: o/ We have a small problem with the tag-release job: since https://review.openstack.org/#/c/364497/1 merged it tries to use zuul-cloner but that is apparently unavailable http://logs.openstack.org/62/6205a619b0949e9b97bb11c42e6a8098d33a3f32/release-post/tag-releases/22669ea/console.html | 07:38 |
| ttx | I'm tempted to revert the patch while we find a solution | 07:39 |
| ttx | unless you can find out why zuul-cloner is not available | 07:39 |
| ttx | I suspect that's because the signing node doesn't have it installed | 07:39 |
| AJaeger | ianw, jhesketh: Here're a few reviews for project-config that should go in IMHO - for newton release work: https://review.openstack.org/364591 https://review.openstack.org/364544 https://review.openstack.org/363978 https://review.openstack.org/363982 - could you help landing these, please? | 07:40 |
| AJaeger | ttx, mmh, fungi mentioned earlier thta the signing node should have zuul-cloner. Let's tell him so that he can double check later ^ | 07:40 |
| *** matrohon has joined #openstack-infra | 07:42 | |
| *** thorst has joined #openstack-infra | 07:42 | |
| AJaeger | ttx, zuul-cloner lives on other systems in /usr/zuul-env/bin/zuul-cloner | 07:42 |
| AJaeger | Do you want to send a patch to change the path - and then test? | 07:42 |
| *** matthewbodkin has joined #openstack-infra | 07:42 | |
| *** nstolyarenko has joined #openstack-infra | 07:43 | |
| ttx | AJaeger: if we are not 99% sure of the fix I'd rather just revert the change and unblock master releases | 07:44 |
| *** nstolyarenko has quit IRC | 07:44 | |
| ttx | since it takes a bit of time to propagate | 07:44 |
| *** amotoki has joined #openstack-infra | 07:44 | |
| ttx | I can propose the revert if you're ok to approve it | 07:45 |
| AJaeger | ttx, we need an infra-root to log into the signing node and confirm the path. | 07:45 |
| AJaeger | ttx, sure I'm fine to approve to unblock you... | 07:45 |
| AJaeger | ttx, reading backscroll: Update th epath - and I change it. | 07:46 |
| *** oanson has joined #openstack-infra | 07:46 | |
| AJaeger | Let me show you what I mean: | 07:46 |
| *** Ravikiran_K has quit IRC | 07:46 | |
| AJaeger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-09-01.log.html#t2016-09-01T18:13:01 | 07:46 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Revert "fix branch handling in clone_repo" https://review.openstack.org/364727 | 07:47 |
| AJaeger | "dhellmann: separate (but related) note... we did eventually get /usr/zuul-env/bin/zuul-cloner onto the signing node so that script can likely be simplified again when you're ready to hack on that" | 07:47 |
| AJaeger | So, dhellmann simplified too much ;) | 07:47 |
| ttx | ah, hmm, that might qualify as 99% sure | 07:47 |
| *** shardy has joined #openstack-infra | 07:47 | |
| ttx | AJaeger: let me propose the path change instead | 07:48 |
| AJaeger | ttx, ok. Will wiat for it and approve directly if it's fine... | 07:48 |
| *** thorst has quit IRC | 07:48 | |
| *** yaume has joined #openstack-infra | 07:49 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 07:50 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 07:50 |
| *** kzaitsev_mb has joined #openstack-infra | 07:50 | |
| *** shardy_afk has quit IRC | 07:51 | |
| *** Julien-zte has joined #openstack-infra | 07:52 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Update zuul-cloner path https://review.openstack.org/364729 | 07:52 |
| AJaeger | ttx, +2A | 07:54 |
| ttx | thx! | 07:54 |
| ttx | it can hardly be worse anyway :) | 07:54 |
| AJaeger | ;) | 07:54 |
| AJaeger | I trust fungi, it will work ;) | 07:55 |
| *** shashank_hegde has quit IRC | 07:57 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** oanson has quit IRC | 08:00 | |
| openstackgerrit | Masayuki Igawa proposed openstack/gertty: Add number of changes to the change list view https://review.openstack.org/364733 | 08:01 |
| *** shashank_hegde has joined #openstack-infra | 08:01 | |
| *** hichihara has quit IRC | 08:03 | |
| *** Julien-zte has quit IRC | 08:05 | |
| *** lucas-dinner is now known as lucasagomes | 08:05 | |
| *** asettle has joined #openstack-infra | 08:07 | |
| *** derekh has joined #openstack-infra | 08:08 | |
| *** bauzas is now known as bauwser | 08:10 | |
| *** ggnel_t has joined #openstack-infra | 08:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Update zuul-cloner path https://review.openstack.org/364729 | 08:11 |
| *** kaisers_ has joined #openstack-infra | 08:12 | |
| *** kzaitsev_mb has quit IRC | 08:21 | |
| *** gongysh has joined #openstack-infra | 08:23 | |
| *** salv-orlando has joined #openstack-infra | 08:25 | |
| openstackgerrit | lijunli proposed openstack-dev/pbr: When generating local console scripts, should give user a choice to decide whether add -O to ‘executable’, such as '/usr/bin/python -O' https://review.openstack.org/364759 | 08:25 |
| *** maishsk has quit IRC | 08:25 | |
| *** javeriak has joined #openstack-infra | 08:26 | |
| *** maishsk has joined #openstack-infra | 08:26 | |
| *** berendt has quit IRC | 08:32 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:34 |
| *** oanson has joined #openstack-infra | 08:35 | |
| *** yolanda has joined #openstack-infra | 08:37 | |
| *** r-mibu has quit IRC | 08:38 | |
| *** dkehn_ has quit IRC | 08:38 | |
| *** r-mibu has joined #openstack-infra | 08:38 | |
| *** dkehn has quit IRC | 08:39 | |
| *** shashank_hegde has quit IRC | 08:42 | |
| *** berendt has joined #openstack-infra | 08:44 | |
| *** dkehn has joined #openstack-infra | 08:45 | |
| *** thorst has joined #openstack-infra | 08:46 | |
| *** _degorenko|afk is now known as degorenko | 08:47 | |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 08:47 |
| *** shardy has quit IRC | 08:48 | |
| *** gongysh has quit IRC | 08:48 | |
| *** berendt has quit IRC | 08:50 | |
| *** notnownikki has joined #openstack-infra | 08:50 | |
| *** thorst has quit IRC | 08:54 | |
| *** javeriak has quit IRC | 08:57 | |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/364775 | 08:57 |
| *** dkehn_ has joined #openstack-infra | 08:57 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 08:58 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic https://review.openstack.org/335981 | 08:58 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 08:58 |
| *** Ravikiran_K has joined #openstack-infra | 08:59 | |
| AJaeger | infra-root, https://github.com/openstack/deb-python-hacking is not setup correctly, there's no content. The repo was created yesterday. Could you set permissions correctly, please? | 09:02 |
| *** Genek has joined #openstack-infra | 09:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Enable release-notes for magnum https://review.openstack.org/364782 | 09:04 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:10 |
| *** jaosorior has joined #openstack-infra | 09:11 | |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 09:11 |
| *** mptacekx has joined #openstack-infra | 09:19 | |
| andreykurilin | AJaeger: hi! Does it possible(if yes - how?) to get privileged access to #openstack-rally chat? it looks like it is managed by openstack-infra | 09:20 |
| *** Hal1 has quit IRC | 09:22 | |
| *** Na3iL has quit IRC | 09:25 | |
| *** sambetts|afk is now known as sambetts | 09:28 | |
| AJaeger | andreykurilin: one of the IRC admins can do this for you - I'm not one. If it's not urgent, better wait until later today | 09:30 |
| AJaeger | andreykurilin: see also http://docs.openstack.org/infra/system-config/irc.html | 09:31 |
| *** _nadya_ has joined #openstack-infra | 09:31 | |
| AJaeger | andreykurilin: and yes, our accessbot has privileged access to #openstack-rally so that our bots can send messages and admins have access | 09:32 |
| andreykurilin | AJaeger: it is not urgent. I just want to modify topic of the chat, so I can wait :) whom should I ping later? | 09:33 |
| AJaeger | If you issue "/msg chanserv access #openstack-rally list", you see who has access. | 09:33 |
| AJaeger | So, boris-42 and SergeyLukjanov should be able to help you | 09:34 |
| andreykurilin | ok, thanks | 09:34 |
| AJaeger | Or one of the others later | 09:34 |
| andreykurilin | thanks a lot | 09:34 |
| *** yaume has quit IRC | 09:35 | |
| *** javeriak has joined #openstack-infra | 09:35 | |
| *** yanyanhu has quit IRC | 09:37 | |
| *** yamamoto_ has quit IRC | 09:38 | |
| *** yaume has joined #openstack-infra | 09:39 | |
| *** Na3iL has joined #openstack-infra | 09:39 | |
| *** zshuo has quit IRC | 09:40 | |
| andreykurilin | AJaeger: btw, where I can look at all available job environment variables? for example I want to access to name of the job from execution script. Does it possible? | 09:41 |
| AJaeger | look at the log file, see the ansible logs for details | 09:46 |
| AJaeger | andreykurilin: but accessing the name of the job is a really bad idea, that might change. | 09:46 |
| AJaeger | andreykurilin: why do you want to do that? | 09:47 |
| *** berendt has joined #openstack-infra | 09:48 | |
| andreykurilin | AJaeger: actually, I'm not interesting on the name of job, I just want to find some variable which can help to determine where test was launched(at local-pc or at upstream-job). Such information will not affect how tests are launched and how how results are processed(I do not want volkswagen-ci), but it can help to build different formats of logs/reports | 09:49 |
| andreykurilin | *interesting in | 09:50 |
| AJaeger | So, check the ansible log for which are set | 09:51 |
| *** thorst has joined #openstack-infra | 09:52 | |
| AJaeger | ZUUL_* are pretty unique to our CI ;) | 09:52 |
| *** javeriak has quit IRC | 09:52 | |
| lifeless | fungi: quite seriously, noone *needs* editables, though they often desire it. | 09:54 |
| *** e0ne has joined #openstack-infra | 09:55 | |
| lifeless | ianw: not sure what you mean by querying how you were installed... | 09:56 |
| andreykurilin | AJaeger: :( okay, I'll parse logs:) | 09:58 |
| *** bhagyashri has quit IRC | 09:58 | |
| AJaeger | lifeless: even with usedevelop=False it fails for me ;( | 09:58 |
| *** thorst has quit IRC | 09:58 | |
| lifeless | AJaeger: :( | 09:59 |
| AJaeger | http://logs.openstack.org/71/364671/2/check/gate-openstackdocstheme-pep8-ubuntu-xenial/1c14d78/console.html#_2016-09-02_05_48_47_336466 | 09:59 |
| AJaeger | the only solution that worked is constraints ;) | 09:59 |
| lifeless | \o/ | 10:00 |
| lifeless | who woulda thought | 10:00 |
| AJaeger | ;) | 10:00 |
| *** zhurong_ has quit IRC | 10:03 | |
| openstackgerrit | Pawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade https://review.openstack.org/364809 | 10:03 |
| *** markvoelker has joined #openstack-infra | 10:10 | |
| *** rtheis has joined #openstack-infra | 10:11 | |
| *** kevinbenton has quit IRC | 10:15 | |
| *** drifterza has quit IRC | 10:17 | |
| *** kevinbenton has joined #openstack-infra | 10:17 | |
| *** tkelsey has joined #openstack-infra | 10:19 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:19 | |
| *** rods has quit IRC | 10:22 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** tkelsey has quit IRC | 10:23 | |
| *** kzaitsev_mb has quit IRC | 10:25 | |
| *** rods has joined #openstack-infra | 10:25 | |
| *** pblaho has joined #openstack-infra | 10:26 | |
| *** markvoelker has joined #openstack-infra | 10:30 | |
| *** caowei has quit IRC | 10:31 | |
| *** mpaolino has quit IRC | 10:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add images profile for openstackci oscc clouds and add it to infracloud https://review.openstack.org/364826 | 10:39 |
| *** markvoelker has quit IRC | 10:40 | |
| rcarrillocruz | mordred , pabelanger : automating mirror images bootstrapping on our new clouds ^ | 10:41 |
| rcarrillocruz | \o/ | 10:41 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 10:45 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic https://review.openstack.org/335981 | 10:45 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 10:45 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic https://review.openstack.org/364830 | 10:45 |
| AJaeger | project-config cores: Here're a few reviews for project-config that should go in IMHO - for newton release work: https://review.openstack.org/364591 https://review.openstack.org/364544 https://review.openstack.org/363978 https://review.openstack.org/363982 - could you help landing these, please? | 10:49 |
| *** matbu is now known as matbu|lch | 10:50 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [Rally] Add new experimental jobs https://review.openstack.org/364836 | 10:55 |
| *** thorst has joined #openstack-infra | 10:56 | |
| *** berendt has quit IRC | 10:56 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 10:57 |
| *** yolanda has quit IRC | 10:57 | |
| *** zul has quit IRC | 10:58 | |
| *** slaweq has quit IRC | 10:58 | |
| *** yolanda has joined #openstack-infra | 11:01 | |
| *** thorst has quit IRC | 11:04 | |
| *** sarob has joined #openstack-infra | 11:05 | |
| *** kzaitsev_mb has quit IRC | 11:07 | |
| *** _nadya_ has quit IRC | 11:07 | |
| *** mptacekx has quit IRC | 11:07 | |
| *** Genek has quit IRC | 11:07 | |
| *** claudiub has quit IRC | 11:07 | |
| *** jgrassler has quit IRC | 11:07 | |
| *** dulek has quit IRC | 11:07 | |
| *** afred312 has quit IRC | 11:07 | |
| *** M-docaedo_vector has quit IRC | 11:07 | |
| *** degorenko has quit IRC | 11:07 | |
| *** lennyb has quit IRC | 11:07 | |
| *** spiffxp has quit IRC | 11:07 | |
| *** wfoster has quit IRC | 11:07 | |
| *** darrenc has quit IRC | 11:07 | |
| *** SotK has quit IRC | 11:07 | |
| *** anteaya has quit IRC | 11:07 | |
| *** sambetts has quit IRC | 11:07 | |
| *** stevebaker has quit IRC | 11:07 | |
| *** eliqiao has quit IRC | 11:07 | |
| *** aderyugin has quit IRC | 11:07 | |
| *** _nadya_ has joined #openstack-infra | 11:09 | |
| *** mptacekx has joined #openstack-infra | 11:09 | |
| *** claudiub has joined #openstack-infra | 11:09 | |
| *** dulek has joined #openstack-infra | 11:09 | |
| *** afred312 has joined #openstack-infra | 11:09 | |
| *** M-docaedo_vector has joined #openstack-infra | 11:09 | |
| *** degorenko has joined #openstack-infra | 11:09 | |
| *** lennyb has joined #openstack-infra | 11:09 | |
| *** spiffxp has joined #openstack-infra | 11:09 | |
| *** wfoster has joined #openstack-infra | 11:09 | |
| *** darrenc has joined #openstack-infra | 11:09 | |
| *** SotK has joined #openstack-infra | 11:09 | |
| *** anteaya has joined #openstack-infra | 11:09 | |
| *** sambetts has joined #openstack-infra | 11:09 | |
| *** stevebaker has joined #openstack-infra | 11:09 | |
| *** eliqiao has joined #openstack-infra | 11:09 | |
| *** aderyugin has joined #openstack-infra | 11:09 | |
| *** sarob has quit IRC | 11:09 | |
| *** rhallisey has joined #openstack-infra | 11:11 | |
| *** rhallisey has quit IRC | 11:11 | |
| *** rhallisey has joined #openstack-infra | 11:13 | |
| *** jgrassler has joined #openstack-infra | 11:14 | |
| *** dingyichen has quit IRC | 11:15 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Add basic docs for python client https://review.openstack.org/362878 | 11:19 |
| *** jkilpatr has joined #openstack-infra | 11:20 | |
| *** javeriak has joined #openstack-infra | 11:20 | |
| *** ramishra has quit IRC | 11:21 | |
| *** ramishra has joined #openstack-infra | 11:23 | |
| *** flepied has quit IRC | 11:25 | |
| *** flepied has joined #openstack-infra | 11:27 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 11:28 |
| *** matbu|lch is now known as matbu | 11:28 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 11:28 |
| *** yolanda has quit IRC | 11:29 | |
| *** matthewbodkin has quit IRC | 11:29 | |
| *** zul has joined #openstack-infra | 11:30 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 11:30 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 11:30 |
| *** florianf has quit IRC | 11:31 | |
| *** spzala has joined #openstack-infra | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** Thelo_ has quit IRC | 11:33 | |
| *** spzala has quit IRC | 11:35 | |
| *** apetrich has quit IRC | 11:37 | |
| *** Thelo_ has joined #openstack-infra | 11:38 | |
| *** baoli has joined #openstack-infra | 11:39 | |
| *** Thelo_ has quit IRC | 11:40 | |
| *** lucasagomes is now known as lucas-hungry | 11:40 | |
| *** Thelo_ has joined #openstack-infra | 11:43 | |
| *** Hal1 has joined #openstack-infra | 11:43 | |
| *** florianf has joined #openstack-infra | 11:45 | |
| *** dprince has joined #openstack-infra | 11:47 | |
| *** gouthamr has joined #openstack-infra | 11:49 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 11:49 |
| *** baoli_ has joined #openstack-infra | 11:54 | |
| *** EricGonczer_ has joined #openstack-infra | 11:56 | |
| *** EricGonczer_ has quit IRC | 11:56 | |
| *** thorst has joined #openstack-infra | 11:57 | |
| *** baoli has quit IRC | 11:57 | |
| *** EricGonczer_ has joined #openstack-infra | 11:59 | |
| *** derekh has quit IRC | 12:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:03 | |
| *** pilgrimstack has quit IRC | 12:03 | |
| *** gongysh has joined #openstack-infra | 12:03 | |
| *** tpsilva has joined #openstack-infra | 12:04 | |
| openstackgerrit | Merged openstack-infra/nodepool: Unsubvert image and flavor caching https://review.openstack.org/364510 | 12:04 |
| *** _sigmavirus24 is now known as sigmavirus | 12:05 | |
| *** sigmavirus has joined #openstack-infra | 12:05 | |
| *** ekarlso_ has joined #openstack-infra | 12:05 | |
| *** pilgrimstack has joined #openstack-infra | 12:05 | |
| *** EricGonczer_ has quit IRC | 12:06 | |
| *** raildo has joined #openstack-infra | 12:06 | |
| *** baoli has joined #openstack-infra | 12:08 | |
| *** baoli__ has joined #openstack-infra | 12:10 | |
| *** ekarlso_ has quit IRC | 12:11 | |
| *** salv-orlando has quit IRC | 12:11 | |
| *** flaper87 has quit IRC | 12:11 | |
| *** baoli_ has quit IRC | 12:12 | |
| *** matthewbodkin has joined #openstack-infra | 12:12 | |
| *** ilyashakhat has joined #openstack-infra | 12:12 | |
| *** asettle has quit IRC | 12:13 | |
| *** baoli has quit IRC | 12:13 | |
| *** zul_ has joined #openstack-infra | 12:15 | |
| *** trown|outtypewww is now known as trown | 12:15 | |
| *** gordc has joined #openstack-infra | 12:15 | |
| *** askb has quit IRC | 12:15 | |
| *** zul_ has quit IRC | 12:16 | |
| *** zul_ has joined #openstack-infra | 12:17 | |
| *** esikachev has quit IRC | 12:17 | |
| *** zul has quit IRC | 12:17 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 12:19 |
| rcarrillocruz | clarkb: https://review.openstack.org/#/c/364868/1 , as discussed the other day, to just run the launcher against a particular clouds from our clouds_layouts.yaml | 12:21 |
| *** javeriak has quit IRC | 12:23 | |
| rcarrillocruz | s/clouds/cloud | 12:23 |
| *** berendt has joined #openstack-infra | 12:24 | |
| *** rlandy has joined #openstack-infra | 12:24 | |
| mordred | #status log restarted nodepool with the latest shade and nodepool changes. all looks well - floating-ips, images and flavors are not being hammered | 12:24 |
| openstackstatus | mordred: finished logging | 12:24 |
| mordred | infra-root: ^^ | 12:24 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 12:25 |
| *** andreas_s has quit IRC | 12:26 | |
| *** rodrigods has quit IRC | 12:27 | |
| *** rodrigods has joined #openstack-infra | 12:27 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Move the clouds_layouts.yaml file https://review.openstack.org/364871 | 12:29 |
| *** flaper87 has joined #openstack-infra | 12:29 | |
| *** flaper87 has quit IRC | 12:29 | |
| *** flaper87 has joined #openstack-infra | 12:29 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart https://review.openstack.org/364874 | 12:30 |
| *** rfolco has joined #openstack-infra | 12:30 | |
| AJaeger | mordred, rcarrillocruz : Here're a few reviews for project-config that should go in IMHO - for newton release work: https://review.openstack.org/364591 https://review.openstack.org/364544 https://review.openstack.org/363978 https://review.openstack.org/363982 - could you help landing these, please? | 12:34 |
| *** electrofelix has joined #openstack-infra | 12:34 | |
| rcarrillocruz | checking | 12:35 |
| *** javeriak has joined #openstack-infra | 12:38 | |
| *** annegentle has joined #openstack-infra | 12:38 | |
| *** lucas-hungry is now known as lucasagomes | 12:38 | |
| *** javeriak has quit IRC | 12:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add cinder-stable-maint to os-brick https://review.openstack.org/363982 | 12:42 |
| *** zul_ has quit IRC | 12:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Add nova-stable-maint to os-vif gerrit ACL https://review.openstack.org/363978 | 12:43 |
| *** jaosorior has quit IRC | 12:43 | |
| *** jaosorior has joined #openstack-infra | 12:43 | |
| AJaeger | thanks, rcarrillocruz | 12:43 |
| *** zul has joined #openstack-infra | 12:43 | |
| *** zul has quit IRC | 12:45 | |
| *** zul has joined #openstack-infra | 12:45 | |
| rcarrillocruz | np | 12:46 |
| *** zhurong has joined #openstack-infra | 12:46 | |
| *** zynzel has joined #openstack-infra | 12:46 | |
| zigo | mordred: https://review.openstack.org/364537 <--- As per our discussion yesterday with fungi. | 12:47 |
| rcarrillocruz | pabelanger: you know what, following what fungi said yesterday... how about deploying chocolate with Mitaka? | 12:47 |
| rcarrillocruz | i mean | 12:47 |
| openstackgerrit | Merged openstack-infra/project-config: fix networking-hyperv release acls https://review.openstack.org/364591 | 12:47 |
| rcarrillocruz | we could start provisioning | 12:47 |
| rcarrillocruz | deploying the cloud | 12:48 |
| rcarrillocruz | etc | 12:48 |
| rcarrillocruz | finish up things during mid cycle | 12:48 |
| rcarrillocruz | and will be a great exercise to redeploy a whole cloud when newton is out the door and we decide it's ok to go with it | 12:48 |
| rcarrillocruz | i see no point on having all this HW idle for now | 12:48 |
| amrith | rcarrillocruz, thanks for your help yesterday. got to the bottom of the problem, merged a fix, all humming along again. | 12:48 |
| mordred | zigo: awesome | 12:48 |
| rcarrillocruz | and putting RC1 or whatever it's risky | 12:49 |
| rcarrillocruz | amrith: you're very welcome! | 12:49 |
| *** alaski is now known as lascii | 12:49 | |
| rcarrillocruz | RC1 Newton that is | 12:49 |
| zigo | mordred: Cheers! | 12:49 |
| amrith | thx rcarrillocruz | 12:49 |
| amrith | mordred, g'morning | 12:49 |
| mordred | morning amrith ! | 12:49 |
| amrith | is today better? | 12:49 |
| zigo | AJaeger: Could you +2 workflow this one too, so we can move ahead and release Newton? :) https://review.openstack.org/#/c/364537/ | 12:49 |
| jhesketh | AJaeger: sorry I was out this evening... looks like rcarrillocruz beat me to it on those project-config patches needed for newton | 12:49 |
| AJaeger | zigo: not right now. Could you please look at https://review.openstack.org/356795 and clean up those problems? | 12:50 |
| zigo | AJaeger: I did have a look, but I'm not sure how. I did remember cleaning-up the upstream repo though. | 12:51 |
| *** gouthamr_ has joined #openstack-infra | 12:51 | |
| openstackgerrit | Merged openstack-infra/project-config: do not error if a branch already exists https://review.openstack.org/364544 | 12:51 |
| zigo | AJaeger: Do I still need to approve the removal of deb-websockify if the repo was cleaned? | 12:52 |
| * zigo checks again with git fsck | 12:52 | |
| *** annegentle has quit IRC | 12:52 | |
| zigo | AJaeger: git fsck says "dangling blob 9e758039948eef26684cd776824a2accf08a295f" | 12:52 |
| fungi | AJaeger: ttx: oh, yep when i reviewed 364497 i missed that it didn't specify the path to zuul-cloner. it was indeed supposed to use /usr/zuul-env/bin/zuul-cloner like on our other nodes | 12:53 |
| zigo | That's not a big issue, it's should be fine no? | 12:53 |
| zigo | Or should I still fix it? | 12:53 |
| skraynev | rcarrillocruz: hi. do you have rights to add me to group of reviewers for new (recently created) project? | 12:53 |
| rcarrillocruz | i do | 12:53 |
| rcarrillocruz | which group is | 12:53 |
| rcarrillocruz | ? | 12:53 |
| fungi | AJaeger: ttx: also reverting wouldn't have been much better since prior to that change it was checking out the wrong branch for some tags | 12:53 |
| ttx | fungi: yes, we got it all fixed | 12:54 |
| ttx | Works almost perfectly now | 12:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart https://review.openstack.org/364874 | 12:54 |
| ttx | Just needs https://review.openstack.org/#/c/364417/ | 12:54 |
| *** kgiusti has joined #openstack-infra | 12:54 | |
| ttx | fungi: then life will be perfect | 12:54 |
| *** jcoufal has joined #openstack-infra | 12:55 | |
| fungi | AJaeger: https://github.com/openstack/deb-python-hacking is replicating successfully now. same as usual, manage-projects got an http error response back from the github project creation api call, so thought it failed and didn't bother attempting to set permissions for gerrit replication into it | 12:55 |
| *** gouthamr has quit IRC | 12:55 | |
| skraynev | rcarrillocruz: groups https://review.openstack.org/#/admin/groups/1547,members https://review.openstack.org/#/admin/groups/1546,members https://review.openstack.org/#/admin/groups/1548,members according patch https://review.openstack.org/#/c/357745/ | 12:55 |
| *** woodster_ has joined #openstack-infra | 12:56 | |
| rcarrillocruz | skraynev: done | 12:57 |
| fungi | andreykurilin: i just made sure that boris-42 (as current ptl) has sufficient permissions to add other admins to #openstack-rally through chanserv now | 12:58 |
| *** kushal has joined #openstack-infra | 12:58 | |
| *** Goneri has joined #openstack-infra | 12:58 | |
| zigo | fungi: Can you +2 workflow so we can go ahead and release Newton ? https://review.openstack.org/364537 | 12:58 |
| *** Thelo_ has quit IRC | 12:59 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 12:59 |
| *** dtardivel has joined #openstack-infra | 12:59 | |
| *** piet has joined #openstack-infra | 13:00 | |
| *** andymaier has joined #openstack-infra | 13:00 | |
| AJaeger | zigo: I don't know what needs to be done - right now that repo is in broken state. Please talk with clarkb and figure it out. | 13:03 |
| zigo | AJaeger: Will do. | 13:03 |
| *** berendt has quit IRC | 13:04 | |
| *** psachin has quit IRC | 13:04 | |
| AJaeger | fungi, thanks for fixing github - then the gerrit/projects upstream cleanup change is fine... | 13:04 |
| AJaeger | bbl | 13:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove experimental bandit job from anchor https://review.openstack.org/364896 | 13:05 |
| dhellmann | AJaeger : thanks for your help with that zuul-cloner issue earlier. | 13:05 |
| * dhellmann makes a note to pay closer attention to paths when fungi says things to him | 13:05 | |
| *** salv-orlando has joined #openstack-infra | 13:06 | |
| fungi | AJaeger: so usedevelop=False works for me _if_ i don't add skipsdist=True like you did in 364671,2 | 13:07 |
| *** xyang1 has joined #openstack-infra | 13:08 | |
| *** jaosorior has quit IRC | 13:09 | |
| *** tongli has joined #openstack-infra | 13:09 | |
| *** signed8bit has joined #openstack-infra | 13:10 | |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare https://review.openstack.org/364370 | 13:11 |
| *** yuval has joined #openstack-infra | 13:12 | |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare https://review.openstack.org/364370 | 13:13 |
| *** asettle has joined #openstack-infra | 13:13 | |
| *** spzala has joined #openstack-infra | 13:16 | |
| *** mriedem has joined #openstack-infra | 13:16 | |
| *** maishsk_ has joined #openstack-infra | 13:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/364775 | 13:18 |
| *** rossella_s has quit IRC | 13:18 | |
| fungi | AJaeger: from the tox logs, it looks like it never bothered to directly install openstackdocstheme from source in a separate step (perhaps because it saw it already installed by one of the dependencies?) | 13:18 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: NPM DSVM jobs are now voting. https://review.openstack.org/363200 | 13:19 |
| *** rossella_s has joined #openstack-infra | 13:19 | |
| *** asettle has quit IRC | 13:19 | |
| *** maishsk has quit IRC | 13:19 | |
| *** maishsk_ is now known as maishsk | 13:19 | |
| *** piet has quit IRC | 13:19 | |
| skraynev | rcarrillocruz: great! thank you :) | 13:21 |
| *** yuval has quit IRC | 13:21 | |
| *** eharney has joined #openstack-infra | 13:22 | |
| rcarrillocruz | np | 13:23 |
| *** zhurong has quit IRC | 13:23 | |
| fungi | ttx: dhellmann: i've just approved 364417 | 13:23 |
| *** mmedvede has quit IRC | 13:23 | |
| Zara | question-- I want to test my rudimentary storyboard commandline interface against storyboard-dev, but I get 'certificate verify failed' errors. I'm guessing it's because the certificates are self-signed. Is there anything we can do to make it play nicely? | 13:25 |
| openstackgerrit | Pawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade https://review.openstack.org/364809 | 13:25 |
| fungi | i need to step away again for more storm prep, but i hope to still be around to work on our rename maintenance planning (and around for the maintenance itself) | 13:25 |
| *** dstanek has quit IRC | 13:25 | |
| openstackgerrit | Aleksey Zvyagintsev proposed openstack-infra/project-config: murano-pkg-check: add coverage job https://review.openstack.org/364907 | 13:26 |
| fungi | Zara: depends a bit on how your tool is implemented, but there should be ways to make it okay with self-signed certs | 13:26 |
| pabelanger | rcarrillocruz: yes, if we can make sure the hardware in infracloud-chocolate is happy but deploying a cloud, I say we actually redeploy at the midcycle to newton RC1 | 13:26 |
| fungi | Zara: is teh source code public somewhere i can skim? | 13:26 |
| dhellmann | fungi : thanks! | 13:26 |
| *** zz_dimtruck has quit IRC | 13:26 | |
| Zara | fungi: sure, it's pretty much a wrapper for the python client, https://review.openstack.org/#/c/348418/ | 13:26 |
| Zara | well, the python client would be more useful to link | 13:27 |
| fungi | dhellmann: i assume the outer bands of this storm have already reached you--hope you're on high ground! | 13:27 |
| *** madorn has quit IRC | 13:27 | |
| Zara | fungi: https://git.openstack.org/cgit/openstack-infra/python-storyboardclient/tree/ | 13:27 |
| *** wendar has quit IRC | 13:27 | |
| *** kencjohnston has quit IRC | 13:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add merge commit ACL for packaging-deb https://review.openstack.org/364537 | 13:27 |
| fungi | Zara: got it, so storyboardclient is raising an exception? | 13:27 |
| AJaeger | fungi, ah - thanks! New test up... | 13:28 |
| AJaeger | fungi, all the best for the storm! Hope it's not too stormy | 13:28 |
| dhellmann | fungi : I think the worst of it is supposed to pass southeast of us, but we're supposed to see some from the edges. I hope you weather it ok. | 13:28 |
| Zara | fungi: yeah, paste here: http://paste.openstack.org/show/566072/ | 13:28 |
| *** mptacekx has quit IRC | 13:29 | |
| rcarrillocruz | bkero: the gotty thing to share terminals over a browser is superawesome | 13:29 |
| rcarrillocruz | thx for the suggestion | 13:29 |
| mrhillsman | morning/evening - just checking if things in cloud8 are ok | 13:29 |
| AJaeger | fungi, indeed: http://logs.openstack.org/71/364671/3/check/gate-openstackdocstheme-pep8-ubuntu-xenial/c1bbf33/console.html#_2016-09-02_13_20_32_660447 THANKS a lot! | 13:29 |
| dhellmann | fungi : it looks like the leading edge has already reached your area | 13:29 |
| fungi | AJaeger: we've already moved most stuff inside (was up at sunrise carrying deck furniture around), so not much else to worry about unless the waters crest our ground floor in which case i need to bring a bunch of things upstairs | 13:29 |
| *** pgadiya has quit IRC | 13:29 | |
| *** wendar has joined #openstack-infra | 13:29 | |
| *** jcoufal has quit IRC | 13:30 | |
| fungi | already have all the storm shutters closed | 13:30 |
| Zara | :0 | 13:30 |
| AJaeger | fungi, hope it's not that high... | 13:30 |
| fungi | they' | 13:31 |
| fungi | re saying the storm surge could be up to a meter above ground level, but i think that's only likely on the ocean side of the island | 13:31 |
| dhellmann | fungi : what do you do with your car? | 13:31 |
| fungi | unless we end up with winds pushing water into the inlet | 13:31 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Tweak 'about' page https://review.openstack.org/361246 | 13:31 |
| *** kencjohnston has joined #openstack-infra | 13:32 | |
| fungi | dhellmann: throw bikes in the trunk, drive cars to a store parking lot about a mile waay that's on higher ground up on the dunes, then bike home and hang the bikes up | 13:32 |
| dhellmann | fungi : smart | 13:32 |
| * fungi remembers he needs to check tire pressure on the bikes :/ | 13:32 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 13:32 |
| fungi | also, it's only a mile, so easy enough walk if the weather hasn't gotten terrible yet | 13:33 |
| *** zz_dimtruck has joined #openstack-infra | 13:33 | |
| fungi | heck, the entire island is only a mile across at the widest point anyway | 13:33 |
| *** zz_dimtruck is now known as dimtruck | 13:33 | |
| *** madorn has joined #openstack-infra | 13:34 | |
| *** dstanek has joined #openstack-infra | 13:34 | |
| * Zara wondered 'if one side of the island is the ocean side, what's on the other side?' then figured it was an island near the mainland and there was less sea on the other side... | 13:35 | |
| openstackgerrit | Merged openstack-infra/project-config: do not fail if there are no constraint updates to push https://review.openstack.org/364417 | 13:35 |
| fungi | most of our flood risk on this side is likely to be over the weekend after the storm passes once all the water it dumps on the mainland flows down into the sound, if it flows in faster than it can flow out the inlet into the ocean | 13:35 |
| fungi | Zara: barrier island, so it separates a modest sized body of water from the atlantic | 13:37 |
| Zara | ah | 13:38 |
| *** senk has quit IRC | 13:38 | |
| fungi | basically a very narrow strip of land (quite literally a sandbar) that stretches for a couple hundred miles | 13:38 |
| openstackgerrit | Doug Hellmann proposed openstack/os-client-config: Update reno for stable/newton https://review.openstack.org/364920 | 13:38 |
| fungi | with a few inlets at various points | 13:38 |
| mordred | Zara: https://goo.gl/maps/5cteQ7tw89z | 13:39 |
| fungi | yep, that | 13:39 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/elastic-recheck: Add query for bug 1548652 https://review.openstack.org/364921 | 13:39 |
| openstack | bug 1548652 in VirtualBMC "calls to vbmc via ipmitool sometimes time out" [High,Confirmed] https://launchpad.net/bugs/1548652 | 13:39 |
| fungi | i'm there at the end of teh albemarle sound where it says "kill devil hills" | 13:39 |
| Zara | :0 that sounds awesome | 13:40 |
| Zara | well, except for the storms | 13:40 |
| *** jcoufal has joined #openstack-infra | 13:40 | |
| fungi | you get used to them | 13:40 |
| fungi | i'm less worried about this one since it's not coming at us directly off the atlantic | 13:40 |
| *** jcoufal has quit IRC | 13:41 | |
| *** links has quit IRC | 13:41 | |
| fungi | actually storm weekends can be kinda nice since they call for mandatory tourist evacuations on most of the islands here ;) | 13:42 |
| Zara | hahaha | 13:42 |
| *** jcoufal has joined #openstack-infra | 13:42 | |
| *** Na3iL has quit IRC | 13:43 | |
| fungi | especially some of the islands to the south of us reachable only by boat, because the ferries have to shut down and they don't want to have a ton of tourists stranded there becoming a nuisance | 13:43 |
| Zara | so a good time for a swim? | 13:43 |
| fungi | can be sometimes, but usually you don't want to wander out into the ocean for a while after one of these passes through | 13:44 |
| fungi | debris, jellyfish, possibly lingering dangerous currents | 13:45 |
| fungi | great for shell hunting though | 13:45 |
| *** akshai has joined #openstack-infra | 13:45 | |
| *** deadnull has joined #openstack-infra | 13:46 | |
| *** kevinbenton has quit IRC | 13:47 | |
| *** hsammic has joined #openstack-infra | 13:47 | |
| Zara | gah, now I want to live on the beach | 13:48 |
| *** hongbin has joined #openstack-infra | 13:48 | |
| fungi | Zara: so looking at your traceback, there are a few options. complicated a bit by the old oslo-incubator code in storyboardclient though | 13:48 |
| Zara | yeah, we're looking to remove that anyway in the future | 13:49 |
| lucasagomes | hi all, if you have time minda taking a quick look at https://review.openstack.org/#/c/362867/ ? It just changes a wildcard in the function that copies the logs in the Ironic jobs to also include the logs from the deployment ramdisk | 13:49 |
| lucasagomes | thank you | 13:49 |
| fungi | i need to look a little closer at the _get_all() method in there to see how hard it would be to pass extra requests parameters into openstack.common.apiclient.client | 13:50 |
| Zara | (got to the 'make a story' step https://storyboard.openstack.org/#!/story/2000707) | 13:50 |
| *** kevinbenton has joined #openstack-infra | 13:50 | |
| fungi | Zara: so anyway, the "nice" solution to this would be to add a parameter passed through down to the requests.get() call to add a cert parameter, then plumb that up through into your cli as an option like --cert-path | 13:51 |
| *** mmedvede has joined #openstack-infra | 13:52 | |
| *** matbu is now known as matbu|brb | 13:52 | |
| fungi | optionally (or additionally) make it possible to do a verify=False in the requests.get() call with a sufficiently scary sounding option name like --insecure | 13:53 |
| fungi | however a workaround would be to set the CA_BUNDLE envvar in the calling environment to a file with a copy of the server cert (i don't recall the exact dance steps there) | 13:54 |
| *** gouthamr_ has quit IRC | 13:54 | |
| *** zhurong has joined #openstack-infra | 13:54 | |
| *** cardeois has joined #openstack-infra | 13:54 | |
| *** e0ne has quit IRC | 13:55 | |
| *** bhunter71 has joined #openstack-infra | 13:55 | |
| *** e0ne has joined #openstack-infra | 13:55 | |
| *** gouthamr has joined #openstack-infra | 13:55 | |
| fungi | oh, ignore what i said about the cert option to requests.get(), that's for specifying a client certificate. the verify option does double-duty either to disable verification or to specify the path to the trusted cert bundle | 13:56 |
| bhunter71 | a question; in gerrit, can you add a user (email address) to a group _before this user has ever logged in_? I've historically required that users log in one time before being granted extra privs, | 13:56 |
| fungi | bhunter71: depends on how you create users in gerrit. if you're talking about review.openstack.org, new users are created on initial login and don't exist before then | 13:57 |
| Zara | fungi: okay, thank you! :) I'll have a poke around. | 13:57 |
| bhunter71 | fungi: I'm not particularly talking about review.openstack.org, but my gerrit acts the same as you describe. | 13:58 |
| fungi | bhunter71: yeah, if you're using automatic account creation with openid authentication then i don't know of any way to pre-add them to a group | 13:58 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart https://review.openstack.org/364874 | 13:59 |
| bhunter71 | fungi; is there a way you know of to pre-create the account , yet use openid auth? | 13:59 |
| *** senk has joined #openstack-infra | 13:59 | |
| *** Julien-zte has joined #openstack-infra | 13:59 | |
| bhunter71 | assuming I know all the details - name, email, etc. | 14:00 |
| *** ccarmack has joined #openstack-infra | 14:00 | |
| rcarrillocruz | bhunter71: i'm afraid not | 14:00 |
| fungi | bhunter71: only if you also know what their openid url will be... then you could precreate the account and add the openid as an account_external_ids row | 14:00 |
| rcarrillocruz | gerrit does not have a bootstrap mechanism | 14:00 |
| rcarrillocruz | you would need to inject it to the DB | 14:00 |
| rcarrillocruz | as fungi said | 14:00 |
| rcarrillocruz | there was a project around that had that code | 14:00 |
| rcarrillocruz | let me search | 14:00 |
| fungi | bhunter71: the way openid auto account creation works, if the user logs in with an openid url gerrit hasn't previously seen, gerrit will create a completely new account for it (even if there is already an existing account with the same name, e-mail addresses, et cetera) | 14:01 |
| rcarrillocruz | bhunter71: https://github.com/forj-oss/redstone/blob/master/puppet/modules/gerrit_config/manifests/createfirstaccount.pp | 14:01 |
| fungi | for similar reasons, we end up with duplicate accounts somewhat frequently because users do things that cause their openid url to change | 14:01 |
| bhunter71 | fungi/rcarrillocruz. Thanks so much, this is very helpful! | 14:02 |
| andreykurilin | fungi: thanks. will ask him | 14:02 |
| *** mat128 is now known as mat128|away | 14:02 | |
| rcarrillocruz | np | 14:03 |
| fungi | bhunter71: so in short, _if_ you control the openid service you're using such that you can predict a user's openid, then you _could_ add it to gerrit in advance | 14:03 |
| *** tesseract- has quit IRC | 14:03 | |
| fungi | bhunter71: or put a different way, if you want to manually associate a known openid with an account, you can simply add it to the account_external_ids table in the database before they log in for the first time | 14:04 |
| bhunter71 | you guys are the freaking best. I <3 #openstack-infra | 14:04 |
| fungi | or you can use an update query later to move that openid to a different account_id | 14:04 |
| bhunter71 | thanks fungi | 14:04 |
| fungi | yw | 14:05 |
| anteaya | fungi: did the release team get everything finished that they need to get finished? | 14:05 |
| *** bin_ has joined #openstack-infra | 14:05 | |
| *** kaisers_ has quit IRC | 14:05 | |
| anteaya | or have we checked with them yet? | 14:06 |
| Zara | fungi: thanks, that worked \o/ | 14:06 |
| fungi | anteaya: yeah, as you've discovered, it's release meeting time right now | 14:07 |
| anteaya | fungi: thanks | 14:07 |
| fungi | np | 14:07 |
| *** vhosakot has joined #openstack-infra | 14:08 | |
| *** jamesdenton has joined #openstack-infra | 14:09 | |
| *** mtanino has joined #openstack-infra | 14:10 | |
| zaro | morning | 14:12 |
| *** tesseract- has joined #openstack-infra | 14:13 | |
| anteaya | morning zaro | 14:14 |
| *** deadnull_ has joined #openstack-infra | 14:14 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Fix ironic-inspector-client tempest job to actually use the client from git https://review.openstack.org/365001 | 14:16 |
| *** rbrndt has joined #openstack-infra | 14:16 | |
| *** javeriak has joined #openstack-infra | 14:16 | |
| *** deadnull has quit IRC | 14:18 | |
| *** Na3iL has joined #openstack-infra | 14:19 | |
| *** kaisers_ has joined #openstack-infra | 14:19 | |
| *** tkelsey has joined #openstack-infra | 14:22 | |
| *** Julien-zte has quit IRC | 14:22 | |
| *** matbu|brb is now known as matbu | 14:23 | |
| *** Julien-zte has joined #openstack-infra | 14:24 | |
| *** fguillot has joined #openstack-infra | 14:25 | |
| *** asettle has joined #openstack-infra | 14:25 | |
| *** annegentle has joined #openstack-infra | 14:26 | |
| *** tkelsey has quit IRC | 14:26 | |
| *** matt-borland has joined #openstack-infra | 14:29 | |
| *** senk has quit IRC | 14:29 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 14:30 |
| *** martinkopec has quit IRC | 14:31 | |
| *** andymaier has quit IRC | 14:33 | |
| *** _nadya_ has quit IRC | 14:34 | |
| pabelanger | rcarrillocruz: are any of the 3 held nodes in nodepool from debugging failures from yesterday? If so, can they be deleted? | 14:34 |
| rcarrillocruz | they can be deleted, yeah | 14:35 |
| pabelanger | cool | 14:35 |
| pabelanger | will do that now | 14:35 |
| *** esberglu has joined #openstack-infra | 14:35 | |
| *** sputnik13_ has joined #openstack-infra | 14:35 | |
| rcarrillocruz | i held one for checking the location of the cache/files/cirros-blah | 14:35 |
| *** inc0 has joined #openstack-infra | 14:35 | |
| inc0 | howdy | 14:35 |
| inc0 | pabelanger, did you try to start gatejobs on our little cloud? | 14:36 |
| *** javeriak_ has joined #openstack-infra | 14:36 | |
| krotscheck | Oh gerrit, how I love thee, let me count the days. | 14:36 |
| krotscheck | ...of waiting. | 14:36 |
| *** e0ne has quit IRC | 14:37 | |
| *** javeriak has quit IRC | 14:37 | |
| inc0 | krotscheck, gertty helps. | 14:37 |
| pabelanger | inc0: not yet. I think we are waiting for releases to be finished for the day | 14:37 |
| *** e0ne has joined #openstack-infra | 14:37 | |
| krotscheck | inc0: tty is for crusty old ops people. | 14:37 |
| krotscheck | ;) | 14:37 |
| inc0 | ok, fair enough, let me know if you need any help | 14:38 |
| inc0 | krotscheck, crusty old ops people know what works. | 14:38 |
| *** eharney has quit IRC | 14:38 | |
| krotscheck | inc0: Yes, but it takes more than one lick to get to the toffee center. | 14:38 |
| *** martinkopec has joined #openstack-infra | 14:38 | |
| * krotscheck will leave you with that mental image. | 14:39 | |
| anteaya | fungi: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 14:39 |
| fungi | thanks for the url reminder, anteaya! | 14:39 |
| anteaya | welcome | 14:39 |
| fungi | also i haven't checked yet, but someone posted a late entry for renames to the infra ml | 14:40 |
| anteaya | oh, I will look | 14:40 |
| fungi | if we get a chance to squeeze that one into the mix | 14:40 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 14:40 |
| anteaya | also gerrit is slow if that isn't already a known thing | 14:40 |
| inc0 | ok...Smaug...we (Kolla) competed with them for Koala mascot. Because apperantly they were renaming and new name was something connected to Koala...so I ask you...how the hell Karbor is connected in any shape or form to Koalas? | 14:40 |
| anteaya | do we have to swear? | 14:41 |
| *** matbu is now known as matbu|mtg | 14:41 | |
| anteaya | can we not just ask a question? | 14:41 |
| fungi | backwards it's "robrak" | 14:42 |
| *** jcoufal has quit IRC | 14:42 | |
| fungi | still no connection | 14:42 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [Rally] Add new experimental jobs https://review.openstack.org/364836 | 14:42 |
| inc0 | swearing is a form of empowering emotion, it is a poetic tool in our disposals. | 14:42 |
| inc0 | let's use it proudly | 14:42 |
| anteaya | fungi: did you have a chance to chat with tonyb at all about the requirements rename patch? https://review.openstack.org/#/c/353371 | 14:42 |
| *** jcoufal has joined #openstack-infra | 14:42 | |
| anteaya | inc0: really, that is an interesting stance | 14:43 |
| * fungi would prefer to keep discussions here less "emotionally empowered" as well ;) | 14:43 | |
| krotscheck | I have a +2A patch with a finally-merged depends-on thats not getting into the gare. Can someone toggle the +A for me? https://review.openstack.org/#/c/356609/ | 14:43 |
| krotscheck | *gate | 14:43 |
| *** kushal has quit IRC | 14:43 | |
| fungi | krotscheck: done | 14:43 |
| krotscheck | fungi: thanks | 14:44 |
| openstackgerrit | Merged openstack-infra/project-config: wait to publish releases.o.o until after tagging https://review.openstack.org/363979 | 14:44 |
| fungi | dhellmann: ttx: ^ merged | 14:44 |
| bauwser | hey -infra, scrolling backwards doesn't show me people reporting having Gerrit issues, but I just got a 502 after 20mins of high latency | 14:45 |
| anteaya | bauwser: thank you, gerrt is slow for me as well | 14:45 |
| anteaya | I don't think any of us have gotten around to looking at cacti.openstack.org yet, you are welcome to if you wish | 14:46 |
| inc0 | anteaya, look, there is also a greek word for it! http://literarydevices.net/solecism/ and if there is ancient greek word for something it has to be wise and true | 14:46 |
| anteaya | inc0: do you really want to waste my time with this? | 14:46 |
| anteaya | you can't see I am busy? | 14:46 |
| anteaya | how's that for emotionally empowered? | 14:47 |
| *** esikachev has joined #openstack-infra | 14:47 | |
| *** links has joined #openstack-infra | 14:47 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update infracloud documentation to remove old geo information https://review.openstack.org/365018 | 14:47 |
| HeOS | Hello, infra-team! If this request has +2, could you please merge it into repository? https://review.openstack.org/#/c/362002/. | 14:48 |
| inc0 | anteaya, well, it's your time;) anyway good point, I have handful of reviews to make, ah glorious rc season | 14:48 |
| HeOS | I'll really appreciate your help. | 14:48 |
| anteaya | infra-root, can someone check javamelody for the gc rate please? cpu usage appears to be up: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 14:48 |
| *** e0ne has quit IRC | 14:49 | |
| *** Julien-zte has quit IRC | 14:51 | |
| anteaya | fungi: let me know once you have decided your rename workflow of choice | 14:52 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 14:52 |
| jeblair | fungi, anteaya: gerrit gc time is high | 14:53 |
| anteaya | jeblair: thank you | 14:53 |
| fungi | yep, i just started pulling that up | 14:53 |
| jeblair | started maybe 30 mins ago and is continuing | 14:53 |
| fungi | also show-caches --show-threads reports we're around 66-67 httpd threads (we set our max to 100) so not responsible for the 5xx errors | 14:54 |
| fungi | i saw a few very small gc time spikes late yesterday but was hoping it would hold off until the maintenance | 14:54 |
| fungi | looks like we made it about a week since the last restart | 14:55 |
| jeblair | http://imgur.com/a/TCnqN | 14:55 |
| fungi | last restarted thursday, august 25 | 14:56 |
| *** kushal has joined #openstack-infra | 14:56 | |
| jeblair | fungi: yeah, *maybe* the spike around 10:00 was a predictor? <shrug> | 14:56 |
| *** piet has joined #openstack-infra | 14:56 | |
| fungi | maybe... | 14:56 |
| irtermite | @all where do I find the schedule of infra jobs that run? I ask, as there is a request to upgrade the switches the Rackspace RPC OSIC cluster is running on. There is a request for Central Time next week Wednesday 00:00 – 06:00 and Thursday 00:00 – 06:00 | 14:56 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 14:56 |
| irtermite | How impactful would this be to your efforts? | 14:56 |
| irtermite | cloudnull: ^^ for awareness | 14:57 |
| cloudnull | cool, Nexus upgrade? | 14:57 |
| cloudnull | better RA support hopfully | 14:57 |
| fungi | irtermite: those times should be relatively quiet for us. especially since the feature freeze rush ends this week | 14:57 |
| openstackgerrit | Merged openstack-infra/project-config: Added npm-docs draft build for eslint-config-openstack. https://review.openstack.org/356609 | 14:58 |
| irtermite | thanks fungi... how did the weather go? | 14:58 |
| fungi | we could preemptively dial our quota there down to 0 just to make things nicer | 14:58 |
| *** rbrndt has quit IRC | 14:58 | |
| anteaya | irtermite: have you seen http://status.openstack.org/zuul/ before? if you look at the bottom you can see the graphs of the job rates | 14:58 |
| fungi | irtermite: the bulk of the storm hasn't reached us yet, and the flood risk is overnight tonight through to tomorrow night | 14:58 |
| dhellmann | fungi : ack, thanks | 14:58 |
| *** maishsk has quit IRC | 14:59 | |
| irtermite | cloudnull: yes... I just CCed you on the email. Fixes: Unexpected switch reloads, VPC and port-channel disruptions, IPv6 HSRP issue. | 14:59 |
| *** mikelk has quit IRC | 14:59 | |
| *** sputnik13_ has quit IRC | 14:59 | |
| *** kzaitsev_mb has quit IRC | 14:59 | |
| anteaya | irtermite: this is the configuration file of all the jobs that can be called upon to run: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml | 14:59 |
| cloudnull | nice! | 14:59 |
| anteaya | but that doesn't give you information on when they are run | 14:59 |
| anteaya | fungi: do you have to move furniture today? | 15:00 |
| fungi | anteaya: i got up at sunrise to do that. already taken care of | 15:00 |
| irtermite | oh no... best of luck to you fungi! | 15:00 |
| AJaeger | project-config cores, could you review a change to add releasenotes for magnum, please? https://review.openstack.org/364782 | 15:00 |
| fungi | irtermite: thanks, this doesn't look like it will be a major impact for us since the eye is coming at us over the mainland rather than over the water | 15:01 |
| anteaya | fungi: awesome, glad you are all set | 15:01 |
| irtermite | thank you anteaya | 15:01 |
| anteaya | irtermite: you're welcome | 15:01 |
| *** rcernin has quit IRC | 15:01 | |
| *** maishsk has joined #openstack-infra | 15:02 | |
| irtermite | well keep us updated fungi! | 15:02 |
| fungi | jeblair: i'm considering restarting gerrit... there are things on the cusp of merging though | 15:02 |
| *** jcoufal has quit IRC | 15:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Use list_servers for polling rather than get_server_by_id https://review.openstack.org/365021 | 15:03 |
| anteaya | +1 restart gerrit | 15:03 |
| *** links has quit IRC | 15:03 | |
| mordred | Shrews, jeblair: if'n you don't mind ^^ I noticed the GetServer poll in the nodepool logs this morning | 15:03 |
| irtermite | how would you prefer to proceed next week? drop quota to 0 or just roll with it and see? We are doing reboots of odd numbers devices first then even after. So... node access will still be up. Honestly, I don't foresee that being a problem with the current quota at all. | 15:04 |
| anteaya | AJaeger: +2 | 15:04 |
| fungi | irtermite: oh, then i'd just leave things running | 15:04 |
| irtermite | cloudnull: +2? | 15:04 |
| cloudnull | I have no +2 | 15:04 |
| fungi | irtermite: generally if our ci loses access to a node it just reschedules the job running there and deletes the old node | 15:04 |
| irtermite | you do to this decision cloudnull ;) | 15:05 |
| irtermite | you hold the keys, if you willl | 15:05 |
| *** Ravikiran_K has quit IRC | 15:05 | |
| irtermite | fungi: perfect | 15:05 |
| cloudnull | how long will the upgrade take ? | 15:05 |
| fungi | dhellmann: ttx: dims: we're looking at a quick gerrit restart to relieve some resource pressure... are you at a lull on release approvals for the moment? | 15:05 |
| irtermite | The plan is (Odds Wednesday 00:00 – 06:00 and Evens Thursday 00:00 – 06:00) | 15:05 |
| *** weshay is now known as weshay_mtg | 15:06 | |
| fungi | odds bodkins | 15:06 |
| *** salv-orlando has quit IRC | 15:06 | |
| *** e0ne has joined #openstack-infra | 15:06 | |
| cloudnull | do we really think it'll take 6 hours? | 15:06 |
| anteaya | hongbin: we are renaming higgins to zun, do you still want the gerrit groups to be called higgens-core and higgens-release? or do you want to update https://review.openstack.org/#/c/329247/6/gerrit/acls/openstack/zun.config | 15:06 |
| irtermite | pft, doubtful | 15:06 |
| dims | fungi : nothing is ready to push a release for. so sure | 15:06 |
| anteaya | dims: thank you | 15:07 |
| cloudnull | personally, I'd say Bill O’Reilly it and "DO IT LIVE!" | 15:07 |
| hongbin | anteaya: ack, will update that | 15:07 |
| anteaya | hongbin: thank you | 15:07 |
| *** jordanP has joined #openstack-infra | 15:07 | |
| anteaya | hongbin: also gerrit is about to be restarted, so wait a minute or two | 15:07 |
| fungi | i'm waiting for one tempest job to complete so zuul can report a couple of merges into gerrit and then we should be at a safe window for a quick gerrit restart | 15:07 |
| anteaya | fungi: ah okay, thanks | 15:07 |
| hongbin | ok | 15:07 |
| dhellmann | fungi : a quick restart is ok with me | 15:08 |
| dhellmann | fungi : I'm still reviewing branch results | 15:08 |
| irtermite | ACK cloudnull and fungi moving forward live this wed and thur | 15:08 |
| irtermite | on it | 15:08 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config: Add zaqar to scenario002 for tripleo ci https://review.openstack.org/365023 | 15:08 |
| *** maishsk has quit IRC | 15:09 | |
| *** kushal has quit IRC | 15:10 | |
| cloudnull | we have other users on this cloud so if the impact is misserable it'll be felt everwhere and there's really no getting around it. we also have the controll plain split out in different cabs so "in theory" as you take devices down and upgrade them the cloud "should" stay up even w/ degrated connectivity. | 15:10 |
| pleia2 | fungi: o/ what do we have for prep re: renames? | 15:10 |
| pleia2 | (whenever you have a moment, no rush) | 15:10 |
| *** dprince has quit IRC | 15:10 | |
| anteaya | pleia2: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 15:10 |
| cloudnull | so yes, irtermite "Do it LIVE!" | 15:10 |
| anteaya | pleia2: and good morning | 15:10 |
| pleia2 | anteaya: thanks | 15:10 |
| anteaya | welcome | 15:10 |
| fungi | okay, zuul's processing these merges now | 15:11 |
| anteaya | pleia2: currently I'm just reviewing the patches | 15:11 |
| pleia2 | anteaya: nods, I'll do the same | 15:11 |
| anteaya | thanks | 15:11 |
| anteaya | pleia2: 3 rename patches and an oopsie fix | 15:11 |
| * pleia2 nods | 15:11 | |
| pleia2 | I recall the oppsie x_x | 15:12 |
| *** matbu|mtg is now known as matbu | 15:12 | |
| anteaya | yeah :( | 15:12 |
| *** martinkopec has quit IRC | 15:13 | |
| * fungi learned his lesson. no restarting puppet early even if it looks like the timing will work out. because sometimes... it doesn't :/ | 15:13 | |
| *** javeriak_ has quit IRC | 15:13 | |
| anteaya | we all have our lessons to learn | 15:13 |
| anteaya | some of them are hard | 15:14 |
| fungi | luckily the correction on that one is to just delete a couple directories while gerrit is offline | 15:14 |
| anteaya | awesome I wondered what the fix was | 15:14 |
| anteaya | are you going to put it in the etherpad at some point so we can see it crossed off? | 15:15 |
| fungi | sure | 15:15 |
| *** yaume has quit IRC | 15:15 | |
| anteaya | thanks | 15:15 |
| *** pcaruana has quit IRC | 15:15 | |
| anteaya | history and all | 15:15 |
| *** apetrich has joined #openstack-infra | 15:15 | |
| *** bnemec is now known as beekneemech | 15:16 | |
| *** caowei has joined #openstack-infra | 15:16 | |
| *** salv-orlando has joined #openstack-infra | 15:16 | |
| *** rkukura has quit IRC | 15:16 | |
| *** sdake has joined #openstack-infra | 15:16 | |
| *** thcipriani|afk is now known as thcipriani | 15:17 | |
| fungi | i think the gerrit slowness is causing zuul to take even longer to handle the fallout from these merges, so keeps getting new merge events piling up | 15:18 |
| pabelanger | looks like we are having some issues deleting nodes in ovh | 15:18 |
| *** matrohon has quit IRC | 15:18 | |
| pabelanger | seems limited to bhs1 | 15:18 |
| fungi | pabelanger: yeah, from the graph it seems like that just started | 15:18 |
| pabelanger | fungi: Ya, checking status.ovh.com now | 15:19 |
| odyssey4me | project-config cores - I'd appreciate a second review on https://review.openstack.org/364286 asap | 15:19 |
| *** dansmith is now known as superdan | 15:19 | |
| fungi | luckily the correction on that one is to just delete a couple directories while gerrit is offline | 15:20 |
| fungi | er, wrong line from my client history | 15:20 |
| anteaya | well if we are having our cascading failures _after_ feature freeze I still count that as the best feature freeze week ever | 15:20 |
| fungi | #status notice The Gerrit service on review.openstack.org is restarting quickly to relieve resource pressure and restore normal performance | 15:20 |
| openstackstatus | fungi: sending notice | 15:20 |
| anteaya | fungi: ha ha ha | 15:20 |
| anteaya | fungi the reruns | 15:20 |
| anteaya | fungi's greatest hits | 15:20 |
| *** esikachev has quit IRC | 15:21 | |
| *** zhurong has quit IRC | 15:21 | |
| pleia2 | lol | 15:21 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting quickly to relieve resource pressure and restore normal performance | 15:21 | |
| fungi | it's on its way back up now | 15:21 |
| anteaya | pleia2: I'm picture fungi as The Beaver talking with Wally and eating cookies | 15:22 |
| pleia2 | anteaya: hahaha | 15:22 |
| EmilienM | thx for the restart, it looks better now | 15:22 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 15:23 |
| fungi | EmilienM: sure, sorry for the delay... wanted to find a moment when zuul wasn't in the middle of trying to merge a change | 15:23 |
| openstackstatus | fungi: finished sending notice | 15:23 |
| *** kushal has joined #openstack-infra | 15:24 | |
| fungi | anteaya: tonyb talked through the requirements rename bits with AJaeger earlier, so should be all set | 15:25 |
| fungi | (just catching back up on conversation from before the gerrit restart) | 15:25 |
| fungi | gonna step away from a couple minutes now that nothing's on fire... will be back shortly | 15:25 |
| *** esikachev has joined #openstack-infra | 15:26 | |
| *** spzala has quit IRC | 15:26 | |
| anteaya | fungi: awesome thank you | 15:26 |
| anteaya | fungi: thanks for the restart | 15:26 |
| odyssey4me | thanks anteaya | 15:27 |
| anteaya | odyssey4me: welcome | 15:27 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Zun: rename higgins to zun (2) https://review.openstack.org/329247 | 15:28 |
| anteaya | hongbin: thank you | 15:29 |
| *** spzala has joined #openstack-infra | 15:29 | |
| *** spzala has quit IRC | 15:29 | |
| *** sputnik13_ has joined #openstack-infra | 15:29 | |
| hongbin | np | 15:29 |
| zigo | pabelanger: http://logs.openstack.org/98/364998/1/check/gate-deb-python-os-client-config-dpkg-debian-jessie/f5951fa/console.html <--- Yet another instance of httpredir.d.o mirror not working. Do you think I could work on doing a "proper" Debian mirroring thing? Where would that be? | 15:30 |
| anteaya | pleia2: can you check and make sure I have the gerrit group info correct on the etherpad, please? | 15:30 |
| pleia2 | anteaya: sure thing | 15:31 |
| anteaya | pleia2: thank you | 15:31 |
| pabelanger | zigo: aside from unsigned packages in our AFS mirror, what is the issue with using them today? | 15:31 |
| zigo | pabelanger: Well, the unsigned thing prevents using it during the sbuild-create. | 15:31 |
| pabelanger | zigo: and no way to work around that? | 15:31 |
| zigo | sbuild-createchroot, that is. | 15:32 |
| zigo | pabelanger: None that I know at least. | 15:32 |
| pabelanger | ok | 15:32 |
| *** esikache1 has joined #openstack-infra | 15:33 | |
| pabelanger | zigo: it is just a time issue right now, I need to spend some time on the apt-ftp script you linked before | 15:33 |
| zigo | pabelanger: I was asking here if there was a way so I could do it... | 15:33 |
| zigo | pabelanger: I'm sure you're super busy! :) | 15:33 |
| *** mrjk has joined #openstack-infra | 15:34 | |
| *** jcoufal has joined #openstack-infra | 15:34 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added documentation publish jobs for the JavaScript SDK https://review.openstack.org/346131 | 15:35 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: npm-docs job is now voting https://review.openstack.org/365033 | 15:35 |
| pleia2 | anteaya: thumbs up | 15:35 |
| pabelanger | zigo: you could write the patch, but we'd need an infra-root to run / watch the script. We'd need to update our mirror_update.pp manifest to use the rsync process, in 2 stages. I think the script you linked from debian will do that | 15:35 |
| *** esikachev has quit IRC | 15:36 | |
| *** kaisers_ has quit IRC | 15:36 | |
| zigo | pabelanger: Where should I write such patch? Any path I should look into in project-config? | 15:36 |
| zigo | Or something new? | 15:37 |
| mordred | the biggest 'feature' we may need to write | 15:37 |
| mordred | is that we do _not_ want to delete files when rsync thinks they shoudl be deleted | 15:37 |
| mordred | we want to register those files $somewhere | 15:37 |
| mordred | and delete them on a 2-3 hour delay | 15:37 |
| pabelanger | Yup | 15:37 |
| mordred | so every time we run the rsync, we need to collect the list of things that rsync wants to delete | 15:37 |
| zigo | mordred: That's already what the mirror thing does. | 15:37 |
| mordred | awesome | 15:37 |
| mordred | as long as it does that, then we're set | 15:38 |
| pabelanger | I think ftpsync does this, I just haven't run it | 15:38 |
| zigo | In fact, it's the master FTP that does that. | 15:38 |
| zigo | It first pushes the update. | 15:38 |
| zigo | Then updates the indexes. | 15:38 |
| zigo | Then after 2 or 3 dak run only, it deletes the old stuff. | 15:38 |
| mordred | sweet | 15:38 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting multi-region functional test to OSA os_swift https://review.openstack.org/364286 | 15:38 |
| *** esikache1 has quit IRC | 15:38 | |
| *** esikachev has joined #openstack-infra | 15:39 | |
| zigo | If we use InRelease (I'm not sure Jessie does, but Xenial should be), then it's completely atomic. | 15:39 |
| zigo | The Release.gpg stuff broke the atomicity. | 15:39 |
| dhellmann | fungi , anteaya : I've reviewed the logs for the branches I created, and they all look good. It should be fine to go ahead with the gerrit maintenance later today. | 15:39 |
| zigo | Only bad FTP mirrors not using the correct ftp-sync scripts have issues. | 15:40 |
| anteaya | dhellmann: thank you! | 15:40 |
| zigo | Unfortunately, many mirrors are like that. :( | 15:40 |
| anteaya | pleia2: ^^ | 15:40 |
| anteaya | AJaeger: are you around today for the rename? | 15:40 |
| persia | For Debian, ftpsync is likely the best. For Ubuntu, ubumirror may be preferable. | 15:40 |
| anteaya | pleia2: and thanks for checking the gerrit group change info | 15:41 |
| pleia2 | dhellmann: thanks! | 15:41 |
| *** martinkopec has joined #openstack-infra | 15:42 | |
| *** sputnik13_ has quit IRC | 15:42 | |
| *** andymaier has joined #openstack-infra | 15:42 | |
| anteaya | pleia2: so right now I'm waiting for fungi to tell me if he wants to use the documented new workflow using ansible playbooks, or the old manual way | 15:42 |
| *** tesseract- has quit IRC | 15:42 | |
| anteaya | pleia2: plus the online reindex will be new as well | 15:42 |
| pleia2 | anteaya: yeah, I was going to ask about that next | 15:42 |
| * pleia2 nods | 15:42 | |
| anteaya | zaro: how did the online reindex test go? | 15:42 |
| pleia2 | I do have to hop out at 19:30 for a lunch meeting I nearly forgot about | 15:43 |
| anteaya | zaro: does it look good to use it for today? | 15:43 |
| pleia2 | but hopefully we'll know one way or the other re: online/offline, and will be either waiting or doing by then | 15:43 |
| anteaya | pleia2: understood, helping with prep is wonderful, thank you | 15:43 |
| anteaya | pleia2: ack | 15:43 |
| anteaya | pleia2: we might be done by the time you have to go | 15:43 |
| zaro | anteaya: the test went great. gerrit was slow during the index but nothing failed. | 15:43 |
| pleia2 | hope so! | 15:43 |
| pleia2 | I am going out for Ramen with one of the women who helped with book editing :d | 15:44 |
| anteaya | zaro: awesome thank you | 15:44 |
| pleia2 | book comes out next week :D | 15:44 |
| zaro | so yeah, i think it's go to go | 15:44 |
| anteaya | pleia2: oh that sounds wonderful | 15:44 |
| *** pelix has joined #openstack-infra | 15:44 | |
| anteaya | pleia2: yay book \o/ | 15:44 |
| AJaeger | anteaya: no, won't be around - I'm not feeling too well, and will leave soon | 15:44 |
| *** xarses has quit IRC | 15:44 | |
| pleia2 | pretty exciting, I hope it doesn't suck | 15:44 |
| pleia2 | haha | 15:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote os_nova/lxd and os_neutron/ovs jobs to voting https://review.openstack.org/365035 | 15:44 |
| anteaya | pleia2: I of course want an autographed copy | 15:44 |
| pleia2 | anteaya: of course :) | 15:44 |
| anteaya | AJaeger: oh my, take care of yourself | 15:44 |
| anteaya | AJaeger: I hope you start feeling better soon | 15:45 |
| anteaya | pleia2: thank you :) | 15:45 |
| anteaya | zaro: wonderful, I hope it works | 15:45 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote os_nova/lxd and os_neutron/ovs jobs to voting https://review.openstack.org/365035 | 15:46 |
| mordred | persia: what's the difference? | 15:46 |
| clarkb | good morning | 15:47 |
| *** electrofelix has quit IRC | 15:47 | |
| anteaya | morning clarkb | 15:48 |
| AJaeger | thanks, anteaya | 15:48 |
| anteaya | clarkb: if you care, renames are a go and the etherpad is here: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 15:48 |
| anteaya | AJaeger: :) | 15:48 |
| anteaya | AJaeger: hope you have an enjoyable weekend | 15:48 |
| anteaya | clarkb: waiting on fungi to say what workflow he wants to use, he'll be back in a bit | 15:49 |
| pabelanger | zigo: mordred: running ftpsync locally to see how it works. | 15:50 |
| * AJaeger hopes that the rename works just fine | 15:50 | |
| pleia2 | thanks AJaeger, feel better | 15:50 |
| *** martinkopec has quit IRC | 15:51 | |
| AJaeger | thanks , pleia2 | 15:51 |
| persia | mordred: Each is the official mirroring script for the relevant distribution, and so they invoke rsync subtly differently, encode different metadata about the distro in source, etc. | 15:51 |
| mordred | persia: lovely | 15:52 |
| persia | But it's been a few years since I was a mirror operator, so I don't remember the details | 15:52 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 15:52 |
| AJaeger | anteaya: Do you want to stack the rename changes on top of each other? | 15:52 |
| *** eharney has joined #openstack-infra | 15:52 | |
| * AJaeger proposes to soft-freeze project-config so that we don't have any merge conflicts | 15:52 | |
| mordred | pabelanger: I'm hoping that the on-disk results of ftp-sync isn't massively different than our current reprepro output dir ... I mena, I'd expect the pool to be largely the same I hope? | 15:52 |
| AJaeger | So, let's only merge critical changes for the release right now... | 15:52 |
| anteaya | AJaeger: I'm just going to wait to hear back from fungi before I do anything else yet | 15:53 |
| anteaya | AJaeger: I don't know as there are any release critical patches outstanding | 15:53 |
| AJaeger | anteaya: We merged everything release critical that I was aware off. | 15:53 |
| anteaya | we got the green light from dhellmann so I take that to mean the release team is clear and isn't waiting on anything | 15:53 |
| pabelanger | mordred: Ya, I'm looking to see if we can only mirror jessie. But expect everything but indexes to change | 15:54 |
| AJaeger | dhellmann, ttx, dims, are there still any changes you like to have merged? Otherwise I propose to soft-freeze project-config | 15:54 |
| anteaya | AJaeger: yeah, so I'm on board with project-config patch freeze | 15:54 |
| pabelanger | mordred: err, nothing to change but indexes | 15:54 |
| anteaya | I won't merge anything else | 15:54 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins https://review.openstack.org/319754 | 15:54 |
| anteaya | I'll let fungi inform the others | 15:54 |
| dims | AJaeger : nothing from me | 15:55 |
| persia | pabelanger: maybe the stamp files will change? (not that this affects clients) | 15:55 |
| anteaya | AJaeger: realease team already said we are good to go | 15:55 |
| *** senk has joined #openstack-infra | 15:55 | |
| anteaya | we don't have to keep asking them | 15:55 |
| AJaeger | ah.. | 15:55 |
| mordred | pabelanger: oh. phew | 15:55 |
| mordred | pabelanger: the first thign you said was concerning :) | 15:55 |
| pabelanger | yar | 15:56 |
| *** matrohon has joined #openstack-infra | 15:56 | |
| Daviey | /win 47 | 15:57 |
| clarkb | my understanding of ftpsync was it does two passes and on the second it removes the packages no logner in the index | 15:57 |
| clarkb | which is what we don't want | 15:57 |
| clarkb | but maybe I was reading debians "your tools must do this" wrongly and thinking that was also what their tools do | 15:57 |
| *** pilgrimstack has quit IRC | 15:58 | |
| *** matrohon has quit IRC | 15:58 | |
| pabelanger | #RSYNC_OPTIONS2="--max-delete=40000 --delay-updates --delete --delete-excluded" | 15:58 |
| pabelanger | that is what happens on stage 2 | 15:58 |
| pabelanger | we have the option to change that | 15:58 |
| *** matrohon has joined #openstack-infra | 15:59 | |
| *** rkukura has joined #openstack-infra | 15:59 | |
| mordred | clarkb: according to zigo - the debian archive already does the delayed delete in the archive itself | 15:59 |
| pabelanger | we could remove the --delete flag, like we do today. | 15:59 |
| clarkb | mordred: greghaynes there is something about the ubuntu minimal images compared to the regular ubuntu cloud iamges that makes virsh shutdown work against regular images and not against dib ubuntu-minimal images. Maybe some missing acpi package? | 15:59 |
| mordred | clarkb: interesting | 16:00 |
| zigo | clarkb: AJaeger asked me to see with you what can be done for deb-websockify which was broken in the upstream repo (but I repaired it since). | 16:00 |
| zigo | clarkb: So, what can I do here? | 16:00 |
| mordred | pabelanger: yah. I mean, my original thought had be too pass -n to the delete command and just collect the output - then parse it and record the files into a little sqlite db | 16:00 |
| mordred | pabelanger: along with timestamp for having been added to the db | 16:01 |
| clarkb | zigo: you need to fix the repo then get a gerrit admin to push the fixed repo into gerrit | 16:01 |
| mordred | pabelanger: then doing the delete on anything wiht a timetamp older than like a day or somethig | 16:01 |
| zigo | clarkb: Well, it's fixed... | 16:01 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins https://review.openstack.org/319754 | 16:01 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module. https://review.openstack.org/319755 | 16:01 |
| zigo | clarkb: git fsck doesn't give the error there used to be. | 16:01 |
| clarkb | zigo: ok please push a revert of the change to remove it and note that a gerrit admin will have to force push the repo contents | 16:02 |
| *** matthewbodkin has quit IRC | 16:02 | |
| AJaeger | anteaya: project rename changes look fine - I didn't check pypi | 16:02 |
| *** tongli has quit IRC | 16:02 | |
| anteaya | AJaeger: thank you for reviewing the patches | 16:02 |
| anteaya | I think pleia2 checked pypi and launchpad | 16:03 |
| anteaya | pleia2: was that for all of them or just one of the patches? | 16:03 |
| clarkb | fungi: jhesketh have we done any investigating into what caused our log disk usage to skyrocket? seems like we had been well under capacity for a long time | 16:03 |
| AJaeger | anteaya: they don't seem to conflict, so no need to base them on top of each other - on the other hand, stacking them should be trivial ;) | 16:03 |
| anteaya | AJaeger: agreed | 16:03 |
| clarkb | maybe we aren't compressing some class of file or some project is uploading massive qcow2s on every job run again? | 16:03 |
| *** rbrndt has joined #openstack-infra | 16:03 | |
| *** sandanar has joined #openstack-infra | 16:05 | |
| pabelanger | mordred: Right, we have some options. | 16:07 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Revert "Remove deb-websockify from projects.yaml" https://review.openstack.org/365047 | 16:07 |
| zigo | clarkb: There you go. | 16:07 |
| pabelanger | mordred: Issue I see right now, it looks to be hard to only mirror jessie, like we do today. Working to see exlude magic can help | 16:07 |
| *** tphummel has joined #openstack-infra | 16:08 | |
| anteaya | pleia2: feel like posting a reminder to the channels about the gerrit maintenance? | 16:08 |
| fungi | dhellmann: thanks for confirming! | 16:08 |
| anteaya | ah here's fungi | 16:08 |
| anteaya | so either one | 16:08 |
| fungi | anteaya: pleia2: i think i'm comfortable trying the documented playbook, though with a few edits. i'll get patches up shortly to document the revision | 16:09 |
| anteaya | fungi: sounds good | 16:09 |
| fungi | clarkb: ^ | 16:09 |
| zigo | mordred: os-client-config fails to build with the sphinx doc calling git: http://logs.openstack.org/98/364998/2/check/gate-deb-python-os-client-config-dpkg-debian-jessie/28018e3/console.html :( | 16:09 |
| pabelanger | clarkb: So, I've pushed ubuntu-xenial to almost all regions, which includes new version of glean. It looks like we might have an issue in rackspace however | 16:09 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added documentation publish jobs for the JavaScript SDK https://review.openstack.org/346131 | 16:09 |
| pabelanger | clarkb: still waiting for a ubuntu-xenial image to come online | 16:10 |
| * fungi is still catching up on scrollback | 16:10 | |
| clarkb | pabelanger: fun fun. It is the odd one out when it comes to using the config drive network info | 16:10 |
| *** roxanaghe has joined #openstack-infra | 16:10 | |
| *** weshay_mtg is now known as weshay | 16:10 | |
| zigo | mordred: Is this fixed in later release? | 16:10 |
| fungi | AJaeger: project-config soft freeze until maintenance (for non-maintenance-related/non-emergency) changes sounds great, thanks | 16:11 |
| dstufft | fungi: sorry, I wasn't feeling great yesterday. What's up | 16:11 |
| pabelanger | clarkb: Ya, I wonder if we are just running glean too soon now | 16:11 |
| pabelanger | clarkb: I'll see if I can get the console working in rackspace | 16:12 |
| openstackgerrit | Dean Troyer proposed openstack/os-client-config: Update reno for stable/newton https://review.openstack.org/364920 | 16:12 |
| *** ilyashakhat_mobi has joined #openstack-infra | 16:12 | |
| fungi | clarkb: yeah, i haven't dug into what may be consuming more log space over the past few months (besides that brief period with ungodly huge keystone logs) | 16:12 |
| *** tonytan4ever has joined #openstack-infra | 16:13 | |
| pleia2 | anteaya: I think we'll do a notification like a half hour before | 16:13 |
| fungi | dstufft: oh, yesterday was mostly me discovering that setuptools 25.0.0 broke the ability to successfully pip install -e something and have it take precedence in the import path after a packaged version of the same something is installed | 16:13 |
| dstufft | fungi: oh, is this the raw path thing? | 16:14 |
| fungi | dstufft: yep | 16:14 |
| pleia2 | anteaya: oh and yeah, checked lp for all (and pypi for the one that needed) | 16:14 |
| anteaya | pleia2: oh I noticed nova folks planning their afternoon | 16:14 |
| *** xarses has joined #openstack-infra | 16:14 | |
| anteaya | pleia2: and I bet they forgot about the maintenance | 16:15 |
| persia | pabelanger: You cannot exclude suites with EXCLUDE= in ftpsync. I always just put up with them, because there aren't very many. | 16:15 |
| anteaya | was considering speaking up in channel but thought a channel wide broadcast would be better | 16:15 |
| mordred | zigo: os-client-config uses reno as part of its sphinx doc run - do my knowledge reno does not yet have git information cached in the source tarball ... | 16:15 |
| fungi | dstufft: there seem to be a couple of alternative solutions (one being the long-standing pr to have pip -e uninstall existing versions before installing) | 16:15 |
| pabelanger | persia: yup, that is what I am reading too. We might have to do the same | 16:15 |
| mordred | dhellmann: ^^ is that still correct? | 16:15 |
| pleia2 | anteaya: not sure I want to send a really early alert just because one team didn't read my email x_x | 16:15 |
| pleia2 | I can send out a reminder email | 16:15 |
| anteaya | pleia2: okay that would work | 16:15 |
| anteaya | and I bet they read it and forgot | 16:16 |
| anteaya | and thanks for checking lp and pypi | 16:16 |
| *** andymaier has quit IRC | 16:16 | |
| mordred | zigo: how do you handle this in the other reno-using projects? | 16:16 |
| pleia2 | ok, email sent | 16:16 |
| mordred | zigo: (it's also possible there is something I'm doing wrong with reno in os-client-config) | 16:17 |
| *** Swami has joined #openstack-infra | 16:17 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Simplify delete by removing unnecessary loop https://review.openstack.org/357990 | 16:17 |
| fungi | dstufft: mostly just curious at this point as to whether https://github.com/pypa/pip/pull/3898 or https://github.com/pypa/pip/pull/3904 have much chance of appearing in a coming pip release to counter that, or whether it's something tox should be patched to work around as a known problem behavior? | 16:17 |
| mordred | zigo: oh! you know what ... | 16:18 |
| anteaya | pleia2: thank you | 16:18 |
| *** shashank_hegde has joined #openstack-infra | 16:18 | |
| mordred | zigo: ok. I think I see the issue | 16:18 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 16:18 |
| zigo | mordred: I thought you would! :) | 16:18 |
| dstufft | fungi: looks like the first one is merged already? | 16:18 |
| *** tonytan_brb has joined #openstack-infra | 16:19 | |
| dtroyer | mordred: was it this? https://review.openstack.org/364920 | 16:19 |
| *** jordanP has quit IRC | 16:19 | |
| dtroyer | nevermind ^^^, that never merged... | 16:20 |
| *** matrohon has quit IRC | 16:20 | |
| *** _nadya_ has joined #openstack-infra | 16:21 | |
| zigo | I added git as build-depends-indep, but then it may fail with no git log. | 16:21 |
| dhellmann | mordred , zigo : the feature is in reno, but not pbr, so yes we are not caching release notes in the tarballs yet | 16:21 |
| openstackgerrit | John Dickinson proposed openstack-infra/project-config: Add openstack-releasenotes-jobs to python-swiftclient https://review.openstack.org/365052 | 16:21 |
| fungi | dstufft: oh, maybe. i'll admit i find gituhb's ui a bit confusing. i should just clone the repo and look with the cli | 16:21 |
| dhellmann | mordred , zigo : that's high on my ocata list | 16:21 |
| *** asettle has quit IRC | 16:21 | |
| fungi | dstufft: aha, there it is... "xavfernandez merged commit 3417fac into pypa:master 15 days ago" | 16:21 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Don't build releasenotes in normal docs build https://review.openstack.org/365053 | 16:22 |
| mordred | zigo, dhellmann: ^^ | 16:22 |
| fungi | dstufft: so i guess once we have new pip and virtualenv releases we should be safe to go back to usedevelop=true in tox again | 16:22 |
| *** tonytan4ever has quit IRC | 16:22 | |
| zigo | mordred: Oh, indeed, I'll add it as a Debian patch, thanks a lot. | 16:22 |
| mordred | dtroyer: ^^ you too - I believe that will fix the problem zigo is having making packages | 16:22 |
| *** _nadya_ has quit IRC | 16:22 | |
| *** zul has quit IRC | 16:23 | |
| *** dims has quit IRC | 16:23 | |
| dstufft | fungi: theortically! | 16:23 |
| fungi | dstufft: thanks for following up! | 16:23 |
| dstufft | I'm not really much of a fan of usedevelop=true | 16:23 |
| dstufft | I can probably see about cutting a release though | 16:24 |
| openstackgerrit | Chris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available https://review.openstack.org/359471 | 16:24 |
| *** gyee has joined #openstack-infra | 16:24 | |
| fungi | dstufft: yea, we have some projects that take a while to install and the devs like local tox envs to have editable installs so they don't have to keep reinstalling... but i agree in our ci it's something we could consider overriding entirely | 16:24 |
| openstackgerrit | Merged openstack-infra/shade: Use list_servers for polling rather than get_server_by_id https://review.openstack.org/365021 | 16:25 |
| *** sambetts is now known as sambetts|afk | 16:25 | |
| dtroyer | mordred: +@… do we need that in stable/newton too? | 16:25 |
| mordred | dtroyer: just pushed the backported patch - yeah, I think we shoudl do | 16:25 |
| clarkb | pabelanger: I can put my local ubuntu-xenial image somewhere if it will help you debug quicker on rax | 16:25 |
| dtroyer | I see it now ;) | 16:25 |
| clarkb | pabelanger: the password is totally insecure though | 16:25 |
| fungi | dstufft: though that leads to potential bad developer experiences if they didn't realize that locally they weren't testing their actual code changes, and then got a nasty surprise failure after pushing code up for ci testing | 16:26 |
| clarkb | I am going to have to pop out just around when the gerrit downtime starts to meet up with other foundation staff to talk about using gerrit (don't worry we can use review-dev) | 16:26 |
| dstufft | fungi: yea :/ | 16:26 |
| jeblair | clarkb: that is a master stroke of scheduling :) | 16:26 |
| fungi | clarkb: or maybe you can just use review.o.o... ideally the outage will be brief and gerrit will just be sluggish for a few hours afterward until the online reindexing completes | 16:27 |
| clarkb | fungi: oh right online reindexing | 16:27 |
| clarkb | lets hope | 16:27 |
| dstufft | I mostly just don't like it because develop installs are materially different than normal installs so it's not out of the realm of possibilities to have something pass in usedevelop=True that would otherwise fail | 16:27 |
| fungi | dstufft: agreed. it's messy | 16:27 |
| anteaya | rename etherpad for those who want to play along: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 16:28 |
| fungi | i've been known to just copy code from my source tree into the virtualenv rather than use editable installs | 16:28 |
| *** tongli has joined #openstack-infra | 16:28 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 16:30 |
| *** esberglu has quit IRC | 16:31 | |
| *** tongli has quit IRC | 16:32 | |
| *** dims has joined #openstack-infra | 16:32 | |
| *** degorenko is now known as _degorenko|afk | 16:33 | |
| dims | AJaeger : how long would the soft freeze on project-config be for? | 16:34 |
| *** dprince has joined #openstack-infra | 16:34 | |
| *** hashar has quit IRC | 16:34 | |
| anteaya | dims: until after the gerrit downtime is complete | 16:35 |
| anteaya | dims: what do you need? | 16:35 |
| *** sandanar_ has joined #openstack-infra | 16:35 | |
| dims | anteaya : ah i thought it was related to release.. :) | 16:35 |
| dims | no worries | 16:35 |
| anteaya | not related at all | 16:35 |
| bkero | rcarrillocruz: have you been able to use gotty ywt? | 16:35 |
| *** zul has joined #openstack-infra | 16:36 | |
| anteaya | solely so that the project config patches for the rename don't need to be rebased | 16:36 |
| *** sputnik13 has joined #openstack-infra | 16:36 | |
| dims | anteaya : i was asking around and found one thing someone wanted (https://review.openstack.org/#/c/359831/) but that can wait till next week for sure | 16:36 |
| *** Apoorva has joined #openstack-infra | 16:36 | |
| *** bin_ has quit IRC | 16:37 | |
| zigo | mordred: It didn't fix it... :/ | 16:37 |
| *** jpich has quit IRC | 16:37 | |
| jeblair | bkero: i find this approach simpler, and it uses only commonly available tools: http://amo-probos.org/post/17 | 16:37 |
| zigo | I'll disable doc build for the moment. | 16:37 |
| anteaya | dims: okay great, yeah freeze will be off by end of the day | 16:37 |
| *** shashank_hegde has quit IRC | 16:37 | |
| *** hashar has joined #openstack-infra | 16:38 | |
| *** eranrom has quit IRC | 16:38 | |
| *** sandanar has quit IRC | 16:38 | |
| bkero | jeblair: im not sure if tcpserver is a commonly installed binary | 16:38 |
| bkero | but certainly more available than gotty. I'm assuming it's in some distro package. | 16:39 |
| pabelanger | clarkb: jeblair: The good news, is our EOFError rate is way down now | 16:39 |
| *** hashar is now known as hasharAway | 16:39 | |
| *** hasharAway is now known as hasharAwayy | 16:39 | |
| clarkb | pabelanger: awesome | 16:39 |
| mordred | zigo: it worked for me locally | 16:40 |
| jeblair | bkero: i said 'available' not 'installed' on purpose. :) it's ucspi-tcp in debuntu. | 16:40 |
| *** jerryz has joined #openstack-infra | 16:41 | |
| mordred | zigo: I made a source tarball with that commit, then untarred that into a location that has no git things anywhere | 16:41 |
| mordred | zigo: then ran the sphinx build | 16:41 |
| jeblair | bkero: it's all just an apt-get away | 16:41 |
| bkero | aye | 16:42 |
| mordred | zigo: can you share the new failure? | 16:42 |
| jeblair | pabelanger: what reduced it? | 16:42 |
| jeblair | pabelanger, clarkb: want to land https://review.openstack.org/364309 ? | 16:43 |
| bkero | jeblair: looks like a fine workable solution with distro tools. I just have a strong distaste for the 'script' binary. Too many uni classes being recorded with it and needing perfect execution to look proper in vim. | 16:43 |
| clarkb | jeblair: looking | 16:43 |
| *** tqtran has joined #openstack-infra | 16:43 | |
| *** eranrom has joined #openstack-infra | 16:43 | |
| *** dtantsur is now known as dtantsur|afk | 16:43 | |
| *** eranrom has quit IRC | 16:43 | |
| clarkb | bkero: but script replay is awesome | 16:43 |
| clarkb | bkero: ever need to do a live demo? thats teh way to do it | 16:43 |
| pabelanger | jeblair: we found openssh-server was getting restarted by glean, so early connection would drop | 16:44 |
| jeblair | pabelanger: ah, right. cool. | 16:44 |
| mordred | pabelanger: \o/ | 16:44 |
| jeblair | pabelanger: shall we abandon the nodepool change then? | 16:44 |
| zigo | mordred: http://logs.openstack.org/98/364998/3/check/gate-deb-python-os-client-config-dpkg-debian-jessie/dd94a1f/console.html | 16:44 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 16:44 |
| bkero | clarkb: I've found it to be more trouble than other methods. Mayne that's PTSD. | 16:44 |
| pabelanger | How ever, it appears rackspace is having issues now | 16:44 |
| jeblair | bkero: it sounds like you were using it for something different. | 16:45 |
| pabelanger | jeblair: I think we can for now to avoid churn | 16:45 |
| zigo | mordred: Oh, my bad, it looks like it didn't have my change somehow. | 16:45 |
| * zigo rebuilds | 16:45 | |
| bkero | jeblair: recording terminal sessions? | 16:45 |
| mordred | zigo: yah - I was about to say... | 16:46 |
| *** annegentle has quit IRC | 16:47 | |
| clarkb | jeblair: I have approved the timeout change | 16:47 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/elastic-recheck: Expand query for bug 1617447 https://review.openstack.org/365064 | 16:47 |
| openstack | bug 1617447 in neutron "InterfaceAttachFailed: Failed to attach network adapter device" [High,Fix released] https://launchpad.net/bugs/1617447 - Assigned to Kevin Benton (kevinbenton) | 16:47 |
| *** senk__ has joined #openstack-infra | 16:48 | |
| jeblair | bkero: yes, recording terminal sessions requiring perfect execution in vim seems slightly different than broadcasting a session where you can always hit "^L" if there is a problem. | 16:48 |
| *** chem is now known as chem|off | 16:48 | |
| *** senk has quit IRC | 16:49 | |
| jeblair | (though i have not encountered such problems. but then, i don't use vim.) | 16:49 |
| *** tongli has joined #openstack-infra | 16:49 | |
| bkero | jeblair: What I meant to say was that I was using script to record terminal sessions to prove I knew how to unix. The professor in question erroneously told us to use 'script' to record them so he could "watch" them in vim. | 16:51 |
| *** rajinir has joined #openstack-infra | 16:51 | |
| bkero | The shortness of my previous explanation was due to phone software keyboard :( | 16:51 |
| jeblair | bkero: that was indeed an error. and i would say that reflects about as well on the suitability of 'script' for this as if the prof had asked you to use 'gotty' so they could watch it in vim. :) | 16:51 |
| bkero | I'd be impressed if he could open a HTTP session in vim | 16:52 |
| *** hsammic has quit IRC | 16:52 | |
| clarkb | bkero: you can pretty easily | 16:52 |
| clarkb | vim http://foo.bar/stuff | 16:52 |
| fungi | you can totally open streaming network sockets in vim | 16:53 |
| fungi | as well as that | 16:53 |
| clarkb | (I do this a lot when dealing with our job logs) | 16:53 |
| fungi | also you can open network sockets in bash as if they were file descriptors too :/ | 16:53 |
| jeblair | bkero: the script/tcpserver thing is really nice -- it's simple and very fast -- no buffering for the audience, and it looks like a real terminal session :) | 16:53 |
| *** caowei has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack-infra/zuul: bump timeout on ssh commands to 30s https://review.openstack.org/364309 | 16:53 |
| *** eranrom has joined #openstack-infra | 16:54 | |
| *** tongli has quit IRC | 16:54 | |
| *** lucasagomes is now known as lucas-dinner | 16:54 | |
| bkero | jeblair: It sounds elegant, although I'm curious how it supports different line length wraps between client and server | 16:54 |
| jeblair | bkero: it works best if everyone sticks to the same window size, fer shure. :) | 16:54 |
| fungi | and terminal type, especially if there are going to be ansi escapes | 16:55 |
| jeblair | "film should be shown in its intended aspect ratio" | 16:55 |
| bkero | oof, true | 16:55 |
| anteaya | jeblair: ++ | 16:55 |
| bkero | yeah, looks like different column lengths completely mangle it | 16:56 |
| bkero | at least it recovers gracefully | 16:56 |
| *** dprince has quit IRC | 16:57 | |
| *** jkilpatr has quit IRC | 16:57 | |
| fungi | obvious caveats aside, it's a remarkably simple solution | 16:57 |
| bkero | jeblair: I'm trying it with 'sl' and it looks like there are some artifacts generated in the listening terminal -- know what that's about? | 16:57 |
| *** ifarkas is now known as ifarkas_afk | 16:58 | |
| bkero | looks like I have 79 cols on server and client | 16:58 |
| *** ZZelle has quit IRC | 17:00 | |
| *** ZZelle has joined #openstack-infra | 17:00 | |
| bkero | http://staff.osuosl.org/~bkero/artifacts.png server on left, client on right | 17:00 |
| pabelanger | clarkb: okay, I have the console working in rax. Did you want to upload your image to rax-ord? | 17:01 |
| *** amotoki has quit IRC | 17:01 | |
| clarkb | pabelanger: its up to you, wasn't sure if that would help or not | 17:01 |
| pabelanger | clarkb: Ya, will same me time rebuilding one so I can login via shell | 17:01 |
| clarkb | ok the question though becomes how do I do the upload as not nodepool :/ | 17:01 |
| clarkb | mordred: ^ is there an easy way or should I just cry | 17:02 |
| jeblair | bkero: i do not know. | 17:02 |
| clarkb | oh I have to convert it to vhd too :/ | 17:03 |
| jeblair | bkero: still happens with just a tail of the script file, so it's not the tcpserver or telnet part of things | 17:03 |
| *** javeriak has joined #openstack-infra | 17:03 | |
| mordred | clarkb: there should be a really easy way | 17:04 |
| clarkb | pabelanger: I will copy it up to nodepool.o.o where it can be converted then figure out the upload from there | 17:04 |
| *** oanson has quit IRC | 17:05 | |
| mordred | clarkb: c=shade.openstack_cloud(cloud='openstackci-osic-cloud1') ; c.create_image(name='foo', filename='path/to/foo.qcow2', wait=True) | 17:05 |
| bkero | jeblair: I'm wondering if it's imperfections in the terminal type | 17:05 |
| bkero | and how it handles control characters | 17:05 |
| pabelanger | clarkb: does recuse mode work with our images and rackspace | 17:05 |
| *** yuval has joined #openstack-infra | 17:06 | |
| mordred | clarkb: or: c=shade.openstack_cloud(cloud='rax', region_name='ORD') ; c.create_image(name='foo', filename='path/to/foo.qcow2', wait=True) | 17:06 |
| mordred | clarkb: you can do the same with ansible - but since it's literally two lines in python, i'd probably do it in the repl :) | 17:06 |
| *** trown is now known as trown|lunch | 17:06 | |
| *** _nadya_ has joined #openstack-infra | 17:07 | |
| mordred | you might also want to do shade.simple_logging(debug=True) before the shade.openstack_cloud call - you'll get _Some_ feedback about activity that way | 17:07 |
| clarkb | mordred: cool thanks | 17:07 |
| *** _nadya_ has quit IRC | 17:07 | |
| clarkb | pabelanger: no idea | 17:08 |
| *** signed8bit is now known as signed8bit_Zzz | 17:08 | |
| *** roxanaghe has quit IRC | 17:10 | |
| *** jerryz has quit IRC | 17:10 | |
| *** signed8bit_Zzz is now known as signed8bit | 17:10 | |
| *** roxanaghe has joined #openstack-infra | 17:10 | |
| *** sandanar__ has joined #openstack-infra | 17:10 | |
| clarkb | pabelanger: the image is going into /opt/test_images where I will convert to raw then back to vhd then do mordreds shade thing | 17:11 |
| *** bryan_att has left #openstack-infra | 17:11 | |
| clarkb | mordred: does create_image not require a container type argument? | 17:11 |
| *** jerryz has joined #openstack-infra | 17:11 | |
| fungi | zaro: what's the command to initiate an online reindex in gerrit? | 17:11 |
| mordred | clarkb: nope. it infers from filename | 17:12 |
| *** ddieterly has joined #openstack-infra | 17:12 | |
| *** shashank_hegde has joined #openstack-infra | 17:12 | |
| *** shashank_hegde has quit IRC | 17:12 | |
| zigo | mordred: It worked! :) | 17:13 |
| mordred | zigo: woot! | 17:13 |
| zigo | https://review.openstack.org/#/c/364998/ | 17:13 |
| fungi | ooh, here's another problem i just noticed with the rename playbook... it seems to assume that the account you're running this in on the puppetmaster has ssh credentials for the gerrit api | 17:13 |
| pabelanger | clarkb: ack | 17:13 |
| fungi | which will almost certainly not be the case | 17:13 |
| mordred | clarkb: it'll also look at what the cloud claims to want ... if you're doing something like uploading a qcow2 to ovh, which claims to want raw, then you might have to be explicit | 17:13 |
| * zigo goes out to eat | 17:13 | |
| *** sandanar_ has quit IRC | 17:14 | |
| *** florianf has quit IRC | 17:15 | |
| *** HeOS has quit IRC | 17:15 | |
| mordred | clarkb: you know - we _could_ have it use libmagic to figure out for sure what type of image you gave it, instead of having it look at extension | 17:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update project rename instructions https://review.openstack.org/365067 | 17:17 |
| fungi | anteaya: pleia2: mordred: ^ | 17:17 |
| anteaya | fungi: thank you | 17:17 |
| * pleia2 reviews | 17:17 | |
| fungi | speaking of which, i'm going to go ahead and make sure our puppet/ansible cron is good and stopped now | 17:18 |
| anteaya | fungi: good idea | 17:18 |
| *** rossella_s has quit IRC | 17:18 | |
| *** bethwhite_ has quit IRC | 17:18 | |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| fungi | current run started at 17:15 utc | 17:19 |
| *** ddieterly has quit IRC | 17:19 | |
| anteaya | how long do you expect it to take? | 17:19 |
| fungi | i've commented it out in the crontab but will check in on it again in a few minutes | 17:19 |
| *** e0ne has quit IRC | 17:19 | |
| fungi | hopefully it'll be well and done in the next 20 minutes | 17:19 |
| clarkb | pabelanger: what region would be best to test in? | 17:19 |
| fungi | i did not comment out the cron for infra-cloud hosts since they should be entirely unrelated to this work | 17:20 |
| * clarkb does ord as I think we have most quota there | 17:20 | |
| * pleia2 nods | 17:20 | |
| pabelanger | clarkb: ord should be good | 17:20 |
| fungi | also when i run the playbook, i'm going to take out the gerrit group renames step and perform those directly on my own | 17:21 |
| anteaya | fungi: ack | 17:21 |
| anteaya | fungi: i put the gerrit group chances in the etherpad and pleia2 reviewed them | 17:21 |
| fungi | at least until puppetmaster's root account has gerrit admin account credentials | 17:21 |
| fungi | which won't be som,ething i'll work on today | 17:22 |
| anteaya | fair | 17:22 |
| fungi | speaking of which, i mostly expect the api calls to perform group renames would not work without adding in some sort of delay | 17:22 |
| fungi | gerrit is not immediately usable as soon as its initscript returns | 17:23 |
| pleia2 | yeah | 17:23 |
| *** sarob has joined #openstack-infra | 17:24 | |
| *** ddieterly has joined #openstack-infra | 17:25 | |
| *** tongli has joined #openstack-infra | 17:25 | |
| *** sarob has quit IRC | 17:25 | |
| clarkb | pabelanger: the upload is running in a screen owned by my user (yet another use of script is to make screen work for other users) you can follow along there. Also the readme in /opt/test_images has the user auth info | 17:25 |
| fungi | oh, i guess this is also missing the command to start online reindexing. i need to look that up since zaro seems to not be around | 17:25 |
| *** mwhahaha has quit IRC | 17:25 | |
| *** sarob has joined #openstack-infra | 17:25 | |
| clarkb | mordred: I want this set of steps to be in the official rax docs | 17:26 |
| clarkb | mordred: their docs are basically here use the gui or consider making a snapshot because its complicated otherwise | 17:26 |
| *** salv-orlando has quit IRC | 17:26 | |
| pabelanger | clarkb: need to teaching, do you know if it is udev that runs to setup config-2 for blkid? | 17:26 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/devstack-gate: Add some missing projects https://review.openstack.org/365071 | 17:27 |
| pabelanger | clarkb: I'm thinking we just need to add another dependency for systemd | 17:27 |
| clarkb | pabelanger: that sounds right | 17:27 |
| clarkb | pabelanger: though in a systemd world its probably some new thing that has replaced udev :P | 17:28 |
| pabelanger | ya | 17:28 |
| pabelanger | going to check an image in osic and see | 17:28 |
| zaro | fungi: ssh -p 29418 fungi@review.openstack.org gerrit index start --force changes | 17:28 |
| *** mwhahaha has joined #openstack-infra | 17:29 | |
| *** ilyashakhat_mobi has quit IRC | 17:29 | |
| *** markvoelker has joined #openstack-infra | 17:29 | |
| fungi | thanks zaro! | 17:29 |
| zaro | we we following an etherpad for this project name change? | 17:29 |
| *** tongli has quit IRC | 17:29 | |
| anteaya | zaro: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 17:29 |
| zaro | thanks | 17:29 |
| anteaya | welcome | 17:30 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Expand query for bug 1617447 https://review.openstack.org/365064 | 17:30 |
| openstack | bug 1617447 in neutron "InterfaceAttachFailed: Failed to attach network adapter device" [High,Fix released] https://launchpad.net/bugs/1617447 - Assigned to Kevin Benton (kevinbenton) | 17:30 |
| *** mat128|away is now known as mat128 | 17:30 | |
| *** ddieterly is now known as ddieterly[away] | 17:31 | |
| anteaya | fungi: do you want me to recheck those 3 patches or not bother? | 17:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove octavia bandit job https://review.openstack.org/365074 | 17:31 |
| fungi | anteaya: what was wrong with them? | 17:32 |
| anteaya | nothing | 17:32 |
| fungi | or just want to make sure they're still passing jobs? | 17:32 |
| anteaya | just wanted to make sure they didn't need a rebase | 17:32 |
| anteaya | yeah that | 17:32 |
| anteaya | pleia2: you want to do the statusbot announcement? | 17:32 |
| clarkb | pabelanger: I have a systemd-udevd service on tumbleweed | 17:32 |
| pleia2 | sure, let's see... | 17:32 |
| AJaeger | clarkb: is EmilienM's change 365071 fine? I thought we don't want to add more to PROJECTS... | 17:33 |
| clarkb | AJaeger: correct those should go in the tests that need those projects | 17:33 |
| pabelanger | clarkb: Ya, I am just reading up on it now | 17:34 |
| *** markvoelker has quit IRC | 17:34 | |
| clarkb | AJaeger: my eventual goal is to get the PROJECTS list down to what devstack runs by default. nova neutron glance cinder swift horizon maybe | 17:34 |
| anteaya | fungi: all send them through check again | 17:34 |
| *** markvoelker has joined #openstack-infra | 17:34 | |
| anteaya | s/all/I'll | 17:34 |
| fungi | okay | 17:34 |
| pabelanger | clarkb: I think we want After=systemd-udevd.service | 17:34 |
| clarkb | AJaeger: beacuse such a large chunk of our job time is setting up repos we will never use | 17:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Use time.sleep() when waiting for workers to run https://review.openstack.org/365075 | 17:34 |
| pabelanger | but need to test | 17:34 |
| clarkb | pabelanger: ya that seems about right | 17:34 |
| clarkb | pabelanger: my iamge exists in ORD now | 17:34 |
| AJaeger | clarkb: gave a -1 ;) | 17:35 |
| AJaeger | clarkb: ok, then I remembered correctly | 17:35 |
| pleia2 | anteaya, fungi - want to take a quick look at what I put in the etherpad for notice? | 17:35 |
| clarkb | pabelanger: and log in details are in /opt/test_images/README | 17:35 |
| pabelanger | clarkb: we might want to also include haveged too | 17:36 |
| pabelanger | clarkb: sure, will check shortly | 17:36 |
| clarkb | pabelanger: that image should have haveged | 17:36 |
| anteaya | pleia2: looks good to me | 17:36 |
| clarkb | oh you mean in the unit file ? | 17:36 |
| pabelanger | ya | 17:36 |
| EmilienM | clarkb: ack | 17:36 |
| pabelanger | clarkb: actually, I don't know if we need the service running | 17:36 |
| pabelanger | I think jeblair said we don't | 17:36 |
| clarkb | pabelanger: I don't think its necessary since the init urandom script turns it off if it is | 17:36 |
| pleia2 | #status notice The infrastructure team is taking Gerrit offline for maintenance, beginning shortly after 18:00 UTC for a potentially 4 hour maintenance window. | 17:36 |
| openstackstatus | pleia2: sending notice | 17:36 |
| pabelanger | clarkb: Ya, that sounds right | 17:37 |
| clarkb | pabelanger: we don't it invokes the process directly and retreives the data | 17:37 |
| *** dtardivel has quit IRC | 17:37 | |
| *** sasi_ has joined #openstack-infra | 17:37 | |
| jeblair | clarkb, pabelanger; the urandam init does not require a running haveged | 17:37 |
| zaro | fungi:fyi, probably need to online reindex accounts as well as changes. | 17:37 |
| zaro | do accounts first | 17:37 |
| anteaya | zaro: what is the command, is it different that the one you gave? | 17:38 |
| jeblair | clarkb, pabelanger: it's fine for the haveged service to start after urandom init runs (indeed, things will be slightly faster if it does :) | 17:38 |
| -openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance, beginning shortly after 18:00 UTC for a potentially 4 hour maintenance window. | 17:38 | |
| *** eranrom has quit IRC | 17:38 | |
| zaro | anteaya: it's on the review of that change | 17:38 |
| fungi | zaro: thanks | 17:38 |
| *** _nadya_ has joined #openstack-infra | 17:38 | |
| anteaya | zaro: ah thank you | 17:38 |
| sasi_ | having problem in building dib image on nodepool -- http://paste.openstack.org/show/566130/ | 17:39 |
| pabelanger | jeblair: thanks | 17:39 |
| *** mriedem is now known as mriedem_afk | 17:39 | |
| *** waht has joined #openstack-infra | 17:39 | |
| clarkb | sasi_: do you have a /var/log/nodepool/images? look inthere for the build logs to see why it failed | 17:39 |
| openstackstatus | pleia2: finished sending notice | 17:39 |
| pleia2 | openstackstatus: you get a cookie! | 17:40 |
| clarkb | sasi_: but it looks like maybe it doesn't have file permissions to create that dir it is complaining about | 17:40 |
| zaro | fungi: step 4.5 - stop gerrit? | 17:40 |
| *** deadnull_ has quit IRC | 17:40 | |
| *** shashank_hegde has joined #openstack-infra | 17:40 | |
| fungi | zaro: that playbook already stops and starts gerrit | 17:40 |
| zaro | gotcha | 17:41 |
| AJaeger | fungi, openstackdocstheme gets more interesting ;( I merged the one change and for the fun of it want to enable constraints on it - but now I get "Could not satisfy constraints for 'openstackdocstheme': installation from path or url cannot be constrained to a version" - see http://logs.openstack.org/99/364499/4/check/gate-openstackdocstheme-docs-ubuntu-xenial/e6bd228/console.html | 17:41 |
| fungi | the ansible script on puppetmaster.o.o is no longer running and is still commented out in the crontab there, so should be all set | 17:42 |
| AJaeger | fungi: I have a working solution, so can ignore this one. No need to dig now into this... | 17:42 |
| anteaya | fungi: thank you | 17:42 |
| AJaeger | fungi: more a FYI | 17:42 |
| *** Na3iL has quit IRC | 17:42 | |
| *** chem|off has quit IRC | 17:43 | |
| sasi_ | Clarkb: no error on the image.log | 17:43 |
| fungi | AJaeger: i think that's usually due to lack of edit-constraints setting the entry in your requirements list appropriately, but in this case it could be complicated by your transitive dependency loop | 17:43 |
| clarkb | sasi_: ya its likely failing early then and its a permissions issue. make sure it can write to that directoy | 17:43 |
| AJaeger | fungi, yeah - I have edit-constraints. I think the transitive loop is the problem... | 17:43 |
| fungi | AJaeger: er, actually edit-constraints modifies the constraints file, which should make that work. so not sure what's happening there | 17:43 |
| AJaeger | yeah, after changing the usedevelop setting it suddenly fails... | 17:44 |
| *** prometheanfire has quit IRC | 17:45 | |
| *** dprince has joined #openstack-infra | 17:45 | |
| sasi_ | clarkb: owner of the /opt/nodepool_dib is nodepool, and it can write into it | 17:45 |
| sasi_ | drwxr-xr-x 2 nodepool nodepool 4096 Sep 2 10:44 nodepool_dib/ | 17:45 |
| *** ccarmack has quit IRC | 17:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Add After=systemd-udev-settle.service for systemd https://review.openstack.org/365079 | 17:48 |
| pabelanger | clarkb: ^ WIP until I can test it | 17:48 |
| *** adrian_otto has joined #openstack-infra | 17:49 | |
| *** tongli has joined #openstack-infra | 17:49 | |
| *** _nadya_ has quit IRC | 17:49 | |
| fungi | anteaya: pleia2: zaro: okay, i think the steps in the etherpad coupled with the proposed change will cover it | 17:49 |
| anteaya | fungi: awesome, thank you | 17:49 |
| fungi | i'm working on the ansible playbook parameter file now | 17:50 |
| *** eranrom has joined #openstack-infra | 17:50 | |
| anteaya | wonderful | 17:50 |
| zaro | fungi: lgtm | 17:50 |
| *** yamahata has joined #openstack-infra | 17:51 | |
| anteaya | fungi: patches finished recheck, no rebases required | 17:51 |
| anteaya | fungi: I haven't stacked them | 17:52 |
| * AJaeger signs off now and hopes that all playbooks play well ;) | 17:53 | |
| anteaya | AJaeger: thank you | 17:53 |
| *** tongli has quit IRC | 17:53 | |
| *** waht has quit IRC | 17:54 | |
| stevebaker | hey, it looks like merging to heat-templates is wedged https://review.openstack.org/#/c/350584/ https://review.openstack.org/#/c/355718/ | 17:55 |
| anteaya | stevebaker: we are about to take gerrit offline to rename some projects | 17:55 |
| stevebaker | ok | 17:56 |
| anteaya | stevebaker: perhaps post maintenance we can come back to this? | 17:56 |
| stevebaker | sure | 17:56 |
| anteaya | stevebaker: thanks :) | 17:56 |
| stevebaker | thank you | 17:56 |
| jesusaur | clarkb: do your log_processor workers ever die with MemoryError? I'm seeing that happen a few times a month | 17:56 |
| clarkb | jesusaur: no, I had to give logstash 2gb of heap though | 17:56 |
| *** eharney_ has joined #openstack-infra | 17:57 | |
| *** eharney has quit IRC | 17:57 | |
| *** ijw has joined #openstack-infra | 17:57 | |
| AJaeger | anteaya: those two reviews by stevebaker really look strange and need investigation - but agreed, gerrit offline is important. | 17:57 |
| AJaeger | really off now ;) | 17:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo: add missing PROJECTS in the list https://review.openstack.org/365081 | 17:58 |
| *** eharney_ is now known as eharney | 17:58 | |
| EmilienM | pabelanger, AJaeger ^ | 17:58 |
| anteaya | I just figured not sense getting into and then gerrit going offline | 17:58 |
| fungi | anteaya: pleia2: zaro: the playbook parameter file is drafted in the etherpad now | 17:58 |
| anteaya | fungi: thank you | 17:58 |
| pleia2 | great | 17:58 |
| *** piet has quit IRC | 17:58 | |
| *** AnarchyAo has joined #openstack-infra | 17:58 | |
| fungi | i'm getting the parameter file and an edited copy of the playbook into place in my homedir on puppetmaster.o.o now | 17:58 |
| anteaya | fungi: looks right to me | 17:58 |
| pabelanger | clarkb: okay, have server online and network fixed | 17:58 |
| pabelanger | clarkb: testing patch now | 17:59 |
| *** ijw has quit IRC | 17:59 | |
| pabelanger | after I installed vim | 17:59 |
| clarkb | pabelanger: ya its minimal :) | 17:59 |
| *** piet has joined #openstack-infra | 17:59 | |
| jesusaur | clarkb: hm, ok. I'm also limiting logstash to 2gb, but i also still have a local elasticsearch with a 1gb heap | 17:59 |
| *** sandanar has joined #openstack-infra | 17:59 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: DNM: hack projects until https://review.openstack.org/#/c/365081/ https://review.openstack.org/365082 | 18:00 |
| zaro | fungi: lgtm | 18:00 |
| * jesusaur will try to play with heap sizes or bump to 8gb instances | 18:00 | |
| pleia2 | woo, lmk when it's alert time :) | 18:00 |
| stevebaker | AJaeger, anteaya: it could be that heat-templates-buildimages is failing - that job can be deleted | 18:01 |
| *** ijw has joined #openstack-infra | 18:01 | |
| fungi | anteaya: pleia2: zaro: i'm all set with the parameter file and updated playbook now | 18:01 |
| anteaya | stevebaker: ah perhaps post maintenance we can begin there | 18:01 |
| anteaya | fungi: awesome, can pleia2 alert folks now? | 18:01 |
| fungi | yep, alert away | 18:01 |
| pleia2 | #status alert Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC. | 18:02 |
| openstackstatus | pleia2: sending alert | 18:02 |
| fungi | who else in infra-root wants to grab steps in the plan at https://etherpad.openstack.org/p/project-renames-Septemeber-2016 ? | 18:02 |
| *** sandanar__ has quit IRC | 18:03 | |
| pabelanger | fungi: As soon as I fix ubuntu-xenial in rackspace, I'll add my name to something | 18:03 |
| fungi | cool | 18:03 |
| pleia2 | I should probably not be the reindexer, since I need to step out in 90 minutes | 18:03 |
| fungi | zaro: is it okay for accounts reindex and changes reindex to be running concurrently? | 18:03 |
| *** ijw has quit IRC | 18:03 | |
| fungi | pleia2: i can handle that part. i'll be around | 18:03 |
| zaro | don't know. i have not tested that scenario so i wouldn't try it. | 18:04 |
| zaro | accounts should be pretty fast | 18:04 |
| pleia2 | oh, anyone handling zuul yet? | 18:04 |
| -openstackstatus- NOTICE: Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC. | 18:04 | |
| *** ChanServ changes topic to "Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC." | 18:04 | |
| fungi | zaro: fair enough. i'll just watch the processes list until accounts finishes | 18:04 |
| pleia2 | I need to look up commands for that | 18:04 |
| *** annegentle has joined #openstack-infra | 18:04 | |
| fungi | pleia2: in https://review.openstack.org/365067 | 18:05 |
| fungi | which is why i linked it | 18:05 |
| pleia2 | right-o | 18:05 |
| fungi | you can probably pull up the docs-draft version now if you want | 18:05 |
| zaro | fungi: watch gerrit show-queue | 18:05 |
| fungi | zaro: oh, right, that one. thanks! ;) | 18:05 |
| pleia2 | I can do zuul whenever we're ready | 18:05 |
| anteaya | I don't see a stop gerrit step, is there a stop gerrit step? | 18:05 |
| zaro | it's in the playbook | 18:06 |
| anteaya | zaro: ah sorry | 18:06 |
| fungi | http://docs-draft.openstack.org/67/365067/1/check/gate-system-config-docs-ubuntu-xenial/8eb737d//doc/build/html/gerrit.html#renaming-a-project | 18:06 |
| anteaya | I do believe we are ready to stop zuul, are we not? | 18:06 |
| pabelanger | clarkb: here is the failure: http://paste.openstack.org/show/566137/ | 18:06 |
| pabelanger | need to figure out why | 18:06 |
| fungi | yep... pleia2: whenever you're ready | 18:06 |
| *** trown|lunch is now known as trown | 18:07 | |
| pleia2 | on it | 18:07 |
| jeblair | i'm around; ping me if you need backup | 18:07 |
| fungi | thanks jeblair! | 18:07 |
| openstackstatus | pleia2: finished sending alert | 18:07 |
| * pleia2 waits for zuul to stop | 18:07 | |
| pleia2 | there | 18:08 |
| clarkb | pabelanger: I think that meams there is no network device with that macaddr in sysfs | 18:08 |
| pleia2 | zuul step all done | 18:08 |
| anteaya | pleia2: wonderful | 18:08 |
| fungi | pleia2: thanks. i'll run the playbook now | 18:09 |
| * pleia2 fingers crossed | 18:09 | |
| pabelanger | clarkb: Hmm, maybe this is a bug with the new release over systemd issue | 18:09 |
| pabelanger | because glean is running | 18:09 |
| pabelanger | just failing | 18:09 |
| fungi | it's running now, but i'm going to carefully inspect its output and then go check behind it as per the plan | 18:10 |
| * pleia2 nods | 18:10 | |
| anteaya | fungi: ack | 18:10 |
| fungi | oof, a failure | 18:10 |
| anteaya | :( | 18:10 |
| fungi | oh geez, i see a typo in the playbook | 18:11 |
| anteaya | oh no :( | 18:11 |
| anteaya | what do you want to do? | 18:11 |
| fungi | however, this thing probably isn't idempotent so i doubt i can just rerun it after i fix it | 18:11 |
| pleia2 | fungi: is 6.1 Gerrit or GitHub? | 18:12 |
| fungi | pleia2: gerrit | 18:12 |
| pleia2 | k | 18:12 |
| jeblair | fungi: can you paste the playbook output? | 18:13 |
| fungi | i'm going to correct the typo and try to surgically remove the steps which have already run | 18:13 |
| fungi | jeblair: yep, doing that now | 18:13 |
| fungi | http://paste.openstack.org/show/566138 | 18:14 |
| mordred | fungi: there is an option with running ansible playbooks to re-run failed steps - as an fyi | 18:14 |
| fungi | i think the nested dirs need trailing slashes | 18:14 |
| jeblair | heh, those plays need names :) | 18:15 |
| mordred | fungi: you can also use "--start-at" - but things need names :) | 18:15 |
| mordred | ansible-playbook foo.yml --start-at-task=START_AT | 18:15 |
| *** ijw has joined #openstack-infra | 18:16 | |
| fungi | hrm, actually no i don't think it's a trauiling slash problem | 18:17 |
| mordred | (I'd be helping more explicitly, but I've got a call in 10 minutes so I don't want to have to bail in the middle - so comments from the peanut gallery is best I've got today - sorry) | 18:18 |
| jeblair | fungi: it's missing a trailing '.git' | 18:18 |
| fungi | oh! | 18:18 |
| fungi | jeblair: yep, i just identified the same :/ | 18:18 |
| fungi | trying to tab-complete those dirnames | 18:18 |
| *** kushal has quit IRC | 18:18 | |
| *** ijw has quit IRC | 18:19 | |
| *** notnownikki has quit IRC | 18:19 | |
| pabelanger | clarkb: mordred: Okay, I see the issue. There is a bug in glean that we need to fix | 18:20 |
| *** ddieterly[away] is now known as ddieterly | 18:20 | |
| fungi | though there were also "mv: cannot stat ‘/opt/lib/gitopenstack/higgins" so the missing trailing / was also a problem | 18:20 |
| *** spzala has joined #openstack-infra | 18:20 | |
| fungi | i'm correcting both, and also for the git0*.o.o step | 18:20 |
| mordred | pabelanger: neat | 18:20 |
| fungi | and rerunning with the steps prior to the failed mv commands removed | 18:21 |
| anteaya | fungi: ack | 18:21 |
| jeblair | fungi: agreed | 18:21 |
| jeblair | my updated copy of the playbook is this: https://etherpad.openstack.org/p/TPGv9SJPaV | 18:21 |
| fungi | it's running again | 18:21 |
| pabelanger | mordred: clarkb: so using http://paste.openstack.org/show/566139/ and trying to configure eth0, there is a KeyError because glean assumes the first ethernet_mac_address in links is eth0, when in fact, it is eth1 in this case | 18:22 |
| fungi | succeeded everywhere except the storyboard step this time | 18:22 |
| fungi | another paste on the way | 18:22 |
| fungi | but this is probably "these projects aren't in storyboard" nonissues | 18:22 |
| clarkb | pabelanger: neat | 18:22 |
| clarkb | pabelanger: so it needs to iterate to find the right one | 18:22 |
| anteaya | fungi: ah, so yay mostly success | 18:23 |
| pabelanger | clarkb: Ya, I just need to see what is the right way to do that | 18:23 |
| fungi | http://paste.openstack.org/show/566140 | 18:23 |
| fungi | Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock | 18:24 |
| mordred | pabelanger: oh stellar! | 18:24 |
| *** pvaneck has joined #openstack-infra | 18:24 | |
| pleia2 | heh | 18:24 |
| pleia2 | is it not finding the mysql creds? | 18:24 |
| fungi | that's what i'm checking | 18:25 |
| mordred | or hostname is maybe not set? | 18:25 |
| *** spzala has quit IRC | 18:25 | |
| pleia2 | hostname should be with the rest of the creds | 18:25 |
| *** e0ne has joined #openstack-infra | 18:25 | |
| *** gyee has quit IRC | 18:25 | |
| jeblair | none of these are in storyboard, though, right? so it doesn't matter for this run? | 18:25 |
| anteaya | jeblair: correct | 18:26 |
| pleia2 | yeah | 18:26 |
| *** yuval has quit IRC | 18:26 | |
| fungi | i bet we didn't puppet /root/.my.cnf and then neglected to copy it when we replaced the sb server | 18:26 |
| mordred | root@storyboard:~# mysql storyboard | 18:26 |
| mordred | ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) | 18:26 |
| pleia2 | yep | 18:26 |
| *** spzala has joined #openstack-infra | 18:26 | |
| jeblair | mordred: reproduced! | 18:26 |
| mordred | I can confirm both the error, and the lack of .my.cnf | 18:26 |
| fungi | yeah, there's no /root/.my.cnf file | 18:26 |
| fungi | anyway, we can take that as something to fix after. i'll remove up through the storyboard.o.o task and start the playbook again | 18:27 |
| fungi | fatal: [review.openstack.org]: UNREACHABLE! => {"changed": false, "msg": "Failed | 18:28 |
| fungi | to connect to the host via ssh.", "unreachable": true} | 18:28 |
| fungi | huh? | 18:28 |
| pleia2 | hm, wrong port? | 18:28 |
| *** salv-orlando has joined #openstack-infra | 18:28 | |
| fungi | remote_user: gerrit2 | 18:29 |
| fungi | a downside of this playbook is that it assumes it can ssh as multiple accounts and into the gerrit api | 18:29 |
| fungi | it only actually has credentials to ssh as root | 18:29 |
| mordred | fungi: yah - we should change away from remote_user and to become_user - which uses sudo | 18:29 |
| *** jkilpatr has joined #openstack-infra | 18:30 | |
| mordred | turns out we're not the only ones to have encountered that problem | 18:30 |
| fungi | mordred: so make remote_user: root and add become_user: gerrit2? | 18:30 |
| mordred | yah - or leave remote_user out | 18:30 |
| *** spzala has quit IRC | 18:30 | |
| fungi | oh, and it has sudo: yes on that task for some strange reason | 18:31 |
| mordred | fungi: we want "become: true\nbecome_user: gerrit2" | 18:31 |
| *** ccarmack has joined #openstack-infra | 18:31 | |
| mordred | http://docs.ansible.com/ansible/become.html | 18:31 |
| *** pelix has quit IRC | 18:32 | |
| fungi | looks like become: true must be assumed if you supply become_user? | 18:32 |
| *** ccarmack has left #openstack-infra | 18:32 | |
| fungi | because it worked for me without become: true | 18:32 |
| mordred | maybe that's because sudo: true was there there | 18:32 |
| fungi | i removed the sudo line completely | 18:32 |
| notmorgan | oh hi. | 18:33 |
| pleia2 | o/ notmorgan | 18:33 |
| fungi | http://paste.openstack.org/show/566141 | 18:33 |
| fungi | so that's the last of the playbook | 18:33 |
| *** ogelbukh has quit IRC | 18:33 | |
| * notmorgan hails the wonderful infra folks from the california south. | 18:33 | |
| *** shashank_hegde has quit IRC | 18:33 | |
| notmorgan | pleia2: sorry i didn't bug ya when i was in the bay area, but was in a rush to get down here (plans changed). Will have more time on the trip back north. | 18:34 |
| anteaya | fungi: yay playbook's done | 18:34 |
| pleia2 | notmorgan: was sorry to miss you, but yes, next time :) | 18:34 |
| fungi | yep, i'll get to checking behind it now to make sure it did... stuff | 18:34 |
| fungi | preferably the stuff we wanted done | 18:34 |
| anteaya | indeed | 18:34 |
| *** ogelbukh has joined #openstack-infra | 18:34 | |
| *** jkilpatr has quit IRC | 18:35 | |
| fungi | it does seem to have started gerrit there on the last run | 18:35 |
| fungi | people can probably begin poking around to make sure it's running correctly | 18:35 |
| fungi | i'm checking filesystem and database now | 18:35 |
| anteaya | yes gerrit is back for me | 18:35 |
| * notmorgan logs in at the right time to see gerrit return. | 18:35 | |
| notmorgan | talk about timing | 18:35 |
| *** shashank_hegde has joined #openstack-infra | 18:35 | |
| fungi | i'll note that smaug doesn't seem to have renamed openstack/smaug-dashboard | 18:35 |
| fungi | was it supposed to be in the list? | 18:36 |
| anteaya | :( | 18:36 |
| jeblair | python-smaugclient exists as well | 18:36 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade: Prevent Negative Caching https://review.openstack.org/363321 | 18:36 |
| *** salv-orlando has quit IRC | 18:36 | |
| fungi | also openstack/python-smaugclient | 18:36 |
| fungi | yeah | 18:36 |
| anteaya | fungi: it was in the patch: https://review.openstack.org/#/c/353304/9/zuul/layout.yaml | 18:37 |
| *** roxanaghe has quit IRC | 18:37 | |
| anteaya | sorry I didn't realize it should have been in the playbook config file | 18:37 |
| fungi | anteaya: ahh, just not in our list in the etherpad i guess | 18:37 |
| anteaya | sigh | 18:37 |
| anteaya | my bad reviewing | 18:37 |
| pleia2 | ugh, sorry | 18:37 |
| jeblair | GitCommandError: 'git fetch https://git.openstack.org/openstack-infra/shade +refs/changes/21/363321/2:refs/changes/21/363321/2' returned with exit code 128 | 18:37 |
| jeblair | stderr: 'fatal: Couldn't find remote ref refs/changes/21/363321/2 | 18:37 |
| fungi | i was taking your list at the top there as a full list of project names we were going to rename, sorry | 18:37 |
| anteaya | also I didn' tknow the ansible config file was not going to use the patch information | 18:37 |
| *** tongli has joined #openstack-infra | 18:38 | |
| anteaya | fungi: yeah, sorry those were just the commit message titles of the patches | 18:38 |
| anteaya | or at least what was put into the wiki | 18:38 |
| fungi | oops. i shouldn't have assumed | 18:38 |
| anteaya | which I copy pasted from the wiki page and was input by the patch owners | 18:38 |
| fungi | okay, i'll see if i can get a working playbook assembled to do those other two | 18:39 |
| anteaya | yeah, sorry i missed that this was info that should have been in the ansible file | 18:39 |
| anteaya | fungi: thanks, sorry | 18:39 |
| jeblair | oh, maybe that just hasn't propogated yet because of the replication backlog on startup | 18:39 |
| jeblair | that ref is in gerrit, so that looks good | 18:40 |
| *** prometheanfire has joined #openstack-infra | 18:40 | |
| *** hasharAwayy is now known as hashar | 18:40 | |
| *** tongli has quit IRC | 18:42 | |
| *** e0ne has quit IRC | 18:42 | |
| *** craige has quit IRC | 18:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add nodeset as a top-level configuration item https://review.openstack.org/365089 | 18:42 |
| jeblair | pleia2: zuul appears to be running | 18:43 |
| pleia2 | ! | 18:44 |
| fungi | zuul's i don't see it running in ps | 18:44 |
| pleia2 | I don't either | 18:44 |
| jeblair | huh. i thought ! was not a command. | 18:44 |
| anteaya | status.openstack.org/zuul shows activity | 18:44 |
| anteaya | something is doing something | 18:44 |
| jeblair | yeah, the status page is extant | 18:45 |
| pleia2 | was I search for the wrong thing in ps? | 18:45 |
| fungi | weiiiird | 18:45 |
| jeblair | is that an apache cache or something? | 18:45 |
| *** e0ne has joined #openstack-infra | 18:45 | |
| pleia2 | I think it must be | 18:45 |
| jeblair | ah yep | 18:45 |
| fungi | okay, panic complete | 18:45 |
| jeblair | http://zuul.openstack.org/status.json returned data, until i shift-reloaded it | 18:46 |
| jeblair | i have not seen it do that before | 18:46 |
| fungi | i'm about to rerun the playbook with the culmination of our planned edits for the two remaining repos | 18:46 |
| anteaya | fungi: lets see what happens this time | 18:46 |
| pleia2 | sounds good | 18:46 |
| jeblair | fungi: you get to test the updates! yay! | 18:46 |
| fungi | for the record, this is the updated playbook http://paste.openstack.org/show/566143 | 18:47 |
| jeblair | fungi: wait a sec | 18:47 |
| fungi | i am holding | 18:47 |
| jeblair | fungi: how did you fix the trailing / problem? | 18:47 |
| jeblair | oh i think i see it | 18:47 |
| fungi | injecting a / between the parameters in the format string | 18:47 |
| jeblair | yep | 18:47 |
| fungi | where there was none previously | 18:48 |
| fungi | done for both review and git0* | 18:48 |
| jeblair | fungi: playbook lgtm. | 18:48 |
| fungi | it's running now | 18:49 |
| fungi | no failures! | 18:49 |
| pleia2 | \o/ | 18:49 |
| anteaya | yay | 18:49 |
| fungi | the output was http://paste.openstack.org/show/566144 | 18:50 |
| anteaya | nice | 18:51 |
| *** jkilpatr has joined #openstack-infra | 18:52 | |
| fungi | filesystem changes on review.o.o lgtm now. checking db next | 18:52 |
| anteaya | yay | 18:52 |
| anteaya | when is the best time to fix the oopsie from last rename window? | 18:55 |
| fungi | i already did | 18:55 |
| anteaya | awesome, thank you | 18:55 |
| *** dprince has quit IRC | 18:55 | |
| *** kushal has joined #openstack-infra | 18:55 | |
| fungi | database looks good... http://paste.openstack.org/show/566145 | 18:56 |
| *** e0ne has quit IRC | 18:56 | |
| *** Thelo_ has joined #openstack-infra | 18:56 | |
| *** dprince has joined #openstack-infra | 18:56 | |
| fungi | checking files on git servers now | 18:56 |
| pleia2 | want to divide and conquer git servers? | 18:56 |
| *** kushal has quit IRC | 18:56 | |
| fungi | i was just going to spot check a couple | 18:57 |
| pleia2 | k | 18:57 |
| fungi | but feel free to check some too | 18:57 |
| fungi | i trust that if ansible says it ran something on each server, it did. i'm more interested that the thing it ran on at least one server was the thing we wanted | 18:57 |
| *** e0ne has joined #openstack-infra | 18:57 | |
| fungi | looks right to me! | 18:58 |
| pleia2 | spot check on 6,7,9 lgtm | 18:58 |
| pleia2 | s/9/8 | 18:59 |
| pleia2 | numbers | 18:59 |
| fungi | i checked 2 and 5 | 18:59 |
| pleia2 | time to bring zuul back? | 18:59 |
| fungi | i think so, yes | 18:59 |
| *** sandanar has quit IRC | 18:59 | |
| pleia2 | ok, doing so | 18:59 |
| pleia2 | reenqueuing gate | 19:00 |
| pleia2 | now check | 19:00 |
| fungi | once you rename gerrit groups, i'll start the account reindex | 19:00 |
| fungi | i worry that doing those out of order might not work correctly | 19:00 |
| pleia2 | ok, all should be reenqueued | 19:00 |
| pleia2 | renaming now | 19:00 |
| fungi | though would likely be fine | 19:01 |
| pleia2 | renames done | 19:01 |
| fungi | since both are being handled within gerrit anyway | 19:01 |
| fungi | okay, getting reindexing going | 19:01 |
| *** sdake has quit IRC | 19:02 | |
| fungi | um, gerrit index start --force accounts seems to also start indexing changes | 19:02 |
| anteaya | oh | 19:02 |
| anteaya | zaro: ^^ | 19:03 |
| zaro | what's the gerrrit queue say? | 19:03 |
| fungi | at least gerrit show-queue returns a ton of entries for "Index all changes of project openstack/..." | 19:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Make sure interface_name is validate for network files https://review.openstack.org/365097 | 19:03 |
| pabelanger | mordred: clarkb: ^ I think that is the correct approach | 19:03 |
| *** yuanying has quit IRC | 19:03 | |
| zaro | hmm, well i didn't notice that when i ran account reindex but probably only because i only changed accounts and not any changes that go along with it. | 19:04 |
| fungi | presumably that's something we can try to recreate on review-dev | 19:04 |
| fungi | anyway, 4379 tasks still in the queue when i checked again just now | 19:05 |
| pleia2 | where did it start? | 19:05 |
| fungi | 0b201ba1 19:02:09.162 Index all changes of project openstack-infra/project-config | 19:06 |
| fungi | i think | 19:06 |
| zaro | it should list all repos | 19:06 |
| fungi | yeah, it does | 19:06 |
| fungi | i mean, i didn't count them all, but the rowcount makes it pretty likely this is all repos | 19:07 |
| *** Jeffrey4l has quit IRC | 19:07 | |
| zaro | is one of the name change ones in the queue? | 19:07 |
| anteaya | do we have any idea how long reindexing is expected to take? | 19:07 |
| fungi | zaro: i'll find out | 19:07 |
| zaro | i think same amount of time as offline index | 19:07 |
| pleia2 | should we ok and do another alert about gerrit being slow? | 19:08 |
| anteaya | 4-5 hours | 19:08 |
| fungi | 9a2e9b97 waiting .... 19:02:09.170 Index all changes of project openstack/karbor | 19:08 |
| fungi | zaro: so, yes | 19:08 |
| anteaya | are there any actions gerrit can't perform successfully until reindex is complete? | 19:08 |
| zaro | ok, once it's off the queue you can do a gerrit search for changes and it should appear. | 19:08 |
| fungi | and no matches for "smaug" in the output anywhere | 19:08 |
| *** sasi_ has quit IRC | 19:09 | |
| *** tongli has joined #openstack-infra | 19:09 | |
| *** tongli has quit IRC | 19:09 | |
| fungi | so anyway, i guess we'll count items 6.2 and 6.3 done | 19:09 |
| zaro | anteaya: no, should work as normal except probably slow | 19:09 |
| anteaya | zaro: okay thank you | 19:09 |
| *** tongli has joined #openstack-infra | 19:09 | |
| *** tongli has quit IRC | 19:09 | |
| zaro | you just can't work on the renamed projects until it's done indexing them | 19:09 |
| anteaya | pleia2: I'm fine with okay and gerrit is slow for about 5 hours | 19:09 |
| anteaya | zaro: that's what I was wondering | 19:09 |
| pleia2 | do I need to OK before another alert? | 19:09 |
| fungi | there are still a few steps (>=7) with no assignees. any latecomers intending to pick any of those up or should i grab some? | 19:09 |
| *** tongli has joined #openstack-infra | 19:09 | |
| *** yuval has joined #openstack-infra | 19:10 | |
| anteaya | pleia2: I think so | 19:10 |
| fungi | pleia2: no need to okay before doing a second alert but i vote we just okay with that message instead | 19:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests https://review.openstack.org/365099 | 19:10 |
| *** ddieterly is now known as ddieterly[away] | 19:10 | |
| fungi | yeah | 19:10 |
| pleia2 | fungi: like: #status ok Gerrit is completing an online re-index, you may encounter slowness until it is complete | 19:10 |
| fungi | yep! | 19:10 |
| pleia2 | #status ok Gerrit is completing an online re-index, you may encounter slowness until it is complete | 19:10 |
| openstackstatus | pleia2: sending ok | 19:10 |
| fungi | that will set everyone's channel topics back to normal, but still echo that message to the channels | 19:11 |
| * pleia2 nods | 19:11 | |
| fungi | i think that's a good compromise for mild degradation but systems online and functioning | 19:11 |
| anteaya | hongbin: best not to work on the renamed projects until online reindexing is complete, guessing at about 5 hours | 19:11 |
| hongbin | anteaya: ack | 19:11 |
| anteaya | yuval: same for karbor ^^ | 19:11 |
| anteaya | hongbin: thank you | 19:11 |
| pleia2 | if we're good to do the github renames, I can pop over there and learn how to do that | 19:11 |
| jeblair | zaro: what happens if you do? | 19:11 |
| yuval | anteaya: :) | 19:12 |
| anteaya | yuval: thank you | 19:12 |
| yuval | anteaya: thank you for the hard work! | 19:12 |
| mordred | pleia2: tl;dr "clickety clickety click. click click. clicketty click." | 19:12 |
| pleia2 | mordred: figured :) | 19:13 |
| jeblair | anteaya, zaro: if i'm understanding zaro correctly -- nothing *bad* will happen if people do 'work' on changes in the project, it's just that searches won't work correctly | 19:13 |
| anteaya | yuval: I'm a bystander, fungi and pleia2 did the heavy lifting with cameos by jeblair and mordred :) | 19:13 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 19:13 | |
| -openstackstatus- NOTICE: Gerrit is completing an online re-index, you may encounter slowness until it is complete | 19:13 | |
| anteaya | jeblair: ah okay, thanks for clarifying | 19:13 |
| zaro | jeblair: i'm guessing that it will work, just can't search | 19:13 |
| anteaya | yeah searches not working is not something I would have worried about | 19:13 |
| jeblair | (so, i think it should be okay to, say, upload changes, perform reviews, etc -- just don't expect search queries, dashboards, etc to work as expected while running. zaro should confirm though. :) | 19:13 |
| anteaya | zaro: ah okay thanks | 19:14 |
| anteaya | hongbin: yuval ^^ | 19:14 |
| yuval | anteaya: ofcourse, I meant it in the plural context | 19:14 |
| fungi | pleia2: yeah, github renaming should be fine now | 19:14 |
| anteaya | yuval: thank you :) | 19:14 |
| pleia2 | on it | 19:14 |
| fungi | jeblair: zaro: i wonder if zuul will get confused query responses until reindexing is complete | 19:14 |
| mordred | fungi: that'll be fun to learn | 19:14 |
| fungi | for those repos, specifically | 19:15 |
| zaro | fungi: queries for thos projects won't work so i'm assume zuul would get confused. | 19:15 |
| fungi | i mean, chances are it would just fail to find a change and so wouldn't enqueue it | 19:15 |
| anteaya | hongbin: yuval if you do see anything funny in gerrit with your repos, please do share, we are curious | 19:15 |
| yuval | anteaya: anything specific I should be looking for? | 19:16 |
| jeblair | fungi, mordred: yep. i expect it to not work but not break. | 19:16 |
| fungi | and might also fail to enqueue changes with a cross-repo depends-on to changes in one of the renamed repos | 19:16 |
| zaro | fungi: curious, what's the cpu util? | 19:16 |
| fungi | zaro: great q! checking now | 19:16 |
| fungi | zaro: though you can also look in cacti.o.o | 19:16 |
| anteaya | yuval: behaviour that doesn't respond as normal, patch urls, comments on patches, searches | 19:16 |
| *** shashank_hegde has quit IRC | 19:16 | |
| openstackstatus | pleia2: finished sending ok | 19:16 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 19:16 |
| jeblair | not as high as a gc loop, apparently. | 19:17 |
| fungi | definitely looks like it's doing stuff, but not slammed | 19:17 |
| pleia2 | github renames done, probably :) | 19:17 |
| anteaya | pleia2: yay | 19:17 |
| pleia2 | (got the two other smaugs too) | 19:17 |
| hongbin | anteaya: search didn't work, patches are gone under the "Outgoing reviews" session. That is what I observed | 19:17 |
| *** ddieterly[away] is now known as ddieterly | 19:17 | |
| *** Nakato has quit IRC | 19:17 | |
| anteaya | hongbin: thanks for sharing | 19:18 |
| * pleia2 patpat former dragons | 19:18 | |
| *** spzala has joined #openstack-infra | 19:18 | |
| zaro | that's surprising. i saw very different behavior on review-dev | 19:18 |
| zaro | i guess tuning on review probably helped this | 19:18 |
| anteaya | zaro: what did you see? | 19:18 |
| *** Nakato has joined #openstack-infra | 19:19 | |
| zaro | CPU utilization like at over 100% | 19:19 |
| pleia2 | fungi: anyone on #7? | 19:19 |
| fungi | pleia2: i haven't picked up any new tasks yet | 19:19 |
| fungi | i was taking a minor break | 19:19 |
| jeblair | fungi: are there more re-index commands to run? | 19:20 |
| fungi | i'm happy to approve those changes, but might be better if someone who actually read them did that | 19:20 |
| fungi | jeblair: apparently not | 19:20 |
| anteaya | when do we merge the project-config patches? | 19:20 |
| jeblair | fungi: oh, now i see the rest of the ones in the queue :) | 19:20 |
| fungi | |grep -c 'Index all changes of project' is reporting 395 to go | 19:21 |
| zaro | renamed projects off queue yet? | 19:21 |
| jeblair | zaro: did that 'sort projects by number of changes for reindexing' patch land? | 19:22 |
| fungi | note the ones it says are currently active indexing: cinder, horizon, keystone, heat, nova, openstack-manuals, neutron... it's going to be a while | 19:22 |
| fungi | jeblair: i don't think we had that change cherry-picked | 19:22 |
| anteaya | when do the project-config patches merge? | 19:22 |
| zaro | jeblair: it landed upstream but cannot be easily backported | 19:22 |
| jeblair | fungi, zaro: well, interestingly, nova ended up near the beginning for some reason | 19:22 |
| jeblair | so that's nice :) | 19:22 |
| fungi | anteaya: when someone approves them. any time now is fine | 19:22 |
| *** spzala has quit IRC | 19:23 | |
| anteaya | okay, shall I go ahead? | 19:23 |
| fungi | anteaya: be my guest! | 19:23 |
| jeblair | maybe something about how '--accounts' ended up triggering them | 19:23 |
| anteaya | okay | 19:23 |
| zaro | hmm, that's odd maybe i'm not remember correctly. lemme check again | 19:23 |
| dhellmann | fungi : is it safe to start processing releases again? we have one bug fix in a lib | 19:23 |
| fungi | dhellmann: it should be safe as long as it's not for one of the projects we just renamed | 19:24 |
| dhellmann | os-vif, so I don't think so | 19:24 |
| fungi | nope, that one's fair game | 19:24 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 19:24 |
| anteaya | approved | 19:24 |
| dhellmann | great, thanks | 19:24 |
| pleia2 | thanks anteaya | 19:24 |
| anteaya | fungi: what should happen with the requirements change at this point? https://review.openstack.org/#/c/353371 | 19:24 |
| fungi | anteaya: i'll see how much of it is safe | 19:25 |
| anteaya | pleia2: I just noticed merge the patches wasn't on the list of items to do | 19:25 |
| anteaya | fungi: thanks | 19:25 |
| *** ilyashakhat_mobi has joined #openstack-infra | 19:25 | |
| zaro | jeblair: nope, it wasn't backported. probably indexes them in random order? | 19:25 |
| *** tkelsey has joined #openstack-infra | 19:25 | |
| pleia2 | anteaya: I assumed that was step 7 | 19:25 |
| *** sarob has quit IRC | 19:25 | |
| fungi | anteaya: i believe 353371 can't be merged until there is a release of that lib | 19:26 |
| anteaya | pleia2: oh | 19:26 |
| pleia2 | but I could be mistaken :) | 19:26 |
| anteaya | it says puppet configuration | 19:26 |
| rcarrillocruz | bkero: i've used it only locally, flawless... | 19:26 |
| fungi | anteaya: note tonyb's inline comments | 19:26 |
| pleia2 | yeah | 19:26 |
| rcarrillocruz | i need to test it from internet, doing nat on my router | 19:26 |
| rcarrillocruz | one thing i see that doesn't work is splitting screens with gnu screen | 19:27 |
| anteaya | fungi: okay thank you, I read that but am grateful you also looked | 19:27 |
| rcarrillocruz | the splits are not seen on the browser | 19:27 |
| rcarrillocruz | i have to try with tmux (i'm a gnu screen user really) | 19:27 |
| fungi | anteaya: basically that change only adds the new name, but won't work until there's a release of it on pypi | 19:27 |
| anteaya | fungi: ah thank you, now I understand | 19:27 |
| anteaya | should step 7 have said project config patches, rather than puppet configuration? | 19:27 |
| fungi | anteaya: then it will pass testing and be able to merge, then requirements updates propagate out with the new name, then a change can be submitted to remove the old name from those files | 19:27 |
| anteaya | ah, okay thank you, sorry I'm slow | 19:28 |
| fungi | anteaya: yeah, i think it's poorly named in our instructions (from back when this was all in openstack-infra/config probably). feel free to leave an inline comment on my process update | 19:28 |
| fungi | once i have a chance to digest this maintenance i'll have a bunch of other updates to make in that change anyway | 19:29 |
| rcarrillocruz | how's gerrit rename going? been off all my afternoon/evening | 19:29 |
| fungi | rcarrillocruz: you missed a bunch of ansible playbook fun | 19:29 |
| rcarrillocruz | huh? | 19:29 |
| *** spzala has joined #openstack-infra | 19:29 | |
| rcarrillocruz | which playbook | 19:29 |
| *** tkelsey has quit IRC | 19:30 | |
| fungi | the project renaming playbook | 19:30 |
| rcarrillocruz | you ran the one that yolanda had in review or something | 19:30 |
| rcarrillocruz | ? | 19:30 |
| anteaya | fungi: done | 19:30 |
| fungi | this was our first attempt to run it in production, yes | 19:30 |
| rcarrillocruz | was it good? | 19:30 |
| fungi | it needs lots of edits, but i think it will be more useful in the future | 19:30 |
| zaro | luckily, the online index part wasn't as exciting | 19:30 |
| pleia2 | we didn't destroy openstack | 19:30 |
| pleia2 | so, success | 19:30 |
| fungi | pleia2: there's always next week! | 19:30 |
| *** vhosakot has quit IRC | 19:31 | |
| pleia2 | hehe | 19:31 |
| rcarrillocruz | yolanda is awesome, wasn't expecting less | 19:31 |
| rcarrillocruz | :-) | 19:31 |
| *** vhosakot has joined #openstack-infra | 19:31 | |
| anteaya | yolanda is awesome ++ | 19:31 |
| pleia2 | aaand I am not lunching because my lunch date is also in ops and she's stuck with an emergency | 19:31 |
| pleia2 | (we may never actually meet) | 19:31 |
| pleia2 | hehe | 19:31 |
| fungi | rcarrillocruz: maybe star 365067 and pitch in after i make some updates in a new patchset based on our findings from today's maintenance | 19:32 |
| rcarrillocruz | pabelanger: mind reviewing https://review.openstack.org/#/c/364868/ and https://review.openstack.org/#/c/364871/ ? | 19:32 |
| anteaya | pleia2: ha ha ha | 19:32 |
| rcarrillocruz | also clarkb , we chatted about it ^ | 19:32 |
| rcarrillocruz | fungi: sure, let me look | 19:32 |
| anteaya | pleia2: glad to have your continued presence | 19:32 |
| *** gouthamr has quit IRC | 19:33 | |
| *** dkehn_ has quit IRC | 19:34 | |
| anteaya | stevebaker: still want to chat? | 19:34 |
| *** spzala has quit IRC | 19:34 | |
| mordred | "pleia2 | we didn't destroy openstack" <-- that's really all I ever hope for on any given day | 19:35 |
| *** dkehn has quit IRC | 19:35 | |
| pleia2 | :D | 19:35 |
| anteaya | in other news I just learned that working groups have presidents | 19:35 |
| anteaya | http://lists.openstack.org/pipermail/user-committee/2016-September/001215.html | 19:35 |
| *** ddieterly is now known as ddieterly[away] | 19:35 | |
| anteaya | I did not know this | 19:35 |
| mordred | anteaya: do they have parliments or congresses? | 19:35 |
| anteaya | my eyes have been opened | 19:35 |
| *** spzala has joined #openstack-infra | 19:36 | |
| anteaya | I don't know but now I am curious | 19:36 |
| *** thorst has quit IRC | 19:36 | |
| fungi | sokath, his eyes uncovered! | 19:36 |
| *** dkehn has joined #openstack-infra | 19:36 | |
| * anteaya looks up the sokath reference | 19:37 | |
| fungi | sorry, terrible st:tng episode | 19:37 |
| *** thorst has joined #openstack-infra | 19:37 | |
| *** markvoelker has quit IRC | 19:37 | |
| fungi | popped into my head when you said "my eyes have been opened" | 19:37 |
| mordred | fungi: s/terrible/amazing/ | 19:37 |
| jeblair | fungi when the disk filled | 19:37 |
| mordred | jeblair, his zuul full | 19:38 |
| *** yuval has quit IRC | 19:38 | |
| fungi | last night it was jhesketh when the disk filled | 19:38 |
| jeblair | fungi, and jhesketh, at tanagra | 19:38 |
| openstackgerrit | Merged openstack-infra/project-config: Rename os-failures to os-faults https://review.openstack.org/364556 | 19:39 |
| *** thorst_ has joined #openstack-infra | 19:39 | |
| pleia2 | jeblair: haha | 19:39 |
| *** vhosakot has quit IRC | 19:39 | |
| *** roxanaghe has joined #openstack-infra | 19:39 | |
| *** ddieterly[away] is now known as ddieterly | 19:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Zun: rename higgins to zun (2) https://review.openstack.org/329247 | 19:40 |
| rcarrillocruz | one dreams | 19:40 |
| openstackgerrit | Merged openstack-infra/project-config: Rename Smaug to Karbor https://review.openstack.org/353304 | 19:40 |
| rcarrillocruz | when we get the finglonger thing | 19:40 |
| rcarrillocruz | we can schedule this kind of things | 19:40 |
| rcarrillocruz | we sit | 19:40 |
| rcarrillocruz | relax | 19:40 |
| rcarrillocruz | and get the results feeded into firehose | 19:40 |
| mordred | a bug in that theoretical language seems to be that it would be rather difficult to tell the story of darmok and jalad at tenagra in the first place if the langauge only works by literary references to other events | 19:40 |
| *** spzala has quit IRC | 19:40 | |
| rcarrillocruz | mtreinish: ^ | 19:40 |
| rcarrillocruz | one dreams! | 19:40 |
| fungi | anteaya: was that all of them merged just now? | 19:40 |
| anteaya | fungi: I'm reading the webpage on it | 19:40 |
| jeblair | i'm going to stand on the 'amazing' side of the line for this episode too... my appreciation for it only increased when i learned there were societies on earth where you have to know the full shared history of proverbs in order to effectively communicate (at least, at a high level) | 19:41 |
| anteaya | fungi: yes, 3 patches, those were them | 19:41 |
| *** thorst has quit IRC | 19:41 | |
| mordred | jeblair: ++ | 19:41 |
| jeblair | mordred: well, i mean, that's what the tachyon langugae compensator is for | 19:41 |
| pabelanger | mordred: clarkb: The reason we didn't catch https://review.openstack.org/#/c/365097/ with glean testing, is because systemd will run glean per interface, where testing does all. We should fix that with a follow patch | 19:41 |
| fungi | jeblair: yes, it's a good episode, i've just seen it too many times for it to be anything other than annoying at this point | 19:41 |
| pleia2 | rcarrillocruz: yes, that's totally how it will be ;) | 19:41 |
| mordred | jeblair: oh. right. I forgot | 19:41 |
| jeblair | fungi: it does go on a bit | 19:42 |
| mordred | pleia2: ++ | 19:42 |
| mordred | gah | 19:42 |
| *** dkehn has quit IRC | 19:42 | |
| mordred | I mean - pleia2 ++ for sure - but I was going for pabelanger ++ | 19:42 |
| pleia2 | all the ++s | 19:42 |
| fungi | anteaya: so it does look to me like they've all merged now | 19:43 |
| fungi | i'm going ahead with reenabling puppet in that case | 19:43 |
| pleia2 | yep | 19:43 |
| rcarrillocruz | mordred: one thing i've noticed with glean and the handling with vlans, which i believe it was not always the case, is that the raw interface gets configured the interface as 'inet manual' , i.e. it doesn't pull from dhcp | 19:43 |
| *** asselin__ has joined #openstack-infra | 19:43 | |
| rcarrillocruz | i believe it would be better if we had the raw interface also do a dhcp | 19:43 |
| *** links has joined #openstack-infra | 19:43 | |
| *** thorst_ has quit IRC | 19:43 | |
| rcarrillocruz | that way if we ever get lose access to the public ip range, we can always ssh from the baremetal node to the private interfaces | 19:44 |
| rcarrillocruz | without the need to go thru ilo textcons, etc | 19:44 |
| fungi | woah, we're down to just 4 projects still reindexing | 19:44 |
| fungi | zaro: ^ | 19:44 |
| fungi | unsurprisingly, they are: neutron, nova, openstack-manuals, tempest | 19:44 |
| *** Na3iL has joined #openstack-infra | 19:44 | |
| zaro | ok, so i assume you can check that querying the renamed project changes are working now | 19:44 |
| *** asselin_ has quit IRC | 19:45 | |
| anteaya | fungi: yes, all merged | 19:45 |
| zaro | you can check logs for 'INFO com.google.gerrit.lucene.OnlineReindexer :' messages. | 19:45 |
| zaro | fungi: when done you'll get something like 'INFO com.google.gerrit.lucene.OnlineReindexer : Reindex to version 14 complete' | 19:46 |
| *** dkehn_ has joined #openstack-infra | 19:46 | |
| *** jklare_ has quit IRC | 19:46 | |
| *** jklare has joined #openstack-infra | 19:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Make builder recognize manual build requests https://review.openstack.org/365112 | 19:46 |
| zaro | yep, i am able to query changes for openstack/zun | 19:47 |
| fungi | zaro: indeed, right now all we have is: | 19:47 |
| fungi | [2016-09-02 19:02:09,155] INFO com.google.gerrit.lucene.OnlineReindexer : Starting online reindex from schema version 14 to 14 | 19:47 |
| fungi | that looks like when i sent the reindex api command | 19:47 |
| *** amitgandhinz has quit IRC | 19:48 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs https://review.openstack.org/365114 | 19:48 |
| *** shardy has joined #openstack-infra | 19:49 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Make builder recognize manual build requests https://review.openstack.org/365112 | 19:49 |
| *** dkehn has joined #openstack-infra | 19:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Make sure interface_name is validate for network files https://review.openstack.org/365097 | 19:51 |
| fungi | and now we're down to just neutron and nova | 19:51 |
| pabelanger | mordred: clarkb: okay, I think that is everything^ until we review, release glean. We don't have ubuntu-xenial in rax | 19:51 |
| fungi | ansible should kick off to run manage-projects in about 8 minutes | 19:51 |
| *** ddieterly is now known as ddieterly[away] | 19:51 | |
| *** annegentle has quit IRC | 19:51 | |
| *** pblaho has quit IRC | 19:53 | |
| anteaya | and it is labour day weekend | 19:53 |
| mordred | anteaya: not to be pedantic ... | 19:54 |
| pleia2 | I know where this is going | 19:54 |
| mordred | anteaya: but in murica, it's labor day weekend ... | 19:54 |
| pleia2 | hehe | 19:54 |
| anteaya | so it is | 19:54 |
| anteaya | I was talking about Canada's labour day weekend | 19:54 |
| mordred | ah! | 19:54 |
| mordred | so you were | 19:54 |
| pleia2 | is that this weekend too? | 19:54 |
| anteaya | indeed it is | 19:54 |
| pleia2 | neat | 19:54 |
| mordred | I'm so used to thanksgivings being on different days | 19:54 |
| anteaya | they are so | 19:54 |
| mordred | it would be cool if thanksgiving had two different spellings | 19:55 |
| fungi | wow, a labour day weekend and a labor day weekend on the same weekend | 19:55 |
| anteaya | no u in thanksgiving | 19:55 |
| mordred | anteaya: we could add one | 19:55 |
| *** madhuvishy is now known as madhuvishy|food | 19:55 | |
| anteaya | fungi: who would have thought | 19:55 |
| fungi | thanks gouging | 19:55 |
| mordred | anteaya: thauksgiving ? | 19:55 |
| pabelanger | thanksgiving in a month! | 19:55 |
| jeblair | thanksmordreding | 19:55 |
| mordred | ++ | 19:55 |
| anteaya | ha ha ha | 19:55 |
| pleia2 | jeblair: still no u | 19:55 |
| mordred | of course - we have boxing day sales in the US now | 19:55 |
| *** links has quit IRC | 19:55 | |
| mordred | so the homogenization continues | 19:55 |
| pabelanger | leifmadsen: brigden fair! ^ | 19:55 |
| anteaya | did you get boxing day from the uk via canada? | 19:56 |
| *** spzala has joined #openstack-infra | 19:56 | |
| mordred | we don't have boxing day | 19:56 |
| mordred | just sales on it | 19:56 |
| anteaya | ah | 19:56 |
| mordred | because selling things | 19:56 |
| anteaya | always need to have a sale | 19:56 |
| mordred | we'll adopt ANYTHING if it can be an excuse for a sale | 19:56 |
| mordred | yup | 19:56 |
| *** jcoufal has quit IRC | 19:56 | |
| *** gyee has joined #openstack-infra | 19:56 | |
| mordred | heck - we'd adopt the metric system if someone could link it to black friday | 19:56 |
| anteaya | ha ha ha | 19:57 |
| anteaya | someone should tell NASA | 19:57 |
| *** mriedem_afk is now known as mriedem | 19:57 | |
| anteaya | that's what it takes | 19:57 |
| * mordred points out that we _did_ adopt the metric system in two places ... coke and Coke | 19:57 | |
| anteaya | I don't understand that reference | 19:57 |
| mordred | coke is sold in grams and kilos. Coke is sold in liters | 19:57 |
| anteaya | ha ha ha | 19:58 |
| mordred | anteaya: cocaine and coca-cola | 19:58 |
| anteaya | okay now I'm caught up | 19:58 |
| anteaya | sorry | 19:58 |
| pleia2 | hehe | 19:58 |
| mordred | a 2-liter bottle is an extremely common thing in the US | 19:58 |
| mordred | and nobody seems to have a problem with it | 19:58 |
| anteaya | the druggies are always at the cutting edge of tech | 19:58 |
| mordred | anteaya: I'm guessing it's because cocaine is an import | 19:58 |
| anteaya | moar Coke | 19:58 |
| mordred | and doing the conversion would be a lot of work | 19:58 |
| anteaya | ha ha ha | 19:58 |
| anteaya | metric for the laxy | 19:59 |
| anteaya | lazy | 19:59 |
| mordred | whereas weed is almost always grown domestically, thus the continued selling in ounces and pounds | 19:59 |
| openstackgerrit | Merged openstack/os-client-config: Don't build releasenotes in normal docs build https://review.openstack.org/365053 | 19:59 |
| anteaya | ha ha ha | 19:59 |
| mordred | but yeah - good for coca-cola for shifting extremely sugary drugs to metric | 19:59 |
| mordred | I'm guessing it makes global commerce easier for them | 19:59 |
| anteaya | no way they did it as a public service | 19:59 |
| *** jcoufal has joined #openstack-infra | 20:00 | |
| mordred | nope | 20:00 |
| dstufft | medication is almost entirely done in metric too | 20:00 |
| mordred | I was just about to say that | 20:00 |
| anteaya | woooo | 20:00 |
| anteaya | inroads | 20:00 |
| anteaya | though I will say I'm hybrid | 20:01 |
| anteaya | tempeture is either | 20:01 |
| mordred | https://en.wikipedia.org/wiki/Metrication_in_the_United_States | 20:01 |
| anteaya | distances is km if longer than 50m and feet and inches for shorter distances | 20:01 |
| anteaya | if I am standing on a bathroom scale it is pounds | 20:01 |
| dstufft | I almost entirely use imperial, not because I think it's better, but because the relative sizes of things is stuck in my head as imperial | 20:02 |
| anteaya | baking can be done in ml/mg/g/kg or imperial | 20:02 |
| anteaya | liquid measure is cups or oz, rarely ml | 20:02 |
| pabelanger | mordred: clarkb: looks like https://review.openstack.org/#/c/293648/ is what broke rackspace, we must be using an old version of glean in the images that are still working | 20:03 |
| anteaya | 12 ft is 12 ft not 4m | 20:03 |
| anteaya | my height is feet and inches not cm and m | 20:03 |
| anteaya | dstufft: yeah, this is why I do temp either way | 20:04 |
| *** akshai has quit IRC | 20:04 | |
| *** openstackgerrit has quit IRC | 20:04 | |
| irtermite | pabelanger: why for you break rackspace? at least Rackspace Private Cloud is solid. ;) | 20:04 |
| anteaya | I always talk to my dad about the weather and he is in farenheit | 20:04 |
| *** openstackgerrit has joined #openstack-infra | 20:04 | |
| fungi | i do all of my baking in metric, mainly because it's convenient that 1ml or 1cc of water is ~ 1g | 20:04 |
| anteaya | fungi: yes | 20:05 |
| fungi | and much of my baking is proportional by weight to the water content | 20:05 |
| dstufft | I have to do math to figure out if 100kg is a lot or not but I know what 220lbs is :V | 20:05 |
| anteaya | my scales are metric, my measure devices are in cups most of them | 20:05 |
| irtermite | well, seeing as how 1m =/= 1yard, that statement is correct no matter how you look at it anteaya ;) | 20:05 |
| anteaya | dstufft: :) | 20:05 |
| mgagne | weather is celsius for us (canada), pool temperature is fahrenheit. :D | 20:05 |
| fungi | heh | 20:05 |
| fungi | that's a fun one | 20:05 |
| anteaya | irtermite: yep | 20:06 |
| anteaya | mgagne: you do pool temp in fahrenheit? | 20:06 |
| irtermite | mgagne: canada is weird to begin with, and that's why we love you, oh hat of the uS | 20:06 |
| anteaya | mgagne: where do you live? | 20:06 |
| mgagne | anteaya: montreal, canada | 20:06 |
| anteaya | mgagne: awesome | 20:06 |
| anteaya | mgagne: I'm in ontario | 20:06 |
| mgagne | =) | 20:06 |
| *** armax has joined #openstack-infra | 20:06 | |
| *** tqtran has quit IRC | 20:07 | |
| *** gouthamr has joined #openstack-infra | 20:07 | |
| anteaya | mgagne: I'll try to remember to ping next time I'm going to be in town | 20:07 |
| anteaya | like when I get EmilienM to come to montreal | 20:07 |
| mgagne | body temp is C too. 100F? ¯\_(ツ)_/¯ it's impossibru, you would be boiling :D | 20:07 |
| anteaya | I can do body temp either way | 20:07 |
| anteaya | starting learning lifeguard things and the first aid bits still used F | 20:08 |
| mgagne | :-/ | 20:08 |
| anteaya | when I started to learn lifeguard things | 20:08 |
| anteaya | long time ago | 20:08 |
| *** spzala has quit IRC | 20:10 | |
| anteaya | fungi: how much is left to reindex? | 20:12 |
| fungi | anteaya: still neutron and nova | 20:12 |
| fungi | dunno how much progress there is on them | 20:13 |
| anteaya | ah thanks | 20:13 |
| fungi | as long as we don't rename nova, or neutron (again), this online reindexing seems to be a great speed-up for us | 20:13 |
| anteaya | ha ha ha | 20:13 |
| anteaya | indeed | 20:13 |
| *** spzala has joined #openstack-infra | 20:14 | |
| anteaya | with added bonus we debugged the playbook plus tested it | 20:14 |
| anteaya | and I know better now what you need for your ansible config file | 20:14 |
| *** ddieterly[away] is now known as ddieterly | 20:14 | |
| anteaya | and updated workflow instructions patch | 20:15 |
| *** ilyashakhat_mobi has quit IRC | 20:15 | |
| *** roxanaghe has quit IRC | 20:15 | |
| *** shashank_hegde has joined #openstack-infra | 20:16 | |
| fungi | yep. we still need some .gitreview patches proposed to the 5 renamed repos | 20:17 |
| fungi | if anybody feels like pushing those up | 20:17 |
| pleia2 | ah, sure | 20:17 |
| dhellmann | clarkb : looking over my notes, I forgot to tell you that it was safe to go ahead with whatever image rebuilding work you were planning (which I think is separate from the gerrit maintenance). We're done with branches. | 20:17 |
| fungi | thanks pleia2! | 20:18 |
| fungi | i think i'm going to crack open a beer and finish some storm prep | 20:18 |
| pleia2 | good idea | 20:18 |
| anteaya | fungi: sounds good, what do you have left to do? | 20:18 |
| *** spzala has quit IRC | 20:18 | |
| fungi | anteaya: no idea--that's what i'm going to go find out next | 20:18 |
| anteaya | ah, good hunting | 20:19 |
| pleia2 | karbor | 20:20 |
| pleia2 | derp | 20:20 |
| pleia2 | window focus | 20:20 |
| pleia2 | (karbor was already done at some point btw) | 20:20 |
| *** kgiusti has left #openstack-infra | 20:20 | |
| *** jamesdenton has quit IRC | 20:21 | |
| fungi | they may be on the ball and already proposing these themselves | 20:21 |
| pleia2 | python-karborclient too | 20:21 |
| fungi | i'm betting karbor-dashboard as well then | 20:21 |
| *** e0ne has quit IRC | 20:21 | |
| pleia2 | seems not | 20:21 |
| pleia2 | gotta keep me on my toes | 20:22 |
| fungi | i so wanted to shorten that one to kardboard | 20:22 |
| *** jcoufal_ has joined #openstack-infra | 20:22 | |
| anteaya | ha ha ha | 20:22 |
| anteaya | clarkb: did you see tim bell was asking about running elastic search on the operators mailing list? | 20:23 |
| anteaya | jesusaur: ^^ | 20:23 |
| pleia2 | alright, all done | 20:23 |
| pabelanger | clarkb: I reverted to a working ubuntu-xenial image in rackspace, it is using an older version of glean that doesn't have the bug | 20:23 |
| *** jcoufal has quit IRC | 20:23 | |
| anteaya | pleia2: thank you | 20:24 |
| clarkb | pabelanger: oj | 20:24 |
| clarkb | er ok | 20:24 |
| pabelanger | clarkb: this is also likely why we cannot launch nodes in rax-iad right now too... glean is broken | 20:24 |
| anteaya | clarkb: http://lists.openstack.org/pipermail/openstack-operators/2016-September/011461.html | 20:24 |
| *** eharney has quit IRC | 20:25 | |
| EmilienM | anteaya: September 13 and 14 i'll be in MTL | 20:25 |
| EmilienM | anteaya: we have an OpenStack Meetup | 20:25 |
| EmilienM | anteaya: I bought train tickets today, feel free to join | 20:25 |
| anteaya | EmilienM: awesome, thank you :) | 20:26 |
| anteaya | I'll see which way the wind blows for those days | 20:27 |
| fungi | tristanC: tonyb: that deathclock on the election page is an awesome feature. do the events advance automatically or do you have to merge changes to switch to the next current event? | 20:27 |
| anteaya | could stand some good monteal food for a bit :) | 20:27 |
| anteaya | EmilienM: does the meetup have a url? | 20:29 |
| *** shashank_hegde has quit IRC | 20:29 | |
| prometheanfire | hi, we (requirements) good to do the smaug migration https://review.openstack.org/353305 | 20:29 |
| prometheanfire | and https://review.openstack.org/353371 | 20:30 |
| anteaya | prometheanfire: are you asking or announcing? | 20:30 |
| pabelanger | clarkb: We might as well update the upstart script for glean too while we are at it. I'm seeing some EOFErrors on them too | 20:30 |
| prometheanfire | asking | 20:30 |
| *** esikachev has quit IRC | 20:30 | |
| prometheanfire | forgot the ? | 20:30 |
| fungi | prometheanfire: 353305 should be safe at any time | 20:30 |
| fungi | prometheanfire: i believe 353371 needs to wait until python-karborclient has a release on pypi | 20:31 |
| prometheanfire | because of the depends | 20:31 |
| prometheanfire | k | 20:31 |
| fungi | since requirements jobs are going to try to install everything listed in the global reqs and fail on that addition | 20:32 |
| prometheanfire | I think we have to wait on the release for all of it | 20:32 |
| prometheanfire | unless karbor (and dashboard) have been published already | 20:32 |
| *** Na3iL has quit IRC | 20:32 | |
| fungi | prometheanfire: the projects.txt file should be fine now... it's only used to determine which projects get requirements update proposals submitted to them i think | 20:32 |
| prometheanfire | ah, right | 20:33 |
| *** tongli has quit IRC | 20:33 | |
| prometheanfire | not all there today | 20:33 |
| fungi | and if 353305 isn't merged, requirements proposals are going to fail (perhaps at best skip the old repos since it can no longer check them out, i forget if it's been made that robust yet) | 20:33 |
| prometheanfire | and yes, that's all that it's used for | 20:34 |
| *** gouthamr has quit IRC | 20:34 | |
| anteaya | fungi: I missed a system config change for smaug to karbor: https://review.openstack.org/#/c/353309/2 cc pleia2 | 20:34 |
| prometheanfire | so the bot knows what to submit to | 20:34 |
| fungi | anteaya: oh, i expect they want 353309 to coincide with a channel move | 20:34 |
| EmilienM | anteaya: I'm not sure :( | 20:35 |
| anteaya | EmilienM: okay thanks | 20:35 |
| EmilienM | anteaya: montrealopenstack.org but it's not updated :( | 20:35 |
| fungi | anteaya: anyway, if they're okay having channel logs cut over from the old channel from the new one right now, then i'm happy to approve that | 20:35 |
| fungi | there aren't any meetings at this time on a friday afaik | 20:35 |
| anteaya | the channel is operative | 20:35 |
| fungi | i don't have bandwidth currently to help them set up channel forwarding from old to new or anything like that at the moment though | 20:36 |
| anteaya | for the channel info it says Flags : GUARD | 20:37 |
| anteaya | I asked in #openstack-karbor if someone could join us and tell us what they want | 20:38 |
| anteaya | based on the names in the channel I'm guessing for many of them it is early saturday morning for them | 20:39 |
| jesusaur | anteaya: i hadn't see that email, thanks for the poke | 20:39 |
| anteaya | jesusaur: sure, I'm feeling tim would enjoy discussing things with you | 20:40 |
| jesusaur | s/see/seen | 20:40 |
| anteaya | heap size and all that | 20:40 |
| *** askb has joined #openstack-infra | 20:41 | |
| *** tonytan_brb has quit IRC | 20:42 | |
| anteaya | EmilienM: indeed the website needs a bit of a refresh | 20:42 |
| EmilienM | anteaya: only a bit? | 20:42 |
| anteaya | EmilienM: is the meetup two days or that the meetup occurs when you are there for those two days | 20:42 |
| anteaya | EmilienM: I'm trying to be nice, hey it exists | 20:42 |
| * anteaya learned extant today, thanks jeblair | 20:43 | |
| EmilienM | anteaya: 13th evening only | 20:43 |
| anteaya | EmilienM: ah thank you | 20:43 |
| EmilienM | it will be from 6 to 8 something like this | 20:43 |
| anteaya | EmilienM: sitting in an office other than that for you? | 20:43 |
| anteaya | EmilienM: awesome thanks | 20:43 |
| EmilienM | anteaya: I guess you can come, I'll do visitor badge :) | 20:43 |
| anteaya | thank you :) | 20:43 |
| anteaya | let me think about it | 20:43 |
| anteaya | been very busy with home things the last two weeks | 20:44 |
| anteaya | haven't thought that far ahead yeat | 20:44 |
| anteaya | yet | 20:44 |
| *** salv-orlando has joined #openstack-infra | 20:44 | |
| openstackgerrit | Merged openstack-infra/system-config: Add images profile for openstackci oscc clouds and add it to infracloud https://review.openstack.org/364826 | 20:45 |
| *** shashank_hegde has joined #openstack-infra | 20:46 | |
| *** thorst has joined #openstack-infra | 20:46 | |
| *** markvoelker has joined #openstack-infra | 20:47 | |
| anteaya | think I'll weekend now | 20:47 |
| pleia2 | enjoy, anteaya | 20:48 |
| pleia2 | thanks for your help | 20:48 |
| *** amitgandhinz has joined #openstack-infra | 20:48 | |
| *** matt-borland has quit IRC | 20:48 | |
| *** trown is now known as trown|outtypewww | 20:49 | |
| *** shashank_hegde has quit IRC | 20:49 | |
| *** AnarchyAo has quit IRC | 20:53 | |
| *** eranrom has quit IRC | 20:53 | |
| *** amitgandhinz has quit IRC | 20:54 | |
| *** tqtran has joined #openstack-infra | 20:54 | |
| *** salv-orlando has quit IRC | 20:54 | |
| *** Goneri has quit IRC | 20:55 | |
| *** piet has quit IRC | 20:56 | |
| jesusaur | the elasticsearch data is backed on cinder volumes, right? cacti only shows disk usage for / | 20:58 |
| *** ddieterly is now known as ddieterly[away] | 20:58 | |
| *** fguillot has quit IRC | 20:59 | |
| *** e0ne has joined #openstack-infra | 21:01 | |
| *** e0ne has quit IRC | 21:03 | |
| *** rajinir has quit IRC | 21:05 | |
| fungi | jesusaur: yep | 21:06 |
| fungi | maybe net-snmp is having trouble finding those filesystems | 21:06 |
| *** eharney has joined #openstack-infra | 21:07 | |
| *** salv-orlando has joined #openstack-infra | 21:08 | |
| *** cardeois has quit IRC | 21:09 | |
| raddaoui | pabelanger, fungi, clarkb networking is fixed as you wanted, mirror can get public ipv4 address assinged directly | 21:09 |
| fungi | raddaoui: oh, that was fast! | 21:10 |
| fungi | thanks! | 21:10 |
| mrhillsman | raddaoui has done a great job seeing this through :) | 21:11 |
| *** roxanaghe has joined #openstack-infra | 21:11 | |
| *** madhuvishy|food is now known as madhuvishy | 21:11 | |
| raddaoui | yeah I didn't think it will be that fast as well | 21:11 |
| *** raildo has quit IRC | 21:11 | |
| raddaoui | thx mrhillsman :) | 21:12 |
| mrhillsman | man thank you lol | 21:12 |
| mrhillsman | making my job 200% easier all the time | 21:12 |
| pabelanger | mordred: clarkb: okay, have glean testing fixed. will push something up later tonight | 21:13 |
| openstackgerrit | Merged openstack/os-client-config: Update reno for stable/newton https://review.openstack.org/364920 | 21:13 |
| *** shardy has quit IRC | 21:14 | |
| *** Thelo_ has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** gyee has quit IRC | 21:16 | |
| *** rtheis has quit IRC | 21:18 | |
| *** sdake has joined #openstack-infra | 21:19 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 21:19 | |
| *** Thelo_ has joined #openstack-infra | 21:20 | |
| *** bauwser is now known as bauzas | 21:20 | |
| *** Hal1 has quit IRC | 21:21 | |
| mordred | raddaoui: wow! nicely done! | 21:21 |
| *** ijw has joined #openstack-infra | 21:21 | |
| * mordred hands raddaoui a mostly unused pie he found that looks probably tasty | 21:21 | |
| *** dprince has quit IRC | 21:22 | |
| *** Thelo_ has quit IRC | 21:24 | |
| *** lascii is now known as alaski | 21:27 | |
| *** roxanaghe has quit IRC | 21:28 | |
| *** roxanaghe has joined #openstack-infra | 21:30 | |
| *** inc0 has quit IRC | 21:31 | |
| *** gyee has joined #openstack-infra | 21:32 | |
| *** Hal1 has joined #openstack-infra | 21:33 | |
| clarkb | raddaoui: tytyty | 21:34 |
| clarkb | pabelanger: cool I am back to my desk now is there anything I can help with that? | 21:34 |
| clarkb | pleia2: fungi anteaya everything seems to have gone ok? anything else I can do to help while other people make sure their homes don't fly away in a hurricane? | 21:36 |
| clarkb | if not I am going to ask for reviews on topic:newton-xenial and https://review.openstack.org/312647 | 21:37 |
| pleia2 | I don't think we confirmed the online reindex was done | 21:37 |
| *** roxanaghe has quit IRC | 21:37 | |
| pleia2 | but seems otherwise finished | 21:37 |
| clarkb | pleia2: cool. Is that just running the show queues command ong errit to see if its done? | 21:37 |
| pleia2 | I think we just look at the log | 21:37 |
| pleia2 | I haven't logged in | 21:38 |
| clarkb | kk /me looks | 21:38 |
| zaro | you can check either | 21:38 |
| pleia2 | o/ zaro | 21:38 |
| zaro | cpu is way down now | 21:38 |
| zaro | meaning its probably done | 21:38 |
| *** shashank_hegde has joined #openstack-infra | 21:39 | |
| clarkb | baaa7f86 19:02:09.171 Index all changes of project openstack/nova | 21:39 |
| clarkb | thats the only one that says index so I think likely all but nova are done | 21:39 |
| *** roxanaghe has joined #openstack-infra | 21:40 | |
| *** ianw has quit IRC | 21:40 | |
| clarkb | I ran gerrit show queue -w | 21:40 |
| *** xarses has quit IRC | 21:41 | |
| clarkb | looks like we may have some broken github repos that are failing replication. I am going to poke around on gerrit and see if I can fix that | 21:41 |
| zaro | i guess it's working even better than i expected | 21:41 |
| zaro | didn't even eat up that much CPU | 21:41 |
| raddaoui | clarkb: anytime :) | 21:41 |
| *** tonytan4ever has joined #openstack-infra | 21:42 | |
| *** signed8bit is now known as signed8bit_Zzz | 21:43 | |
| clarkb | ya looks like they don't have the gerrit team on them with write perms | 21:43 |
| clarkb | so gonna add that and see if I can get the gerrit queues to empty | 21:44 |
| *** tpsilva has quit IRC | 21:44 | |
| pleia2 | was this ones I renamed today, or just random other ones? | 21:44 |
| *** ijw has quit IRC | 21:45 | |
| *** rfolco has quit IRC | 21:45 | |
| jeblair | it's the deb repos | 21:46 |
| clarkb | pleia2: jeblair ya its the deb repos | 21:46 |
| jeblair | they didn't get set up right because jeepyb exceeded the github api limit because they were all created at once | 21:46 |
| pleia2 | ah | 21:47 |
| pleia2 | right | 21:47 |
| *** tonytan4ever has quit IRC | 21:48 | |
| *** amitgandhinz has joined #openstack-infra | 21:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests https://review.openstack.org/365099 | 21:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable test_failed_change_at_head https://review.openstack.org/363821 | 21:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add nodeset as a top-level configuration item https://review.openstack.org/365089 | 21:52 |
| *** rlandy has quit IRC | 21:52 | |
| *** tphummel has quit IRC | 21:52 | |
| *** gordc has quit IRC | 21:54 | |
| *** Goneri has joined #openstack-infra | 21:55 | |
| *** amitgandhinz has quit IRC | 21:55 | |
| *** ilyashakhat_mobi has quit IRC | 21:57 | |
| *** gluytium has quit IRC | 21:58 | |
| *** ddieterly[away] has quit IRC | 21:58 | |
| *** Goneri has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack-infra/zuul: Use {{ ansible_host }} for ssh-keyscan https://review.openstack.org/352627 | 22:00 |
| *** ijw has joined #openstack-infra | 22:01 | |
| *** ijw has quit IRC | 22:02 | |
| *** gluytium has joined #openstack-infra | 22:03 | |
| jeblair | pabelanger: you might want to see the tortured comment i left on https://review.openstack.org/331499 | 22:05 |
| *** tphummel has joined #openstack-infra | 22:05 | |
| *** hashar has quit IRC | 22:05 | |
| *** jcoufal_ has quit IRC | 22:06 | |
| *** jcoufal has joined #openstack-infra | 22:07 | |
| *** xyang1 has quit IRC | 22:09 | |
| jeblair | mordred: http://www.nytimes.com/2016/09/02/world/europe/bags-of-cocaine-worth-56-million-are-found-at-coca-cola-factory-in-france.html | 22:10 |
| *** ianw has joined #openstack-infra | 22:11 | |
| *** dimtruck is now known as zz_dimtruck | 22:12 | |
| *** xarses has joined #openstack-infra | 22:12 | |
| clarkb | I think reindexing is done | 22:13 |
| *** ijw has joined #openstack-infra | 22:15 | |
| *** david-lyle has quit IRC | 22:15 | |
| *** ijw has quit IRC | 22:15 | |
| *** ijw has joined #openstack-infra | 22:16 | |
| *** david-lyle has joined #openstack-infra | 22:17 | |
| * fungi checks | 22:19 | |
| fungi | yep! | 22:19 |
| fungi | [2016-09-02 19:02:09,155] INFO com.google.gerrit.lucene.OnlineReindexer : Starting online reindex from schema version 14 to 14 | 22:19 |
| fungi | [2016-09-02 22:07:48,331] INFO com.google.gerrit.lucene.OnlineReindexer : Reindex to version 14 complete | 22:20 |
| fungi | [2016-09-02 22:07:48,331] INFO com.google.gerrit.lucene.OnlineReindexer : Using schema version 14 | 22:20 |
| jeblair | yay we're at version 14! | 22:20 |
| fungi | so only took 3 hours | 22:20 |
| jeblair | versioun 14 is dead! long live version 14! | 22:20 |
| fungi | not bad at all | 22:20 |
| jeblair | oops, there's that extra u from earlier | 22:20 |
| *** pradk has quit IRC | 22:20 | |
| pleia2 | \o/ | 22:21 |
| * fungi vanishes again | 22:21 | |
| clarkb | jeblair: btw 312647 seemed fine after a recheck. | 22:21 |
| zaro | hhallelujah! can have a good weekend | 22:21 |
| clarkb | the queue is also clear now | 22:22 |
| clarkb | I got those github projects sorted | 22:22 |
| pleia2 | online reindex is good | 22:22 |
| *** javeriak has quit IRC | 22:22 | |
| jeblair | clarkb: yep! logs look good | 22:23 |
| jeblair | +3 | 22:23 |
| pabelanger | jeblair: Agreed, part of the reason why I haven't moved forward with it too. I think we can revisit once zuulv3 has landed | 22:25 |
| pabelanger | clarkb: just pushing up patch now | 22:25 |
| *** shashank_hegde has quit IRC | 22:27 | |
| *** tkelsey has joined #openstack-infra | 22:28 | |
| clarkb | we launched over 30k jobs per day a couple days last week | 22:29 |
| jeblair | time to update the slides! :) | 22:30 |
| pleia2 | nice | 22:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd https://review.openstack.org/365097 | 22:31 |
| pabelanger | clarkb: ^ | 22:31 |
| *** tkelsey has quit IRC | 22:32 | |
| *** ijw has quit IRC | 22:32 | |
| *** jcoufal_ has joined #openstack-infra | 22:32 | |
| *** ddieterly has joined #openstack-infra | 22:32 | |
| tonyb | fungi: the events iin the table get marked as past and the timer automagically moves to the next one IIRC | 22:33 |
| *** jcoufal has quit IRC | 22:33 | |
| zaro | alright, good work today infra team. feeling good after that online index so i'm gonna do early weekend :) | 22:34 |
| clarkb | pabelanger: I am going to need a whole lot more comments :) I have no idea why skipping an interface would be what we want there | 22:35 |
| *** ddieterly has quit IRC | 22:36 | |
| pabelanger | clarkb: Its a hack right now, we should likely ask mordred how to handle that better. Basically the test fixtures are setup to provider multiple interfaces, when we run against a single interface, tests fail because the fixtures still reference the other eth interfaces, but glean only build eth0 | 22:37 |
| tonyb | #success election content moved from the wiki to a gerrit workflow! Thanks tristanC and everyone on the Infra team that made it happen! | 22:37 |
| openstackstatus | tonyb: Added success to Success page | 22:37 |
| clarkb | pabelanger: right but what is the fix for rax? | 22:37 |
| clarkb | pabelanger: basically the change doesn't explain what was broken before or how it fixes it so I am lost | 22:38 |
| *** claudiub has quit IRC | 22:38 | |
| pabelanger | clarkb: sure, I can update the commit message. The fix is line 371 in cmd.py. We didn't 'continue' in the top level loop, only the sub loop. | 22:39 |
| clarkb | pabelanger: basically glean is full of hacks and none of them are documented. When I run into new ones that I have to debug I try to push a change up to explain what is going on like with the multiple json keys for network info. So would be good to be very clear what is going on in the code there that makes rax and not rax work | 22:40 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update the QA team meeting chair https://review.openstack.org/364065 | 22:40 |
| pabelanger | clarkb: yes, let me push up some comments in line | 22:40 |
| clarkb | pabelanger: thansk. Also reading the old code its basically just a noop | 22:41 |
| clarkb | pabelanger: all it does is go through each of the mac addrs from the config drive and see if they arein sysfs but doesn't do anything based on that? | 22:41 |
| clarkb | yay fun bugs I see the problem now | 22:42 |
| clarkb | but ya a comments saying "If we can't map the interfaces mac addresses to sysfs we skip this interface entirely" would probably help maybe with an example/explanation for why this might occur if we know why | 22:42 |
| pabelanger | clarkb: Right | 22:42 |
| pabelanger | Will add that now | 22:43 |
| clarkb | the intent of that code was to continue the outer loop but only continued the inner loop | 22:43 |
| pabelanger | ya, that's how it originally worked | 22:43 |
| pabelanger | but when raw_macs what added, it got broken | 22:43 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions} https://review.openstack.org/365163 | 22:44 |
| clarkb | gotcha | 22:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd https://review.openstack.org/365097 | 22:46 |
| pabelanger | clarkb: open to suggestions for line 191 in test_glean.py too | 22:47 |
| *** shashank_hegde has joined #openstack-infra | 22:47 | |
| *** yamahata has quit IRC | 22:47 | |
| *** markvoelker has quit IRC | 22:47 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions} https://review.openstack.org/365163 | 22:48 |
| clarkb | pabelanger: might be slightyl clearer to have those checks in another sub function | 22:48 |
| zigo | pabelanger: clarkb: Could you please accept these 2 repos? ^ | 22:48 |
| clarkb | pabelanger: but other wise I think thats the right way to filter that | 22:49 |
| clarkb | pabelanger: because in the case of just setting eth0 the other interfaces won't be touched and we would fail otherwise | 22:49 |
| *** yamahata has joined #openstack-infra | 22:49 | |
| *** snarwade has quit IRC | 22:50 | |
| pabelanger | zigo: in the middle of something atm, I'll have to look in a bit | 22:50 |
| zigo | clarkb: Do you have time? | 22:50 |
| zigo | It's yet another blocker... :/ | 22:50 |
| zigo | fungi: You maybe? | 22:51 |
| clarkb | pabelanger: left a nit but you have to repush anyways because pep8 failed | 22:51 |
| zigo | fungi: https://review.openstack.org/365163 | 22:51 |
| zigo | mordred: https://review.openstack.org/365163 | 22:51 |
| *** amitgandhinz has joined #openstack-infra | 22:51 | |
| *** mmedvede has quit IRC | 22:51 | |
| clarkb | pabelanger: so hopeflly we can address the nit :) | 22:51 |
| clarkb | pabelanger: and seems like the overall issue is that systemd invokes glean per interface rather than doing a whole bunch at once. Which is what led to that issue. Makes sense | 22:52 |
| *** snarwade has joined #openstack-infra | 22:52 | |
| *** Apoorva has quit IRC | 22:53 | |
| *** ijw has joined #openstack-infra | 22:55 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd https://review.openstack.org/365097 | 22:55 |
| *** amitgandhinz has quit IRC | 22:55 | |
| *** ijw has quit IRC | 22:56 | |
| pabelanger | clarkb: right, I wonder if we should revisit that for systemd at a later time. Based on the noise we seen from neutron yesterday | 22:56 |
| pabelanger | _shurgs_ | 22:56 |
| clarkb | ya if we triggered on "bring up host networking" and just did it rather than trigger on every if up | 22:57 |
| pabelanger | Ya, something to play with another day. | 22:58 |
| jeblair | [i really miss the named loops in perl. eg, "last INTERFACE"] | 22:58 |
| *** ijw has joined #openstack-infra | 22:59 | |
| zigo | jeblair: https://review.openstack.org/365163 <--- Can you +2 workflow that one, so we can move ahead ? | 23:00 |
| jeblair | zigo: yep | 23:01 |
| zigo | Thanks so much. | 23:01 |
| *** rbrndt has quit IRC | 23:01 | |
| *** jeblair is now known as corvus | 23:01 | |
| *** asettle has joined #openstack-infra | 23:05 | |
| *** asettle has quit IRC | 23:10 | |
| clarkb | pabelanger: +2 probably best to get that in next week rather than try and shove it in and rebuild this late in the week | 23:11 |
| *** jcoufal has joined #openstack-infra | 23:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions} https://review.openstack.org/365163 | 23:15 |
| *** jcoufal_ has quit IRC | 23:16 | |
| *** mmedvede has joined #openstack-infra | 23:18 | |
| *** openstackgerrit has quit IRC | 23:18 | |
| *** openstackgerrit has joined #openstack-infra | 23:18 | |
| *** signed8bit_Zzz is now known as signed8bit | 23:19 | |
| *** spzala has joined #openstack-infra | 23:19 | |
| *** signed8bit is now known as signed8bit_Zzz | 23:19 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add networking-ovn meeting https://review.openstack.org/363906 | 23:20 |
| *** spzala has quit IRC | 23:23 | |
| pabelanger | clarkb: I am okay with that | 23:26 |
| *** thorst has quit IRC | 23:30 | |
| *** Swami has quit IRC | 23:32 | |
| *** thorst has joined #openstack-infra | 23:32 | |
| *** spzala has joined #openstack-infra | 23:34 | |
| *** jamesdenton has joined #openstack-infra | 23:37 | |
| *** ijw has quit IRC | 23:40 | |
| *** tqtran has quit IRC | 23:41 | |
| *** ijw has joined #openstack-infra | 23:41 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use fallocate for swapfile if on ext https://review.openstack.org/312647 | 23:44 |
| *** spzala has quit IRC | 23:44 | |
| *** ijw has quit IRC | 23:45 | |
| *** asettle has joined #openstack-infra | 23:46 | |
| *** thcipriani is now known as thcipriani|afk | 23:47 | |
| *** senk__ has quit IRC | 23:51 | |
| *** senk has joined #openstack-infra | 23:51 | |
| *** amitgandhinz has joined #openstack-infra | 23:52 | |
| *** amitgandhinz has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
