| clarkb | completely unrelated but for those of you attending the midcycle sprint are you planning to shuttle/train/drive? | 00:00 |
|---|---|---|
| jeblair | is there a train? | 00:00 |
| pabelanger | I'd do train | 00:00 |
| pabelanger | didn't know it was an option | 00:00 |
| clarkb | jeblair: ya you can flughafen to mannheim and mannheim to wiesloch-walldorf then walk a mile or so depending on your hotel | 00:01 |
| clarkb | or catch a more local train and walk less depending on your hotel | 00:01 |
| jeblair | how about that | 00:01 |
| clarkb | captain train says it will take about an hour and 7 minutes (excluding walking) | 00:01 |
| *** jerryz has quit IRC | 00:02 | |
| pabelanger | clarkb: cost? | 00:02 |
| clarkb | pabelanger: ~ 25 euro | 00:02 |
| clarkb | the shuttle looks like 60 euro one way | 00:02 |
| pabelanger | cool | 00:02 |
| clarkb | which is what prompted me to look at other options | 00:03 |
| pabelanger | ya, going to do train then | 00:03 |
| jeblair | ++train | 00:03 |
| *** julim has quit IRC | 00:03 | |
| jeblair | oh capitain train changed their name | 00:03 |
| clarkb | my flight lands at 1:15pm Sunday the 18th fwiw | 00:04 |
| clarkb | if anyone else knows their flight details already we could be the annoying people on the train | 00:04 |
| *** zhurong has quit IRC | 00:04 | |
| pabelanger | I'm in 06:15am localtime | 00:05 |
| pabelanger | over night flight | 00:05 |
| pabelanger | clarkb: do you have the URL handy for the train? | 00:06 |
| jeblair | pabelanger: https://www.captaintrain.com | 00:07 |
| *** aeng has joined #openstack-infra | 00:07 | |
| clarkb | yup ^ | 00:07 |
| pabelanger | great,thanks | 00:08 |
| *** Julien-zte has joined #openstack-infra | 00:08 | |
| clarkb | and you want frankfurt flughafen fernbf not the regional train station at the airport | 00:08 |
| clarkb | though you can probably also just show up and book there as well | 00:09 |
| clarkb | (I have done this a non zero number of times) | 00:09 |
| *** gildub_ has joined #openstack-infra | 00:09 | |
| *** elo has quit IRC | 00:10 | |
| pabelanger | ya, looks like it runs regularly | 00:10 |
| *** kzaitsev_mb has quit IRC | 00:11 | |
| cloudnull | evenings | 00:11 |
| pabelanger | clarkb: ya, I'll pick up a ticket once I land. seems to depart ever 1hr | 00:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 00:16 |
| *** Jeffrey4l has joined #openstack-infra | 00:16 | |
| *** bswartz has quit IRC | 00:17 | |
| *** raunak has quit IRC | 00:18 | |
| cloudnull | pabelanger: did the updated image w/ the networking bits ever move forward? is it possible to revert https://review.openstack.org/#/c/356382/ or should we also wait on the urandom bits before ? | 00:18 |
| *** mriedem has joined #openstack-infra | 00:18 | |
| pabelanger | cloudnull: just kicked off a new DIB build now | 00:19 |
| pabelanger | we had some git repo issues today | 00:19 |
| *** thorst_ has joined #openstack-infra | 00:19 | |
| cloudnull | selfishly, i'd to see it back in action, because in OpenStack-Ansible we're waiting on trusty gates. | 00:19 |
| cloudnull | ^ completely selfish -- just +/- curious -- no worries regardless | 00:20 |
| cloudnull | :) | 00:20 |
| pabelanger | should take another hour to build, once finished we can upload again and revert. | 00:20 |
| pabelanger | don't see an issue | 00:20 |
| pabelanger | if we land 356706 tonight too, I'll kick off an image build | 00:20 |
| pabelanger | oh, we should land 356637 too | 00:21 |
| pabelanger | that will fix multi-node jobs | 00:21 |
| *** gyee_ has quit IRC | 00:21 | |
| *** Julien-zte has quit IRC | 00:22 | |
| cloudnull | we can wait, tbt the gate is running decently and wait times have not been bad. | 00:23 |
| cloudnull | so no pressure | 00:23 |
| cloudnull | let least not from me | 00:23 |
| pabelanger | ya, this week has been way better then last! | 00:24 |
| cloudnull | indeed it has | 00:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 00:25 |
| ianw | yay! fedora24 not crashing (not working either, but i'll take it) http://logs.openstack.org/63/356763/1/experimental/gate-tempest-dsvm-platform-fedora24/01ef6d9/ | 00:29 |
| *** pilgrimstack has joined #openstack-infra | 00:31 | |
| jeblair | ianw: what kernel does f24 have? | 00:31 |
| jeblair | pabelanger: is it correct to both 'before' and 'wants' network-pre.target? | 00:32 |
| *** bswartz has joined #openstack-infra | 00:32 | |
| *** tphummel has joined #openstack-infra | 00:32 | |
| ianw | jeblair: 4.6.6+ianw | 00:33 |
| jeblair | i don't speak systemd. i mean, being before something while also wanting it seems sort of poetic, which is nice, but that doesn't seem like lennart's style, so i dunno. | 00:33 |
| *** rbuzatu has joined #openstack-infra | 00:34 | |
| *** sdake has joined #openstack-infra | 00:35 | |
| *** pilgrimstack has quit IRC | 00:35 | |
| jeblair | cloudnull, ianw: then we may want to add f24 to https://review.openstack.org/356706 | 00:35 |
| jeblair | but we can do it later | 00:35 |
| *** armax has quit IRC | 00:36 | |
| ianw | ok, i've only been vaguely follow ... for unbound and ipv6 right? | 00:37 |
| jeblair | ianw: just unbound being slow to start on boot -- the parent change (already merged) contains the essay that explains all: https://review.openstack.org/356697 | 00:38 |
| * jeblair dines | 00:39 | |
| *** rbuzatu has quit IRC | 00:39 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/puppet-mediawiki: Add 'autopatrol' group https://review.openstack.org/356313 | 00:41 |
| *** roxanaghe has quit IRC | 00:42 | |
| *** dimtruck is now known as zz_dimtruck | 00:45 | |
| *** jamielennox is now known as jamielennox|away | 00:48 | |
| *** gus has joined #openstack-infra | 00:48 | |
| jroll | jeblair: that's quite the essay, I enjoyed it :) | 00:48 |
| *** amitgandhinz has joined #openstack-infra | 00:49 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Add initialize-urandom to fedora 24 https://review.openstack.org/356818 | 00:49 |
| cloudnull | ianw jeblair ^ | 00:49 |
| apuimedo | pabelanger: is it possible to make devstack process correctly requirements.txt that point to a git master branch instead of a fixed release? | 00:50 |
| *** itisha has quit IRC | 00:50 | |
| apuimedo | I'm asking because we develop openstack/kuryr-libnetwork against the master of openstack/kuryr | 00:51 |
| apuimedo | while openstack/kuryr is not stable | 00:51 |
| apuimedo | (we have been splitting the repos) | 00:51 |
| ianw | apuimedo: LIBS_FROM_GIT? | 00:52 |
| apuimedo | ianw: yup | 00:52 |
| apuimedo | we're developing a both the lib and the server at the same time | 00:53 |
| *** raunak has joined #openstack-infra | 00:53 | |
| apuimedo | and they are both in prerelease state | 00:53 |
| *** amitgandhinz has quit IRC | 00:53 | |
| ianw | apuimedo: the server is installed from git, though? only the clients we're pip installing? | 00:53 |
| apuimedo | one option is to keep nagging for releases | 00:53 |
| *** tqtran has quit IRC | 00:54 | |
| apuimedo | ianw: devstack gets the server plugin from git, the requirements file of which points to a pip git url | 00:54 |
| ianw | apuimedo: umm, not quite following, can you link to the plugin? | 00:57 |
| apuimedo | sure | 00:58 |
| *** shashank_hegde has quit IRC | 00:58 | |
| *** rfolco has joined #openstack-infra | 00:58 | |
| *** rfolco has quit IRC | 00:58 | |
| *** spzala has joined #openstack-infra | 00:59 | |
| *** sdake has quit IRC | 00:59 | |
| apuimedo | requirements line (which will point soon to openstack/kuryr instead of that fork) https://github.com/openstack/kuryr-libnetwork/blob/master/requirements.txt | 00:59 |
| apuimedo | plugin https://github.com/openstack/kuryr-libnetwork/blob/master/devstack/plugin.sh | 00:59 |
| apuimedo | ianw: ^^ | 00:59 |
| *** sdake has joined #openstack-infra | 01:02 | |
| *** spzala has quit IRC | 01:03 | |
| *** xarses has joined #openstack-infra | 01:03 | |
| *** jamielennox|away is now known as jamielennox | 01:04 | |
| *** nwkarsten has joined #openstack-infra | 01:05 | |
| ianw | apuimedo: if you just drop the @ bit of the -e line, that would mean just install from master? | 01:06 |
| apuimedo | right | 01:07 |
| *** zhurong has joined #openstack-infra | 01:10 | |
| *** sdake has quit IRC | 01:11 | |
| ianw | apuimedo: so, that doesn't work? | 01:12 |
| *** hichihara has quit IRC | 01:12 | |
| apuimedo | I'll try now without the branch selection | 01:12 |
| apuimedo | ianw: "error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'-e git:/'"" | 01:14 |
| *** spzala has joined #openstack-infra | 01:15 | |
| *** pt_15 has quit IRC | 01:16 | |
| ianw | apuimedo: what you to change the line to? | 01:17 |
| *** malabaris has joined #openstack-infra | 01:17 | |
| *** malabaris has left #openstack-infra | 01:19 | |
| apuimedo | sorry, I had a typo | 01:19 |
| apuimedo | the real error is | 01:19 |
| pabelanger | jeblair: Yes. Services that want to be run before the network is configured should place Before=network-pre.target and also set Wants=network-pre.target to pull it in. According to: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ | 01:19 |
| *** thorst_ has quit IRC | 01:19 | |
| apuimedo | http://paste.openstack.org/show/560292/ | 01:20 |
| apuimedo | and the content of requirements.txt is | 01:20 |
| *** thorst_ has joined #openstack-infra | 01:20 | |
| apuimedo | ianw: http://paste.openstack.org/show/560293/ | 01:21 |
| apuimedo | note that pip install -r requirements.txt works | 01:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 01:21 |
| *** aeng has quit IRC | 01:23 | |
| apuimedo | ianw: funnily enough, if I change the egg name from kuryr-lib to kuryr_lib, then the error is just | 01:25 |
| apuimedo | Could not find a version that satisfies the requirement kuryr_lib (from kuryr-libnetwork==0.1.0.dev517) (from versions: ) | 01:25 |
| apuimedo | No matching distribution found for kuryr_lib (from kuryr-libnetwork==0.1.0.dev517) | 01:25 |
| *** pvaneck has quit IRC | 01:28 | |
| *** gouthamr_ has joined #openstack-infra | 01:28 | |
| *** thorst_ has quit IRC | 01:29 | |
| *** Sukhdev has quit IRC | 01:30 | |
| *** zz_dimtruck is now known as dimtruck | 01:31 | |
| *** gouthamr has quit IRC | 01:32 | |
| *** pilgrimstack has joined #openstack-infra | 01:33 | |
| ianw | apuimedo: the "-" is not helping -> The "project name" component of the url suffix "egg=<project name>-<version>" is used by pip in its dependency logic to identify the project prior to pip downloading and analyzing the metadata. The optional "version" component of the egg name is not functionally important. | 01:36 |
| apuimedo | yeah | 01:36 |
| apuimedo | I'm fixing that now | 01:36 |
| ianw | but, odd ... | 01:36 |
| *** aeng has joined #openstack-infra | 01:36 | |
| apuimedo | now I'm using kuryr_lib | 01:36 |
| apuimedo | but I don't know why pbr still tries to find a version | 01:36 |
| apuimedo | if no version is specified, it should use the latest | 01:37 |
| apuimedo | shouldn't it? | 01:37 |
| *** pilgrimstack has quit IRC | 01:37 | |
| *** annegentle has joined #openstack-infra | 01:38 | |
| *** spzala has quit IRC | 01:38 | |
| *** spzala has joined #openstack-infra | 01:39 | |
| *** edmondsw has quit IRC | 01:40 | |
| *** hockeynut has joined #openstack-infra | 01:42 | |
| *** spzala has quit IRC | 01:43 | |
| *** annegentle has quit IRC | 01:45 | |
| *** raunak has quit IRC | 01:48 | |
| apuimedo | ianw: https://review.openstack.org/#/c/356824/ | 01:48 |
| ianw | apuimedo: hmm, yeah run it with verbose and it goes off looking for it on pypi | 01:48 |
| apuimedo | this fixes the egg name | 01:48 |
| openstackgerrit | Merged openstack-infra/shade: Use "image" as argument for Glance V1 upload error path https://review.openstack.org/355715 | 01:49 |
| *** amitgandhinz has joined #openstack-infra | 01:49 | |
| *** tqtran has joined #openstack-infra | 01:51 | |
| *** amitgandhinz has quit IRC | 01:54 | |
| *** tqtran has quit IRC | 01:56 | |
| *** rbuzatu has joined #openstack-infra | 01:56 | |
| *** nwkarsten has quit IRC | 01:57 | |
| *** Apoorva has quit IRC | 01:57 | |
| *** nwkarsten has joined #openstack-infra | 01:57 | |
| *** yanyanhu has joined #openstack-infra | 01:58 | |
| *** jimbaker has quit IRC | 02:00 | |
| *** tbarron is now known as tbarron|afk | 02:00 | |
| *** vinaypotluri has quit IRC | 02:01 | |
| *** nwkarsten has quit IRC | 02:02 | |
| *** rbuzatu has quit IRC | 02:03 | |
| *** jimbaker has joined #openstack-infra | 02:04 | |
| *** jimbaker has quit IRC | 02:04 | |
| *** jimbaker has joined #openstack-infra | 02:04 | |
| *** jerryz has joined #openstack-infra | 02:15 | |
| *** mriedem has quit IRC | 02:18 | |
| *** rlandy has quit IRC | 02:21 | |
| *** thorst_ has joined #openstack-infra | 02:22 | |
| pabelanger | ianw: cloudnull: ubuntu-trusty is now uploading to osic-cloud1, going to get the revert patch ready | 02:22 |
| cloudnull | ooohh :) | 02:22 |
| *** thorst_ has quit IRC | 02:23 | |
| *** thorst_ has joined #openstack-infra | 02:24 | |
| pabelanger | ianw: cloudnull: also started ubuntu-xenail rebuilt, it won't fix dns but it will include ipv6 sysctl fix | 02:24 |
| pabelanger | ianw: You'll likely need to make sure that is upload to osic-cloud1 once finished building | 02:24 |
| cloudnull | nice! | 02:25 |
| cloudnull | that should at least help jobs that make into the nodepool | 02:25 |
| cloudnull | 's/jobs/instances/' | 02:26 |
| * cloudnull is tired. needs to not be typing right now | 02:26 | |
| *** baoli has joined #openstack-infra | 02:30 | |
| *** fguillot has quit IRC | 02:32 | |
| *** caowei has joined #openstack-infra | 02:32 | |
| *** thorst_ has quit IRC | 02:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Disable trusty nodes on OSIC" https://review.openstack.org/356834 | 02:33 |
| pabelanger | cloudnull: ianw: jhesketh: okay, we can revert now ^ | 02:33 |
| *** pilgrimstack has joined #openstack-infra | 02:33 | |
| jhesketh | pabelanger: looking (sorry I haven't been following hte channel) | 02:34 |
| pabelanger | jhesketh: let me include the actual commit that fixes osic | 02:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Disable trusty nodes on OSIC" https://review.openstack.org/356834 | 02:37 |
| *** pilgrimstack has quit IRC | 02:37 | |
| stevemar | anyone know swapnil / coolsvaps irc name? | 02:39 |
| stevemar | i would hvae thought it was one of those | 02:39 |
| pabelanger | jhesketh: bascially, cloudnull noticed we needed https://review.openstack.org/#/c/356204/ for ipv6 nodes in osic-cloud1. It was affecting ubuntu-trusty pretty bad (slow git clones I believe) so we disabled them in osic-cloud1 this morning | 02:39 |
| pabelanger | Finially finished a new DIB with the patch, and it was just uploaded to osic-cloud1 | 02:40 |
| *** psachin has quit IRC | 02:40 | |
| pabelanger | working on ubuntu-xenial DIB now, but still another hour or so away from being uploaded | 02:40 |
| *** shashank_hegde has joined #openstack-infra | 02:40 | |
| *** eandersson_ has joined #openstack-infra | 02:40 | |
| jhesketh | pabelanger: okay, lgtm... I'll +W and keep an eye | 02:41 |
| jhesketh | (on the trusty fix) | 02:42 |
| pabelanger | great, thanks | 02:42 |
| *** jamielennox is now known as jamielennox|away | 02:42 | |
| *** gongysh has joined #openstack-infra | 02:42 | |
| cloudnull | nice thanks pabelanger :) | 02:42 |
| * cloudnull fingers cross that we're done with that forever | 02:43 | |
| *** changzhi has joined #openstack-infra | 02:44 | |
| *** coolsvap has joined #openstack-infra | 02:45 | |
| *** yuanying_ has quit IRC | 02:46 | |
| *** jimbaker has quit IRC | 02:47 | |
| *** eandersson_ has quit IRC | 02:47 | |
| cloudnull | if there are some RH folks that wouldnt mind reviewing https://review.openstack.org/#/c/356456/ -- it'd be great to see the same fix added in for the infra image simply be a default element in DIB. | 02:50 |
| *** amitgandhinz has joined #openstack-infra | 02:50 | |
| *** jimbaker has joined #openstack-infra | 02:51 | |
| *** jimbaker has quit IRC | 02:51 | |
| *** jimbaker has joined #openstack-infra | 02:51 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/groups-static-pages: Remove groups.json https://review.openstack.org/356843 | 02:52 |
| *** yuanying has joined #openstack-infra | 02:52 | |
| *** amitgandhinz has quit IRC | 02:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Disable trusty nodes on OSIC" https://review.openstack.org/356834 | 02:59 |
| *** spzala has joined #openstack-infra | 03:00 | |
| *** rwsu has joined #openstack-infra | 03:02 | |
| *** nwkarsten has joined #openstack-infra | 03:04 | |
| *** julim has joined #openstack-infra | 03:05 | |
| *** spzala has quit IRC | 03:05 | |
| *** hockeynut has quit IRC | 03:06 | |
| *** baoli has quit IRC | 03:07 | |
| *** jamielennox|away is now known as jamielennox | 03:07 | |
| *** yamahata has quit IRC | 03:07 | |
| *** asettle has joined #openstack-infra | 03:08 | |
| *** Goneri has quit IRC | 03:10 | |
| *** csomerville has joined #openstack-infra | 03:13 | |
| *** cody-somerville has quit IRC | 03:13 | |
| *** gongysh has quit IRC | 03:14 | |
| *** aeng has quit IRC | 03:14 | |
| *** dingyichen has quit IRC | 03:16 | |
| *** asettle has quit IRC | 03:16 | |
| *** raunak has joined #openstack-infra | 03:16 | |
| *** pahuang has quit IRC | 03:16 | |
| *** yuanying has quit IRC | 03:18 | |
| *** nwkarsten has quit IRC | 03:18 | |
| *** yuanying has joined #openstack-infra | 03:19 | |
| *** julim has quit IRC | 03:19 | |
| *** nwkarsten has joined #openstack-infra | 03:19 | |
| *** cody-somerville has joined #openstack-infra | 03:20 | |
| *** rbuzatu has joined #openstack-infra | 03:20 | |
| *** coolsvap is now known as _coolsvap_ | 03:21 | |
| *** csomerville has quit IRC | 03:22 | |
| *** nwkarsten has quit IRC | 03:23 | |
| *** rbuzatu has quit IRC | 03:24 | |
| *** r-mibu has quit IRC | 03:24 | |
| *** r-mibu has joined #openstack-infra | 03:25 | |
| *** raunak has quit IRC | 03:25 | |
| *** Sukhdev has joined #openstack-infra | 03:28 | |
| *** raunak has joined #openstack-infra | 03:30 | |
| *** thorst_ has joined #openstack-infra | 03:30 | |
| *** vikrant has joined #openstack-infra | 03:32 | |
| *** nwkarsten has joined #openstack-infra | 03:33 | |
| *** gouthamr_ has quit IRC | 03:34 | |
| *** pilgrimstack has joined #openstack-infra | 03:35 | |
| *** tonytan4ever has joined #openstack-infra | 03:36 | |
| *** thorst_ has quit IRC | 03:37 | |
| *** vikrant has quit IRC | 03:37 | |
| *** pilgrimstack has quit IRC | 03:39 | |
| *** vikrant has joined #openstack-infra | 03:41 | |
| *** raunak has quit IRC | 03:41 | |
| *** yuanying has quit IRC | 03:42 | |
| *** yuanying has joined #openstack-infra | 03:42 | |
| *** furlongm has quit IRC | 03:43 | |
| *** furlongm has joined #openstack-infra | 03:43 | |
| *** yuanying has quit IRC | 03:43 | |
| *** valderrv has quit IRC | 03:44 | |
| *** links has joined #openstack-infra | 03:45 | |
| *** raunak has joined #openstack-infra | 03:45 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** amitgandhinz has joined #openstack-infra | 03:51 | |
| *** tqtran has joined #openstack-infra | 03:52 | |
| timrc | yolanda, rcarrillocruz: Hey so I'm coming to Barcelona. How many gallons of sangria do you think we'll drink? jk | 03:52 |
| *** amitgandhinz has quit IRC | 03:56 | |
| *** tqtran has quit IRC | 03:56 | |
| *** vikrant is now known as vikrant|brb | 03:58 | |
| *** spzala has joined #openstack-infra | 04:01 | |
| *** tonytan4ever has quit IRC | 04:05 | |
| *** spzala has quit IRC | 04:07 | |
| openstackgerrit | vikas choudhary proposed openstack-infra/project-config: Fix dsvm installation job https://review.openstack.org/346717 | 04:18 |
| *** vikrant|brb is now known as vikrant| | 04:18 | |
| *** vikrant| is now known as vikrant | 04:18 | |
| pabelanger | ianw: if you have a moment, maybe you can figure out why ubuntu-xenial dibs are failing: http://nodepool.openstack.org/dib.ubuntu-xenial.log | 04:29 |
| *** amotoki has joined #openstack-infra | 04:33 | |
| *** thorst_ has joined #openstack-infra | 04:34 | |
| *** amotoki has quit IRC | 04:38 | |
| *** thorst_ has quit IRC | 04:42 | |
| *** vgridnev has joined #openstack-infra | 04:42 | |
| ianw | pabelanger: ok | 04:42 |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test https://review.openstack.org/354566 | 04:45 |
| *** Sukhdev has quit IRC | 04:46 | |
| ianw | pabelanger: hmmm, yeah that numfmt error .. | 04:51 |
| *** amitgandhinz has joined #openstack-infra | 04:52 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Set PS4 to provide additional debug info https://review.openstack.org/271434 | 04:52 |
| *** amitgandhinz has quit IRC | 04:56 | |
| *** narayrak has joined #openstack-infra | 04:59 | |
| *** spzala has joined #openstack-infra | 05:00 | |
| *** amotoki has joined #openstack-infra | 05:00 | |
| *** narayrak has quit IRC | 05:02 | |
| *** jaosorior has joined #openstack-infra | 05:03 | |
| *** amotoki has quit IRC | 05:05 | |
| *** spzala has quit IRC | 05:06 | |
| *** tonytan4ever has joined #openstack-infra | 05:06 | |
| *** vgridnev has quit IRC | 05:06 | |
| jianghuaw | mordred: Thanks for letting my know that zuul-v3 is still working in progress. Could you share the release plan? I would just want to make my own plan basing on it. As it seems I have to upgrade my CI env to zuul-v3, that it can support multi-nodes with different images. | 05:07 |
| *** elo has joined #openstack-infra | 05:07 | |
| *** vgridnev has joined #openstack-infra | 05:11 | |
| *** tonytan4ever has quit IRC | 05:11 | |
| *** raunak has quit IRC | 05:12 | |
| *** raunak has joined #openstack-infra | 05:16 | |
| fifieldt | hi infra | 05:20 |
| fifieldt | anyone have easy log access to the stuff backing wiki.o.o ? I was just POSTing to https://wiki.openstack.org/wiki/Special:UserRights and it gave a blank page -- probably a HTTP 500 or so | 05:20 |
| *** rbuzatu has joined #openstack-infra | 05:21 | |
| *** raunak has quit IRC | 05:22 | |
| *** eandersson_ has joined #openstack-infra | 05:24 | |
| *** jimbaker has quit IRC | 05:25 | |
| *** rbuzatu has quit IRC | 05:26 | |
| openstackgerrit | vikas choudhary proposed openstack-infra/project-config: Fix dsvm installation job https://review.openstack.org/346717 | 05:27 |
| *** jogo has quit IRC | 05:28 | |
| *** amotoki has joined #openstack-infra | 05:28 | |
| *** jimbaker has joined #openstack-infra | 05:28 | |
| *** jimbaker has quit IRC | 05:28 | |
| *** jimbaker has joined #openstack-infra | 05:28 | |
| *** vgridnev has quit IRC | 05:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add TripleO RSS Feed from Juan Antonio Osorio's blog https://review.openstack.org/356898 | 05:30 |
| *** vgridnev has joined #openstack-infra | 05:31 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Use temp file for du calculations https://review.openstack.org/356917 | 05:31 |
| ianw | pabelanger: this might be part of it ... | 05:31 |
| *** amotoki has quit IRC | 05:32 | |
| *** asettle has joined #openstack-infra | 05:35 | |
| *** rajinir has quit IRC | 05:35 | |
| *** pilgrimstack has joined #openstack-infra | 05:35 | |
| *** rbuzatu has joined #openstack-infra | 05:36 | |
| *** r-mibu has quit IRC | 05:37 | |
| *** tphummel has quit IRC | 05:37 | |
| *** asettle has quit IRC | 05:39 | |
| *** thorst_ has joined #openstack-infra | 05:40 | |
| *** pilgrimstack has quit IRC | 05:40 | |
| *** jogo has joined #openstack-infra | 05:42 | |
| *** vgridnev has quit IRC | 05:44 | |
| *** vgridnev has joined #openstack-infra | 05:47 | |
| *** r-mibu has joined #openstack-infra | 05:47 | |
| *** thorst_ has quit IRC | 05:47 | |
| *** abregman has joined #openstack-infra | 05:50 | |
| *** amitgandhinz has joined #openstack-infra | 05:53 | |
| *** roxanaghe has joined #openstack-infra | 05:55 | |
| *** amitgandhinz has quit IRC | 05:57 | |
| *** Cormite has joined #openstack-infra | 05:57 | |
| *** pilgrimstack has joined #openstack-infra | 05:58 | |
| *** pilgrimstack has quit IRC | 05:58 | |
| *** roxanaghe has quit IRC | 05:59 | |
| *** Hal1 has joined #openstack-infra | 06:00 | |
| *** Hal has quit IRC | 06:00 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Pass TRIPLEO_ROOT directory to heat_deploy_times.sh https://review.openstack.org/356946 | 06:01 |
| *** spzala has joined #openstack-infra | 06:03 | |
| *** vgridnev has quit IRC | 06:03 | |
| *** vgridnev has joined #openstack-infra | 06:03 | |
| *** masterLog1x has joined #openstack-infra | 06:05 | |
| *** jimbaker has quit IRC | 06:06 | |
| *** spzala has quit IRC | 06:07 | |
| yolanda | hi timrc , it will be awesome to meet again | 06:07 |
| *** eandersson_ has quit IRC | 06:07 | |
| *** jimbaker has joined #openstack-infra | 06:10 | |
| *** jimbaker has quit IRC | 06:10 | |
| *** jimbaker has joined #openstack-infra | 06:10 | |
| *** vgridnev has quit IRC | 06:10 | |
| *** _nadya_ has joined #openstack-infra | 06:12 | |
| *** _nadya_ has quit IRC | 06:12 | |
| *** rcernin has joined #openstack-infra | 06:12 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** amotoki has joined #openstack-infra | 06:22 | |
| *** amotoki has quit IRC | 06:26 | |
| *** amotoki has joined #openstack-infra | 06:29 | |
| *** masterLog1x has quit IRC | 06:31 | |
| *** _nadya_ has joined #openstack-infra | 06:31 | |
| *** _nadya_ has quit IRC | 06:38 | |
| *** javeriak has joined #openstack-infra | 06:40 | |
| yolanda | good morning | 06:41 |
| *** thorst_ has joined #openstack-infra | 06:45 | |
| *** amotoki has quit IRC | 06:51 | |
| *** adriant has quit IRC | 06:52 | |
| *** thorst_ has quit IRC | 06:52 | |
| *** pcaruana has joined #openstack-infra | 06:52 | |
| *** amotoki has joined #openstack-infra | 06:56 | |
| *** armax has joined #openstack-infra | 06:57 | |
| *** yamamoto has joined #openstack-infra | 07:00 | |
| *** amotoki has quit IRC | 07:04 | |
| *** ganesan has joined #openstack-infra | 07:06 | |
| *** elo has quit IRC | 07:08 | |
| *** xarses has quit IRC | 07:08 | |
| *** xarses has joined #openstack-infra | 07:09 | |
| *** elo has joined #openstack-infra | 07:10 | |
| *** vgridnev has joined #openstack-infra | 07:12 | |
| *** pilgrimstack has joined #openstack-infra | 07:16 | |
| *** tesseract- has joined #openstack-infra | 07:17 | |
| *** armax has quit IRC | 07:18 | |
| *** e0ne has joined #openstack-infra | 07:18 | |
| *** nmagnezi has joined #openstack-infra | 07:21 | |
| *** matrohon has joined #openstack-infra | 07:26 | |
| *** Qiming_ has joined #openstack-infra | 07:27 | |
| *** Qiming_ has quit IRC | 07:28 | |
| *** florianf has joined #openstack-infra | 07:36 | |
| *** esikachev has joined #openstack-infra | 07:37 | |
| *** _nadya_ has joined #openstack-infra | 07:38 | |
| *** yamamoto has quit IRC | 07:38 | |
| *** jpich has joined #openstack-infra | 07:38 | |
| *** rbuzatu has quit IRC | 07:39 | |
| *** rvba has joined #openstack-infra | 07:40 | |
| *** rvba has quit IRC | 07:40 | |
| *** rvba has joined #openstack-infra | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Corrected success-pattern for npm-based doc builds https://review.openstack.org/356606 | 07:40 |
| *** roxanaghe has joined #openstack-infra | 07:43 | |
| *** matrohon has quit IRC | 07:43 | |
| *** matthewbodkin has joined #openstack-infra | 07:44 | |
| *** dizquierdo has joined #openstack-infra | 07:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Move networking-ovn grenade CI job from experimental to check https://review.openstack.org/356474 | 07:45 |
| *** shashank_hegde has quit IRC | 07:46 | |
| *** roxanaghe has quit IRC | 07:47 | |
| *** thorst_ has joined #openstack-infra | 07:50 | |
| *** sshnaidm|afk has quit IRC | 07:50 | |
| rcarrillocruz | timrc: ohh, that's awesome. Sangria drinking levels is proportional to the fruit included in the jar, the sweeter, the more 'omg, i love this, another glass please!' | 07:52 |
| rcarrillocruz | :P | 07:52 |
| *** abregman has quit IRC | 07:53 | |
| *** amitgandhinz has joined #openstack-infra | 07:54 | |
| *** tqtran has joined #openstack-infra | 07:54 | |
| *** Gene_ has joined #openstack-infra | 07:55 | |
| *** amotoki has joined #openstack-infra | 07:55 | |
| *** Gene_ has quit IRC | 07:55 | |
| *** thorst_ has quit IRC | 07:56 | |
| openstackgerrit | mariam john proposed openstack-infra/project-config: Update trove job to install trove tempest plugin https://review.openstack.org/356999 | 07:57 |
| *** amotoki_ has joined #openstack-infra | 07:57 | |
| *** Genek has joined #openstack-infra | 07:58 | |
| *** pilgrimstack has quit IRC | 07:58 | |
| *** tqtran has quit IRC | 07:59 | |
| *** amitgandhinz has quit IRC | 07:59 | |
| *** dizquierdo has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** amotoki has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** sshnaidm has joined #openstack-infra | 08:02 | |
| *** spzala has joined #openstack-infra | 08:04 | |
| *** jimbaker has quit IRC | 08:05 | |
| *** ganesan has quit IRC | 08:08 | |
| *** ihrachys has joined #openstack-infra | 08:08 | |
| *** spzala has quit IRC | 08:08 | |
| *** jimbaker has joined #openstack-infra | 08:09 | |
| *** jimbaker has quit IRC | 08:09 | |
| *** jimbaker has joined #openstack-infra | 08:09 | |
| *** mhickey has joined #openstack-infra | 08:09 | |
| *** lennyb has quit IRC | 08:13 | |
| *** berendt has joined #openstack-infra | 08:17 | |
| *** pilgrimstack has joined #openstack-infra | 08:18 | |
| *** jgrassler has joined #openstack-infra | 08:20 | |
| *** pilgrimstack has quit IRC | 08:20 | |
| *** sslypushenko has joined #openstack-infra | 08:22 | |
| *** dosaboy_ has joined #openstack-infra | 08:23 | |
| *** dosaboy_ has quit IRC | 08:24 | |
| *** dosaboy has quit IRC | 08:24 | |
| sslypushenko | Hi, folks! Can I ask for review/merge of this patch https://review.openstack.org/#/c/353861/? I has pretty much a lot of +1's | 08:24 |
| *** eandersson_ has joined #openstack-infra | 08:26 | |
| *** pilgrimstack has joined #openstack-infra | 08:26 | |
| *** vikasc has joined #openstack-infra | 08:27 | |
| *** lennyb has joined #openstack-infra | 08:27 | |
| *** Goneri has joined #openstack-infra | 08:29 | |
| vikasc | hello folks, Can you please review https://review.openstack.org/#/c/346717/ ? | 08:29 |
| *** dosaboy has joined #openstack-infra | 08:29 | |
| *** armax has joined #openstack-infra | 08:31 | |
| *** javeriak has quit IRC | 08:33 | |
| *** jerryz has quit IRC | 08:34 | |
| *** ihrachys has quit IRC | 08:36 | |
| *** vgridnev has quit IRC | 08:37 | |
| *** jaosorior has quit IRC | 08:37 | |
| ianw | pabelanger: success! f24 -> http://logs.openstack.org/06/343506/2/experimental/gate-tempest-dsvm-platform-fedora24/5fb9c73/ | 08:37 |
| *** pilgrimstack has quit IRC | 08:38 | |
| ianw | pabelanger: still working on the ubutnu-xenial build. it is *not* totally due to -> https://review.openstack.org/356917. i'm fiddling directly on nodepool to try and figure it out. maybe egrep is segfaulting? | 08:38 |
| *** yaume has joined #openstack-infra | 08:39 | |
| *** asettle has joined #openstack-infra | 08:39 | |
| *** yamamoto has joined #openstack-infra | 08:39 | |
| *** vgridnev has joined #openstack-infra | 08:40 | |
| apuimedo | thanks vikasc | 08:40 |
| *** rbuzatu has joined #openstack-infra | 08:40 | |
| vikasc | apuimedo, no worries | 08:40 |
| *** rbuzatu has quit IRC | 08:44 | |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** nwkarsten has quit IRC | 08:48 | |
| *** esikachev has quit IRC | 08:49 | |
| *** dizquierdo has joined #openstack-infra | 08:52 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Custom Jenkins vhost https://review.openstack.org/259954 | 08:52 |
| *** yamamoto has quit IRC | 08:52 | |
| *** pilgrimstack has joined #openstack-infra | 08:53 | |
| *** abregman has joined #openstack-infra | 08:54 | |
| *** pilgrimstack has quit IRC | 08:55 | |
| *** thorst_ has joined #openstack-infra | 08:55 | |
| *** amitgandhinz has joined #openstack-infra | 08:55 | |
| *** pilgrimstack has joined #openstack-infra | 08:55 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Temporarily add rabbit keys to hiera https://review.openstack.org/357021 | 08:55 |
| azvyagintsev | yolanda could you please merge https://review.openstack.org/#/c/353861/ ? | 08:56 |
| *** yamamoto has joined #openstack-infra | 08:56 | |
| openstackgerrit | liyuanzhen proposed openstack-infra/jenkins-job-builder: Not limit stevedore to 1.8.0 https://review.openstack.org/357023 | 08:56 |
| *** yamamoto has quit IRC | 08:56 | |
| *** yamamoto has joined #openstack-infra | 08:56 | |
| *** Cormite has quit IRC | 08:57 | |
| *** roxanaghe has joined #openstack-infra | 08:58 | |
| *** amitgandhinz has quit IRC | 08:59 | |
| *** dkehn_ has quit IRC | 09:00 | |
| *** dkehn has quit IRC | 09:00 | |
| *** pilgrimstack has quit IRC | 09:00 | |
| sslypushenko | yolanda: +1 to request for https://review.openstack.org/#/c/353861/ )) | 09:01 |
| *** esikachev has joined #openstack-infra | 09:02 | |
| *** thorst_ has quit IRC | 09:02 | |
| *** spzala has joined #openstack-infra | 09:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/357032 | 09:06 |
| *** _coolsvap_ is now known as coolsvap | 09:06 | |
| *** e0ne has quit IRC | 09:07 | |
| *** dkehn has joined #openstack-infra | 09:07 | |
| *** d0ugal has quit IRC | 09:08 | |
| *** spzala has quit IRC | 09:09 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** ihrachys has joined #openstack-infra | 09:10 | |
| *** d0ugal has joined #openstack-infra | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:12 | |
| *** yamamoto has quit IRC | 09:12 | |
| *** d0ugal_ has joined #openstack-infra | 09:17 | |
| *** d0ugal_ has quit IRC | 09:18 | |
| *** d0ugal_ has joined #openstack-infra | 09:18 | |
| *** dkehn_ has joined #openstack-infra | 09:18 | |
| *** Genek has quit IRC | 09:19 | |
| *** dtardivel has joined #openstack-infra | 09:19 | |
| *** d0ugal has quit IRC | 09:20 | |
| *** d0ugal_ has quit IRC | 09:20 | |
| *** d0ugal has joined #openstack-infra | 09:20 | |
| *** roxanaghe has quit IRC | 09:21 | |
| *** pilgrimstack has joined #openstack-infra | 09:24 | |
| *** ihrachys has quit IRC | 09:24 | |
| *** sambetts|afk is now known as sambetts | 09:25 | |
| *** electrofelix has joined #openstack-infra | 09:28 | |
| *** yamamoto has joined #openstack-infra | 09:30 | |
| *** yamamoto has quit IRC | 09:30 | |
| *** yamamoto has joined #openstack-infra | 09:30 | |
| *** yamamoto has quit IRC | 09:30 | |
| *** yamamoto has joined #openstack-infra | 09:31 | |
| openstackgerrit | Merged openstack-dev/pbr: Replace OpenStack LLC with OpenStack Foundation https://review.openstack.org/347299 | 09:31 |
| wznoinsk | mmedvede: around? | 09:32 |
| *** sshnaidm has quit IRC | 09:32 | |
| *** acoles_ is now known as acoles | 09:34 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** sshnaidm has joined #openstack-infra | 09:36 | |
| *** _degorenko|afk is now known as degorenko | 09:38 | |
| *** berendt has quit IRC | 09:40 | |
| *** berendt has joined #openstack-infra | 09:44 | |
| frickler | have there been known issues earlier with uploading test results? see the post_failures e.g. here https://review.openstack.org/338435 | 09:45 |
| *** pilgrimstack has quit IRC | 09:46 | |
| *** jordanP has joined #openstack-infra | 09:47 | |
| vgridnev | team, can someone help me with approving this one? https://review.openstack.org/#/c/354700/ thanks in advance | 09:48 |
| *** tosky has joined #openstack-infra | 09:49 | |
| vgridnev | it's really frustrating that patch is | 09:50 |
| vgridnev | long reviewed | 09:50 |
| vgridnev | thanks a lot! | 09:51 |
| *** skraynev__ has quit IRC | 09:51 | |
| *** furlongm has quit IRC | 09:54 | |
| *** furlongm has joined #openstack-infra | 09:54 | |
| *** asettle has quit IRC | 09:55 | |
| *** asettle has joined #openstack-infra | 09:55 | |
| *** amitgandhinz has joined #openstack-infra | 09:56 | |
| *** ihrachys has joined #openstack-infra | 09:56 | |
| *** eranrom has quit IRC | 09:56 | |
| *** eranrom has joined #openstack-infra | 09:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:57 | |
| *** zhurong has quit IRC | 09:58 | |
| *** d0ugal has quit IRC | 09:58 | |
| *** jimbaker has quit IRC | 09:58 | |
| *** yanyanhu has quit IRC | 09:59 | |
| *** amitgandhinz has quit IRC | 10:00 | |
| *** yamamoto has joined #openstack-infra | 10:01 | |
| *** yamamoto has quit IRC | 10:01 | |
| *** jimbaker has joined #openstack-infra | 10:02 | |
| *** jimbaker has quit IRC | 10:02 | |
| *** jimbaker has joined #openstack-infra | 10:02 | |
| *** dizquierdo has quit IRC | 10:03 | |
| *** d0ugal has joined #openstack-infra | 10:04 | |
| *** spzala has joined #openstack-infra | 10:05 | |
| *** sdague has joined #openstack-infra | 10:06 | |
| *** ihrachys has quit IRC | 10:07 | |
| *** javeriak has joined #openstack-infra | 10:09 | |
| *** spzala has quit IRC | 10:10 | |
| *** kzaitsev_mb has quit IRC | 10:12 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set public and private IP addresses for compute002.vanilla https://review.openstack.org/357066 | 10:12 |
| *** amotoki_ has quit IRC | 10:14 | |
| *** berendt has quit IRC | 10:14 | |
| *** berendt has joined #openstack-infra | 10:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute004.vanilla ip addresses https://review.openstack.org/357070 | 10:15 |
| *** amotoki has joined #openstack-infra | 10:18 | |
| *** skraynev has joined #openstack-infra | 10:18 | |
| *** dizquierdo has joined #openstack-infra | 10:20 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute006.vanilla addresses https://review.openstack.org/357073 | 10:21 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute007 IP addresses https://review.openstack.org/357075 | 10:22 |
| *** dizquierdo has quit IRC | 10:28 | |
| *** vgridnev has quit IRC | 10:30 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add neutron-lib source periodic job to networking-ovn https://review.openstack.org/357079 | 10:30 |
| *** berendt has quit IRC | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** ociuhandu has quit IRC | 10:33 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute009.vanilla IP addresses https://review.openstack.org/357081 | 10:34 |
| *** jimbaker has quit IRC | 10:35 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute010.vanilla IP addresses https://review.openstack.org/357084 | 10:36 |
| *** esikachev has quit IRC | 10:36 | |
| *** yaume has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute011.vanilla IP addresses https://review.openstack.org/357087 | 10:38 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute012.vanilla IP addresses https://review.openstack.org/357088 | 10:39 |
| *** jimbaker has joined #openstack-infra | 10:39 | |
| *** jimbaker has quit IRC | 10:39 | |
| *** jimbaker has joined #openstack-infra | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| *** rbuzatu has joined #openstack-infra | 10:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute013.vanilla IP addresses https://review.openstack.org/357091 | 10:42 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute014.vanilla IP addresses https://review.openstack.org/357092 | 10:43 |
| *** berendt has joined #openstack-infra | 10:44 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add neutron-lib source periodic job to networking-ovn https://review.openstack.org/357079 | 10:44 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute015.vanilla IP addresses https://review.openstack.org/357093 | 10:44 |
| *** roxanaghe has joined #openstack-infra | 10:45 | |
| *** amotoki has quit IRC | 10:45 | |
| *** esikachev has joined #openstack-infra | 10:45 | |
| *** amotoki_ has joined #openstack-infra | 10:46 | |
| *** rbuzatu has quit IRC | 10:46 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute016.vanilla IP addresses https://review.openstack.org/357094 | 10:46 |
| flip214 | when using logstash.openstack.org, how do I get to an URL with the filter that I built? | 10:46 |
| flip214 | "share" on top right says "query=", "save" doesn't give me an URL either | 10:47 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute017.vanilla IP addresses https://review.openstack.org/357096 | 10:48 |
| *** Genek has joined #openstack-infra | 10:49 | |
| *** roxanaghe has quit IRC | 10:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute018.vanilla IP addresses https://review.openstack.org/357098 | 10:50 |
| *** vgridnev has joined #openstack-infra | 10:51 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute019.vanilla IP addresses https://review.openstack.org/357100 | 10:52 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute020.vanilla IP addresses https://review.openstack.org/357102 | 10:53 |
| *** coolsvap is now known as _coolsvap_ | 10:55 | |
| *** amitgandhinz has joined #openstack-infra | 10:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:57 | |
| *** eranrom has quit IRC | 10:57 | |
| *** eranrom has joined #openstack-infra | 10:57 | |
| *** eranrom_ has joined #openstack-infra | 10:58 | |
| *** eranrom has quit IRC | 10:58 | |
| *** oanson has joined #openstack-infra | 11:00 | |
| *** amitgandhinz has quit IRC | 11:01 | |
| *** jkilpatr has quit IRC | 11:01 | |
| *** thorst_ has joined #openstack-infra | 11:04 | |
| *** esikachev has quit IRC | 11:06 | |
| ianw | pabelanger: ok, not much closer on this ubuntu-xenail fail. have a look at my local edits to /usr/local/bin/disk-image-builder (search ianw) | 11:06 |
| *** spzala has joined #openstack-infra | 11:06 | |
| flip214 | trying to fetch logs.openstack.org files via wget could make use of some optimizations... | 11:06 |
| flip214 | "Last-modified header missing -- time-stamps turned off. | 11:07 |
| flip214 | and so an interrupted transfer needs to fetch all files anew | 11:07 |
| zigo | pabelanger: For when you wake up: repo signing worked, look here: http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton-backports/ \o/ | 11:07 |
| zigo | However, how do we get the public key? | 11:07 |
| *** caowei has quit IRC | 11:07 | |
| zigo | Could you drop the public-key.gpg somewhere in the repo? | 11:07 |
| ianw | pabelanger: outputting the image size report fails. the final "cat" seems to bail out when it dumps. i've put it under strace and i'll need to output of that to keep debugging. other builds hvae started, and it's late here, so i'll ooop back on this in the morning | 11:08 |
| *** oanson has quit IRC | 11:09 | |
| *** spzala has quit IRC | 11:11 | |
| *** BobBall is now known as BobBall_AWOL | 11:11 | |
| openstackgerrit | Merged openstack-infra/groups: Custom membership fields and reporting https://review.openstack.org/352504 | 11:13 |
| *** lucasagomes is now known as lucas-hungry | 11:13 | |
| *** dizquierdo has joined #openstack-infra | 11:17 | |
| *** rbuzatu has joined #openstack-infra | 11:22 | |
| *** spzala has joined #openstack-infra | 11:23 | |
| *** spzala has quit IRC | 11:23 | |
| *** dizquierdo has quit IRC | 11:24 | |
| *** asettle has quit IRC | 11:25 | |
| *** asettle has joined #openstack-infra | 11:26 | |
| *** Genek has quit IRC | 11:27 | |
| *** vgridnev has quit IRC | 11:29 | |
| azvyagintsev | yolanda: ping https://review.openstack.org/#/c/353861/ :) folks answer 'yes' | 11:29 |
| *** berendt has quit IRC | 11:29 | |
| *** vgridnev has joined #openstack-infra | 11:29 | |
| *** Genek has joined #openstack-infra | 11:30 | |
| *** berendt has joined #openstack-infra | 11:30 | |
| *** ramishra has quit IRC | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** vgridnev has quit IRC | 11:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 11:33 |
| *** vgridnev has joined #openstack-infra | 11:33 | |
| *** ramishra has joined #openstack-infra | 11:33 | |
| *** nwkarsten has joined #openstack-infra | 11:35 | |
| *** changzhi has quit IRC | 11:36 | |
| *** changzhi has joined #openstack-infra | 11:37 | |
| *** jkilpatr has joined #openstack-infra | 11:38 | |
| zigo | ianw: Do you know if it is possible to have just *one* more ready Jessie node, waiting for incoming jobs? | 11:39 |
| *** ociuhandu has joined #openstack-infra | 11:39 | |
| zigo | That would help us *a lot*. | 11:39 |
| openstackgerrit | Merged openstack-infra/system-config: Set public and private IP addresses for compute002.vanilla https://review.openstack.org/357066 | 11:40 |
| *** pilgrimstack has joined #openstack-infra | 11:43 | |
| *** changzhi has quit IRC | 11:43 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add non-voting coverage job to networking-ovn https://review.openstack.org/357138 | 11:44 |
| *** kzaitsev_mb has quit IRC | 11:44 | |
| *** Goneri has quit IRC | 11:44 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add non-voting coverage job to networking-ovn https://review.openstack.org/357138 | 11:45 |
| *** nwkarsten has quit IRC | 11:45 | |
| *** roxanaghe has joined #openstack-infra | 11:46 | |
| *** armax has quit IRC | 11:46 | |
| *** ayoung has quit IRC | 11:50 | |
| *** roxanaghe has quit IRC | 11:50 | |
| *** dprince has joined #openstack-infra | 11:52 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Zun: Add python-zunclient to dsvm https://review.openstack.org/357144 | 11:53 |
| ianw | zigo: you we can up the nodes, see project-config:openstack-infra/nodepool/nodepool.yaml config | 11:55 |
| *** armax has joined #openstack-infra | 11:55 | |
| *** tqtran has joined #openstack-infra | 11:56 | |
| *** Genek has quit IRC | 11:57 | |
| *** amitgandhinz has joined #openstack-infra | 11:57 | |
| *** Goneri has joined #openstack-infra | 12:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:00 | |
| *** tqtran has quit IRC | 12:00 | |
| *** amitgandhinz has quit IRC | 12:02 | |
| *** annegentle has joined #openstack-infra | 12:02 | |
| *** rbuzatu has quit IRC | 12:09 | |
| *** e0ne has joined #openstack-infra | 12:09 | |
| *** woodster_ has joined #openstack-infra | 12:10 | |
| tobiash | hi, is it possible in nodepool to use a single image with multiple flavours in different labels? | 12:14 |
| tobiash | as far as I can see in the docs the flavor is attached to the image itself | 12:14 |
| *** baoli has joined #openstack-infra | 12:15 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| *** ansmith has joined #openstack-infra | 12:15 | |
| *** baoli_ has joined #openstack-infra | 12:16 | |
| *** nwkarsten has joined #openstack-infra | 12:16 | |
| *** Julien-zte has joined #openstack-infra | 12:19 | |
| *** lucas-hungry is now known as lucasagomes | 12:19 | |
| *** baoli has quit IRC | 12:20 | |
| *** ihrachys has joined #openstack-infra | 12:20 | |
| *** berendt has quit IRC | 12:22 | |
| *** javeriak has quit IRC | 12:22 | |
| *** tpsilva has joined #openstack-infra | 12:22 | |
| *** nwkarsten has quit IRC | 12:23 | |
| *** spzala has joined #openstack-infra | 12:24 | |
| *** nwkarsten has joined #openstack-infra | 12:24 | |
| *** matthewbodkin has quit IRC | 12:24 | |
| mordred | tobiash: that's right, we do not currently have much in the way of multi-flavor support | 12:25 |
| tobiash | mordred: would a patch in this direction be accepted? | 12:25 |
| tobiash | then I'd try to implement this as I have a use case for this in my environment | 12:26 |
| *** gouthamr has joined #openstack-infra | 12:26 | |
| *** gordc has joined #openstack-infra | 12:27 | |
| tobiash | I'm currently thinking of specifying min-ram also in the labels section which could override the image min-ram | 12:28 |
| sdague | http://logs.openstack.org/14/329514/7/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7795b10/console.html - internap mirror / network issues? | 12:28 |
| *** nwkarsten has quit IRC | 12:28 | |
| *** boogibugs has quit IRC | 12:28 | |
| sdague | http://logs.openstack.org/14/329514/7/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7795b10/console.html#_2016-08-18_12_26_55_784536 | 12:29 |
| *** spzala has quit IRC | 12:29 | |
| *** fguillot has joined #openstack-infra | 12:29 | |
| sdague | related - http://logs.openstack.org/37/352837/1/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/a99a5f5/logs/grenade.sh.txt.gz#_2016-08-18_11_09_12_578 | 12:29 |
| *** fguillot has quit IRC | 12:30 | |
| sdague | looks like the only 2 fails in the gate right now are internap network | 12:30 |
| *** mdrabe has joined #openstack-infra | 12:30 | |
| *** pradk has joined #openstack-infra | 12:31 | |
| *** ihrachys has quit IRC | 12:31 | |
| mordred | tobiash: we're in a little bit of an awkward moment to accept a patch like that, as we're currently working on a large feature branch in service of zuul v3 ... | 12:32 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add TripleO RSS Feed from Juan Antonio Osorio's blog https://review.openstack.org/356898 | 12:32 |
| *** ihrachys has joined #openstack-infra | 12:32 | |
| mordred | tobiash: I think we should get jeblair's take on such a patch before you write it ... it might also be that we're going to have that use case covered in zuul v3, or it might be a thing that we need to add to the list for v3 and that we'd welcome the help implementing | 12:32 |
| *** asettle has quit IRC | 12:34 | |
| *** asettle has joined #openstack-infra | 12:34 | |
| sdague | yeh, http://logs.openstack.org/01/340601/34/check/gate-tempest-dsvm-postgres-full-ubuntu-xenial/893cdd5/logs/devstacklog.txt.gz - also internap network | 12:35 |
| *** thiagop has joined #openstack-infra | 12:36 | |
| mordred | sdague: wow, it's neat when network stuff works poorly inside of a cloudregion | 12:37 |
| tobiash | mordred: ok, then I'll go for multiple image definitions for now as a workaround | 12:37 |
| *** annegentle has quit IRC | 12:37 | |
| tobiash | thanks for the info | 12:37 |
| mordred | tobiash: yah - sorry about the awkward timing part - but I think it's a worthwhile request | 12:38 |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** jcoufal has joined #openstack-infra | 12:40 | |
| tobiash | mordred: btw, is there a rough time schedule for zuul v3 somewhere? Didn't find much info about that topic apart from the blueprint and the patches in gerrit. | 12:40 |
| *** berendt has joined #openstack-infra | 12:41 | |
| mordred | tobiash: "as soon as we can get it ready" ... we've got some people working on it full time now, so we're hoping that it's not more than a couple of months out - but it's a big beastie, so hard to predict the first release | 12:42 |
| sdague | mordred: so far from perfect because the data we have, but I think basically networking issues there starting about 8 hours ago | 12:43 |
| sdague | http://logstash.openstack.org/#dashboard/file/logstash.json?query=node_provider%3A%5C%22internap-nyj01%5C%22%20AND%20message%3A%5C%22Job%20complete%5C%22%20AND%20voting%3A1%20AND%20build_name%3A*dsvm*%20AND%20build_queue%3Agate | 12:43 |
| sdague | we get to a 40% fail rate in the gate on dsvm jobs (which do reasonable amount of network ops) starting about then | 12:43 |
| *** burgerk has joined #openstack-infra | 12:43 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Group contanct report csv dump filter by group status https://review.openstack.org/357175 | 12:44 |
| sdague | previously, it was negligible | 12:44 |
| mordred | mgagne: ^^ | 12:44 |
| *** zhurong has joined #openstack-infra | 12:45 | |
| *** rodrigods has quit IRC | 12:48 | |
| *** rodrigods has joined #openstack-infra | 12:48 | |
| *** pilgrimstack has quit IRC | 12:49 | |
| *** gouthamr is now known as gouthamr_afk | 12:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: ovb: create dummy eth1 https://review.openstack.org/357181 | 12:51 |
| *** e0ne has quit IRC | 12:52 | |
| *** kgiusti has joined #openstack-infra | 12:52 | |
| *** julim has joined #openstack-infra | 12:53 | |
| *** nwkarsten has joined #openstack-infra | 12:53 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Remove diskimage-builder repo elements folder from ELEMENTS_PATH https://review.openstack.org/357183 | 12:54 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Up the debian-jessie min-ready https://review.openstack.org/357184 | 12:54 |
| *** e0ne has joined #openstack-infra | 12:55 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Add vlan_id variable with value 2551 https://review.openstack.org/357187 | 12:57 |
| *** jtomasek has quit IRC | 12:57 | |
| *** amitgandhinz has joined #openstack-infra | 12:58 | |
| *** esikachev has joined #openstack-infra | 12:58 | |
| *** ihrachys has quit IRC | 12:59 | |
| *** raildo has joined #openstack-infra | 13:00 | |
| *** amotoki_ has quit IRC | 13:00 | |
| *** weshay_ has joined #openstack-infra | 13:00 | |
| *** pilgrimstack has joined #openstack-infra | 13:00 | |
| *** ihrachys has joined #openstack-infra | 13:02 | |
| asselin | rcarrillocruz, is this right? http://paste.openstack.org/show/560795/ | 13:02 |
| asselin | should the loop_var be item_cloud? | 13:02 |
| *** amitgandhinz has quit IRC | 13:02 | |
| *** nwkarsten has quit IRC | 13:04 | |
| asselin | rcarrillocruz, here's my input command: http://paste.openstack.org/show/560796/ | 13:04 |
| *** rbuzatu has joined #openstack-infra | 13:04 | |
| *** esikache1 has joined #openstack-infra | 13:04 | |
| *** nwkarsten has joined #openstack-infra | 13:04 | |
| *** pilgrimstack has quit IRC | 13:06 | |
| *** matthewbodkin has joined #openstack-infra | 13:06 | |
| *** amotoki has joined #openstack-infra | 13:06 | |
| *** esikachev has quit IRC | 13:06 | |
| *** oanson has joined #openstack-infra | 13:08 | |
| *** psilvad has joined #openstack-infra | 13:08 | |
| kzaitsev_mb | folks, DSS Cinder CI is commenting on Murano patches (and says that our patches are in merge failed) О_о | 13:11 |
| *** esikache1 has quit IRC | 13:11 | |
| kzaitsev_mb | сan anyone point me how to debug this situation =) | 13:11 |
| rcarrillocruz | not sure , i haven't used the launch_node playbook in a long time, it's still in review | 13:11 |
| rcarrillocruz | the include/with_items nested | 13:11 |
| rcarrillocruz | and loop_var stuff | 13:11 |
| rcarrillocruz | doesn't work on ansible < 2.1.1.0 | 13:12 |
| rcarrillocruz | you may want to check that | 13:12 |
| rcarrillocruz | other than that, put a debug on the var before the line it fails to see what's up | 13:12 |
| *** inc0 has joined #openstack-infra | 13:12 | |
| rcarrillocruz | - debug: var=vars | 13:12 |
| asselin | rcarrillocruz, ok i'm on ansible==2.1.1.0 | 13:12 |
| asselin | , but yeah I upgraded cloud-launcher and still using the previous playbooks | 13:12 |
| rcarrillocruz | ^ shows everything | 13:12 |
| kzaitsev_mb | https://review.openstack.org/#/c/355369/ this commit for example | 13:12 |
| *** xyang1 has joined #openstack-infra | 13:12 | |
| *** ihrachys has quit IRC | 13:13 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute004.vanilla ip addresses https://review.openstack.org/357070 | 13:14 |
| *** mhickey has quit IRC | 13:14 | |
| *** ihrachys has joined #openstack-infra | 13:16 | |
| *** links has quit IRC | 13:16 | |
| asselin | kzaitsev_mb, in addition to reporting here: try e-mailing them: DSS Cinder CIbudik_06@list.ru | 13:17 |
| kzaitsev_mb | asselin: thanks, will do ) | 13:17 |
| rcarrillocruz | erm | 13:17 |
| rcarrillocruz | yeah | 13:18 |
| *** ihrachys has quit IRC | 13:18 | |
| rcarrillocruz | i just got a vote from DSS Cinder CI on the puppet-infracloud | 13:18 |
| rcarrillocruz | https://review.openstack.org/#/c/357187/ | 13:18 |
| rcarrillocruz | or well, a comment rather | 13:18 |
| *** Julien-zte has quit IRC | 13:18 | |
| *** berendt has quit IRC | 13:18 | |
| cloudnull | mornings all | 13:19 |
| sdague | man in the middle error in the gate - http://logs.openstack.org/27/348427/4/gate/gate-tempest-dsvm-full-devstack-plugin-ceph/97a875e/_zuul_ansible/ansible_log.txt ??? | 13:19 |
| *** amitgandhinz has joined #openstack-infra | 13:20 | |
| mordred | sdague: is it possible that something in that job changed the ssh host key? | 13:20 |
| sdague | mordred: don't know, it's super early | 13:21 |
| *** mdrabe has quit IRC | 13:21 | |
| sdague | that ansible log is basically all the logs that exist | 13:21 |
| mordred | yah | 13:21 |
| cloudnull | how has the re-enablement of trusty on OSIC been going? Anything still needing to be worked on / looked at there? | 13:22 |
| odyssey4me | Odd - I'm starting to see 'DSS Cinder CI' comment on some of our patches... it's clearly external CI, but I am wondering why they're subscribed to our repo. Is there a page somewhere where I can find the owner of the CI to contact? | 13:22 |
| mordred | cloudnull: jeblair wrote the entropy pump yesterday | 13:22 |
| mordred | cloudnull: I believe pabelanger has the systemd unit file for it up for review | 13:22 |
| cloudnull | yup, which was great work jeblair! | 13:23 |
| cloudnull | cool | 13:23 |
| mordred | oh - actually - it looks like the disable trusty patch revert has been merged | 13:24 |
| azvyagintsev | Folks, can we merge https://review.openstack.org/#/c/353861/ ? | 13:24 |
| *** mriedem has joined #openstack-infra | 13:25 | |
| mordred | but this isn't merged: https://review.openstack.org/#/c/356706 | 13:25 |
| mordred | so I'm honestly not sure what the status is | 13:25 |
| odyssey4me | heh, this is a pretty cool tool which I had no idea existed: http://ci-watch.tintri.com/home | 13:25 |
| *** Julien-zte has joined #openstack-infra | 13:26 | |
| *** matt-borland has joined #openstack-infra | 13:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins https://review.openstack.org/357217 | 13:26 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute006.vanilla addresses https://review.openstack.org/357073 | 13:27 |
| *** amotoki has quit IRC | 13:27 | |
| *** mhickey has joined #openstack-infra | 13:28 | |
| cloudnull | mordred: last night it was found that the init-urandom bits were needed in fedora too. -- https://review.openstack.org/#/q/topic:initialize-urandom -- believe ianw found the issue, I just made the PR. | 13:28 |
| *** vikrant has quit IRC | 13:30 | |
| sdague | gerrit reference problem? | 13:31 |
| *** dmitryme has joined #openstack-infra | 13:31 | |
| sdague | https://review.openstack.org/#/c/356694 - I can't seem to -d that change | 13:31 |
| sdague | http://paste.openstack.org/show/560800/ | 13:31 |
| *** mdrabe has joined #openstack-infra | 13:33 | |
| cloudnull | welp, no news is good news :) | 13:33 |
| * cloudnull knocks on wood | 13:33 | |
| cloudnull | i'll be around today should any news pop up | 13:33 |
| openstackgerrit | Merged openstack/diskimage-builder: Change DIB_IPA_CERT resulting file name https://review.openstack.org/355440 | 13:35 |
| mordred | sdague: those change numbers are different | 13:35 |
| sdague | gah, never mind | 13:35 |
| *** esikache1 has joined #openstack-infra | 13:35 | |
| sdague | I need more coffee | 13:35 |
| mordred | sdague: :) | 13:35 |
| mordred | sdague: me too | 13:35 |
| *** vgridnev has quit IRC | 13:36 | |
| rcarrillocruz | so am I | 13:36 |
| *** amitgandhinz has quit IRC | 13:36 | |
| rcarrillocruz | brb | 13:36 |
| EmilienM | could I have some feedback on https://review.openstack.org/#/c/356675/ please? | 13:36 |
| *** amitgandhinz has joined #openstack-infra | 13:36 | |
| *** amotoki has joined #openstack-infra | 13:37 | |
| *** esberglu has joined #openstack-infra | 13:39 | |
| *** pilgrimstack has joined #openstack-infra | 13:39 | |
| *** BigWillie has joined #openstack-infra | 13:39 | |
| *** caowei has joined #openstack-infra | 13:40 | |
| *** hongbin has joined #openstack-infra | 13:41 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 13:41 |
| *** dimtruck is now known as zz_dimtruck | 13:41 | |
| *** zz_dimtruck is now known as dimtruck | 13:41 | |
| *** spzala has joined #openstack-infra | 13:42 | |
| *** _coolsvap_ is now known as coolsvap | 13:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Use ip6tables if nodepool is using IPv6 https://review.openstack.org/356637 | 13:43 |
| openstackgerrit | Merged openstack-infra/project-config: Resolve DNS soon in configure_mirror.sh https://review.openstack.org/355047 | 13:43 |
| sslypushenko | rcarrillocruz: Can I ask you to review https://review.openstack.org/#/c/353861/ and merge it if it is work for you | 13:43 |
| mrhillsman | pabelanger | 13:45 |
| pleia2 | rcarrillocruz: I can go through these patches to set the infracloud ips, but is there a document I should be matching them against? | 13:45 |
| rcarrillocruz | pleia2: well, i think it's fine that i autoapprove them | 13:45 |
| mmedvede | wznoinsk: I am around now | 13:45 |
| rcarrillocruz | i've done so a couple | 13:45 |
| rcarrillocruz | but if you want | 13:45 |
| rcarrillocruz | i'm basiclly adding 6 do the computexxx number | 13:46 |
| rcarrillocruz | meaning | 13:46 |
| rcarrillocruz | compute000 => 15.184.64.6 and 10.10.16.6 | 13:46 |
| pleia2 | ok, easy enough | 13:46 |
| rcarrillocruz | reason is because first 3 IPs are reserved for HPE net folks | 13:46 |
| rcarrillocruz | .4 is baremetal | 13:46 |
| pleia2 | aha | 13:46 |
| rcarrillocruz | .5 is controller | 13:46 |
| rcarrillocruz | thanks | 13:47 |
| *** ihrachys has joined #openstack-infra | 13:47 | |
| zigo | Hi mordred! Could you please approve this? https://review.openstack.org/#/c/357184/ Waiting 30 minutes between each commit is really a no-brainer for us, it's really terrible, I'm sure this will help a lot. | 13:47 |
| openstackgerrit | Miguel Angel Ajo proposed openstack-infra/project-config: Enable QoS on neutron for python-openstacksdk. https://review.openstack.org/357230 | 13:49 |
| *** ihrachys has quit IRC | 13:50 | |
| dmsimard | Just saw a mirror error, is it a one time thing ? http://logs.openstack.org/74/356774/5/gate/gate-packstack-python27-ubuntu-xenial/cdc2ac9/console.html#_2016-08-18_13_16_28_267945 | 13:50 |
| *** dimtruck is now known as zz_dimtruck | 13:51 | |
| *** ihrachys has joined #openstack-infra | 13:53 | |
| *** kaisers1 has joined #openstack-infra | 13:53 | |
| *** asselin_ has joined #openstack-infra | 13:54 | |
| timrc | yolanda: Definitely :) | 13:54 |
| pleia2 | rcarrillocruz: just realized this one doesn't update to be provisioning_ipv4_address, should it? https://review.openstack.org/#/c/357091/1/hiera/group/infracloud.yaml | 13:54 |
| timrc | rcarrillocruz: ++ re: The Sangria fruit-to-liquid ratio. | 13:54 |
| rcarrillocruz | it should , thanks! | 13:54 |
| rcarrillocruz | timrc: you taking additional days to sightseeing? | 13:55 |
| *** asselin has quit IRC | 13:56 | |
| timrc | rcarrillocruz: I need to figure that all out. I probably should :) | 13:57 |
| rcarrillocruz | you really should :-) | 13:57 |
| *** edtubill has joined #openstack-infra | 13:57 | |
| timrc | It'd be before the Summit though. I don't want to miss Halloween with my kids. | 13:57 |
| *** ayoung has joined #openstack-infra | 13:57 | |
| *** itisha has joined #openstack-infra | 13:57 | |
| *** xarses has quit IRC | 13:57 | |
| rcarrillocruz | ah yeah, you have to prepare another arduino/rpi project this year ;-) | 13:57 |
| *** xarses has joined #openstack-infra | 13:58 | |
| timrc | Yeah! I fall pretty short when it comes to actual engineering though :P | 13:58 |
| *** timello has quit IRC | 13:59 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute013.vanilla IP addresses https://review.openstack.org/357091 | 14:00 |
| yolanda | timrc, i have it very easy this time. At a driving distance :) | 14:00 |
| yolanda | but i will stay for 3 nights there as well | 14:01 |
| *** burgerk has quit IRC | 14:02 | |
| sslypushenko | Hi, folks! Can anybody told whom should be asked for review and merge https://review.openstack.org/#/c/353861/? | 14:03 |
| pleia2 | rcarrillocruz: Inline comments: https://review.openstack.org/#/c/357084/ https://review.openstack.org/#/c/357098/ | 14:03 |
| pleia2 | rcarrillocruz: Needs provisioning_ipv4_address fix: https://review.openstack.org/#/c/357075/ https://review.openstack.org/#/c/357102/ | 14:04 |
| rcarrillocruz | thx | 14:04 |
| pleia2 | rcarrillocruz: Both: https://review.openstack.org/#/c/357093/ | 14:04 |
| pleia2 | then we should be ok :) | 14:04 |
| pleia2 | approved the rest | 14:04 |
| *** DrifterZA has joined #openstack-infra | 14:04 | |
| *** sjmc7 has joined #openstack-infra | 14:04 | |
| *** elo has quit IRC | 14:04 | |
| *** caowei has quit IRC | 14:04 | |
| openstackgerrit | Nathan Bohman proposed openstack-infra/shade: Don't cast image propertiess to str https://review.openstack.org/346103 | 14:05 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Temporarily add rabbit keys to hiera https://review.openstack.org/357021 | 14:06 |
| *** caowei has joined #openstack-infra | 14:06 | |
| *** weshay_ is now known as weshay_mtg | 14:06 | |
| odyssey4me | OK, this is going to be a problem. Every single patch we submit, or recheck we issue, is getting an immediate fail from 'DSS Cinder CI <budik_06@list.ru>'. It's effectively spam. See https://review.openstack.org/356864 for a sample. Is there a way we can disable this external CI or somewhere where we can try to figure out who the Ci administrator is? | 14:09 |
| *** rbrndt has joined #openstack-infra | 14:09 | |
| asselin_ | rcarrillocruz, any ideas? http://paste.openstack.org/show/560804/ | 14:09 |
| *** zz_dimtruck is now known as dimtruck | 14:10 | |
| *** xarses has quit IRC | 14:10 | |
| *** edtubill has quit IRC | 14:10 | |
| *** e0ne has quit IRC | 14:10 | |
| rcarrillocruz | i suspect your launch playbook does not have volumes | 14:11 |
| asselin_ | odyssey4me, infra-root can disable the account. You can try e-mail them at their e-mail address provided to gerrit. Looks like a new account. DSS Cinder CI budik_06@list.ru | 14:11 |
| rcarrillocruz | https://review.openstack.org/#/c/320466/15/playbooks/launch_node.yml | 14:11 |
| *** ihrachys has quit IRC | 14:11 | |
| rcarrillocruz | otherwise you'd have an empty 'volumes' dict on what you pasted | 14:11 |
| asselin_ | rcarrillocruz, that's true, no volumes | 14:11 |
| pleia2 | asselin_: sure, I'll have a look | 14:12 |
| pleia2 | er, odyssey4me :) | 14:12 |
| odyssey4me | thanks pleia2 | 14:13 |
| *** _nadya_ has quit IRC | 14:13 | |
| asselin_ | kzaitsev_mb reported as well ^^ | 14:13 |
| * pleia2 nods | 14:13 | |
| odyssey4me | I'll mail the ML asking them to fix the scope of their tests, and forward the mail to them too. But if there's anything you can do from an infra standpoint to scope their account that'd be great. | 14:14 |
| pleia2 | odyssey4me: that would be helpful, paper trail and all | 14:14 |
| *** ihrachys has joined #openstack-infra | 14:14 | |
| pleia2 | I'll reply to your email when I disable the account (after finding it) | 14:14 |
| *** e0ne has joined #openstack-infra | 14:14 | |
| asselin_ | rcarrillocruz, so how do I 'not' specify any extra volume? | 14:14 |
| rcarrillocruz | you shouldn't have, make sure your launch_node.yml does contain line 14 | 14:15 |
| odyssey4me | pleia2 http://lists.openstack.org/pipermail/openstack-dev/2016-August/101887.html | 14:15 |
| pleia2 | odyssey4me: ty | 14:15 |
| rcarrillocruz | that line will add an empty volumes if nothing is passed as variable | 14:15 |
| *** ihrachys has quit IRC | 14:16 | |
| *** edtubill has joined #openstack-infra | 14:17 | |
| *** sree has joined #openstack-infra | 14:17 | |
| asselin_ | rcarrillocruz, actually i'm out of sync a bit with your patch....using before you added volumes: http://paste.openstack.org/show/560807/ | 14:17 |
| *** gildub has quit IRC | 14:18 | |
| *** gildub_ has quit IRC | 14:18 | |
| rcarrillocruz | yeah, what i suspected | 14:18 |
| *** Julien-zte has quit IRC | 14:18 | |
| *** timello has joined #openstack-infra | 14:18 | |
| *** ihrachys has joined #openstack-infra | 14:18 | |
| asselin_ | rcarrillocruz, any idea when this will be merged upstream? | 14:19 |
| rcarrillocruz | not sure, i need reviewers | 14:19 |
| rcarrillocruz | pabelanger said he wanted to test it out | 14:20 |
| *** asettle has quit IRC | 14:20 | |
| rcarrillocruz | the role itself he used it to bring up the bluebox resources with success | 14:20 |
| *** edtubill has quit IRC | 14:20 | |
| rcarrillocruz | but by feeding a resources.yml, not the playbook launch node | 14:20 |
| rcarrillocruz | i'll paste you | 14:21 |
| *** asettle has joined #openstack-infra | 14:21 | |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/system-config/tree/cloud_launcher/clouds_layouts.yml | 14:21 |
| *** asettle has quit IRC | 14:21 | |
| *** rfolco has joined #openstack-infra | 14:21 | |
| rcarrillocruz | the idea is to have some day a mechanism that runs the launcher post-merge whenever those layouts change | 14:21 |
| *** asettle has joined #openstack-infra | 14:21 | |
| *** vgridnev has joined #openstack-infra | 14:21 | |
| asselin_ | oh I like that | 14:22 |
| rcarrillocruz | but i need to fix a couple things first to what i have in mind for doing post-resource provisioning | 14:22 |
| rcarrillocruz | the os_server module does not have an idempotent return status | 14:22 |
| *** valderrv has joined #openstack-infra | 14:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute019.vanilla IP addresses https://review.openstack.org/357100 | 14:22 |
| rcarrillocruz | it will return 'changed' regardless if it created anything or not | 14:22 |
| rcarrillocruz | i need to fix that | 14:22 |
| rcarrillocruz | and i also i want to return what the task did | 14:22 |
| rcarrillocruz | 'created' | 14:22 |
| rcarrillocruz | 'deleted' | 14:22 |
| rcarrillocruz | 'updated' | 14:22 |
| rcarrillocruz | cos depending on that meta-result, we can run tasks based on it | 14:23 |
| rcarrillocruz | where i'm going with ^ is having something like: | 14:23 |
| *** baoli_ has quit IRC | 14:23 | |
| *** baoli has joined #openstack-infra | 14:24 | |
| rcarrillocruz | http://paste.openstack.org/show/560809/ | 14:24 |
| *** edtubill has joined #openstack-infra | 14:24 | |
| rcarrillocruz | something along those lines | 14:24 |
| rcarrillocruz | define pre/post create/delete tasks defined in the resources model | 14:25 |
| jascott1 | hi! | 14:25 |
| asselin_ | rcarrillocruz, i like the direction | 14:25 |
| rcarrillocruz | and for thost pre/post create/delete/update we need to get what the os_server is doing any time | 14:25 |
| rcarrillocruz | it will take some time, i have now being completely pulled by the infracloud | 14:26 |
| rcarrillocruz | asselin_: you saw the 'scaling' thing on the launcher | 14:26 |
| rcarrillocruz | ? | 14:26 |
| asselin_ | rcarrillocruz, yes, I think it merged right? I didn't get to try it out yet. I need to launch a single node before doing clusters | 14:27 |
| rcarrillocruz | yup, and it's tested on the gate too | 14:27 |
| *** edtubill has quit IRC | 14:27 | |
| asselin_ | rcarrillocruz, I see you took this out. any particular reason? "- { role: set_hostname }" | 14:28 |
| *** raunak has joined #openstack-infra | 14:28 | |
| sdague | mordred / pleia2 so, internap, 40% gate failure rate | 14:28 |
| sdague | who can investigate? | 14:28 |
| rcarrillocruz | hmm, i'd say no, that should be still there | 14:31 |
| rcarrillocruz | feel free to push a patchset | 14:31 |
| *** annegentle has joined #openstack-infra | 14:31 | |
| asselin_ | rcarrillocruz, ok, let me get it working and i'll update | 14:31 |
| openstackgerrit | Nathan Bohman proposed openstack-infra/shade: Don't cast image propertiess to str https://review.openstack.org/346103 | 14:32 |
| rcarrillocruz | ++ | 14:33 |
| *** edtubill has joined #openstack-infra | 14:34 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Prettify Task-Status-Changes in Recent Events https://review.openstack.org/357257 | 14:34 |
| *** mhickey has quit IRC | 14:34 | |
| *** jed56 has quit IRC | 14:35 | |
| azvyagintsev | Foolks , https://review.openstack.org/#/c/353861/ can we merge it ? sdague mordred odyssey4me yolanda | 14:36 |
| *** kszukielojc has joined #openstack-infra | 14:39 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fail fast if a testenv couldn't be created https://review.openstack.org/341428 | 14:39 |
| kszukielojc | Hi all can we get a +2 for https://review.openstack.org/#/c/353861 ? | 14:39 |
| bswartz | having a weird gerrit problem: http://paste.openstack.org/show/560811/ | 14:39 |
| *** Julien-zte has joined #openstack-infra | 14:40 | |
| odyssey4me | azvyagintsev I think you mean to ask the infra project-config admins. I'm not one of those. :) | 14:40 |
| azvyagintsev | broadcast :) | 14:40 |
| *** raunak has quit IRC | 14:41 | |
| *** edtubill has quit IRC | 14:41 | |
| *** kzaitsev_mb has quit IRC | 14:41 | |
| *** pilgrimstack has quit IRC | 14:41 | |
| *** d0ugal has quit IRC | 14:41 | |
| *** pilgrimstack has joined #openstack-infra | 14:41 | |
| dansmith | sdague: mordred: pleia2: at 40% fail, does it even make sense to have internap in the pool? I would think it's doing more harm than good at that point | 14:42 |
| pabelanger | zigo: great. I'll have to talk to fungi about the public key, but don't think it will be an issue | 14:42 |
| *** xarses has joined #openstack-infra | 14:42 | |
| zigo | pabelanger: I solved it by adding the public key block within the code, but I'd prefer downloading it so you'd be free to update it. | 14:42 |
| zigo | pabelanger: In the mean while, can you +2 this one? https://review.openstack.org/#/c/357184/ | 14:42 |
| zigo | :) | 14:43 |
| *** rbuzatu has quit IRC | 14:43 | |
| zigo | pabelanger: Another issue. It looks like reprepro only accept packages that are comming with a .changes. Am I right? | 14:43 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Fix bridge creation in no vlan use case https://review.openstack.org/357261 | 14:43 |
| fungi | zigo: pabelanger: if you're talking about publishing the public signing key somewhere, that's in the works but pending https://review.openstack.org/353692 since we want to make sure it's served via https | 14:43 |
| *** edtubill has joined #openstack-infra | 14:44 | |
| *** Julien-zte has quit IRC | 14:44 | |
| zigo | FYI, I'm fighting to add a backport of dpkg so that dpkg-source ignores the .gitreview, which otherwise blocks us for building... :/ | 14:46 |
| *** burgerk has joined #openstack-infra | 14:46 | |
| dansmith | sdague: pleia2: mordred: I see all kinds of network failures in those logs, ntp syncs, apt, pip, etc.. | 14:46 |
| pabelanger | zigo: yes, only .changes right now. Obviously that could be updated | 14:46 |
| *** edtubill has quit IRC | 14:47 | |
| zigo | pabelanger: How can this be fixed, and then how will it gather the .deb files? | 14:47 |
| *** mhickey has joined #openstack-infra | 14:47 | |
| *** rbuzatu has joined #openstack-infra | 14:47 | |
| *** ihrachys has quit IRC | 14:47 | |
| zigo | pabelanger: The problem is when I'm downloading from official jessie-backports and trying to push to the jessie-newton-backports repo... | 14:47 |
| *** edtubill has joined #openstack-infra | 14:47 | |
| *** eeiden has quit IRC | 14:47 | |
| pabelanger | zigo: you are not building them? | 14:47 |
| zigo | pabelanger: Do you think I should attempt to rebuild? | 14:48 |
| *** vinaypotluri has joined #openstack-infra | 14:48 | |
| zigo | I'm currently not. | 14:48 |
| *** wcriswell has quit IRC | 14:48 | |
| *** stewie925 has quit IRC | 14:48 | |
| *** d0ugal has joined #openstack-infra | 14:48 | |
| *** d0ugal has quit IRC | 14:48 | |
| *** d0ugal has joined #openstack-infra | 14:48 | |
| pabelanger | Ya, today the pipeline expects packages to be built by openstack | 14:48 |
| *** _ari_ has joined #openstack-infra | 14:49 | |
| zigo | It should be trivial to rebuild, I should just change stuff in packages-list (ie: just add -d jessie-backports instead of --download-from-jessie-backports) | 14:49 |
| pabelanger | we could add another jobs to just import existing packages, but there maybe some objections to that | 14:49 |
| zigo | I'll attempt and see if it builds correctly. | 14:49 |
| pabelanger | okay | 14:49 |
| *** skipp has quit IRC | 14:49 | |
| *** rajinir has joined #openstack-infra | 14:50 | |
| bswartz | nevermind my earlier question I discovered it's my fault | 14:51 |
| *** dmellado|off is now known as dmellado | 14:52 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins https://review.openstack.org/357217 | 14:52 |
| openstackgerrit | Merged openstack-infra/project-config: Up the debian-jessie min-ready https://review.openstack.org/357184 | 14:57 |
| *** tonytan4ever has joined #openstack-infra | 14:57 | |
| *** sree has quit IRC | 14:58 | |
| *** edtubill has quit IRC | 14:58 | |
| *** nmagnezi has quit IRC | 14:58 | |
| *** gouthamr_afk is now known as gouthamr | 14:59 | |
| *** _nadya_ has joined #openstack-infra | 14:59 | |
| clarkb | zigo: did you see that the deb-websockify repo is "broken" and needs to be corrected before we can import it into gerrit? | 14:59 |
| *** kgiusti is now known as kgiusti|meeting | 14:59 | |
| *** edtubill has joined #openstack-infra | 14:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute017.vanilla IP addresses https://review.openstack.org/357096 | 14:59 |
| zigo | clarkb: I didn't, but shouldn't the CI catch it? | 15:00 |
| zigo | clarkb: In what way is it broken? | 15:00 |
| *** tbarron|afk is now known as tbarron | 15:00 | |
| *** salv-orlando has joined #openstack-infra | 15:00 | |
| clarkb | zigo: no, the problem is in importing it into gerrit so no CI can run on it. The problem is it uses zero padded file modes which gerrit does not allow. You can see this as a warning if you git fsck --full | 15:00 |
| *** yamamoto has quit IRC | 15:01 | |
| clarkb | zigo: https://www.perforce.com/blog/150109/git-out-vulnerability-out-fire describes how to fix it. If you do that and get the repo updated to a state you are happy with we can finish the import process | 15:01 |
| *** sree has joined #openstack-infra | 15:01 | |
| *** sree has quit IRC | 15:02 | |
| mordred | zigo: also, the example script has this line: git fast-export --all | (cd ../myFixedRepo; git fast-import)1 | 15:02 |
| mordred | the "1" isn't part of the script - it's a footnote indicator - which I think is a bit lame | 15:02 |
| *** sree has joined #openstack-infra | 15:02 | |
| *** hockeynut has joined #openstack-infra | 15:04 | |
| *** bhunter71 has joined #openstack-infra | 15:04 | |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| *** devkulkarni has joined #openstack-infra | 15:04 | |
| electrofelix | waynr zxiiro: I think tomorrow is the last scheduled virtual sprint, but I'm not sure if we're ready with the remaining patches to review? | 15:05 |
| zxiiro | electrofelix: oh shoot i totally forgot. It's been a busy week for me. | 15:05 |
| zigo | clarkb: It's fixed now. | 15:06 |
| zxiiro | electrofelix: i'll poke at the remaining patches but ti sounds like we may need at least another session? | 15:06 |
| zigo | mordred: Fixed. | 15:06 |
| zigo | How can the import be attempted again? | 15:06 |
| *** sree has quit IRC | 15:06 | |
| zigo | Will gerrit try again by itself? | 15:06 |
| electrofelix | zxiiro: I think they may not have been lined up, so I was thinking we just use the same timeslot to line them up, and work on the TODO's outstanding assuming that works for waynr and we can then have a later one to focus on landing and remaining nits? | 15:08 |
| *** timello has quit IRC | 15:08 | |
| *** d0ugal has quit IRC | 15:09 | |
| *** yamamoto has quit IRC | 15:09 | |
| *** edtubill has quit IRC | 15:09 | |
| zxiiro | electrofelix: yeah lets do that | 15:10 |
| *** markusry has joined #openstack-infra | 15:10 | |
| zxiiro | electrofelix: I'm heading to LinuxCon tomorrow but I'll be around during our meeting time | 15:10 |
| zxiiro | electrofelix: seems like only 13 patches remaining | 15:11 |
| zxiiro | electrofelix: I'll spend the afternoon going over them and helping rebase if needed | 15:11 |
| electrofelix | zxiiro: cheers, I think two of the early ones have alternative architectural approaches that could be better, which is why it might be difficult to land them tomorrow, but at least we can have any immediate conflicts solved | 15:13 |
| *** elo has joined #openstack-infra | 15:13 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OSA Xenial jobs to voting https://review.openstack.org/357283 | 15:14 |
| *** edtubill has joined #openstack-infra | 15:15 | |
| *** eeiden has joined #openstack-infra | 15:15 | |
| *** coolsvap is now known as _coolsvap_ | 15:16 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OSA Xenial jobs to voting https://review.openstack.org/357283 | 15:16 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update Delivery Pipeline plugin to use convert xml https://review.openstack.org/346988 | 15:16 |
| *** csomerville has joined #openstack-infra | 15:18 | |
| *** cody-somerville has quit IRC | 15:18 | |
| *** stewie925 has joined #openstack-infra | 15:19 | |
| clarkb | sdague: dansmith what are we looking at to get that 40% number? is this a single test failing? an aggregate of all tests? any example failures? | 15:19 |
| *** skipp has joined #openstack-infra | 15:20 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for CodeCover plugin https://review.openstack.org/346974 | 15:20 |
| dansmith | clarkb: I dunno about the number, but just looking at a bunch of failures, they seem to be a variety of things.. clarkb like this non-fatal one: http://logs.openstack.org/50/348050/3/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/a56ddae/console.html#_2016-08-18_13_59_50_344972 | 15:20 |
| dansmith | and then... | 15:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for enabling empty results https://review.openstack.org/244588 | 15:20 |
| dansmith | clarkb: http://logs.openstack.org/50/348050/3/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/a56ddae/logs/grenade.sh.txt.gz#_2016-08-18_14_09_41_415 | 15:20 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Temporary allow merges for sahara-tests-release https://review.openstack.org/357290 | 15:20 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Revert "Temporary allow merges for sahara-tests-release" https://review.openstack.org/357291 | 15:20 |
| *** ihrachys has joined #openstack-infra | 15:21 | |
| *** mmedvede has quit IRC | 15:21 | |
| dansmith | clarkb: this one gets way farther, but similar fail: http://logs.openstack.org/84/319384/11/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/db0cdbc/logs/grenade.sh.txt.gz#_2016-08-18_13_46_57_851 | 15:21 |
| clarkb | dansmith: the ntpd issue is being worked on. We need to change how that works and its different for every distro :/ | 15:21 |
| dansmith | i.e. network stuff | 15:21 |
| *** rcernin has quit IRC | 15:22 | |
| clarkb | looks like talking to the mirror host timed out which was the actual fail | 15:22 |
| dansmith | clarkb: okay, I figured that was related to this since it seems to be random network fails, but fair enough | 15:22 |
| dansmith | right | 15:22 |
| clarkb | dansmith: it has to do with how and when ntp is willing to step vs skew the clock and its a mess. Probably not network related | 15:22 |
| *** ihrachys has quit IRC | 15:22 | |
| dansmith | okay | 15:22 |
| dansmith | clarkb: ntpdate first? | 15:23 |
| clarkb | dansmith: except ntpdate is deprecated and should not be used | 15:23 |
| dansmith | clarkb: this one got killed early, apt fail: http://logs.openstack.org/14/329514/7/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7795b10/console.html#_2016-08-18_12_26_55_784536 | 15:23 |
| dansmith | clarkb: ....riiight... :P | 15:23 |
| clarkb | but yes maybe we should go back to ntpdate temporarily until we can get the other bits sorted | 15:23 |
| *** pilgrimstack has quit IRC | 15:24 | |
| clarkb | dansmith: ya it apparently was a fork of ntp code from the land before time and is buggy and insecure | 15:24 |
| dansmith | clarkb: yeah, but alas, I dunno how else to programmatically get the clock close enough for ntpd, but that's not important | 15:24 |
| *** d0ugal has joined #openstack-infra | 15:24 | |
| *** mhickey has quit IRC | 15:24 | |
| dansmith | here's a non-grenade pip-based fail: http://logs.openstack.org/63/264263/33/gate/gate-tempest-dsvm-ironic-multitenant-network/d6e3b6d/logs/devstacklog.txt.gz#_2016-08-18_11_23_31_673 | 15:25 |
| dansmith | clarkb: so I dunno about the 40% number, but the logstash query I'm looking at shows about 15% ratio of FAILURE to SUCCESS | 15:26 |
| dansmith | that's just the gate, so maybe it's higher if you factor in check | 15:26 |
| clarkb | ok, I will have to look at mirror server logs to see if we can narrow down the case of the time outs | 15:26 |
| mordred | clarkb: it all seems to be internap | 15:26 |
| mordred | clarkb: sdague earlier was musing that there were internap network issues | 15:26 |
| *** pilgrimstack has joined #openstack-infra | 15:28 | |
| *** vhosakot has joined #openstack-infra | 15:28 | |
| *** amitgandhinz has quit IRC | 15:28 | |
| *** sdake_ has joined #openstack-infra | 15:29 | |
| *** amitgandhinz has joined #openstack-infra | 15:29 | |
| *** pilgrimstack has quit IRC | 15:29 | |
| *** abregman_ has joined #openstack-infra | 15:30 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add valid_dict parameter to convert xml https://review.openstack.org/345144 | 15:30 |
| *** e0ne has quit IRC | 15:30 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Rework the task list layout https://review.openstack.org/357306 | 15:30 |
| *** hieulq_ has joined #openstack-infra | 15:31 | |
| *** jianghuaw-m has joined #openstack-infra | 15:31 | |
| *** abregman has quit IRC | 15:32 | |
| *** Swami has joined #openstack-infra | 15:32 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove handle_entity_children for convert xml https://review.openstack.org/349045 | 15:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker https://review.openstack.org/357308 | 15:33 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update LogStash plugin to use convert xml https://review.openstack.org/348599 | 15:34 |
| *** mmedvede has joined #openstack-infra | 15:34 | |
| *** zhurong has quit IRC | 15:34 | |
| toabctl | somebody there to have a look at https://review.openstack.org/#/c/354141/ ? | 15:36 |
| *** mhickey has joined #openstack-infra | 15:36 | |
| clarkb | mordred: did anyone manage to take a look at the mirror host logs to see if there was any more info there? (I am trying to filter through scrollback but not noticing much) | 15:37 |
| mordred | clarkb: no - I haven't gotten to the part of my morning where I'm useful yet | 15:38 |
| dansmith | clarkb: I was going to look to see if I see a bulk of similar failures for other providers | 15:38 |
| dansmith | clarkb: otherwise seems like the mirror logs are probably not that relevant, right? | 15:38 |
| *** jianghuaw-m has quit IRC | 15:39 | |
| *** Apoorva has joined #openstack-infra | 15:39 | |
| clarkb | dansmith: possibly. If something was breaking afs globally then we should still see that in internap for example but ya it would give us better data to see if it was a problem in other providers too | 15:39 |
| *** esikache1 has quit IRC | 15:40 | |
| *** kgiusti|meeting is now known as kgiusti | 15:41 | |
| mordred | clarkb: the logstash query from sdague earlier was: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=node_provider:%5C%22internap-nyj01%5C%22%20AND%20message:%5C%22Job%20complete%5C%22%20AND%20voting:1%20AND%20build_name:*dsvm*%20AND%20build_queue:gate | 15:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker https://review.openstack.org/357308 | 15:41 |
| *** wcriswell has joined #openstack-infra | 15:43 | |
| dansmith | I'm trying to alter that query a bit to figure out if this is more than internap, but I suck at logstash so it's taking me a bit | 15:43 |
| pabelanger | jeblair: mordred: clarkb: It looks like ubuntu-xenail builds in diskimage-builder are broken for the moment, ianw was looking at it last night. So until we fix it, we cannot land our current patches for osic-cloud1 | 15:43 |
| clarkb | pabelanger: have a paste of the error handy? | 15:44 |
| clarkb | the internap mirror did have a hung process yesterday according to inaccurate dmesg -T output | 15:44 |
| cloudnull | pabelanger: if i can help in some way let me know | 15:45 |
| pabelanger | clarkb: a little large: http://nodepool.openstack.org/dib.ubuntu-xenial.log | 15:45 |
| *** jerryz has joined #openstack-infra | 15:45 | |
| *** mmedvede has quit IRC | 15:45 | |
| dansmith | clarkb: mordred: yeah, that pip timeout error seems to only dig up internap for me | 15:46 |
| *** javeriak has joined #openstack-infra | 15:46 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for Growl plugin https://review.openstack.org/350838 | 15:48 |
| fungi | yikes, rootfs utilization on mirror.regionone.osic-cloud1.openstack.org is climbing rapidly | 15:48 |
| fungi | we probably need to put the afs cache there on a separate volume before it asplodes | 15:49 |
| clarkb | is it not on a separate volume? | 15:49 |
| *** pilgrimstack has joined #openstack-infra | 15:50 | |
| fungi | not that i can see in cacti. checking locally in a sec | 15:50 |
| fungi | though more likely this is apache logs | 15:50 |
| jeblair | last time i looked, or cache use was very small | 15:50 |
| fungi | we generate crazy amounts of (mostly useless?) apache log volume | 15:50 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update Build Name Setter plugin to use convert xml https://review.openstack.org/351899 | 15:50 |
| jeblair | it's 12g on that host | 15:50 |
| *** chem has quit IRC | 15:50 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update cobertura to use convert xml https://review.openstack.org/351891 | 15:50 |
| david-lyle | sorry for the noise, but any chance for a second +2 on https://review.openstack.org/356757 it's fixes a blocking gate issue for searchlight, it's just making a job non-voting in project-config | 15:50 |
| jeblair | and it's on its own volume | 15:51 |
| fungi | yeah, apache logs are 32G there currently according to du | 15:51 |
| jeblair | (which is enormous) | 15:51 |
| *** javeriak has quit IRC | 15:51 | |
| *** javeriak has joined #openstack-infra | 15:52 | |
| fungi | snmpd has been running there since May23 | 15:52 |
| fungi | so probably predates the cache volume and just needs a restart to pick it up | 15:52 |
| fungi | i'll do that now | 15:52 |
| *** chem has joined #openstack-infra | 15:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin https://review.openstack.org/351771 | 15:53 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: adding support for the xcode builder plugin https://review.openstack.org/146918 | 15:53 |
| fungi | i wonder if we should do something to scale down either our logging or log retention for apache on our mirrors | 15:53 |
| *** edtubill has quit IRC | 15:53 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter https://review.openstack.org/351384 | 15:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development https://review.openstack.org/352599 | 15:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf https://review.openstack.org/352600 | 15:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3 https://review.openstack.org/357319 | 15:54 |
| *** jerryz has quit IRC | 15:54 | |
| *** edtubill has joined #openstack-infra | 15:55 | |
| *** tqtran has joined #openstack-infra | 15:57 | |
| *** pcm_ has quit IRC | 15:58 | |
| *** weshay_mtg is now known as wesha_ | 15:59 | |
| *** wesha_ is now known as weshay_ | 15:59 | |
| fungi | clarkb: what hung process did we have yesterday on the internap mirror? dmesg says it was >2 weeks ago, and while i don't entirely trust its timestamps i don't see similar messages in our syslog going back as far as we're retaining it (a week) | 15:59 |
| *** pc_m has joined #openstack-infra | 16:00 | |
| fungi | the most recent one i see mentioned by dmesg was wednesday, august 3 | 16:00 |
| *** tonytan4ever has quit IRC | 16:00 | |
| *** edtubill has quit IRC | 16:01 | |
| *** tonytan4ever has joined #openstack-infra | 16:01 | |
| *** tqtran has quit IRC | 16:02 | |
| clarkb | oh sorry that was august 3rd nevermind | 16:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development https://review.openstack.org/352599 | 16:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf https://review.openstack.org/352600 | 16:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3 https://review.openstack.org/357319 | 16:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config https://review.openstack.org/357323 | 16:02 |
| clarkb | need more caffeine | 16:02 |
| fabo | clarkb: who can merge a merge request these days? I have a mr open since with a +2 since 4 months now | 16:03 |
| fabo | clarkb: https://review.openstack.org/#/c/98437/ | 16:03 |
| clarkb | fungi: mgagne http://paste.openstack.org/show/560822/ thats a client side pip request log saying the request timed out and apache claiming the transfer succeeded but transfered not enough bytes by my count | 16:03 |
| *** oanson has quit IRC | 16:03 | |
| fabo | clarkb: or do you know Darragh nick if he's online | 16:03 |
| clarkb | fungi: mgagne it almost looks like apache thought it was done there for some reason? | 16:03 |
| fungi | fabo: electrofelix | 16:03 |
| fabo | thanks | 16:03 |
| fabo | electrofelix: follow up to https://review.openstack.org/#/c/98437/ ;) let me know what's the next steps to get it merged please | 16:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3 https://review.openstack.org/357326 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config https://review.openstack.org/357327 | 16:05 |
| fungi | fabo: the main jenkins-job-builder core reviewers according to https://review.openstack.org/#/admin/groups/194,members are hashar (not in irc at the moment), electrofelix, zaro, zxiiro and waynr | 16:05 |
| *** sree has joined #openstack-infra | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:06 | |
| fungi | clarkb: logs indicating a short response can, i believe, indicate an abnormally terminated transfer (like client closing the connection early for example) | 16:06 |
| *** hpe-hj has joined #openstack-infra | 16:07 | |
| fungi | seems like we dug into similar failures in the past related to our git farm | 16:07 |
| clarkb | fungi: mgagne ya we have seen similar in rax when the bw limitations kick in and they close connections for us. Any chance internap does similar? | 16:07 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3145&rra_id=all does show plateaued graphs though | 16:07 |
| clarkb | *doesn't show | 16:08 |
| electrofelix | fabo: I'd forgotten about that one, doesn't show up on my review dashboard since I've already +2'ed it, I'll do a quick look and hopefully zxiiro can review/approve | 16:08 |
| waynr | fabo fungi not sure i'll be able to get to that review today, possible tomorrow though during the JJB sprint | 16:08 |
| *** vgridnev has quit IRC | 16:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config https://review.openstack.org/357327 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3 https://review.openstack.org/357326 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development https://review.openstack.org/357329 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf https://review.openstack.org/357330 | 16:08 |
| *** valderrv has quit IRC | 16:09 | |
| *** gyee has joined #openstack-infra | 16:09 | |
| fabo | waynr: that's fine, it's in the queue for 2 years now :) I'm not at +/- 1 day | 16:10 |
| *** sree has quit IRC | 16:10 | |
| fabo | waynr: just by curiosity, is there's some info about JJB sprint? what are the topics and/or expected outcome? | 16:11 |
| *** yamamoto has quit IRC | 16:11 | |
| *** cody-somerville has joined #openstack-infra | 16:11 | |
| fabo | electrofelix: thanks! | 16:11 |
| fungi | clarkb: yeah, that graph doesn't look like we're seeing a rate limit kick in, more likely some piece of gear is getting overloaded or we've got a noisy neighbor chewing up host nic buffer space or something like that | 16:13 |
| *** csomerville has quit IRC | 16:13 | |
| *** yamahata has joined #openstack-infra | 16:14 | |
| zxiiro | electrofelix: there's a pretty bad bug in the "disabled" job configuration. It accepts any input as "true" rather than checking if the user passed true|false :( | 16:14 |
| zxiiro | electrofelix: just got burned by it. I'll look into submitting a patch later today | 16:14 |
| zaro | morning | 16:14 |
| electrofelix | zxiiro: sorry which one is that? | 16:14 |
| zxiiro | we tried to make our job templates have the disabled field configurable and it ended up disabling all of our jobs | 16:14 |
| *** shashank_hegde has joined #openstack-infra | 16:14 | |
| electrofelix | fabo: so we've been running a sprint every 2nd week, email about it was sent to openstack-infra, although maybe it should go to openstack-dev these days as well? | 16:15 |
| electrofelix | fabo: see https://wiki.openstack.org/wiki/VirtualSprints#JJB_V2.0_API_Sprint | 16:15 |
| zxiiro | electrofelix: here's the lines of code causing the issue https://github.com/openstack-infra/jenkins-job-builder/blob/master/jenkins_jobs/modules/general.py#L130-L135 | 16:15 |
| *** tphummel has joined #openstack-infra | 16:15 | |
| electrofelix | zxiiro: so you didn't use '{obj:disabled}' to ensure the original object was passed through instead of a string? | 16:16 |
| openstackgerrit | Dmitry Ilyin proposed openstack-infra/project-config: Enable voting cheack for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 16:16 |
| *** sdake_ has quit IRC | 16:16 | |
| *** ihrachys has joined #openstack-infra | 16:16 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix issue that cause JCloud instance to be set https://review.openstack.org/353211 | 16:16 |
| zxiiro | electrofelix: we did but the problem is we have a defaults configured disabled: False | 16:16 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update cfp builder to use convert xml https://review.openstack.org/353198 | 16:16 |
| zxiiro | electrofelix: it seems defaults area always returned as strings | 16:16 |
| electrofelix | zxiiro: nuts | 16:17 |
| *** DrifterZA has quit IRC | 16:17 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update findbugs_settings to use convert xml https://review.openstack.org/353204 | 16:17 |
| electrofelix | I thought that was changed? | 16:17 |
| clarkb | pabelanger: ianw http://paste.openstack.org/show/560825/ appears to be the relevant part of the dib log. Looks like we are du'ing things to see where our disk image size is when it traps the exit handler | 16:17 |
| clarkb | greghaynes: ^ have you seen that before? | 16:17 |
| fabo | electrofelix: thanks for the pointers and good to know | 16:17 |
| zxiiro | electrofelix: obj:disabled only works if you configure it in the project section of the config and not as a default value | 16:17 |
| clarkb | I wonder if du is exiting non zero? | 16:17 |
| *** amitgandhinz has quit IRC | 16:18 | |
| zxiiro | electrofelix: so I think the right thing to do is convert bool to string always for this field and parse the string data | 16:18 |
| *** amitgandhinz has joined #openstack-infra | 16:18 | |
| greghaynes | clarkb: thats post-failure | 16:19 |
| electrofelix | zxiiro: if we just convert it to newer style would it do the right thing? | 16:19 |
| greghaynes | clarkb: is that what youre looking for? something failing in cleanup | 16:19 |
| zxiiro | electrofelix: what do you mean by newer style? | 16:19 |
| greghaynes | clarkb: that line at the top trap_cleanup means the exit trap is starting so something exit 1'd before then | 16:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 16:19 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update maven-builder to use convert xml https://review.openstack.org/352671 | 16:19 |
| electrofelix | zxiiro: XML.SubElement(xml, 'disabled').text = str(disabled.lower()) | 16:19 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update runscope plugin to use convert xml https://review.openstack.org/352652 | 16:19 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update docker build publish to use convert xml https://review.openstack.org/352647 | 16:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 16:19 |
| zxiiro | electrofelix: yes that was what i was thinking | 16:20 |
| electrofelix | zxiiro: remove the nested if else? | 16:20 |
| electrofelix | leave the top level one though | 16:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options https://review.openstack.org/312121 | 16:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: gitlab trigger: Update Gitlab to >=1.3.0 https://review.openstack.org/333113 | 16:20 |
| clarkb | greghaynes: ya what was running in line one is the du to capture where file usage went | 16:20 |
| zxiiro | electrofelix: we need to retain the "None" functionality though | 16:20 |
| clarkb | greghaynes: there are many lines of that so I trimmed it down to one | 16:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: test non-default value of attach-build-log https://review.openstack.org/356236 | 16:20 |
| zxiiro | electrofelix: because we had a request that this field should keep the existing settting if the user does not pass it | 16:20 |
| electrofelix | zxiiro: yeap, that's what I was thinking ;) | 16:20 |
| *** mmedvede has joined #openstack-infra | 16:20 | |
| zxiiro | electrofelix: i'll fix it and put a comment so that we remember | 16:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: add compress-log option to compress log https://review.openstack.org/354138 | 16:20 |
| greghaynes | ah | 16:20 |
| *** Goneri has quit IRC | 16:20 | |
| *** abregman_ has quit IRC | 16:20 | |
| *** mtanino has joined #openstack-infra | 16:20 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Search for VLAN devices on config drive to create bridge on it https://review.openstack.org/357339 | 16:21 |
| clarkb | greghaynes: that happens in bin/disk-image-create | 16:22 |
| rcarrillocruz | yolanda: ^ | 16:22 |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Security update for Google Analytics, Panels, Panelizer https://review.openstack.org/357340 | 16:22 |
| clarkb | greghaynes: eval ${du_output_show} <<< "$du_output" that is the line that appears to cause the trap to trigger | 16:23 |
| clarkb | greghaynes: assuming all output got flushed | 16:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker https://review.openstack.org/357308 | 16:23 |
| clarkb | greghaynes: does dib run with pipefail on or off? | 16:24 |
| greghaynes | clarkb: off in that script IIRC | 16:24 |
| greghaynes | clarkb: looks like theres grep fun | 16:24 |
| greghaynes | clarkb: so probably the thing to do is bash right before there and play with runnign du_output_show | 16:25 |
| clarkb | greghaynes: the greps happen before I think du_output_show's command is what is likely breaking | 16:25 |
| greghaynes | du_output_show is two greps? I am confused | 16:25 |
| greghaynes | oh, you mean the sorts and whatnot | 16:26 |
| *** _nadya_ has quit IRC | 16:26 | |
| clarkb | its sort | numfmt | 16:26 |
| clarkb | tehre is no grep there | 16:26 |
| greghaynes | ah, right, the greps are in a conditional we dont set | 16:26 |
| *** tesseract- has quit IRC | 16:26 | |
| greghaynes | yea, regardless youre just going to want to bash and run that command which I think youve got | 16:26 |
| greghaynes | see what the actual error is | 16:27 |
| *** sambetts is now known as sambetts|afk | 16:27 | |
| greghaynes | I kind of want to remove set -e around this area too. Its really silly to fail when printing out trace info | 16:27 |
| clarkb | I also think we can drop the numfmt entirely and just use du -h and sort -h which may make this simpler and less flaky | 16:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Enable shared networks for te-broker https://review.openstack.org/357308 | 16:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker https://review.openstack.org/357341 | 16:29 |
| *** dronshaw has joined #openstack-infra | 16:30 | |
| greghaynes | clarkb: since these are all pipes im somewhat confused what would fail and still output partially - sort requires a full pass before it can print anything... | 16:30 |
| greghaynes | and it looks like we set +o in that block | 16:30 |
| *** Apoorva has quit IRC | 16:30 | |
| greghaynes | clarkb: so it must be numfmt? | 16:31 |
| *** matthewbodkin has quit IRC | 16:31 | |
| clarkb | ya I am guessing that numfmt isn't liking an input to it | 16:31 |
| clarkb | that it can't convert properly | 16:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 16:31 |
| clarkb | eg field 1 isn't a number | 16:32 |
| *** dronshaw has left #openstack-infra | 16:32 | |
| *** pilgrimstack has quit IRC | 16:32 | |
| *** caowei has quit IRC | 16:32 | |
| *** baoli has quit IRC | 16:32 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Search for VLAN devices on config drive to create bridge on it https://review.openstack.org/357339 | 16:32 |
| clarkb | Exit status is 0 if all input numbers were successfully converted. By default, numfmt will stop at the first conversion error with exit status 2. | 16:32 |
| greghaynes | hrm, we got exitval 1 | 16:33 |
| greghaynes | oh wait, I misread from before - we are going to do the greps that get appended to du_output_command | 16:34 |
| greghaynes | so its possible that numfmt fails then because no pipefail the grep's fail | 16:34 |
| *** ccamacho has quit IRC | 16:34 | |
| greghaynes | can you run with a bash in ther? Itll take me a while to do a real nodepool image | 16:35 |
| clarkb | oh we do append those greps I just double checked the echo output to make sure we go into that block | 16:35 |
| zxiiro | fabo: i'll take a look at your patch this afternoon | 16:35 |
| clarkb | greghaynes: I will likely try to reproduce locally | 16:35 |
| greghaynes | ok | 16:35 |
| *** hieulq_ has quit IRC | 16:35 | |
| *** nwkarsten has quit IRC | 16:35 | |
| greghaynes | I like the use du -h btw. I think this is ianw code so he may have thoughts | 16:36 |
| *** ihrachys has quit IRC | 16:36 | |
| *** nwkarsten has joined #openstack-infra | 16:36 | |
| fungi | the downside to du -h is that it's not easily sortable | 16:36 |
| *** armax has quit IRC | 16:37 | |
| clarkb | greghaynes: check out echo "9.5MiB" | grep -v '\..MiB' | 16:38 |
| clarkb | fungi: it is, works fine with sort -h | 16:38 |
| *** ihrachys has joined #openstack-infra | 16:38 | |
| greghaynes | huh | 16:38 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Add support for configuring split-stack networks https://review.openstack.org/357342 | 16:38 |
| clarkb | greghaynes: that returns 1 because it inverts the match. Now to see if we get that if any fail to maatch | 16:38 |
| clarkb | greghaynes: I would expect 0 if any match but whatever time to test | 16:38 |
| *** jheroux has joined #openstack-infra | 16:38 | |
| openstackgerrit | Dmitry Ilyin proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 16:38 |
| mordred | Shrews, cloudnull, clarkb: ^^ this is needed to deal with multi-node in OSIC for proper categorization of the public quality of v4 and v6 split networks | 16:39 |
| fungi | clarkb: hah, i never knew about sort -h | 16:39 |
| *** mhickey has quit IRC | 16:39 | |
| clarkb | greghaynes: ya in a stream if any match grep should give us 0 so the invert probably isn't breaking us | 16:39 |
| mordred | Shrews, cloudnull, clarkb (I'm ^^ the occ change, not the project-config change) | 16:39 |
| mordred | notmorgan: you may also be interested | 16:40 |
| *** amotoki has quit IRC | 16:40 | |
| *** nwkarsten has quit IRC | 16:40 | |
| greghaynes | clarkb: my bet is on numfmt fail and then grep fails with no input | 16:40 |
| greghaynes | clarkb: seems really easy to make happen - make a file with a newline in it | 16:40 |
| *** itisha has quit IRC | 16:40 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:40 | |
| greghaynes | er, filename | 16:40 |
| clarkb | greghaynes: ha ok | 16:40 |
| *** ihrachys has quit IRC | 16:41 | |
| clarkb | greghaynes: so maybe we want --invalid=warn passed to numfmt | 16:41 |
| *** jpich has quit IRC | 16:41 | |
| greghaynes | clarkb: Yep, and if that goes to stderr we have an easy way to debug actually | 16:42 |
| *** lucasagomes is now known as lucas-dinner | 16:42 | |
| wznoinsk | hi all | 16:42 |
| clarkb | or use a zero byte delimter | 16:42 |
| clarkb | or do both | 16:42 |
| * clarkb tests | 16:42 | |
| *** jordanP has quit IRC | 16:43 | |
| *** sdake has joined #openstack-infra | 16:43 | |
| *** ihrachys has joined #openstack-infra | 16:44 | |
| Shrews | mordred: reviewed. issues found | 16:44 |
| mordred | Shrews: woot. thanks! | 16:45 |
| wznoinsk | I have a problem with dhcp-all-interfaces element, it kicks off at the network-interface (getting up) but only then it adds the /etc/network/interfaces auto iface/dhcp details and it's doing it after static-network-up-emitted | 16:46 |
| wznoinsk | it's causing cloud-init-nonet seeing static-network-up-emitted and progressing to init phase where not all the interfaces are yet up | 16:47 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Add support for configuring split-stack networks https://review.openstack.org/357342 | 16:47 |
| Shrews | mordred: and another | 16:47 |
| wznoinsk | (disk image builder) ^ | 16:47 |
| mordred | bah | 16:47 |
| *** Na3iL has joined #openstack-infra | 16:47 | |
| Shrews | mordred: oh, you got it too | 16:47 |
| greghaynes | wznoinsk: youre using cloud-init and dhcp-all-interfaces? | 16:48 |
| greghaynes | wznoinsk: any reason youre using both? I'd expect cloud-init to dhcp for you | 16:49 |
| fungi | wznoinsk: interesting. i don't think we're using those elements in the minimal image types for our ci... also, there is a #openstack-dib irc channel now | 16:49 |
| *** sdague has quit IRC | 16:49 | |
| greghaynes | that too :) | 16:49 |
| wznoinsk | fungi: yes, I'm on that # but here's a bigger audience... | 16:50 |
| fungi | it still has that "new channel smell" and everything | 16:50 |
| wznoinsk | sure, will move tto there | 16:50 |
| clarkb | greghaynes: du -a -c -x --null ./ | sort -nr --zero-terminated | numfmt --to=iec-i --padding=7 --suffix=B --field=1 --from-unit=1024 --zero-terminated --invalid=warn | egrep --null-data 'MiB|GiB|TiB|PiB' | grep --null-data -v '\..MiB' | tr '\0' '\n' | 16:50 |
| *** raunak has joined #openstack-infra | 16:50 | |
| clarkb | that seems to work just in a random dir on my local machine | 16:50 |
| openstackgerrit | Merged openstack-infra/storyboard: Create timeline events for boards and worklists https://review.openstack.org/350146 | 16:50 |
| fungi | wznoinsk: i guess my point was, if people are basing the images for their ci on our nodepool configuration, they're unlikely to be using that combination of elements | 16:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Have buildZooKeeperHosts take a ConfigValue object https://review.openstack.org/357347 | 16:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a script to print the ZK tree https://review.openstack.org/357348 | 16:51 |
| fungi | greghaynes: you might want to add your shiny new channel to https://wiki.openstack.org/wiki/IRC at some point | 16:52 |
| greghaynes | fungi: ah, good tip | 16:52 |
| *** d34dh0r53 is now known as RichardLongus | 16:52 | |
| *** ccamacho has joined #openstack-infra | 16:52 | |
| *** ccamacho has left #openstack-infra | 16:52 | |
| wznoinsk | fungi: actually that's not in CI context anymore, we just finishing on creating our own dev cloud, the one I'm now working on is the baremetal one, kolla, ironic, dib, cloud-init and other flowers, I've taken the dhcp-all-elements from ironic guide I think | 16:53 |
| wznoinsk | greghaynes: I have to check cloud-init doing dhcp for interfaces | 16:54 |
| *** eandersson_ has quit IRC | 16:54 | |
| fungi | wznoinsk: oh, got it. then yeah, that might be relevant to what we're doing on infra-cloud but rcarrillocruz and crinkle would probably be able to say for sure | 16:54 |
| wznoinsk | moving to #openstack-dib if anyone interested | 16:54 |
| *** shashank_hegde has quit IRC | 16:54 | |
| *** annegentle has quit IRC | 16:54 | |
| fungi | though we're not using kolla or cloud-init | 16:55 |
| fungi | glean is being leveraged pretty heavily on our infra-cloud deployment instead of cloud-init | 16:55 |
| mgagne | clarkb: which flavor are you using? | 16:56 |
| *** tosky has quit IRC | 16:56 | |
| *** vgridnev has joined #openstack-infra | 16:56 | |
| mgagne | clarkb: I think it's 100mbps for A1 series and 50mbps for B1.1 and B1.2 and 1gbps for the rest | 16:56 |
| *** annegentle has joined #openstack-infra | 16:57 | |
| dhellmann | does anyone know what "DSS Cinder CI" is or why it's voting on changes in the releases repository? | 16:57 |
| dhellmann | or commenting, at least | 16:57 |
| *** asettle has quit IRC | 16:58 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a notice when worklists or boards are private or archived https://review.openstack.org/357354 | 16:58 |
| *** RichardLongus is now known as d34dh0r53 | 16:58 | |
| dmsimard | dhellmann: I've seen it start commenting on Packstack reviews today too, nonsense afaict .. complained about non-mergeable review while it was entirely fine | 16:58 |
| clarkb | mgagne: A1.2 | 16:58 |
| *** ihrachys has quit IRC | 16:58 | |
| dmsimard | No clue where it comes from or why it would want to do CI on Packstack. | 16:58 |
| *** bethwhite__ has quit IRC | 16:58 | |
| *** asettle has joined #openstack-infra | 16:58 | |
| mgagne | 100mbps then | 16:58 |
| fungi | dhellmann: dmsimard: pleia2 disabled the account earlier today | 16:59 |
| *** derekh has quit IRC | 16:59 | |
| dhellmann | fungi, pleia2 : ok, thanks! | 16:59 |
| fungi | we assume it was misconfigured and reporting on all changes in gerrit | 16:59 |
| dmsimard | fair assumption, it was mine as well | 16:59 |
| dhellmann | that's what I figured | 16:59 |
| *** yamahata has quit IRC | 17:00 | |
| *** tkelsey has joined #openstack-infra | 17:00 | |
| *** vgridnev has quit IRC | 17:00 | |
| *** yamahata has joined #openstack-infra | 17:00 | |
| clarkb | mgagne: ok, any chance its getting a little overzealous and rate limiting closer to 75mbps? | 17:00 |
| clarkb | mgagne: we appear to peak at about 60-75mbps | 17:01 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Increase the quota in the OSIC cloud1 https://review.openstack.org/357356 | 17:01 |
| mgagne | clarkb: I hope not, it's the default bw limiting feature in Nova/libvirt | 17:01 |
| clarkb | greghaynes: I am waiting for a local build to hopefully reproduce the xenial issue the nI can update dib with my null and warn stuff | 17:01 |
| cloudnull | IDK if we're ready for that yet, or if we want to wait for the DIB issues to be sorted, but there's the change. | 17:01 |
| clarkb | mgagne: do you knwo if its possible to check for that in logs on the hypervisor side? | 17:02 |
| *** piet has joined #openstack-infra | 17:02 | |
| clarkb | cloudnull: I think we want to get these image builds sorted first | 17:02 |
| cloudnull | ++ | 17:02 |
| *** jordanP has joined #openstack-infra | 17:02 | |
| fungi | clarkb: keep in mind that those peaks in cacti are a 5-minute average. if it's hitting 100mbps for one minute and 60mbps for the other four minutes in a sample... | 17:02 |
| greghaynes | clarkb: has there been thought given to adding image builds in to CI for you all? | 17:03 |
| *** sarob has joined #openstack-infra | 17:03 | |
| clarkb | greghaynes: and it looks like we can set the DIB_SHOW_IMAGE_USAGE var to work around it in the mean time | 17:03 |
| *** asettle has quit IRC | 17:03 | |
| *** sarob has quit IRC | 17:03 | |
| *** valderrv_ has joined #openstack-infra | 17:03 | |
| clarkb | greghaynes: there has been but it would be far too slow to gate on I think | 17:03 |
| clarkb | fungi: good point | 17:03 |
| *** electrofelix has quit IRC | 17:03 | |
| *** sarob has joined #openstack-infra | 17:03 | |
| *** tonytan_brb has joined #openstack-infra | 17:04 | |
| fungi | my estimates were slightly off... (100+69*4)/5=75.2 but you get my point | 17:04 |
| mgagne | clarkb: will do some tests on my side first | 17:04 |
| clarkb | greghaynes: we would need to hack it so that dib could use the local preexisting caches as much as possible | 17:04 |
| clarkb | mgagne: thank you | 17:04 |
| greghaynes | clarkb: hrm. Yea thats what I was just wondering - is that time mostly grabbing git repos? | 17:04 |
| zaro | yolanda: hi, i think there's some confusion with https://review.openstack.org/#/c/355587 | 17:04 |
| clarkb | greghaynes: even just updating the git repos is slow | 17:04 |
| greghaynes | because its actually pretty easy to point dib at the right place for that | 17:04 |
| clarkb | greghaynes: when you have >1k of them | 17:04 |
| fungi | clarkb: we could throw ntop or sar or something on there and then chart a more fine-grained burst rate | 17:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 17:05 |
| mgagne | clarkb: but I suggest you upgrade to something in the B series so you get more bandwidth, I'm sure you will hit 100mbps very soon anyway and A series won't answer your bw needs | 17:05 |
| clarkb | greghaynes: 1366 repos to update according ot my local build | 17:05 |
| clarkb | mgagne: ok | 17:05 |
| greghaynes | hah | 17:05 |
| *** Apoorva has joined #openstack-infra | 17:05 | |
| pabelanger | okay, we have the new DNS for osic-cloud8 but still missing SSL, so ^ is WIP until that is fixed | 17:05 |
| *** sree has joined #openstack-infra | 17:05 | |
| pabelanger | for ssl, mrhillsman is helping get that only | 17:05 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute009.vanilla IP addresses https://review.openstack.org/357081 | 17:06 |
| clarkb | mgagne: looks like its 500mbps not 50mbps for the B series? am I reading the flavor show output correctly | 17:06 |
| mgagne | yea, 500mbps | 17:06 |
| *** sdague has joined #openstack-infra | 17:06 | |
| fungi | that's probably plenty of overhead then | 17:06 |
| mgagne | clarkb: yea, typo earlier | 17:06 |
| *** tonytan4ever has quit IRC | 17:06 | |
| mgagne | clarkb: 100mbps for A1 series and 500mbps <-- for B1.1 and B1.2 and 1gbps for the rest | 17:06 |
| clarkb | great looks like B1.2 has 8GB of ram but the same disk and vcpus and 500mbps over 100mbps when compared to A1.2 so maybe we update to that | 17:07 |
| sdague | clarkb: I got the 40% number as the failure rate for dsvm jobs on internap since the first failure at 1am EST | 17:07 |
| sdague | for the gate | 17:08 |
| sdague | which should be 0. | 17:08 |
| mgagne | clarkb: it also has more CPU allocation (3:1 vs 1:1) IIRC | 17:08 |
| *** vgridnev has joined #openstack-infra | 17:08 | |
| sdague | and every spot check was pip or apt related | 17:08 |
| clarkb | pabelanger: you redid the rax one do you also want to rebuild the internap one? I would like to get to the bottom of this dib issue befure I context switch too heavily into something else | 17:08 |
| clarkb | sdague: ya beginning to think its probably network bw rate limiting in internap like we had in rackspace | 17:08 |
| *** timello has joined #openstack-infra | 17:09 | |
| *** vgridnev has quit IRC | 17:09 | |
| mgagne | clarkb: I'm not sure libvirt logs anything related to bw limiting when it kicks | 17:09 |
| *** sree has quit IRC | 17:10 | |
| pabelanger | clarkb: just catching up, we need a new mirror in internap? | 17:10 |
| *** vgridnev has joined #openstack-infra | 17:10 | |
| clarkb | pabelanger: yes I think we need a larger flavor. We are rate limited to 100mbps on the A1.2 flavor and are seeing many failures in internap. APache logs show that transfers are completing before all bits are transfered | 17:10 |
| pabelanger | clarkb: sure, I can launch a replacement today | 17:11 |
| clarkb | pabelanger: the B1.2 flavor will give us 500mbps has the asme number of cpus and disk but has 4x the memory | 17:11 |
| *** timello has quit IRC | 17:11 | |
| *** timello has joined #openstack-infra | 17:12 | |
| *** devkulkarni1 has joined #openstack-infra | 17:12 | |
| pabelanger | clarkb: okay, need to lunch first, then will dive into it | 17:12 |
| mrhillsman | yeah we need to update the endpoints to use SSL - got someone on that | 17:12 |
| mrhillsman | hope is they will be done shortly | 17:13 |
| *** mminesh has quit IRC | 17:13 | |
| clarkb | greghaynes: DIB_SHOW_IMAGE_USAGE doesn't seem to be set by nodepool. Any idea why we are executing that code at all? | 17:13 |
| mgagne | clarkb: we are investigating the networking bw limit issue. a coworker thinks it could be related to something else. | 17:13 |
| *** abadger1999 has joined #openstack-infra | 17:14 | |
| clarkb | mgagne: ok thank you for checking and let us know what you find | 17:14 |
| clarkb | pabelanger: ^ | 17:14 |
| *** devkulkarni has quit IRC | 17:14 | |
| *** mminesh has joined #openstack-infra | 17:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Stop getting extra flavor specs where they're useless https://review.openstack.org/355252 | 17:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Allow image and flavor by name for create_server https://review.openstack.org/355251 | 17:14 |
| fungi | either way, we've got 5-minute averages within 75% of the cap for that flavor, so should still upgreade | 17:14 |
| fungi | upgrade | 17:14 |
| *** shashank_hegde has joined #openstack-infra | 17:15 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Remove duplicate code to setup gerrit local replication https://review.openstack.org/355587 | 17:15 |
| abadger1999 | hey dhellmann, I was looking at using python-reno and wondered if you could answer one usage question... If my project has a mainline (devel) and branches stable-X.Y from there. Would we want to clear out the release-notes directory in devel every time we branch a new stable-X.Y? | 17:17 |
| *** esberglu has quit IRC | 17:17 | |
| *** mmedvede has quit IRC | 17:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add osic-cloud8 to openstack-infra https://review.openstack.org/357364 | 17:18 |
| *** mmedvede has joined #openstack-infra | 17:18 | |
| *** weshay_ has quit IRC | 17:20 | |
| pabelanger | fungi: mrhillsman: ^ adds our nodepool configuration for cloud-8, so one SSL is confirmed to be working. We can launch an AFS server, setup our DNS, then land above | 17:21 |
| pabelanger | I'll poke around a grafana next to make sure osic-cloud8 will be rendered | 17:21 |
| *** electrofelix has joined #openstack-infra | 17:21 | |
| *** valderrv_ has quit IRC | 17:21 | |
| *** electrofelix has quit IRC | 17:22 | |
| *** jerryz has joined #openstack-infra | 17:24 | |
| *** furlongm has quit IRC | 17:24 | |
| *** Hal1 has quit IRC | 17:24 | |
| *** furlongm has joined #openstack-infra | 17:24 | |
| *** valderrv_ has joined #openstack-infra | 17:24 | |
| greghaynes | clarkb: looks like it is set in nodepool builder.py | 17:24 |
| *** tonytan_brb is now known as tonytan4ever | 17:25 | |
| *** Hal has joined #openstack-infra | 17:26 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:26 | |
| mrhillsman | cool | 17:26 |
| dhellmann | abadger1999 : why don't we talk in #openstack-release to keep this channel clear for infra | 17:26 |
| mrhillsman | danke | 17:26 |
| fungi | pabelanger: mrhillsman: weareopenstack.org is an amusing domain... not using osic.org for this one? whois indicates the domain registrant is hidden | 17:26 |
| abadger1999 | I've pinged dhellman in the openstack channels too for a question on usage.dhthanks | 17:26 |
| *** abadger1999 has left #openstack-infra | 17:27 | |
| clarkb | greghaynes: gah that means my local test needs to be restarted to set it myself | 17:27 |
| *** tosky has joined #openstack-infra | 17:27 | |
| greghaynes | clarkb: :( | 17:27 |
| mrhillsman | fungi hehe | 17:27 |
| *** gyee has quit IRC | 17:28 | |
| *** Apoorva has quit IRC | 17:28 | |
| *** shashank_hegde has quit IRC | 17:28 | |
| mrhillsman | it is temporary, did not want to increase delay waiting on that subdomain since it may have to go through a few hands | 17:28 |
| fungi | mrhillsman: might want to be careful with that domain... wouldn't want to frighten the foundation legal team | 17:28 |
| fungi | not that i'll say anything | 17:28 |
| mrhillsman | hopefully one day it will come to be what i envision | 17:29 |
| mrhillsman | nothing for profit so not worried :) | 17:29 |
| *** stewie925 has quit IRC | 17:30 | |
| mrhillsman | and nothing to take away from openstack itself | 17:30 |
| *** dprince has quit IRC | 17:30 | |
| *** vgridnev has quit IRC | 17:30 | |
| *** _nadya_ has joined #openstack-infra | 17:30 | |
| *** esberglu has joined #openstack-infra | 17:31 | |
| fungi | mrhillsman: cool | 17:32 |
| *** stew925 has joined #openstack-infra | 17:33 | |
| *** valderrv_ has quit IRC | 17:34 | |
| *** degorenko is now known as _degorenko|afk | 17:35 | |
| fungi | annegentle: loquacities: it seems rackspace opened a support ticket to discuss trimming down docs.openstack.org, indicating that it's got more files than any other site in cloudsites... does any of the old accumulated content get audited for removal periodically, or are you only removing files people explicitly ask to have deleted? | 17:35 |
| annegentle | fungi we are pretty good at trimming, but it is a big site | 17:35 |
| mordred | fungi: we keep having more $things than other things | 17:35 |
| annegentle | fungi and that's funny that it's their biggest | 17:36 |
| fungi | "As it stands the site uses ~594,000 files, causing the account to be at a grand total of ~853,000 files. This is the largest single site on the storage device in our environment at this time. While for our customers we don't bill for inode consumption, it is still a very finite resource in Cloud Sites and we try our best to monitor and help find ways to reduce the total consumption to keep our | 17:36 |
| fungi | storage devices as happy as we can. | 17:36 |
| annegentle | fungi FTP in and do a cursory check | 17:36 |
| fungi | " | 17:36 |
| annegentle | fungi heh. wow. | 17:36 |
| *** baoli has joined #openstack-infra | 17:37 | |
| fungi | they opened the ticket on the 12th and pinged it for an update today. i only happened to notice because i logged into that tenant to poke at dns and happened to see that there was an open support ticket according to the dashboard | 17:37 |
| annegentle | fungi do you have a suggestion for archiving say, docs.openstack.org/diablo if they are not willing to? | 17:38 |
| annegentle | fungi I would want to have archival copies other than archive.org | 17:39 |
| *** bryan_att has joined #openstack-infra | 17:39 | |
| JayF | is docs.openstack.org a cloud sites site or a cloud files + CDN site? | 17:39 |
| *** tkelsey has quit IRC | 17:39 | |
| annegentle | fungi bexar and cactus are only containing index.html | 17:39 |
| * jroll wonders if openstack pricing will change with the spinout of cloud sites | 17:39 | |
| fungi | JayF: cloudsites | 17:39 |
| annegentle | JayF No idea. jroll we wonder also | 17:39 |
| *** tqtran has joined #openstack-infra | 17:39 | |
| jroll | yeah, probably time to think about moving it anyway | 17:39 |
| mgagne | clarkb: so I'm getting 93.4MB/s when uploading to server and 100MB/s when downloading from server. I think bw limiting isn't the issue here (where it would cap at 70MB/s instead of 100MB/s) | 17:40 |
| *** psilvad has quit IRC | 17:40 | |
| mgagne | clarkb: I suspect you are hitting peaks which aren't shown in the graph. | 17:40 |
| annegentle | jroll yep, we want to | 17:40 |
| fungi | agreed, we've already been having discussions about speeding up the afs-backed docs.o.o server plan | 17:40 |
| JayF | fungi: perhaps moving the generated, static docs to a cloud files CDN site on a different hostname would help? | 17:40 |
| jroll | ok, awesome | 17:40 |
| clarkb | mgagne: ya as fungi says its a 5 minute average so there are likely peaks in there where we need more bw | 17:40 |
| mgagne | The server I tested with is hosted on the exact same compute node as the mirror. I tested with another server on a different compute node and VM so routers aren't bypassed. | 17:41 |
| mgagne | VM -> VLAN | 17:41 |
| annegentle | JayF https://review.openstack.org/#/c/276482/ | 17:42 |
| *** gouthamr is now known as gouthamr_afk | 17:42 | |
| *** jcook has joined #openstack-infra | 17:42 | |
| annegentle | JayF that's the proposed solution | 17:42 |
| mordred | ++ | 17:42 |
| mordred | all it needs is someone hacking on it | 17:42 |
| *** gouthamr_afk is now known as gouthamr-afk | 17:42 | |
| sjmc7 | hi all, i’d appreciate a +A on https://review.openstack.org/#/c/356757/ to unblock our gate if anyone’s got a second. simple review to make a job non-voting | 17:43 |
| *** elo has quit IRC | 17:43 | |
| fungi | annegentle: loquacities: here's the meat of the ticket they opened http://paste.openstack.org/show/560837 | 17:44 |
| *** hockeynut has quit IRC | 17:44 | |
| fungi | worth mention, it echoes JayF's idea of switching to cloudfiles | 17:45 |
| *** valderrv_ has joined #openstack-infra | 17:45 | |
| dansmith | clarkb: mgagne: is this because for internap workers we use a local mirror that itself is getting maxed out? | 17:45 |
| clarkb | dansmith: yes that is what I think is happening | 17:45 |
| JayF | fungi: big downside of cloud files: no SSL support | 17:45 |
| clarkb | so we will replace ut with bigver instance | 17:46 |
| mordred | and also no rsync support | 17:46 |
| annegentle | fungi good to know. JayF I want https :) | 17:46 |
| JayF | fungi: well, SSL is supported; you just can't give it a cert to make the hostname happy | 17:46 |
| dansmith | clarkb: I wonder if it would make sense to do some shaping so that we provide fairness instead of starving one or two connections to the point of timeout? | 17:46 |
| fungi | JayF: well, bit that we'd probably do it anyway, but we don't have https on docs.o.o right now (and we will be able to add it with the afs plan) | 17:46 |
| annegentle | mordred and something like rsync | 17:46 |
| mordred | annegentle: yup | 17:46 |
| odyssey4me | cloudnull doesn't turbolift does rsync-like functionality for cloudfiles? cc mordred | 17:46 |
| odyssey4me | *do | 17:46 |
| fungi | er, not that we'd probably do it anyway i mean | 17:46 |
| *** kzaitsev_mb has quit IRC | 17:46 | |
| cloudnull | yup | 17:46 |
| mordred | not sure. in general though we have a lovely AFS cell that we use for stuff like this that has all of the things it needs | 17:47 |
| openstackgerrit | Merged openstack-infra/system-config: Test Gerrit CI comment regex for rel="nofollow" https://review.openstack.org/355188 | 17:47 |
| odyssey4me | in that case, perhaps we should implement a turbolift publisher for infra, so that docs can be properly mirrored from builds? | 17:47 |
| annegentle | fungi the redirects part is interesting, it might be good to get rid of .htaccess, but doesn't the proposal suggest yet more files? And still we need the archive solution. | 17:47 |
| mordred | so I don't really think there is much value in exploring tools that exist to publish into cloudfiles | 17:47 |
| cloudnull | shameless plug https://pypi.python.org/pypi/turbolift/ | 17:47 |
| fungi | dansmith: i doubt we have any control over the rate limiting gear in internap | 17:47 |
| dansmith | fungi: no, but we have control over our own nic :) | 17:47 |
| fungi | dansmith: fair enough, and that could work for egress on that machine, which is the bulk of the traffic | 17:48 |
| *** dprince has joined #openstack-infra | 17:48 | |
| mordred | (no offense intended to any o those tools) | 17:48 |
| annegentle | fungi also everything in /search is from 2011 so probably safe to say we're not using that mysql database for search. | 17:48 |
| jroll | mordred: ++ | 17:48 |
| mordred | annegentle: ++ | 17:48 |
| dansmith | fungi: if you leave your nic unrestricted and it tries to send at link speed, but are limited to 100mbit, then you're just going to get the upstream router killing your connections with dropped packets | 17:48 |
| fungi | dansmith: agreed | 17:48 |
| dansmith | fungi: right, if the problem is egress from the mirror, we can easily work on that | 17:48 |
| *** psilvad has joined #openstack-infra | 17:48 | |
| *** shashank_hegde has joined #openstack-infra | 17:48 | |
| annegentle | fungi so basically yep looks like we can trim, but give us an archive idea | 17:48 |
| fungi | dansmith: i had originally misinterpreted your suggestion as switching internap's packet shaping to use red or similar | 17:49 |
| dansmith | fungi: nay | 17:49 |
| jeblair | fungi: should i dust off the afs docs plan and propose it for next meeting? | 17:49 |
| fungi | jeblair: i think that would be wise | 17:49 |
| mordred | jeblair: I think so - it seems like the world is conspiring to make it time for that to get real work | 17:49 |
| annegentle | heh | 17:49 |
| jeblair | okay, i will do so | 17:49 |
| fungi | thanks! | 17:50 |
| fungi | annegentle: i'll try to summarize in a response on that ticket | 17:50 |
| odyssey4me | project-config cores, I'd appreciate reviews of https://review.openstack.org/357283 asap please | 17:50 |
| annegentle | fungi make sure you get loquacities in though before | 17:50 |
| annegentle | fungi or Andreas | 17:51 |
| annegentle | fungi since I don't know everything :) | 17:51 |
| fungi | annegentle: yep, i won't get to updating it before she's awake anyway | 17:51 |
| *** rcernin has joined #openstack-infra | 17:51 | |
| fungi | annegentle: and i want to avoid bothering andreas on his much-deserved vacation ;) | 17:51 |
| annegentle | fungi oh for sure | 17:51 |
| *** weshay has joined #openstack-infra | 17:52 | |
| jeblair | apparently i said "there are two approaches we can take" and then listed 3 things and no one called me on it. | 17:53 |
| mordred | jeblair: I think the difference between 2 and 3 is within the margin of error | 17:53 |
| fungi | jeblair: fuzzy math | 17:53 |
| annegentle | jeblair snerk | 17:53 |
| jeblair | mordred: yeah, that would have been my rejoinder :) | 17:53 |
| jeblair | also, half of the future in #2 has come to pass, so it needs a rethink with v2.5 | 17:54 |
| *** devkulkarni1 has quit IRC | 17:55 | |
| *** baoli has quit IRC | 17:56 | |
| *** Apoorva has joined #openstack-infra | 17:56 | |
| *** baoli has joined #openstack-infra | 17:56 | |
| jeblair | like -- put afs creds on launchers and modify the jjb->ansible translation to just sync right into afs on the launcher. | 17:56 |
| *** kzaitsev_mb has joined #openstack-infra | 17:57 | |
| clarkb | greghaynes: :( I don't have enough disk space for this reproduction build /me makes a bigger VM | 17:58 |
| clarkb | maybe I should start with just disabling the output in the nodepool builder | 17:58 |
| clarkb | then we can get this image built | 17:58 |
| fungi | jeblair: giving launchers afs write permission seems safe to me | 17:59 |
| *** acoles is now known as acoles_ | 17:59 | |
| *** annegentle has quit IRC | 17:59 | |
| fungi | as opposed to what we ended up needing to do with the swift publishing, granting write access to job nodes (which was way more complicated to make safe) | 17:59 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Stop running DIB usage report https://review.openstack.org/357393 | 18:00 |
| clarkb | pabelanger: fungi jeblair greghaynes ^ thats the temporary workaround for the xenial image build fails I think | 18:00 |
| clarkb | well maybe not temporary we should just set this per image rather than always | 18:00 |
| *** kzaitsev_mb has quit IRC | 18:01 | |
| jeblair | clarkb: wfm. also, we could just 'stat' the file and print it in nodepool. :) | 18:01 |
| clarkb | jeblair: it runs a fancy formatted du thing so you get a breakdown by dir and file for anything > 10MB | 18:02 |
| *** esikachev has joined #openstack-infra | 18:02 | |
| *** inc0 has quit IRC | 18:02 | |
| fungi | and apparently has portability issues | 18:03 |
| *** xarses has quit IRC | 18:03 | |
| *** valderrv_ has quit IRC | 18:03 | |
| rhedlind | Anyone know the first version of nodepool that dropped support for snapshot images and went to exclusively using diskimage builder? I would like to downgrade to the version before that. :) | 18:04 |
| clarkb | rhedlind: I don't think it has happened yet | 18:05 |
| *** Sukhdev has joined #openstack-infra | 18:05 | |
| fungi | rhedlind: clarkb: though i believe the intent is for nodepool 0.3.0 to be that last version you can count on for some things like snapshot build support and lack of zookeeper dependency | 18:06 |
| *** elo has joined #openstack-infra | 18:06 | |
| fungi | odds are that stuff will change before the next release | 18:07 |
| * mordred agrees with fungi and clarkb | 18:07 | |
| *** mfedosin has joined #openstack-infra | 18:07 | |
| rhedlind | mine is complaining that I don't have a images-dir in my config, so I figured it had already happened | 18:08 |
| pabelanger | clarkb: Wouldn't it make more sense to pass that in via nodepool.yaml in the future? | 18:08 |
| *** jtomasek has joined #openstack-infra | 18:08 | |
| *** mmedvede- has joined #openstack-infra | 18:08 | |
| fungi | rhedlind: did you pin to 0.3.0 already, or are you consuming from master branch tip? | 18:09 |
| clarkb | pabelanger: yup thats what the commit message says | 18:09 |
| clarkb | pabelanger: basically we don't need to bake it into nodepool particularly while it is buggy. And you can always set it via the nodepool config | 18:09 |
| pabelanger | clarkb: cool, I should read that | 18:09 |
| pabelanger | clarkb: ++ | 18:09 |
| rhedlind | I am at 0.3.0, not sure what dev though | 18:09 |
| *** nwkarsten has joined #openstack-infra | 18:10 | |
| fungi | rhedlind: it got recommended in a few places... http://lists.openstack.org/pipermail/openstack-dev/2016-July/098992.html | 18:10 |
| *** yamahata has quit IRC | 18:10 | |
| mfedosin | hi folks! I proposed a patch in project-config some time ago: https://review.openstack.org/#/c/352519/ so, could you review it if you have time? | 18:10 |
| mfedosin | thanks in advance | 18:10 |
| fungi | rhedlind: we expect disruptive changes on the master branch after the 0.3.0 tag | 18:11 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Make disk usage reporting more robust https://review.openstack.org/357401 | 18:11 |
| clarkb | greghaynes: ^ thats an untested start. Need to build a smaller image so that things don't break locally when I run out of disk | 18:12 |
| pleia2 | harddrive in my laptop just died (it's been unwell for a couple days) | 18:12 |
| rhedlind | fungi: I did use nodepool==0.3.0 when running pip install | 18:12 |
| *** rbrndt has quit IRC | 18:13 | |
| pleia2 | so I'm going to step away for the afternoon (and mostluly gone tomorrow for pre-conference anyway) | 18:13 |
| jeblair | pleia2: i have some spares but they probably won't fit in your laptop | 18:14 |
| *** kzaitsev_mb has joined #openstack-infra | 18:14 | |
| pleia2 | jeblair: I'm in Philly anyway :) | 18:14 |
| pleia2 | my husband is coming in tomorrow to give me a spare, woo | 18:14 |
| jeblair | pleia2: oh noes, they probably don't have fry's there! | 18:14 |
| jeblair | pleia2: oh good :) | 18:15 |
| *** furlongm has quit IRC | 18:15 | |
| *** furlongm has joined #openstack-infra | 18:15 | |
| fungi | rhedlind: interesting. do you mind popping the full error up on http://paste.openstack.org/ and linking it in here? | 18:16 |
| *** zul has quit IRC | 18:16 | |
| fungi | i'm skimming the state of the code at 0.3.0 to see why it would care, and which commit started making it care | 18:16 |
| fungi | my guess is the config validator started expecting it | 18:17 |
| jeblair | it's probably okay to just set it to /tmp if you're not going to use it | 18:17 |
| rhedlind | fungi: http://paste.openstack.org/show/560844/ | 18:19 |
| fungi | the only place (in 0.3.0 anyway) that i see expecting it is in builder.py | 18:19 |
| fungi | ahh, yep | 18:19 |
| fungi | so nodepoold is starting a builder thread that isn't needed? | 18:19 |
| dansmith | fungi: pabelanger: either of you willing to run arbitrary code on the internap mirror as root from a pastebin I provide? :) | 18:20 |
| fungi | oh, or maybe the snapshot-based builds also happen out of the builder | 18:20 |
| *** annegentle has joined #openstack-infra | 18:20 | |
| dansmith | fungi: pabelanger er I mean, I have some fairness and limiting commands that are dead simple which would be nice to test on that to avoid stalling individual connections at the peaks :) | 18:20 |
| mgagne | like curl paste.openstack.org | sudo sh - ? :D | 18:21 |
| dansmith | mgagne: exactly :P | 18:21 |
| fungi | dansmith: i'm happy to evaluate it at least. i've done some packet shaping on linux in the past | 18:21 |
| dansmith | fungi: http://pastebin.com/kJLpmubR | 18:21 |
| fungi | it | 18:21 |
| fungi | grrr | 18:21 |
| dansmith | fungi: limit outbound to 1mbit below your limit, provide fairness per flow | 18:21 |
| dansmith | fungi: change the DEV accordingly of course | 18:21 |
| fungi | it's worth considering whether we should look into implementing it on more than just that mirror | 18:21 |
| *** kzaitsev_mb has quit IRC | 18:21 | |
| dansmith | fungi: definitely if we have other places where we're limited, and we could adjust this to be specific to mirror traffic, etc but this is the simplest test | 18:22 |
| dansmith | fungi: and you can do this to show details about it afterwards: tc -s -d class show dev $DEV | 18:22 |
| dansmith | fungi: which will show you balance between all the flows, etc | 18:23 |
| *** zul has joined #openstack-infra | 18:23 | |
| *** pvaneck has joined #openstack-infra | 18:23 | |
| fungi | yep, i'm just refreshing my reading of the tc manpage | 18:23 |
| dansmith | fungi: tested locally with iperf | 18:23 |
| fungi | this looks sane to me. clarkb: you have any opinion on it? | 18:24 |
| pabelanger | fungi: rhedlind: are you doing nodepoold --no-builder ? | 18:24 |
| rhedlind | fungi: looks like the exception went in on Jan 12: https://github.com/openstack-infra/nodepool/commit/9253ae99e91e07cf5446029866ca5e7fa37e8fc8#diff-787275e08e2ebedbbf2a2b0ca0af8c80 | 18:24 |
| rhedlind | pabelanger: no, just nodepoold -d | 18:24 |
| pabelanger | rhedlind: if you add the switch, that will stop the builder process from spawning | 18:25 |
| rhedlind | pabelanger: don't I need it for snapshot images though? | 18:25 |
| pabelanger | no | 18:25 |
| pabelanger | well | 18:25 |
| *** pilgrimstack has joined #openstack-infra | 18:25 | |
| clarkb | fungi: not really other than it would be nice if libvirt was better about advertising appropriate line speeds | 18:25 |
| pabelanger | I should we did snapshots still in nodepoold | 18:26 |
| pabelanger | s/should/thought | 18:26 |
| clarkb | but virtio probably means more direct access to things rather than a fake full duplex 100Mbps interface | 18:26 |
| clarkb | greghaynes: I have a build going now without a lot of infra's extra stuff to at least check my changes work properly and will see if I can reproduce with a reduced image spec | 18:27 |
| greghaynes | clarkb: sounds like fun ;) | 18:29 |
| clarkb | Journal size too big for filesystem. | 18:30 |
| clarkb | thats a nwe one | 18:30 |
| greghaynes | too big!? | 18:30 |
| *** persia has quit IRC | 18:30 | |
| clarkb | thats what it says | 18:30 |
| greghaynes | I guess that means the journal is just plain larger than the fs? | 18:30 |
| clarkb | let me make a paste with more context | 18:30 |
| *** tphummel has quit IRC | 18:31 | |
| greghaynes | we do statically set a journal of some size IIRC, basically its a way to handle the fact that folks usually resize up the images so a larger journal is wanted | 18:31 |
| clarkb | http://paste.openstack.org/show/560845/ | 18:32 |
| *** persia has joined #openstack-infra | 18:32 | |
| clarkb | I wonder if this is crashed due to staleness from my previous build that ran out of disk | 18:34 |
| clarkb | like loop0 is unhappy | 18:34 |
| greghaynes | hrm, you can check losetup -a | 18:35 |
| tosky | hi, if anyone could approve this patch which temporarily allows for merges in sahara-tests by sahara-tests cores, we (from Sahara) would quickly finish a code import: https://review.openstack.org/#/c/357290/ | 18:35 |
| greghaynes | see if a loop device is still mapped to an image.raw | 18:35 |
| tosky | (https://review.openstack.org/#/c/357291/ is the follow-up to disable that once the merges are done) | 18:36 |
| *** tphummel has joined #openstack-infra | 18:36 | |
| clarkb | I rebooted to clear out any temporary mounts or loopbacks that didn't get cleaned up | 18:36 |
| greghaynes | clarkb: yea, that reads like p1 is a crazy small partition | 18:36 |
| greghaynes | ok | 18:36 |
| clarkb | because I am lazy and reboot is really fast when you don't wait for urandom | 18:36 |
| *** hockeynut has joined #openstack-infra | 18:36 | |
| greghaynes | heh | 18:36 |
| *** jheroux has quit IRC | 18:36 | |
| *** ayoung has quit IRC | 18:37 | |
| pabelanger | rcarrillocruz: mordred: have time to review the cloud_launcher stack? https://review.openstack.org/#/c/354852 Want to get that landed so I can add osic-cloud8 support | 18:39 |
| pabelanger | rcarrillocruz: mordred: I've tested it a few times and works | 18:40 |
| fungi | dansmith: clarkb: the requested 99mbit htb and sfq have been applied to eth0 on mirror.nyj01.internap.o.o so let's see if further failures fitting this pattern emerge | 18:42 |
| dansmith | fungi: woot | 18:42 |
| dansmith | fungi: can you run that status command and pastebin it? just because I'm curious | 18:42 |
| fungi | note this is temporary, but if we see positive results we should discuss puppeting it with known bw caps for all our mirrors | 18:42 |
| clarkb | we should also consider increasing the bw if we are hitting the limits. Particularly at 100mbps | 18:43 |
| fungi | dansmith: http://paste.openstack.org/show/560846/ | 18:43 |
| *** mfedosin has quit IRC | 18:43 | |
| fungi | clarkb: yep, i still think we should bump the flavor on the internap mirror regardless | 18:43 |
| dansmith | clarkb: we can use the backlog thing to tell how far oversubscribed we are.. it could be that the occasional peak is just disrupting enough to be a problem | 18:43 |
| *** honza is now known as honza|afk | 18:43 | |
| pabelanger | fungi: clarkb: I'm ready to do that work now, launching a new server | 18:44 |
| *** sjmc7 has left #openstack-infra | 18:44 | |
| fungi | dansmith: yeah, in theory if we safely buffer through brief bursts then it may no longer have any adverse impact | 18:44 |
| dansmith | fungi: right, that's what I'm thinking, but we'll see if it pans out that way | 18:44 |
| fungi | the real risk is when your bursts are too sustained, too large, or too close together to be able to safely buffer | 18:45 |
| fungi | but even then, you ideally get the least disruptive requests sacrificed rather than having it obliterate a bunch of established flows | 18:45 |
| *** yamahata has joined #openstack-infra | 18:46 | |
| fungi | in my former isp/hosting life i spent way too much time tuning packet shaping rules because our owners were cheapskates who wanted to squeeze by on underprovisioned/oversubscribed hardware and links wherever possible | 18:46 |
| fungi | it was one of those "how far can you push it before the customers will notice" sorts of situations | 18:47 |
| *** dtardivel has quit IRC | 18:47 | |
| *** ociuhandu has quit IRC | 18:47 | |
| jeblair | fungi: the *remaining* customers, that is. :) | 18:47 |
| fungi | exactly | 18:47 |
| dansmith | fungi: the fairness queuing is really good practice anyway, just to avoid one thing DoSing everything else | 18:48 |
| fungi | heck, we still had isdn customers in 2005, so some contracts you just can't shake | 18:48 |
| dansmith | you want the DoSer to fail, not everything else | 18:48 |
| clarkb | greghaynes: ok more info, that same du is used to set the image size :) and its 64MB which tells me I probably have a bug in my change \o/ | 18:48 |
| jkilpatr | Who maintains the ansible linters gate? | 18:48 |
| fungi | dansmith: yep | 18:48 |
| greghaynes | hahah | 18:48 |
| greghaynes | clarkb: in theory CI should have failed... | 18:48 |
| fungi | clarkb: or we're getting some awesome compression now | 18:49 |
| *** rbrndt has joined #openstack-infra | 18:49 | |
| greghaynes | ok, I have to meeting for a bit | 18:50 |
| *** jheroux has joined #openstack-infra | 18:52 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 18:52 |
| *** kzaitsev_mb has joined #openstack-infra | 18:52 | |
| *** amitgandhinz has quit IRC | 18:53 | |
| fungi | clarkb: dansmith: we actually missed an snmp sample at 17:50 so there is almost certainly something disturbing our network there (could of course have been a huge burst i guess) http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3145&rra_id=all | 18:54 |
| dansmith | fungi: spike on the graph below around the same time | 18:55 |
| dansmith | oh | 18:55 |
| dansmith | different time scale hehe | 18:55 |
| openstackgerrit | Merged openstack/os-client-config: Add support for configuring split-stack networks https://review.openstack.org/357342 | 18:56 |
| fungi | similar missing sample in osic around 16:00 http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4943&rra_id=all | 18:56 |
| dansmith | fungi: note the big increase in traffic right at the end of jul | 18:56 |
| dansmith | fungi: which would explain this becoming more of a problem | 18:56 |
| fungi | dansmith: yeah, probably somebody repeatedly rebasing huge series of nova changes ;) | 18:56 |
| dansmith | those damn nova people, I swear | 18:56 |
| *** zz_ja has quit IRC | 18:57 | |
| fungi | actually, i think the gap on the internap graph represents two missing samples back to back | 18:57 |
| dansmith | yeah looks like it | 18:58 |
| jkilpatr | pabelanger, hey what version of ansible-linters are you using? My project (browbeat) has a playbook failing that works fine when I install ansible-linters from pip | 18:58 |
| *** _nadya_ has quit IRC | 18:58 | |
| clarkb | my vote is move forward with a B1.2 isntance with 5x the bw | 18:58 |
| fungi | agreed | 18:58 |
| *** jtomasek has quit IRC | 18:59 | |
| clarkb | jkilpatr: have a link to the log? I am hoping we use tox to run that linter... | 18:59 |
| fungi | though this provides a good experiment for how we might reduce the impact of brief spikes in providers where we know we're rate limited | 18:59 |
| jkilpatr | clarkb, http://logs.openstack.org/59/356659/6/check/gate-browbeat-linters-ubuntu-xenial/f962d3f/console.html | 18:59 |
| *** zz_ja has joined #openstack-infra | 18:59 | |
| *** jheroux has quit IRC | 19:00 | |
| clarkb | jkilpatr: yup it uses tox -elinters and the version installed is http://logs.openstack.org/59/356659/6/check/gate-browbeat-linters-ubuntu-xenial/f962d3f/console.html#_2016-08-18_16_57_08_462017 so thats completely controlled by your project | 19:00 |
| pabelanger | jkilpatr: test-requirements.txt will have the version, but it is uncapped. So it will pull down the latest version | 19:00 |
| *** _sarob has joined #openstack-infra | 19:01 | |
| *** sarob has quit IRC | 19:02 | |
| pabelanger | clarkb: fungi: ya, using B1.2 | 19:02 |
| jkilpatr | pabelanger, clarkb ok same version local and remote, "find . -type f -regex '.*.y[a]?ml' -print0 | xargs -t -n1 -0 ansible-lint" runs with some warnings but exits successfully when I run it locally. Which seems to be all tox does | 19:02 |
| clarkb | jkilpatr: have you tried running tox to reproduce? | 19:03 |
| pabelanger | ya | 19:03 |
| pabelanger | tox -elinters should give the same results at the gate | 19:03 |
| clarkb | or -re linters if you have run it before (this will start you with a clean venv) | 19:04 |
| fungi | modulo any missing packages or constraints applied | 19:04 |
| jkilpatr | clarkb, guess that's the next step. I was hoping it was just linters. Down the rabbit hole I go | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Stop getting extra flavor specs where they're useless https://review.openstack.org/355252 | 19:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Allow image and flavor by name for create_server https://review.openstack.org/355251 | 19:05 |
| mordred | ok. those should work now | 19:05 |
| *** fifieldt has quit IRC | 19:06 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: builders: add 'publish over ssh' support as a build step https://review.openstack.org/98437 | 19:06 |
| *** amitgandhinz has joined #openstack-infra | 19:08 | |
| *** kzaitsev_mb has quit IRC | 19:08 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:10 | |
| * clarkb is really confused by dib and has reverted the new patch and is trying the last release to convince himself that dib works :P | 19:10 | |
| clarkb | and of course it does work | 19:11 |
| *** julim has quit IRC | 19:13 | |
| *** jheroux has joined #openstack-infra | 19:14 | |
| *** julim has joined #openstack-infra | 19:14 | |
| *** timello has quit IRC | 19:15 | |
| *** salv-orl_ has joined #openstack-infra | 19:15 | |
| *** fifieldt has joined #openstack-infra | 19:18 | |
| *** salv-orlando has quit IRC | 19:18 | |
| *** vhosakot has quit IRC | 19:18 | |
| *** vhosakot has joined #openstack-infra | 19:18 | |
| pabelanger | clarkb: looks like we may need to update mount_volume.sh, internap is complaining: http://paste.openstack.org/show/560849/ | 19:20 |
| pabelanger | clarkb: I guess that mean, when I create a new volume, it already has a FS? | 19:21 |
| *** timello has joined #openstack-infra | 19:22 | |
| *** Na3iL has quit IRC | 19:22 | |
| *** kzaitsev_mb has quit IRC | 19:22 | |
| clarkb | pabelanger: the problem appears to be that make_swap.sh is grabbing /dev/vdb before the volume mount gets it | 19:23 |
| *** vhosakot has quit IRC | 19:23 | |
| cinerama | greghaynes: we're meeting today, yes? | 19:24 |
| *** vhosakot has joined #openstack-infra | 19:24 | |
| pabelanger | clarkb: let me see why that is | 19:24 |
| *** kszukielojc has quit IRC | 19:25 | |
| clarkb | pabelanger: looks like make_swap just assumes it can use /dev/vdb | 19:25 |
| *** markusry has quit IRC | 19:26 | |
| pabelanger | clarkb: ya | 19:27 |
| *** kzaitsev_mb has joined #openstack-infra | 19:27 | |
| pabelanger | I don't think shade wait for the volume to be mounted | 19:27 |
| pabelanger | so we might be racing | 19:27 |
| *** Hal has quit IRC | 19:28 | |
| clarkb | pabelanger: I don't think shade will but launch node should | 19:29 |
| *** _nadya_ has joined #openstack-infra | 19:29 | |
| clarkb | pabelanger: but I think /dev/vdb is the volume we have just conflicted our two disk management scripts | 19:29 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-logstash: Make the java heap size configurable https://review.openstack.org/357426 | 19:29 |
| pabelanger | clarkb: Ah, you are right | 19:31 |
| *** matt-borland has quit IRC | 19:31 | |
| pabelanger | we mount the volume in launch-node | 19:31 |
| pabelanger | err, attach | 19:31 |
| clarkb | pabelanger: so probably the work around here is to just remove the /dev/vdb check from make_swap.sh | 19:33 |
| *** dgonzalez has quit IRC | 19:34 | |
| *** matt-borland has joined #openstack-infra | 19:34 | |
| *** kzaitsev_mb has quit IRC | 19:34 | |
| *** e0ne has joined #openstack-infra | 19:34 | |
| pabelanger | clarkb: okay, trying that now | 19:35 |
| *** _nadya_ has quit IRC | 19:37 | |
| zaro | zxiiro: are you by chance using the gerrit trigger plugin? | 19:38 |
| *** esikachev has quit IRC | 19:38 | |
| *** nwkarsten has quit IRC | 19:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:39 | |
| *** nwkarsten has joined #openstack-infra | 19:39 | |
| *** nwkarste_ has joined #openstack-infra | 19:40 | |
| *** florianf has quit IRC | 19:41 | |
| rcarrillocruz | pabelanger: sorry, was out | 19:42 |
| rcarrillocruz | sorted out that change i see | 19:42 |
| rcarrillocruz | could you pls review https://review.openstack.org/#/c/357183/ and https://review.openstack.org/#/c/357187/ ? | 19:42 |
| rcarrillocruz | should be quick | 19:42 |
| rcarrillocruz | conf changes | 19:42 |
| pabelanger | +2 | 19:43 |
| *** nwkarsten has quit IRC | 19:44 | |
| clarkb | pabelanger: I think the underlying issue with that make_swap and make_volume interaction is that ephemeral disks (where we want to put swap and /opt) and volumes can share the same /dev device names. And not only that but they can be different in different clouds :( | 19:44 |
| *** ociuhandu has joined #openstack-infra | 19:45 | |
| *** dgonzalez has joined #openstack-infra | 19:45 | |
| rcarrillocruz | thx | 19:46 |
| pabelanger | clarkb: ya, I'm going to just bring up the server first | 19:46 |
| pabelanger | see what is happening, as I am blind right now | 19:47 |
| clarkb | I am not sure that nova enforces a naming scheme across hypervisors | 19:47 |
| clarkb | one thing low on my list is to switch to using the uuid based lable stuff with cinder volumes as its more reliable to map a volume to its device in linux | 19:47 |
| pabelanger | rcarrillocruz: https://review.openstack.org/#/c/354851 just need a +2 on that to land the stack | 19:47 |
| clarkb | its possible that ephemeral devices show up in a more readily identifiable way like that too | 19:48 |
| pabelanger | right | 19:48 |
| *** rbuzatu has quit IRC | 19:48 | |
| rcarrillocruz | +A | 19:48 |
| *** piet has quit IRC | 19:48 | |
| mtreinish | clarkb: does https://review.openstack.org/#/c/351344/ look sane to you. jesusaur and I were looking at it before, but there were some open questions (which I don't remember) | 19:48 |
| rcarrillocruz | it would be good if in the mid-cycle we documented the various differences on how our providers show the cinder volumes | 19:48 |
| *** kzaitsev_mb has quit IRC | 19:49 | |
| rcarrillocruz | as i want to put that make swap thing on the launcher playbook | 19:49 |
| jesusaur | mtreinish: clarkb: iirc, the discussion before had to do with how to tell if the plugin is already installed to avoid running the install command on every puppet run | 19:51 |
| clarkb | mtreinish: we are trying to move away from alll that overparameterized stuff and just pass in fiels using their conf.d setup | 19:51 |
| clarkb | mtreinish: jesusaur so I don't think we would want to do it that way and instead just update our 99-output appropriately | 19:51 |
| clarkb | mtreinish: jesusaur then have an install_plugin manifest that the consumers of logstash module can use to install the necessary plugins | 19:52 |
| clarkb | so that looks like define logstash::install_plugin() in puppet-logstash | 19:52 |
| clarkb | then in system-config use that to install mqtt plugin and edit the output file to use it | 19:52 |
| jesusaur | clarkb: ++ | 19:53 |
| *** Hal has joined #openstack-infra | 19:53 | |
| jesusaur | i had forgotten about the new conf.d setup | 19:53 |
| mtreinish | heh, I didn't even know that was a thing | 19:54 |
| *** xarses has joined #openstack-infra | 19:54 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-askbot: Add explicit configuration support for Akismet https://review.openstack.org/347020 | 19:55 |
| fungi | crinkle: thanks! ^ updated per your comment | 19:55 |
| zxiiro | zaro: yes we still use it heavily. Zuul doesn't do what we want so we haven't moved to it yet | 19:55 |
| *** andrey-mp has joined #openstack-infra | 19:57 | |
| *** Apoorva_ has joined #openstack-infra | 19:59 | |
| andrey-mp | Hi! Is it right channel to ask about IRC? I need to add log saving for openstack-ec2api channel... Are there instructions about this? | 20:00 |
| clarkb | andrey-mp: http://docs.openstack.org/infra/system-config/irc.html should cover the details for that | 20:01 |
| fungi | andrey-mp: http://docs.openstack.org/infra/system-config/irc.html | 20:01 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute010.vanilla IP addresses and enabling it as it is functional https://review.openstack.org/357084 | 20:01 |
| fungi | clarkb beat me to the draw | 20:01 |
| andrey-mp | thank you! | 20:01 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute007 IP addresses https://review.openstack.org/357075 | 20:02 |
| *** Apoorva has quit IRC | 20:03 | |
| fungi | andrey-mp: what it doesn't mention there is that after dealing with the instructions in the access section, you need to add your channel to http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml#n120 | 20:03 |
| fungi | to get channel logging | 20:03 |
| andrey-mp | fungi: thank you! this is that i looking for | 20:04 |
| pabelanger | clarkb: http://74.217.28.58/ | 20:04 |
| pabelanger | clarkb: I did a 2-step process for the volume | 20:04 |
| pabelanger | worked with out an issue | 20:04 |
| pabelanger | I'll kick over the dns here in a few minutes | 20:05 |
| *** _ari_ has quit IRC | 20:05 | |
| zaro | zxiiro: i'm trying out gerrit-trigger ver 2.22.1 with core 1.609.3 but i can't seem to make it work at all. which version are you using? | 20:05 |
| crinkle | fungi: yay | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 20:08 |
| pabelanger | #status log mirror.nyj01.internap.openstack.org replacement server now online, DNS has been updated to 74.217.28.58 | 20:08 |
| openstackstatus | pabelanger: finished logging | 20:08 |
| *** ayoung has joined #openstack-infra | 20:09 | |
| fungi | thanks pabelanger! | 20:09 |
| pabelanger | fungi: no problem | 20:09 |
| clarkb | pabelanger: thanks and ya two step is ok for things that dont immediately try to usr the volume like es | 20:09 |
| clarkb | es is tricky because we have to make aure volume is mounted priper before puppet runs | 20:09 |
| pabelanger | I'll check back in 60mins to confirm traffic has stopped on the original server | 20:09 |
| clarkb | otherwise sadness | 20:10 |
| zxiiro | zaro: i got jenkins core 1.651.3 and GT 2.21.1 | 20:10 |
| zxiiro | zaro: what isn't working? | 20:10 |
| pabelanger | clarkb: ya, it did work for the other mirror we did, so 50% right now | 20:10 |
| fungi | pabelanger: since you seem to be really into a mirror maintenance headspace at the moment, any thoughts on what we should do about the absurd amounts of apache logs we accumulate, for example on the current osic mirror? | 20:10 |
| zxiiro | zaro: the job's not passing in the GERRIT_* properties? | 20:10 |
| *** gyee has joined #openstack-infra | 20:11 | |
| fungi | pabelanger: if we don't do something about it, we're on track to fill up the rootfs there with apache access logs sometime today or early tomorrow | 20:11 |
| pabelanger | wow | 20:11 |
| pabelanger | just seen | 20:12 |
| fungi | i mean, i can delete some old logs, compress/rotate, whatever, but unless we scale back on either the log volume or retention we are going to be right back here in short order | 20:12 |
| pabelanger | fungi: at first glance, maybe more the 7 day log retention? | 20:12 |
| pabelanger | /more/move | 20:12 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Change default for FORCE_CONFIG_DRIVE to False https://review.openstack.org/357443 | 20:13 |
| fungi | that seems reasonable to me. i could even be convinced to scale back to 4 days access log retention (and maybe keep or increase the current error log retention) | 20:13 |
| *** javeriak has quit IRC | 20:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:13 | |
| pabelanger | let me see how it works today | 20:13 |
| pabelanger | ya, it looks to be unmanaged right now | 20:16 |
| pabelanger | the default is rotate 52 | 20:16 |
| *** andrey-mp has left #openstack-infra | 20:16 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Drop postgresql jobs in gate https://review.openstack.org/357444 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method https://review.openstack.org/319752 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs. https://review.openstack.org/319753 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins https://review.openstack.org/319754 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser. https://review.openstack.org/319623 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes https://review.openstack.org/319622 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules. https://review.openstack.org/319625 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings. https://review.openstack.org/319624 | 20:17 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Make builder.Jenkins inherit from jenkins.Jenkins https://review.openstack.org/319751 | 20:17 |
| *** bhunter71 has quit IRC | 20:19 | |
| *** psachin has joined #openstack-infra | 20:20 | |
| *** _coolsvap_ has quit IRC | 20:21 | |
| dansmith | fungi: sdague: the one failure on internap recently was not network-related, FYI if you go back to look | 20:23 |
| sdague | dansmith: ok, cool | 20:23 |
| fungi | dansmith: excellent | 20:23 |
| sdague | fungi: it does look like by about 10am it self corrected on internap | 20:23 |
| *** afred312_ has joined #openstack-infra | 20:23 | |
| sdague | the failure window is 1am - 10am EST | 20:23 |
| fungi | dansmith: sdague: keep in mind as of a few minutes ago that mirror has been replaced with an instance using a flavor with a much higher bandwidth cap | 20:24 |
| *** esikachev has joined #openstack-infra | 20:24 | |
| sdague | however, anything that prevents future fails would be great | 20:24 |
| dansmith | fungi: gah | 20:24 |
| *** BigWillie has quit IRC | 20:24 | |
| *** kzaitsev_mb has quit IRC | 20:24 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm-docs draft build for eslint-config-openstack. https://review.openstack.org/356609 | 20:24 |
| *** afred312 has quit IRC | 20:24 | |
| fungi | so if the issues were related to rate limiting, they should cease unless we increase activity 5x over | 20:24 |
| *** kgiusti has left #openstack-infra | 20:25 | |
| jeblair | fungi: so good until next thursday? | 20:25 |
| fungi | jeblair: bingo! | 20:25 |
| prometheanfire | so... https://review.openstack.org/#/c/310865/ ? | 20:26 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Created DSVM Job for NPM Projects https://review.openstack.org/348056 | 20:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Update logrotate for mirror server https://review.openstack.org/357453 | 20:28 |
| pabelanger | fungi: ^ that should give us 7 day retention | 20:28 |
| jkilpatr | clarkb, where can I get the tox.ini the ansible-linters test is using? its not the same as the one in the browbeat repo | 20:29 |
| pabelanger | clarkb: so we need to restart nodepool today to might up your patch? | 20:32 |
| pabelanger | jkilpatr: it should be | 20:32 |
| *** xarses_ has joined #openstack-infra | 20:33 | |
| jkilpatr | pabelanger, ignore me, need to rebase this branch, old repo sorry | 20:33 |
| pabelanger | jkilpatr: Ya, I just added it this week | 20:34 |
| *** baoli has quit IRC | 20:34 | |
| clarkb | pabelanger: ya but only the builder | 20:34 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Add vlan_id variable with value 2551 https://review.openstack.org/357187 | 20:34 |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** Jeffrey4l has joined #openstack-infra | 20:35 | |
| *** jcoufal_ has joined #openstack-infra | 20:35 | |
| fungi | gotta go grab some dinner, will return to continue reviewing wiki puppet in a couple hours | 20:35 |
| *** xarses has quit IRC | 20:36 | |
| pabelanger | clarkb: okay, cool Was going to say, if we need to restart nodepoold, we'd pull in a few things, like new shade | 20:36 |
| *** TravT has joined #openstack-infra | 20:36 | |
| zaro | zxiiro: can't get it to trigger or report anything back to gerrit. | 20:37 |
| TravT | hello. i'm wondering if I can get some help with approving marking a Searchlight job as non-votiing | 20:37 |
| TravT | https://review.openstack.org/#/c/356757/ | 20:37 |
| *** jcoufal has quit IRC | 20:37 | |
| zaro | zxiiro: maybe i'm setting up wrong? i configured the server, setup a job, and added a trigger for comment-added, change-abandoned and change-restored. i see the events on gerrit side but nothing happens on jenkins side | 20:38 |
| zxiiro | zaro: my first guess if it's not trigger might be https://wiki.jenkins-ci.org/display/JENKINS/Plugins+affected+by+fix+for+SECURITY-170 but I think you're on an old enough version taht you're not affected | 20:40 |
| *** vhosakot has quit IRC | 20:41 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add an example logging.conf for development https://review.openstack.org/352599 | 20:41 |
| zxiiro | zaro: just in case you have a bad job config you can compare with this https://github.com/opendaylight/releng-builder/blob/master/jjb/releng-macros.yaml#L143-L163 | 20:42 |
| zaro | zxiiro: thanks. will try with 1.651.3 now | 20:43 |
| *** _nadya_ has joined #openstack-infra | 20:43 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add a fake-secure.conf https://review.openstack.org/352600 | 20:44 |
| *** baoli has joined #openstack-infra | 20:44 | |
| *** armax has joined #openstack-infra | 20:45 | |
| *** rbuzatu has joined #openstack-infra | 20:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:48 | |
| pabelanger | clarkb: going to enqueue 357393 into gate, so we can start up on xenail builds | 20:49 |
| *** ansmith has quit IRC | 20:49 | |
| pabelanger | it still hasn't passed check yet | 20:49 |
| clarkb | pabelanger: ok | 20:49 |
| clarkb | I am still fighting with dib to get a working bug fix | 20:50 |
| clarkb | I am learning much about bash | 20:50 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute020.vanilla IP addresses https://review.openstack.org/357102 | 20:52 |
| *** vhosakot has joined #openstack-infra | 20:52 | |
| *** rbuzatu has quit IRC | 20:52 | |
| ianw | clarkb: did you see the strace output for the hack i put in? i'm not sure it corresponds to the null problem | 20:53 |
| ianw | i'm just looking now | 20:53 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Fix disabled always returning true https://review.openstack.org/357491 | 20:53 |
| ianw | 3879 write(1, "ib_tmp/dib_build.y8yDOp9f/built/"..., 11127) = -1 EAGAIN (Resource temporarily unavailable) | 20:53 |
| ianw | that is from "cat" | 20:54 |
| *** _nadya_ has quit IRC | 20:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Have buildZooKeeperHosts accept a config object https://review.openstack.org/357347 | 20:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a script to print the ZK tree https://review.openstack.org/357348 | 20:54 |
| ianw | clarkb: have a look in /opt/dib_tmp/dib_build.y8yDOp9f/du_output.tmp.strace | 20:55 |
| *** SumitNaiksatam has quit IRC | 20:55 | |
| clarkb | ianw: I didn't | 20:56 |
| ianw | clarkb: my concern is that *maybe* this is more about a large amount of output and the way nodepool is capturing it | 20:56 |
| openstackgerrit | Merged openstack-infra/nodepool: Stop running DIB usage report https://review.openstack.org/357393 | 20:56 |
| *** spzala has quit IRC | 20:56 | |
| clarkb | ianw: the large amount of output from du? I am definitely struggling with that when trying to debug locally I can't export du_output for example | 20:56 |
| ianw | clarkb: so yeah, the du_output as a var was a bad choice, it's like 100mb | 20:56 |
| ianw | clarkb: but i proposed https://review.openstack.org/#/c/356917/ | 20:57 |
| ianw | clarkb: but /usr/local/bin/disk-image-create on nodepool itself currently has a little hack i put in last night my time | 20:57 |
| clarkb | ianw: cool I also think we could/should just do du -h | sort -h | 20:57 |
| ianw | clarkb: have a look at it, search for #ianw | 20:57 |
| *** kzaitsev_mb has quit IRC | 20:57 | |
| armax | mordred: is this ready to roll? https://review.openstack.org/#/c/352594/ | 20:58 |
| clarkb | ianw: and avoid numfmt entirely | 20:58 |
| ianw | clarkb: "luckily" dib doesn't clean up properly and left behind -> /opt/dib_tmp/dib_build.y8yDOp9f/ where you can see each of the steps | 20:58 |
| *** gouthamr-afk is now known as gouthamr | 20:58 | |
| ianw | clarkb: the problem is, it still happens if you output numfmt to a file, then just "cat" that file | 20:59 |
| *** raildo has quit IRC | 20:59 | |
| clarkb | ah so its the output of the data itself not in the processing | 20:59 |
| mordred | armax: oh - funny story, I was just poking you in #openstack-neutron | 20:59 |
| ianw | clarkb: yep | 20:59 |
| armax | mordred: lol | 20:59 |
| armax | mordred: power of preemption | 21:00 |
| clarkb | ianw: and likely only reproduceable when you have 1366 git repos full of files as part of the du | 21:00 |
| armax | mordred: I just happened to look at my email, sheer coincendence | 21:00 |
| ianw | clarkb: which is yeah, why i wonder if we're in full pipes in python subprocess land, or something ... | 21:00 |
| ianw | clarkb: but why only xenial? i mean it's been working fine for ages | 21:00 |
| ianw | that was where i got to last night :) | 21:00 |
| zaro | zxiiro: still not working. did you need to do the workaround for 1.651.3? | 21:01 |
| *** dprince has quit IRC | 21:02 | |
| clarkb | ianw: maybe as want to restrict the depth that du lists too? | 21:02 |
| clarkb | s/as/we/ | 21:02 |
| zxiiro | zaro nope with that version you done need a workaround. | 21:02 |
| zxiiro | zaro actually I just remembered something | 21:03 |
| ianw | clarkb: well, we could do that. i think my concern more is that if dib can kill itself by outputing a large amount of text, that's going to bite us somewhere else sooner or later | 21:03 |
| zxiiro | zaro: don't use the comment added and add a regex trigger just to test. see if you can make it trigger with regex | 21:03 |
| clarkb | ianw: yup, though this could be specific to fd 1 (stdout) being a pipe attached to python | 21:03 |
| *** julim has quit IRC | 21:04 | |
| zxiiro | zaro if that works it means the comment added trigger needs tweaking. if I recall the default categories don't work out of the box | 21:04 |
| ianw | clarkb: yeah, i'm almost certain it is :) | 21:04 |
| *** e0ne has quit IRC | 21:04 | |
| *** beisner is now known as beisner-biab | 21:05 | |
| ianw | clarkb: i wonder if the thread running the subprocess doesn't collect it's output into the log file for a while, how much pipe buffer we have there | 21:06 |
| zaro | zxiiro: i already have comment added trigger setup to 'recheck' same as your jjb build job. nothing is triggering | 21:06 |
| zaro | zxiiro: actually i've added patchset-created, comment-added, abandoned, and restored all setup. | 21:07 |
| clarkb | ianw: it does while True: process.stdout.readline(); logline | 21:07 |
| *** kzaitsev_mb has joined #openstack-infra | 21:08 | |
| *** Apoorva_ has quit IRC | 21:08 | |
| clarkb | ianw: and it reads until it gets empty string then does a process.wait() | 21:08 |
| clarkb | ianw: is it possible it is somehow reading an empty string, stopping to read off the buffer and just waits until the subprocess crashes? | 21:08 |
| *** Apoorva has joined #openstack-infra | 21:08 | |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| *** yamamoto has quit IRC | 21:09 | |
| ianw | clarkb: right, but if that loop doesn't get scheduled for a while, the data is backing up | 21:09 |
| ianw | from the strace, cat get EAGAIN | 21:09 |
| *** aeng has joined #openstack-infra | 21:09 | |
| zaro | zxiiro: should jobs report back to gerrit if you just manually trigger the jobs? | 21:10 |
| clarkb | there shouldn't be much thread contention there unless we are overlapping uplaods and builds (which weshouldn't be doing as it is build first then upload, I guess if uplaods continue into then ext day maybe we would see that) | 21:10 |
| ianw | clarkb: yeah, and it was totally repeatable last night when it was quiet, so i admit it's a long shot :) | 21:11 |
| *** jkilpatr has quit IRC | 21:11 | |
| *** hockeynut has quit IRC | 21:12 | |
| clarkb | pabelanger: the change to nodepool to stop setting the flag for the du is merged and installed. We do have a few image uplaods running not sure if you want to wait for those to finish or just go ahead and restart the builder and do xeniakl | 21:12 |
| greghaynes | ianw: huh, does cat not handle EAGAIN? | 21:15 |
| *** kzaitsev_mb has quit IRC | 21:16 | |
| *** thorst_ has quit IRC | 21:16 | |
| ianw | greghaynes: here's the strace of cat -> http://paste.openstack.org/show/560862/ | 21:16 |
| clarkb | I wonder if this is related to the python subprocess warning to use communicate instead of stdin/stdout/stderr.read to avoid deadlock due to fuilling buffers | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| greghaynes | ianw: looks like a no, then | 21:16 |
| clarkb | though thats deadlock not eagain | 21:16 |
| greghaynes | so yea, looks like just full buffers | 21:16 |
| greghaynes | we can write du out to a tempfile and work on that? | 21:17 |
| clarkb | greghaynes: thats what the cat was for, it was cating the tempfile at the end to put the data in the dib output | 21:17 |
| greghaynes | oh, hah it just cat'd so much? | 21:17 |
| clarkb | ya that is what ianw is saying | 21:18 |
| *** matt-borland has quit IRC | 21:18 | |
| clarkb | basically nodepool can't keep up with dib when it writes this data out and ofr some reason only affects the xenial builds | 21:18 |
| ianw | greghaynes: yeah, sorry i was debugging it last night, i put some local edits into our disk-image-create on nodepool | 21:18 |
| greghaynes | clearly this isnt ready for the internet if it cant handle that many cats | 21:18 |
| jesusaur | cats have clogged the pipes? sounds like the internet to me... | 21:19 |
| ianw | greghaynes: i was thinking the same thing, bad formatting, but i put in http://paste.openstack.org/show/560863/ | 21:19 |
| pabelanger | clarkb: Ya, lets restart and get ubuntu-xenial into osic-cloud1. That gives us more resources | 21:20 |
| ianw | and the final cat was still failing | 21:20 |
| pabelanger | clarkb: then we can land cloudnull change to bump max-servers | 21:20 |
| cloudnull | :) | 21:21 |
| clarkb | pabelanger: we will also need mordreds fix for the private IP in shade/nodepool I think | 21:21 |
| *** pilgrimstack has quit IRC | 21:21 | |
| clarkb | pabelanger: so that multinode continues to work | 21:21 |
| pabelanger | clarkb: I thought the ip6tables was the fix for that? | 21:21 |
| pabelanger | or did I miss something | 21:21 |
| clarkb | pabelanger: it is but no new image means no fix in | 21:21 |
| clarkb | well its part of the fix | 21:21 |
| greghaynes | ianw: that was failing when run under nodepool? | 21:21 |
| pabelanger | clarkb: okay, we should pull that in once we build ubuntu-xenial | 21:22 |
| ianw | greghaynes: yes -- only when run under nodepool. run those commands by hand and it works | 21:22 |
| pabelanger | we can also to ubuntu-trusty if needed | 21:22 |
| ianw | "The data read is buffered in memory, so do not use this method if the data size is large or unlimited." <--- "large" ... that's helpful | 21:22 |
| clarkb | pabelanger: the new concern is that once that ip6tables fix is in then we will have ready nodes for multinode in osic and then ovs will fail to make the vxlan connections over the private network because ipv6 | 21:22 |
| pabelanger | clarkb: ah, I see what you are saying | 21:23 |
| clarkb | pabelanger: we can avoid that by removing osic from the 2-node labels | 21:23 |
| greghaynes | ianw: I wonder if we should bash loop over it | 21:23 |
| clarkb | pabelanger: so thats a simple fix | 21:23 |
| *** Goneri has joined #openstack-infra | 21:23 | |
| greghaynes | ianw: basically read echo loop | 21:23 |
| pabelanger | clarkb: sure, we can do that | 21:23 |
| ianw | greghaynes: i think we need to fix it in nodepool, because although this triggers it, it's just going to be something else flooding output that kills it next time | 21:24 |
| pabelanger | clarkb: has mordred done the fix and we just need a new release of shade? | 21:24 |
| greghaynes | ianw: yea, fair | 21:24 |
| *** baoli has quit IRC | 21:24 | |
| mordred | pabelanger: I'm still working on that fix | 21:24 |
| clarkb | pabelanger: https://review.openstack.org/#/c/357342/ looks like it was an occ fix and it merged so we just need a release? | 21:25 |
| clarkb | mordred: ^ is that all thats left? or do you need to bubble things up through shade and nodepool now? | 21:25 |
| mordred | patch for shade coming soon | 21:25 |
| mordred | yah | 21:25 |
| *** baoli has joined #openstack-infra | 21:25 | |
| *** bhunter71 has joined #openstack-infra | 21:25 | |
| mordred | nodepool shold not need any patches | 21:25 |
| *** edmondsw has quit IRC | 21:25 | |
| mordred | and I'm 3/4 of the way done with the shade patch | 21:25 |
| *** mriedem has left #openstack-infra | 21:25 | |
| cloudnull | clarkb mordred pabelanger: to unblock multi-node gates, can a tenant network be used as a secondary nic allowing the OVS magic to happen on that device? | 21:25 |
| clarkb | cloudnull: only if shade will report that as the private IP properly | 21:26 |
| *** mriedem has joined #openstack-infra | 21:26 | |
| * cloudnull has no idea how the multi-node gate is built | 21:26 | |
| mordred | I think just waiting for this patch is likely better | 21:26 |
| mordred | it's not a big one | 21:26 |
| clarkb | cloudnull: the existing instances gets a private ipv4 and public ipv6 already but list the public ipv6 as the private network which won't work because we need private v4 | 21:26 |
| mordred | I just need to finish it, which I will do as soon asI stop runnin gmy mouth here | 21:26 |
| clarkb | cloudnull: so as long as the ipv4 addrs there are routable between osic isntances we will be fine with mordred's fix | 21:27 |
| cloudnull | ah ok. makes sense | 21:27 |
| *** beisner-biab is now known as beisner | 21:28 | |
| clarkb | cloudnull: the v4 addrs we do get should be routable between instances ya? | 21:28 |
| cloudnull | yes. they should be. | 21:29 |
| clarkb | perfect | 21:29 |
| * cloudnull goes to make sure. | 21:29 | |
| clarkb | none of this would be an issue if we could just make tunnels over v6 | 21:29 |
| ianw | clarkb / greghaynes: i wonder if http://bugs.python.org/issue1652 is involved ... | 21:29 |
| cloudnull | confirmed | 21:30 |
| *** kzaitsev_mb has joined #openstack-infra | 21:30 | |
| cloudnull | if shade can detect the v4 addr you should be good to go | 21:30 |
| clarkb | cloudnull: ty ty | 21:31 |
| clarkb | ianw: heh resolution says fixed but they clearly did not fix it :/ | 21:31 |
| *** jheroux has quit IRC | 21:31 | |
| *** markusry has joined #openstack-infra | 21:32 | |
| greghaynes | ianw: hrm, that could cause the full buffers if something else is killing the fd | 21:32 |
| greghaynes | although wed still need to know what is killing the fd | 21:33 |
| *** jcoufal_ has quit IRC | 21:34 | |
| *** baoli has quit IRC | 21:35 | |
| *** annegentle has quit IRC | 21:36 | |
| *** annegentle has joined #openstack-infra | 21:37 | |
| greghaynes | ianw: so for 'fixing in nodepool', we could change the stdout=PIPE to write out to a file and then read that in later? | 21:40 |
| greghaynes | or write out to a fifo | 21:40 |
| *** psachin has quit IRC | 21:44 | |
| clarkb | pabelanger: ok want me to go ahead and restart the nodepool builder and start a xenial build? | 21:45 |
| *** pradk has quit IRC | 21:45 | |
| *** furlongm has quit IRC | 21:45 | |
| *** furlongm has joined #openstack-infra | 21:46 | |
| *** valderrv has joined #openstack-infra | 21:46 | |
| mtreinish | heh, mutt 1.7.0 was installed in my daily package upgrade. But, there aren't release notes or anything posted on the official website yet. (just a tarball on the ftp server this morning) | 21:46 |
| Krenair | fungi, pleia2: so... gate-infra-puppet-apply-ubuntu-precise failed despite succeeding earlier? | 21:46 |
| ianw | greghaynes: we want it streaming ... i think that would be the same as using communicate() and just dumping the resulting output | 21:46 |
| * mtreinish is trying to figure out everything that's changed | 21:47 | |
| *** annegentle has quit IRC | 21:48 | |
| openstackgerrit | Merged openstack-infra/puppet-snmpd: Remove initscript https://review.openstack.org/356090 | 21:51 |
| *** kzaitsev_mb has quit IRC | 21:52 | |
| *** xarses_ has quit IRC | 21:53 | |
| greghaynes | ianw: yea, the fifo would be equivalent. I'm actually pretty surprised that this is an issue... all the python process should be doing is reading from that one fd | 21:54 |
| greghaynes | and then theres the one logging thread its feeding to | 21:54 |
| *** vhosakot has quit IRC | 21:54 | |
| clarkb | and the gearman thread | 21:54 |
| greghaynes | yea theres other threads but if ianw did a one off test they all should be relatively quiet | 21:54 |
| *** jerryz has quit IRC | 21:55 | |
| *** annegentle has joined #openstack-infra | 21:55 | |
| clarkb | well I don't hear any objects so I am going to go ahead and restart the nodepool builder to get the code change then build xenial and upload it | 21:55 |
| clarkb | or oh wait maybe pabelanger already did that? | 21:56 |
| greghaynes | ianw: I'm having a hard time coming up with a good way to prove that the read/log loop is what is clogging up the works | 21:56 |
| clarkb | oh cool its running | 21:56 |
| greghaynes | ianw: one simple fix might be to change that stdout.readline() to a read(), although I think that would be minor | 21:57 |
| *** amitgandhinz has quit IRC | 21:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:58 | |
| *** mriedem has quit IRC | 21:58 | |
| *** amitgandhinz has joined #openstack-infra | 21:58 | |
| pabelanger | clarkb: ya, I kicked it just after nodepool updated | 21:58 |
| clarkb | pabelanger: awesome | 21:58 |
| clarkb | I am going to push a change to remove osic from the 2-node labels as a precaution | 21:59 |
| clarkb | then when we get shade and nodepool updated we can add osic back to those labels | 21:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Update logrotate for mirror server https://review.openstack.org/357453 | 22:01 |
| *** vinaypotluri has quit IRC | 22:01 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron experimental job for scenario tests running on LinuxBridge https://review.openstack.org/357511 | 22:01 |
| *** nwkarste_ has quit IRC | 22:01 | |
| *** tbarron is now known as tbarron|afk | 22:02 | |
| *** nwkarsten has joined #openstack-infra | 22:02 | |
| *** gordc has quit IRC | 22:02 | |
| *** annegentle has quit IRC | 22:03 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Remove OSIC from 2-node label types https://review.openstack.org/357512 | 22:05 |
| clarkb | pabelanger: ^ thats the precaution chnage | 22:05 |
| *** thorst_ has joined #openstack-infra | 22:06 | |
| *** nwkarsten has quit IRC | 22:07 | |
| pabelanger | clarkb: great | 22:07 |
| *** vinaypotluri has joined #openstack-infra | 22:07 | |
| ianw | greghaynes: yeah, i don't know at this point. i might try to write a small thing to try repro | 22:07 |
| pabelanger | also looks like the original mirror.nyj01.internap.openstack.org server has stopped getting traffic | 22:07 |
| pabelanger | going to delete it | 22:08 |
| *** kzaitsev_mb has quit IRC | 22:09 | |
| *** burgerk has quit IRC | 22:10 | |
| *** thorst_ has quit IRC | 22:10 | |
| *** thorst_ has joined #openstack-infra | 22:14 | |
| *** xyang1 has quit IRC | 22:14 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 22:15 |
| mordred | cloudnull: congratulations - you have your own unittest in shade now :) | 22:15 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Do not add c-api,c-bak on subnode with Ironic https://review.openstack.org/352909 | 22:15 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 22:15 |
| *** thiagop has quit IRC | 22:16 | |
| *** hpe-hj has quit IRC | 22:16 | |
| *** esberglu has quit IRC | 22:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Also mirror source packages for debian / ubuntu https://review.openstack.org/354818 | 22:16 |
| *** tpsilva has quit IRC | 22:18 | |
| *** mdrabe has quit IRC | 22:19 | |
| *** julim has joined #openstack-infra | 22:19 | |
| *** Goneri has quit IRC | 22:20 | |
| zaro | zxiiro: i configured it out. i neglected to configure the "Gerrit project" section. I just assumed that it would trigger on any project but you must specify a specific one. Also that config (from UI view) was offset to make it look like it's a subset of the Dynamic trigger config. | 22:23 |
| zxiiro | zaro: ah right. You can make it for any project by setting your project to be "**" with the ANT pattern | 22:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 22:24 |
| mordred | clarkb, cloudnull, Shrews: ^^ | 22:24 |
| *** asettle has joined #openstack-infra | 22:24 | |
| mordred | that will not pass tests until the new occ is cut | 22:24 |
| mordred | but there's the code | 22:25 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 22:25 |
| clarkb | mordred: can you review https://review.openstack.org/#/c/357512/ while we wait for that to happen? | 22:25 |
| clarkb | mordred: will prevent job breakage in the interim | 22:25 |
| mordred | clarkb: +A | 22:25 |
| *** nwkarsten has joined #openstack-infra | 22:26 | |
| *** erlon has quit IRC | 22:27 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Remove Neutron Postgres job from the Neutron check queue https://review.openstack.org/357519 | 22:27 |
| jhesketh | Morning | 22:28 |
| *** nwkarsten has quit IRC | 22:30 | |
| *** thorst_ has quit IRC | 22:31 | |
| *** asettle has quit IRC | 22:32 | |
| *** thorst_ has joined #openstack-infra | 22:32 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Add network entries for OSIC https://review.openstack.org/357522 | 22:32 |
| pabelanger | clarkb: going to be a while for ubuntu-xenial image, slow git clones | 22:32 |
| pabelanger | on nodepool.o.o | 22:32 |
| mordred | clarkb: ^^ actually, we should land that to | 22:32 |
| mordred | clarkb: ^^ actually, we should land that too | 22:32 |
| clarkb | pabelanger: ya I was noticing thati n the log tail | 22:33 |
| pabelanger | mordred: What happens when osic-cloud8 comes online? Do the profiles in os-client-config need to change? | 22:34 |
| mordred | well... at that point I should probably add an osic8 profile (and probably should have called this one osic1, but hindsight) | 22:34 |
| mordred | pabelanger: we can also just put the details directly into our clouds.yaml file | 22:34 |
| clarkb | ya I am thinking that might actually be ebtter given what I now know of osics make many clouds model | 22:35 |
| mordred | k. I'll abandon that patch then | 22:35 |
| clarkb | occ probably doesn't need to track all of the variations. AIUI there are rax private cloud clouds, master built clouds, $otherbranch clouds, etc | 22:35 |
| pabelanger | mordred: ya, that's what I did for nodepool clouds.yaml | 22:37 |
| pabelanger | https://review.openstack.org/#/c/356703/ | 22:38 |
| mordred | http://paste.openstack.org/show/560868/ <-- clarkb cloudnull pabelanger that's a server entry from osic with the shade and occ patches applied | 22:38 |
| mordred | you should note that public_v4 is empty, private_v4 has an address and public_v6 has an address | 22:39 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute011.vanilla IP addresses https://review.openstack.org/357087 | 22:39 |
| pabelanger | looks good | 22:39 |
| mordred | hrm. although interface_ip is ... ah - my local machine does not have ipv6 - so yes, interface_ip is correct to be blank :) | 22:39 |
| *** thorst_ has quit IRC | 22:40 | |
| fungi | Krenair: sounds like a nondeterministic/false negative. maybe a race condition in a test? i saw one earlier on another change of mine... i'll dig into the logs | 22:40 |
| mordred | by forcing the local ipv6 flag, I ran the above again and got an ipv6 in the interface_ip | 22:41 |
| Krenair | i've been seeing this with a tests around designate too. everything's fine until someone approves the change, the jenkins becomes unhappy :( | 22:41 |
| *** yamamoto has joined #openstack-infra | 22:42 | |
| fungi | Krenair: oh, on https://review.openstack.org/352055, what's the deal with the lines starting with "?" in the settings file? i was trying to find a php language reference to explain that syntax... seems similar to a trinary operator but that would be "?:" so i'm still confused | 22:43 |
| Krenair | fungi, there is no ? in that diff | 22:45 |
| *** kzaitsev_mb has joined #openstack-infra | 22:45 | |
| fungi | Krenair: weird... maybe gertty is misrepresenting the diff. i'll check the gerrit webui for comparison | 22:46 |
| Krenair | sure you're not looking at a tab character or something? | 22:46 |
| *** rlandy has quit IRC | 22:46 | |
| fungi | Krenair: oh! it is indeed replacing a hard tab with a ? for some reason... | 22:47 |
| *** tonytan4ever has quit IRC | 22:47 | |
| fungi | jeblair: ^ not sure if that's intentional behavior in gertty or not? | 22:47 |
| mtreinish | fungi: heh, I've had unicode chars display as ? in my terminal in the past | 22:48 |
| Krenair | yeah we use tabs for indentation in mediawiki | 22:48 |
| fungi | mtreinish: yep, if you use the replace resolution to ascii in python it definitely does that, but i've not seen it happen for hard tabs | 22:49 |
| mtreinish | like I remember a very confusing experience where cdent used a unicode check mark in channel and it was a question mark on my end | 22:49 |
| jeblair | fungi: it is an unintentional behavior in gertty ... for some reason it doesn't come up often for us :) | 22:49 |
| mordred | jeblair: :) | 22:50 |
| *** rbuzatu has joined #openstack-infra | 22:50 | |
| fungi | jeblair: yeah, mostly just surprised me, since i thought it was showing addition of a literal "?" character instead of it being a placeholder for another byte | 22:50 |
| jeblair | we can totally fix it | 22:51 |
| fungi | across teams, we (infra) actually have a higher volume of files with hard tabs in them than most, i expect | 22:51 |
| jeblair | me may have to figure out how many spaces equal a tab. | 22:51 |
| *** pahuang has joined #openstack-infra | 22:51 | |
| jeblair | prepare the bikeshed! | 22:51 |
| fungi | clearly the answer is "3" | 22:52 |
| fungi | (note, god uses three-space tabs) | 22:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add dual stack network config for OSIC https://review.openstack.org/357531 | 22:52 |
| mordred | pabelanger, clarkb: ^^ there's the clouds.yaml changes for us | 22:52 |
| *** rbrndt has quit IRC | 22:53 | |
| *** rbrndt has joined #openstack-infra | 22:53 | |
| *** rbrndt has quit IRC | 22:54 | |
| pabelanger | mordred: thanks +2 | 22:54 |
| *** ccamacho1 has joined #openstack-infra | 22:54 | |
| *** rbuzatu has quit IRC | 22:54 | |
| *** vikasc has quit IRC | 22:54 | |
| *** beagles is now known as beagles_biaw | 22:56 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144 https://review.openstack.org/357532 | 22:58 |
| *** vikasc has joined #openstack-infra | 22:59 | |
| *** Apoorva_ has joined #openstack-infra | 23:00 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144 https://review.openstack.org/357532 | 23:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add dual stack network config for OSIC https://review.openstack.org/357531 | 23:02 |
| fungi | Krenair: as for the gate-infra-puppet-apply-ubuntu-precise failure i think we hit a job timeout | 23:02 |
| mordred | pabelanger: ^^ just made an update - I tested that file with existing release of shade/occ | 23:02 |
| mordred | and we need to add a thing to not confused old code | 23:02 |
| fungi | Krenair: i've rechecked the failed change, but am digging deeper on why our apply jobs for precise have started doing this very recently | 23:03 |
| *** Apoorva has quit IRC | 23:03 | |
| fungi | chances are this is tied into the ipv6 privacy addresses issue in osic | 23:04 |
| fungi | the job ran there, and definitely had an additional global v6 address according to the network diagnostics at the start of the console log, and was unable to traceroute6 to git.o.o | 23:05 |
| fungi | we _suspect_ that having privacy addresses enabled is resulting in performance issues with git remote operations from osic nodes | 23:06 |
| *** dimtruck is now known as zz_dimtruck | 23:06 | |
| fungi | and our puppet apply jobs do a lot of remote git operations because of needing to clone all our puppet modules | 23:07 |
| *** sshnaidm is now known as sshnaidm|afk | 23:07 | |
| *** hongbin has quit IRC | 23:09 | |
| fungi | yeah, console log shows it updating repos quickly up until 21:14:01 when it started to work on puppe | 23:11 |
| fungi | t-bugdaystats | 23:11 |
| fungi | and took it until 21:17:11 just to check the branches on the origin | 23:11 |
| fungi | and until 21:18:14 to decide what branch to use | 23:12 |
| *** sdake_ has joined #openstack-infra | 23:12 | |
| fungi | and then puppet-mysql_backup similarly took $4 minutes | 23:12 |
| fungi | er, ~4 minutes | 23:12 |
| fungi | yeah, each repo past there took 4 minutes until the job was prematurely terminated for reaching our job timeout | 23:13 |
| *** sdake has quit IRC | 23:13 | |
| clarkb | fungi: fwiw the xenial image build is being very slow to update repos as well | 23:13 |
| *** sdague has quit IRC | 23:14 | |
| *** ccamacho1 has quit IRC | 23:14 | |
| fungi | this sounds consistent with the "4 minute" git remote times sdague was citing earlier | 23:15 |
| clarkb | dib ins't being that slow | 23:16 |
| clarkb | so I guess thats a good thing | 23:16 |
| fungi | clarkb: what's our egress bw cap supposed to be on git.o.o eth0? | 23:16 |
| *** jordanP has quit IRC | 23:16 | |
| fungi | the daily graph shows 5-minute samples reaching 500.89mbps | 23:17 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all | 23:17 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute013.vanilla IP addresses https://review.openstack.org/357091 | 23:17 |
| clarkb | fungi: the rxtx factor is 2500 so I think 1.25Gbe | 23:17 |
| *** armax has quit IRC | 23:17 | |
| clarkb | er 1250mbps | 23:17 |
| fungi | okay, so still well below 50% | 23:17 |
| *** chlong has quit IRC | 23:20 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute014.vanilla IP addresses https://review.openstack.org/357092 | 23:20 |
| fungi | individual git servers don't seem to be spiking above 1-minute averages of 70mbps | 23:21 |
| fungi | just checked all 8 of them | 23:21 |
| fungi | so distribution is remarkably even | 23:21 |
| *** fguillot has joined #openstack-infra | 23:26 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Remove diskimage-builder repo elements folder from ELEMENTS_PATH https://review.openstack.org/357183 | 23:27 |
| fungi | wow, we've been hovering arounf 1.5kjps today according to the graph | 23:29 |
| fungi | er, around | 23:29 |
| fungi | er, and 1.5kjph | 23:30 |
| mgagne | kilojoules per second | 23:30 |
| fungi | kilojobs per hour is what i meant ;) | 23:30 |
| mgagne | that's a lot of energy :D | 23:30 |
| *** thorst_ has joined #openstack-infra | 23:31 | |
| fungi | probably uses a lot of energy | 23:31 |
| mgagne | that's impressive still | 23:31 |
| *** tosky has quit IRC | 23:31 | |
| *** rcernin has quit IRC | 23:32 | |
| *** adriant has joined #openstack-infra | 23:34 | |
| sridhar_ram | infra-team: I'm looking for help in skipping a gate job in a specific branch (stable/mitaka). This is related to https://review.openstack.org/#/c/353136/ where relnotes jobs is now running but we didn't start adding relnotes (and support that in tox.ini) until newton (in the current master). How can skip relnotes job for tacker in stable/mitaka ? | 23:34 |
| sridhar_ram | I notices projects.yaml has a flag is branch-override .. is that it ? | 23:35 |
| sridhar_ram | s/notices/noticed/ | 23:35 |
| mordred | fungi: wow. 1.5kjph is a lot of jobs | 23:35 |
| fungi | sridhar_ram: that's not a flag... it's a parameter some jobs use to indicate that when running on any branch they should test the code from the overridden branch | 23:35 |
| fungi | sridhar_ram: it would need to be supported in the job-template you're instantiating though | 23:36 |
| *** markusry has quit IRC | 23:36 | |
| fungi | pabelanger: note the comment from cloudnull on 357453 patchset 1 about setting maxage and copytruncate | 23:37 |
| fungi | pabelanger: but i'm going to go ahead and approve now because we're down to just a sliver still available on the osic mirror's rootfs | 23:37 |
| tonyb | sridhar_ram: It's prety easy to specify the branch to run a job on .... let me find a good example | 23:38 |
| sridhar_ram | fungi: there is a job-template only for dsvm job .. there is nothing else | 23:38 |
| sridhar_ram | tonyb: thanks | 23:38 |
| sridhar_ram | fungi: tonyb: what baffles me is, this job showed up only for this above patchset .. another patchset that recently merged to stable/mitaka didn't pick up this job .. https://review.openstack.org/#/c/353159/ | 23:39 |
| fungi | sridhar_ram: yeah, tonyb is presumably going to get you an example of specifying in zuul/layout.yaml which branches jobs should be triggered on | 23:39 |
| *** stew925 has quit IRC | 23:39 | |
| *** skipp has quit IRC | 23:39 | |
| tonyb | fungi: Yeah. | 23:39 |
| sridhar_ram | tonyb: fungi: thanks folks! | 23:39 |
| fungi | which is not at all the same thing as overriding what branches should be used in jobs when they run | 23:39 |
| tonyb | sridhar_ram: it's possible that another project-config change altered the behaviour | 23:40 |
| *** eeiden has quit IRC | 23:40 | |
| *** eeiden has joined #openstack-infra | 23:40 | |
| *** wcriswell has quit IRC | 23:41 | |
| tonyb | sridhar_ram: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1713 | 23:41 |
| * sridhar_ram is looking up | 23:41 | |
| tonyb | sridhar_ram: that basically says only run this job if the branch is NOT stable/liberty or stable/mitaka | 23:41 |
| tonyb | sridhar_ram: I'll look at what changed to cause it. It's possible that more thn tacker is affected | 23:44 |
| sridhar_ram | tonyb: thanks a bunch.. | 23:44 |
| sridhar_ram | tonyb: meanwhile, i'll push this change to zuul/layout to guard against such in the future | 23:45 |
| tonyb | sridhar_ram: yeah good plan | 23:45 |
| *** Swami has quit IRC | 23:45 | |
| *** thorst_ has quit IRC | 23:46 | |
| *** thorst_ has joined #openstack-infra | 23:47 | |
| *** tonytan4ever has joined #openstack-infra | 23:47 | |
| *** stewie925_ has joined #openstack-infra | 23:50 | |
| *** kzaitsev_mb has quit IRC | 23:50 | |
| stewie925_ | I have internet connection with my ubuntu - sudo install commands keep getting failed to get new connections - even though I could ping google.com | 23:51 |
| *** rbuzatu has joined #openstack-infra | 23:51 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:52 | |
| *** tonytan4ever has quit IRC | 23:53 | |
| *** zhurong has joined #openstack-infra | 23:53 | |
| clarkb | stewie925_: is it possible you are attempting to talk to a mirror that isn't functioning? or that you have a firewall between you and the mirror? | 23:53 |
| fungi | pabelanger: i've removed apache logs >8 days old on the osic mirror for now to get us breathing room until logrotate kicks again. freed up ~5gb so should be plenty for now | 23:55 |
| *** thorst_ has quit IRC | 23:55 | |
| *** mriedem has joined #openstack-infra | 23:55 | |
| *** rbuzatu has quit IRC | 23:56 | |
| clarkb | xenial image build is finally past the git caching | 23:58 |
| *** gildub_ has joined #openstack-infra | 23:59 | |
| *** gildub has joined #openstack-infra | 23:59 | |
| stewie925_ | hi clarkb: I just tried sudo apt-get install and it was blazing fast - success | 23:59 |
| openstackgerrit | Merged openstack-infra/project-config: Remove OSIC from 2-node label types https://review.openstack.org/357512 | 23:59 |
| stewie925_ | maybe something with this pudb | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!