Thursday, 2016-08-18

clarkbcompletely unrelated but for those of you attending the midcycle sprint are you planning to shuttle/train/drive?00:00
jeblairis there a train?00:00
pabelangerI'd do train00:00
pabelangerdidn't know it was an option00:00
clarkbjeblair: ya you can flughafen to mannheim and mannheim to wiesloch-walldorf then walk a mile or so depending on your hotel00:01
clarkbor catch a more local train and walk less depending on your hotel00:01
jeblairhow about that00:01
clarkbcaptain train says it will take about an hour and 7 minutes (excluding walking)00:01
*** jerryz has quit IRC00:02
pabelangerclarkb: cost?00:02
clarkbpabelanger: ~ 25 euro00:02
clarkbthe shuttle looks like 60 euro one way00:02
clarkbwhich is what prompted me to look at other options00:03
pabelangerya, going to do train then00:03
*** julim has quit IRC00:03
jeblairoh capitain train changed their name00:03
clarkbmy flight lands at 1:15pm Sunday the 18th fwiw00:04
clarkbif anyone else knows their flight details already we could be the annoying people on the train00:04
*** zhurong has quit IRC00:04
pabelangerI'm in 06:15am localtime00:05
pabelangerover night flight00:05
pabelangerclarkb: do you have the URL handy for the train?00:06
jeblairpabelanger: https://www.captaintrain.com00:07
*** aeng has joined #openstack-infra00:07
clarkbyup ^00:07
*** Julien-zte has joined #openstack-infra00:08
clarkband you want frankfurt flughafen fernbf not the regional train station at the airport00:08
clarkbthough you can probably also just show up and book there as well00:09
clarkb(I have done this a non zero number of times)00:09
*** gildub_ has joined #openstack-infra00:09
*** elo has quit IRC00:10
pabelangerya, looks like it runs regularly00:10
*** kzaitsev_mb has quit IRC00:11
pabelangerclarkb: ya, I'll pick up a ticket once I land. seems to depart ever 1hr00:13
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed
*** Jeffrey4l has joined #openstack-infra00:16
*** bswartz has quit IRC00:17
*** raunak has quit IRC00:18
cloudnullpabelanger: did the updated image w/ the networking bits ever move forward? is it possible to revert or should we also wait on the urandom bits before ?00:18
*** mriedem has joined #openstack-infra00:18
pabelangercloudnull: just kicked off a new DIB build now00:19
pabelangerwe had some git repo issues today00:19
*** thorst_ has joined #openstack-infra00:19
cloudnullselfishly, i'd to see it back in action, because in OpenStack-Ansible we're waiting on trusty gates.00:19
cloudnull^ completely selfish -- just +/- curious -- no worries regardless00:20
pabelangershould take another hour to build, once finished we can upload again and revert.00:20
pabelangerdon't see an issue00:20
pabelangerif we land 356706 tonight too, I'll kick off an image build00:20
pabelangeroh, we should land 356637 too00:21
pabelangerthat will fix multi-node jobs00:21
*** gyee_ has quit IRC00:21
*** Julien-zte has quit IRC00:22
cloudnullwe can wait, tbt the gate is running decently and wait times have not been bad.00:23
cloudnullso no pressure00:23
cloudnulllet least not from me00:23
pabelangerya, this week has been way better then last!00:24
cloudnullindeed it has00:24
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8
ianwyay!  fedora24 not crashing (not working either, but i'll take it)
*** pilgrimstack has joined #openstack-infra00:31
jeblairianw: what kernel does f24 have?00:31
jeblairpabelanger: is it correct to both 'before' and 'wants'
*** bswartz has joined #openstack-infra00:32
*** tphummel has joined #openstack-infra00:32
ianwjeblair: 4.6.6+ianw00:33
jeblairi don't speak systemd.  i mean, being before something while also wanting it seems sort of poetic, which is nice, but that doesn't seem like lennart's style, so i dunno.00:33
*** rbuzatu has joined #openstack-infra00:34
*** sdake has joined #openstack-infra00:35
*** pilgrimstack has quit IRC00:35
jeblaircloudnull, ianw: then we may want to add f24 to
jeblairbut we can do it later00:35
*** armax has quit IRC00:36
ianwok, i've only been vaguely follow ... for unbound and ipv6 right?00:37
jeblairianw: just unbound being slow to start on boot -- the parent change (already merged) contains the essay that explains all:
* jeblair dines00:39
*** rbuzatu has quit IRC00:39
openstackgerritTom Fifield proposed openstack-infra/puppet-mediawiki: Add 'autopatrol' group
*** roxanaghe has quit IRC00:42
*** dimtruck is now known as zz_dimtruck00:45
*** jamielennox is now known as jamielennox|away00:48
*** gus has joined #openstack-infra00:48
jrolljeblair: that's quite the essay, I enjoyed it :)00:48
*** amitgandhinz has joined #openstack-infra00:49
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Add initialize-urandom to fedora 24
cloudnullianw jeblair ^00:49
apuimedopabelanger: is it possible to make devstack process correctly requirements.txt that point to a git master branch instead of a fixed release?00:50
*** itisha has quit IRC00:50
apuimedoI'm asking because we develop openstack/kuryr-libnetwork against the master of openstack/kuryr00:51
apuimedowhile openstack/kuryr is not stable00:51
apuimedo(we have been splitting the repos)00:51
ianwapuimedo: LIBS_FROM_GIT?00:52
apuimedoianw: yup00:52
apuimedowe're developing a both the lib and the server at the same time00:53
*** raunak has joined #openstack-infra00:53
apuimedoand they are both in prerelease state00:53
*** amitgandhinz has quit IRC00:53
ianwapuimedo: the server is installed from git, though?  only the clients we're pip installing?00:53
apuimedoone option is to keep nagging for releases00:53
*** tqtran has quit IRC00:54
apuimedoianw: devstack gets the server plugin from git, the requirements file of which points to a pip git url00:54
ianwapuimedo: umm, not quite following, can you link to the plugin?00:57
*** shashank_hegde has quit IRC00:58
*** rfolco has joined #openstack-infra00:58
*** rfolco has quit IRC00:58
*** spzala has joined #openstack-infra00:59
*** sdake has quit IRC00:59
apuimedorequirements line (which will point soon to openstack/kuryr instead of that fork)
apuimedoianw: ^^00:59
*** sdake has joined #openstack-infra01:02
*** spzala has quit IRC01:03
*** xarses has joined #openstack-infra01:03
*** jamielennox|away is now known as jamielennox01:04
*** nwkarsten has joined #openstack-infra01:05
ianwapuimedo: if you just drop the @ bit of the -e line, that would mean just install from master?01:06
*** zhurong has joined #openstack-infra01:10
*** sdake has quit IRC01:11
ianwapuimedo: so, that doesn't work?01:12
*** hichihara has quit IRC01:12
apuimedoI'll try now without the branch selection01:12
apuimedoianw: "error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'-e git:/'""01:14
*** spzala has joined #openstack-infra01:15
*** pt_15 has quit IRC01:16
ianwapuimedo: what you to change the line to?01:17
*** malabaris has joined #openstack-infra01:17
*** malabaris has left #openstack-infra01:19
apuimedosorry, I had a typo01:19
apuimedothe real error is01:19
pabelangerjeblair: Yes. Services that want to be run before the network is configured should place and also set to pull it in. According to:
*** thorst_ has quit IRC01:19
apuimedoand the content of requirements.txt is01:20
*** thorst_ has joined #openstack-infra01:20
apuimedonote that pip install -r requirements.txt works01:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed
*** aeng has quit IRC01:23
apuimedoianw: funnily enough, if I change the egg name from kuryr-lib to kuryr_lib, then the error is just01:25
apuimedoCould not find a version that satisfies the requirement kuryr_lib (from kuryr-libnetwork==0.1.0.dev517) (from versions: )01:25
apuimedoNo matching distribution found for kuryr_lib (from kuryr-libnetwork==0.1.0.dev517)01:25
*** pvaneck has quit IRC01:28
*** gouthamr_ has joined #openstack-infra01:28
*** thorst_ has quit IRC01:29
*** Sukhdev has quit IRC01:30
*** zz_dimtruck is now known as dimtruck01:31
*** gouthamr has quit IRC01:32
*** pilgrimstack has joined #openstack-infra01:33
ianwapuimedo: the "-" is not helping -> The "project name" component of the url suffix "egg=<project name>-<version>" is used by pip in its dependency logic to identify the project prior to pip downloading and analyzing the metadata. The optional "version" component of the egg name is not functionally important.01:36
apuimedoI'm fixing that now01:36
ianwbut, odd ...01:36
*** aeng has joined #openstack-infra01:36
apuimedonow I'm using kuryr_lib01:36
apuimedobut I don't know why pbr still tries to find a version01:36
apuimedoif no version is specified, it should use the latest01:37
apuimedoshouldn't it?01:37
*** pilgrimstack has quit IRC01:37
*** annegentle has joined #openstack-infra01:38
*** spzala has quit IRC01:38
*** spzala has joined #openstack-infra01:39
*** edmondsw has quit IRC01:40
*** hockeynut has joined #openstack-infra01:42
*** spzala has quit IRC01:43
*** annegentle has quit IRC01:45
*** raunak has quit IRC01:48
ianwapuimedo: hmm, yeah run it with verbose and it goes off looking for it on pypi01:48
apuimedothis fixes the egg name01:48
openstackgerritMerged openstack-infra/shade: Use "image" as argument for Glance V1 upload error path
*** amitgandhinz has joined #openstack-infra01:49
*** tqtran has joined #openstack-infra01:51
*** amitgandhinz has quit IRC01:54
*** tqtran has quit IRC01:56
*** rbuzatu has joined #openstack-infra01:56
*** nwkarsten has quit IRC01:57
*** Apoorva has quit IRC01:57
*** nwkarsten has joined #openstack-infra01:57
*** yanyanhu has joined #openstack-infra01:58
*** jimbaker has quit IRC02:00
*** tbarron is now known as tbarron|afk02:00
*** vinaypotluri has quit IRC02:01
*** nwkarsten has quit IRC02:02
*** rbuzatu has quit IRC02:03
*** jimbaker has joined #openstack-infra02:04
*** jimbaker has quit IRC02:04
*** jimbaker has joined #openstack-infra02:04
*** jerryz has joined #openstack-infra02:15
*** mriedem has quit IRC02:18
*** rlandy has quit IRC02:21
*** thorst_ has joined #openstack-infra02:22
pabelangerianw: cloudnull: ubuntu-trusty is now uploading to osic-cloud1, going to get the revert patch ready02:22
cloudnullooohh :)02:22
*** thorst_ has quit IRC02:23
*** thorst_ has joined #openstack-infra02:24
pabelangerianw: cloudnull: also started ubuntu-xenail rebuilt, it won't fix dns but it will include ipv6 sysctl fix02:24
pabelangerianw: You'll likely need to make sure that is upload to osic-cloud1 once finished building02:24
cloudnullthat should at least help jobs that make into the nodepool02:25
* cloudnull is tired. needs to not be typing right now02:26
*** baoli has joined #openstack-infra02:30
*** fguillot has quit IRC02:32
*** caowei has joined #openstack-infra02:32
*** thorst_ has quit IRC02:32
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Disable trusty nodes on OSIC"
pabelangercloudnull: ianw: jhesketh: okay, we can revert now ^02:33
*** pilgrimstack has joined #openstack-infra02:33
jheskethpabelanger: looking (sorry I haven't been following hte channel)02:34
pabelangerjhesketh: let me include the actual commit that fixes osic02:34
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Disable trusty nodes on OSIC"
*** pilgrimstack has quit IRC02:37
stevemaranyone know swapnil / coolsvaps irc name?02:39
stevemari would hvae thought it was one of those02:39
pabelangerjhesketh: bascially, cloudnull noticed we needed for ipv6 nodes in osic-cloud1.  It was affecting ubuntu-trusty pretty bad (slow git clones I believe) so we disabled them in osic-cloud1 this morning02:39
pabelangerFinially finished a new DIB with the patch, and it was just uploaded to osic-cloud102:40
*** psachin has quit IRC02:40
pabelangerworking on ubuntu-xenial DIB now, but still another hour or so away from being uploaded02:40
*** shashank_hegde has joined #openstack-infra02:40
*** eandersson_ has joined #openstack-infra02:40
jheskethpabelanger: okay, lgtm... I'll +W and keep an eye02:41
jhesketh(on the trusty fix)02:42
pabelangergreat, thanks02:42
*** jamielennox is now known as jamielennox|away02:42
*** gongysh has joined #openstack-infra02:42
cloudnullnice thanks pabelanger :)02:42
* cloudnull fingers cross that we're done with that forever02:43
*** changzhi has joined #openstack-infra02:44
*** coolsvap has joined #openstack-infra02:45
*** yuanying_ has quit IRC02:46
*** jimbaker has quit IRC02:47
*** eandersson_ has quit IRC02:47
cloudnullif there are some RH folks that wouldnt mind reviewing -- it'd be great to see the same fix added in for the infra image simply be a default element in DIB.02:50
*** amitgandhinz has joined #openstack-infra02:50
*** jimbaker has joined #openstack-infra02:51
*** jimbaker has quit IRC02:51
*** jimbaker has joined #openstack-infra02:51
openstackgerritTom Fifield proposed openstack-infra/groups-static-pages: Remove groups.json
*** yuanying has joined #openstack-infra02:52
*** amitgandhinz has quit IRC02:55
openstackgerritMerged openstack-infra/project-config: Revert "Disable trusty nodes on OSIC"
*** spzala has joined #openstack-infra03:00
*** rwsu has joined #openstack-infra03:02
*** nwkarsten has joined #openstack-infra03:04
*** julim has joined #openstack-infra03:05
*** spzala has quit IRC03:05
*** hockeynut has quit IRC03:06
*** baoli has quit IRC03:07
*** jamielennox|away is now known as jamielennox03:07
*** yamahata has quit IRC03:07
*** asettle has joined #openstack-infra03:08
*** Goneri has quit IRC03:10
*** csomerville has joined #openstack-infra03:13
*** cody-somerville has quit IRC03:13
*** gongysh has quit IRC03:14
*** aeng has quit IRC03:14
*** dingyichen has quit IRC03:16
*** asettle has quit IRC03:16
*** raunak has joined #openstack-infra03:16
*** pahuang has quit IRC03:16
*** yuanying has quit IRC03:18
*** nwkarsten has quit IRC03:18
*** yuanying has joined #openstack-infra03:19
*** julim has quit IRC03:19
*** nwkarsten has joined #openstack-infra03:19
*** cody-somerville has joined #openstack-infra03:20
*** rbuzatu has joined #openstack-infra03:20
*** coolsvap is now known as _coolsvap_03:21
*** csomerville has quit IRC03:22
*** nwkarsten has quit IRC03:23
*** rbuzatu has quit IRC03:24
*** r-mibu has quit IRC03:24
*** r-mibu has joined #openstack-infra03:25
*** raunak has quit IRC03:25
*** Sukhdev has joined #openstack-infra03:28
*** raunak has joined #openstack-infra03:30
*** thorst_ has joined #openstack-infra03:30
*** vikrant has joined #openstack-infra03:32
*** nwkarsten has joined #openstack-infra03:33
*** gouthamr_ has quit IRC03:34
*** pilgrimstack has joined #openstack-infra03:35
*** tonytan4ever has joined #openstack-infra03:36
*** thorst_ has quit IRC03:37
*** vikrant has quit IRC03:37
*** pilgrimstack has quit IRC03:39
*** vikrant has joined #openstack-infra03:41
*** raunak has quit IRC03:41
*** yuanying has quit IRC03:42
*** yuanying has joined #openstack-infra03:42
*** furlongm has quit IRC03:43
*** furlongm has joined #openstack-infra03:43
*** yuanying has quit IRC03:43
*** valderrv has quit IRC03:44
*** links has joined #openstack-infra03:45
*** raunak has joined #openstack-infra03:45
*** yuanying has joined #openstack-infra03:48
*** amitgandhinz has joined #openstack-infra03:51
*** tqtran has joined #openstack-infra03:52
timrcyolanda, rcarrillocruz: Hey so I'm coming to Barcelona.  How many gallons of sangria do you think we'll drink? jk03:52
*** amitgandhinz has quit IRC03:56
*** tqtran has quit IRC03:56
*** vikrant is now known as vikrant|brb03:58
*** spzala has joined #openstack-infra04:01
*** tonytan4ever has quit IRC04:05
*** spzala has quit IRC04:07
openstackgerritvikas choudhary proposed openstack-infra/project-config: Fix dsvm installation job
*** vikrant|brb is now known as vikrant|04:18
*** vikrant| is now known as vikrant04:18
pabelangerianw: if you have a moment, maybe you can figure out why ubuntu-xenial dibs are failing:
*** amotoki has joined #openstack-infra04:33
*** thorst_ has joined #openstack-infra04:34
*** amotoki has quit IRC04:38
*** thorst_ has quit IRC04:42
*** vgridnev has joined #openstack-infra04:42
ianwpabelanger: ok04:42
openstackgerritYanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test
*** Sukhdev has quit IRC04:46
ianwpabelanger: hmmm, yeah that numfmt error ..04:51
*** amitgandhinz has joined #openstack-infra04:52
openstackgerritMerged openstack-infra/devstack-gate: Set PS4 to provide additional debug info
*** amitgandhinz has quit IRC04:56
*** narayrak has joined #openstack-infra04:59
*** spzala has joined #openstack-infra05:00
*** amotoki has joined #openstack-infra05:00
*** narayrak has quit IRC05:02
*** jaosorior has joined #openstack-infra05:03
*** amotoki has quit IRC05:05
*** spzala has quit IRC05:06
*** tonytan4ever has joined #openstack-infra05:06
*** vgridnev has quit IRC05:06
jianghuawmordred: Thanks for letting my know that zuul-v3 is still working in progress. Could you share the release plan? I would just want to make my own plan basing on it. As it seems I have to upgrade my CI env to zuul-v3, that it can support multi-nodes with different images.05:07
*** elo has joined #openstack-infra05:07
*** vgridnev has joined #openstack-infra05:11
*** tonytan4ever has quit IRC05:11
*** raunak has quit IRC05:12
*** raunak has joined #openstack-infra05:16
fifieldthi infra05:20
fifieldtanyone have easy log access to the stuff backing wiki.o.o ? I was just POSTing to and it gave a blank page -- probably a HTTP 500 or so05:20
*** rbuzatu has joined #openstack-infra05:21
*** raunak has quit IRC05:22
*** eandersson_ has joined #openstack-infra05:24
*** jimbaker has quit IRC05:25
*** rbuzatu has quit IRC05:26
openstackgerritvikas choudhary proposed openstack-infra/project-config: Fix dsvm installation job
*** jogo has quit IRC05:28
*** amotoki has joined #openstack-infra05:28
*** jimbaker has joined #openstack-infra05:28
*** jimbaker has quit IRC05:28
*** jimbaker has joined #openstack-infra05:28
*** vgridnev has quit IRC05:28
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add TripleO RSS Feed from Juan Antonio Osorio's blog
*** vgridnev has joined #openstack-infra05:31
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use temp file for du calculations
ianwpabelanger: this might be part of it ...05:31
*** amotoki has quit IRC05:32
*** asettle has joined #openstack-infra05:35
*** rajinir has quit IRC05:35
*** pilgrimstack has joined #openstack-infra05:35
*** rbuzatu has joined #openstack-infra05:36
*** r-mibu has quit IRC05:37
*** tphummel has quit IRC05:37
*** asettle has quit IRC05:39
*** thorst_ has joined #openstack-infra05:40
*** pilgrimstack has quit IRC05:40
*** jogo has joined #openstack-infra05:42
*** vgridnev has quit IRC05:44
*** vgridnev has joined #openstack-infra05:47
*** r-mibu has joined #openstack-infra05:47
*** thorst_ has quit IRC05:47
*** abregman has joined #openstack-infra05:50
*** amitgandhinz has joined #openstack-infra05:53
*** roxanaghe has joined #openstack-infra05:55
*** amitgandhinz has quit IRC05:57
*** Cormite has joined #openstack-infra05:57
*** pilgrimstack has joined #openstack-infra05:58
*** pilgrimstack has quit IRC05:58
*** roxanaghe has quit IRC05:59
*** Hal1 has joined #openstack-infra06:00
*** Hal has quit IRC06:00
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Pass TRIPLEO_ROOT directory to
*** spzala has joined #openstack-infra06:03
*** vgridnev has quit IRC06:03
*** vgridnev has joined #openstack-infra06:03
*** masterLog1x has joined #openstack-infra06:05
*** jimbaker has quit IRC06:06
*** spzala has quit IRC06:07
yolandahi timrc , it will be awesome to meet again06:07
*** eandersson_ has quit IRC06:07
*** jimbaker has joined #openstack-infra06:10
*** jimbaker has quit IRC06:10
*** jimbaker has joined #openstack-infra06:10
*** vgridnev has quit IRC06:10
*** _nadya_ has joined #openstack-infra06:12
*** _nadya_ has quit IRC06:12
*** rcernin has joined #openstack-infra06:12
*** yamamoto has quit IRC06:21
*** amotoki has joined #openstack-infra06:22
*** amotoki has quit IRC06:26
*** amotoki has joined #openstack-infra06:29
*** masterLog1x has quit IRC06:31
*** _nadya_ has joined #openstack-infra06:31
*** _nadya_ has quit IRC06:38
*** javeriak has joined #openstack-infra06:40
yolandagood morning06:41
*** thorst_ has joined #openstack-infra06:45
*** amotoki has quit IRC06:51
*** adriant has quit IRC06:52
*** thorst_ has quit IRC06:52
*** pcaruana has joined #openstack-infra06:52
*** amotoki has joined #openstack-infra06:56
*** armax has joined #openstack-infra06:57
*** yamamoto has joined #openstack-infra07:00
*** amotoki has quit IRC07:04
*** ganesan has joined #openstack-infra07:06
*** elo has quit IRC07:08
*** xarses has quit IRC07:08
*** xarses has joined #openstack-infra07:09
*** elo has joined #openstack-infra07:10
*** vgridnev has joined #openstack-infra07:12
*** pilgrimstack has joined #openstack-infra07:16
*** tesseract- has joined #openstack-infra07:17
*** armax has quit IRC07:18
*** e0ne has joined #openstack-infra07:18
*** nmagnezi has joined #openstack-infra07:21
*** matrohon has joined #openstack-infra07:26
*** Qiming_ has joined #openstack-infra07:27
*** Qiming_ has quit IRC07:28
*** florianf has joined #openstack-infra07:36
*** esikachev has joined #openstack-infra07:37
*** _nadya_ has joined #openstack-infra07:38
*** yamamoto has quit IRC07:38
*** jpich has joined #openstack-infra07:38
*** rbuzatu has quit IRC07:39
*** rvba has joined #openstack-infra07:40
*** rvba has quit IRC07:40
*** rvba has joined #openstack-infra07:40
openstackgerritMerged openstack-infra/project-config: Corrected success-pattern for npm-based doc builds
*** roxanaghe has joined #openstack-infra07:43
*** matrohon has quit IRC07:43
*** matthewbodkin has joined #openstack-infra07:44
*** dizquierdo has joined #openstack-infra07:44
openstackgerritMerged openstack-infra/project-config: Move networking-ovn grenade CI job from experimental to check
*** shashank_hegde has quit IRC07:46
*** roxanaghe has quit IRC07:47
*** thorst_ has joined #openstack-infra07:50
*** sshnaidm|afk has quit IRC07:50
rcarrillocruztimrc: ohh, that's awesome. Sangria drinking levels is proportional to the fruit included in the jar, the sweeter, the more 'omg, i love this, another glass please!'07:52
*** abregman has quit IRC07:53
*** amitgandhinz has joined #openstack-infra07:54
*** tqtran has joined #openstack-infra07:54
*** Gene_ has joined #openstack-infra07:55
*** amotoki has joined #openstack-infra07:55
*** Gene_ has quit IRC07:55
*** thorst_ has quit IRC07:56
openstackgerritmariam john proposed openstack-infra/project-config: Update trove job to install trove tempest plugin
*** amotoki_ has joined #openstack-infra07:57
*** Genek has joined #openstack-infra07:58
*** pilgrimstack has quit IRC07:58
*** tqtran has quit IRC07:59
*** amitgandhinz has quit IRC07:59
*** dizquierdo has quit IRC07:59
*** zzzeek has quit IRC08:00
*** amotoki has quit IRC08:00
*** zzzeek has joined #openstack-infra08:01
*** sshnaidm has joined #openstack-infra08:02
*** spzala has joined #openstack-infra08:04
*** jimbaker has quit IRC08:05
*** ganesan has quit IRC08:08
*** ihrachys has joined #openstack-infra08:08
*** spzala has quit IRC08:08
*** jimbaker has joined #openstack-infra08:09
*** jimbaker has quit IRC08:09
*** jimbaker has joined #openstack-infra08:09
*** mhickey has joined #openstack-infra08:09
*** lennyb has quit IRC08:13
*** berendt has joined #openstack-infra08:17
*** pilgrimstack has joined #openstack-infra08:18
*** jgrassler has joined #openstack-infra08:20
*** pilgrimstack has quit IRC08:20
*** sslypushenko has joined #openstack-infra08:22
*** dosaboy_ has joined #openstack-infra08:23
*** dosaboy_ has quit IRC08:24
*** dosaboy has quit IRC08:24
sslypushenkoHi, folks! Can I ask for review/merge of this patch I has pretty much a lot of +1's08:24
*** eandersson_ has joined #openstack-infra08:26
*** pilgrimstack has joined #openstack-infra08:26
*** vikasc has joined #openstack-infra08:27
*** lennyb has joined #openstack-infra08:27
*** Goneri has joined #openstack-infra08:29
vikaschello folks, Can you please review ?08:29
*** dosaboy has joined #openstack-infra08:29
*** armax has joined #openstack-infra08:31
*** javeriak has quit IRC08:33
*** jerryz has quit IRC08:34
*** ihrachys has quit IRC08:36
*** vgridnev has quit IRC08:37
*** jaosorior has quit IRC08:37
ianwpabelanger: success!  f24 ->
*** pilgrimstack has quit IRC08:38
ianwpabelanger: still working on the ubutnu-xenial build.  it is *not* totally due to ->  i'm fiddling directly on nodepool to try and figure it out.  maybe egrep is segfaulting?08:38
*** yaume has joined #openstack-infra08:39
*** asettle has joined #openstack-infra08:39
*** yamamoto has joined #openstack-infra08:39
*** vgridnev has joined #openstack-infra08:40
apuimedothanks vikasc08:40
*** rbuzatu has joined #openstack-infra08:40
vikascapuimedo, no worries08:40
*** rbuzatu has quit IRC08:44
*** derekh has joined #openstack-infra08:47
*** nwkarsten has quit IRC08:48
*** esikachev has quit IRC08:49
*** dizquierdo has joined #openstack-infra08:52
openstackgerritMerged openstack-infra/puppet-openstackci: Custom Jenkins vhost
*** yamamoto has quit IRC08:52
*** pilgrimstack has joined #openstack-infra08:53
*** abregman has joined #openstack-infra08:54
*** pilgrimstack has quit IRC08:55
*** thorst_ has joined #openstack-infra08:55
*** amitgandhinz has joined #openstack-infra08:55
*** pilgrimstack has joined #openstack-infra08:55
openstackgerrityolanda.robla proposed openstack-infra/system-config: Temporarily add rabbit keys to hiera
azvyagintsevyolanda could you please merge ?08:56
*** yamamoto has joined #openstack-infra08:56
openstackgerritliyuanzhen proposed openstack-infra/jenkins-job-builder: Not limit stevedore to 1.8.0
*** yamamoto has quit IRC08:56
*** yamamoto has joined #openstack-infra08:56
*** Cormite has quit IRC08:57
*** roxanaghe has joined #openstack-infra08:58
*** amitgandhinz has quit IRC08:59
*** dkehn_ has quit IRC09:00
*** dkehn has quit IRC09:00
*** pilgrimstack has quit IRC09:00
sslypushenkoyolanda:  +1 to request for  ))09:01
*** esikachev has joined #openstack-infra09:02
*** thorst_ has quit IRC09:02
*** spzala has joined #openstack-infra09:04
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** _coolsvap_ is now known as coolsvap09:06
*** e0ne has quit IRC09:07
*** dkehn has joined #openstack-infra09:07
*** d0ugal has quit IRC09:08
*** spzala has quit IRC09:09
*** yamamoto has quit IRC09:10
*** ihrachys has joined #openstack-infra09:10
*** d0ugal has joined #openstack-infra09:12
*** yamamoto has joined #openstack-infra09:12
*** yamamoto has quit IRC09:12
*** d0ugal_ has joined #openstack-infra09:17
*** d0ugal_ has quit IRC09:18
*** d0ugal_ has joined #openstack-infra09:18
*** dkehn_ has joined #openstack-infra09:18
*** Genek has quit IRC09:19
*** dtardivel has joined #openstack-infra09:19
*** d0ugal has quit IRC09:20
*** d0ugal_ has quit IRC09:20
*** d0ugal has joined #openstack-infra09:20
*** roxanaghe has quit IRC09:21
*** pilgrimstack has joined #openstack-infra09:24
*** ihrachys has quit IRC09:24
*** sambetts|afk is now known as sambetts09:25
*** electrofelix has joined #openstack-infra09:28
*** yamamoto has joined #openstack-infra09:30
*** yamamoto has quit IRC09:30
*** yamamoto has joined #openstack-infra09:30
*** yamamoto has quit IRC09:30
*** yamamoto has joined #openstack-infra09:31
openstackgerritMerged openstack-dev/pbr: Replace OpenStack LLC with OpenStack Foundation
wznoinskmmedvede: around?09:32
*** sshnaidm has quit IRC09:32
*** acoles_ is now known as acoles09:34
*** yamamoto has quit IRC09:36
*** sshnaidm has joined #openstack-infra09:36
*** _degorenko|afk is now known as degorenko09:38
*** berendt has quit IRC09:40
*** berendt has joined #openstack-infra09:44
fricklerhave there been known issues earlier with uploading test results? see the post_failures e.g. here
*** pilgrimstack has quit IRC09:46
*** jordanP has joined #openstack-infra09:47
vgridnevteam, can someone help me with approving this one? thanks in advance09:48
*** tosky has joined #openstack-infra09:49
vgridnevit's really frustrating that patch is09:50
vgridnevlong reviewed09:50
vgridnevthanks a lot!09:51
*** skraynev__ has quit IRC09:51
*** furlongm has quit IRC09:54
*** furlongm has joined #openstack-infra09:54
*** asettle has quit IRC09:55
*** asettle has joined #openstack-infra09:55
*** amitgandhinz has joined #openstack-infra09:56
*** ihrachys has joined #openstack-infra09:56
*** eranrom has quit IRC09:56
*** eranrom has joined #openstack-infra09:56
*** kzaitsev_mb has joined #openstack-infra09:57
*** zhurong has quit IRC09:58
*** d0ugal has quit IRC09:58
*** jimbaker has quit IRC09:58
*** yanyanhu has quit IRC09:59
*** amitgandhinz has quit IRC10:00
*** yamamoto has joined #openstack-infra10:01
*** yamamoto has quit IRC10:01
*** jimbaker has joined #openstack-infra10:02
*** jimbaker has quit IRC10:02
*** jimbaker has joined #openstack-infra10:02
*** dizquierdo has quit IRC10:03
*** d0ugal has joined #openstack-infra10:04
*** spzala has joined #openstack-infra10:05
*** sdague has joined #openstack-infra10:06
*** ihrachys has quit IRC10:07
*** javeriak has joined #openstack-infra10:09
*** spzala has quit IRC10:10
*** kzaitsev_mb has quit IRC10:12
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set public and private IP addresses for compute002.vanilla
*** amotoki_ has quit IRC10:14
*** berendt has quit IRC10:14
*** berendt has joined #openstack-infra10:15
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute004.vanilla ip addresses
*** amotoki has joined #openstack-infra10:18
*** skraynev has joined #openstack-infra10:18
*** dizquierdo has joined #openstack-infra10:20
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute006.vanilla addresses
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute007 IP addresses
*** dizquierdo has quit IRC10:28
*** vgridnev has quit IRC10:30
openstackgerritRichard Theis proposed openstack-infra/project-config: Add neutron-lib source periodic job to networking-ovn
*** berendt has quit IRC10:31
*** yamamoto has joined #openstack-infra10:32
*** ociuhandu has quit IRC10:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute009.vanilla IP addresses
*** jimbaker has quit IRC10:35
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute010.vanilla IP addresses
*** esikachev has quit IRC10:36
*** yaume has quit IRC10:36
*** yamamoto has quit IRC10:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute011.vanilla IP addresses
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute012.vanilla IP addresses
*** jimbaker has joined #openstack-infra10:39
*** jimbaker has quit IRC10:39
*** jimbaker has joined #openstack-infra10:39
*** yamamoto has joined #openstack-infra10:41
*** rbuzatu has joined #openstack-infra10:42
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute013.vanilla IP addresses
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute014.vanilla IP addresses
*** berendt has joined #openstack-infra10:44
openstackgerritRichard Theis proposed openstack-infra/project-config: Add neutron-lib source periodic job to networking-ovn
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute015.vanilla IP addresses
*** roxanaghe has joined #openstack-infra10:45
*** amotoki has quit IRC10:45
*** esikachev has joined #openstack-infra10:45
*** amotoki_ has joined #openstack-infra10:46
*** rbuzatu has quit IRC10:46
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute016.vanilla IP addresses
flip214when using, how do I get to an URL with the filter that I built?10:46
flip214"share" on top right says "query=", "save" doesn't give me an URL either10:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute017.vanilla IP addresses
*** Genek has joined #openstack-infra10:49
*** roxanaghe has quit IRC10:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute018.vanilla IP addresses
*** vgridnev has joined #openstack-infra10:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute019.vanilla IP addresses
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute020.vanilla IP addresses
*** coolsvap is now known as _coolsvap_10:55
*** amitgandhinz has joined #openstack-infra10:56
*** kzaitsev_mb has joined #openstack-infra10:57
*** eranrom has quit IRC10:57
*** eranrom has joined #openstack-infra10:57
*** eranrom_ has joined #openstack-infra10:58
*** eranrom has quit IRC10:58
*** oanson has joined #openstack-infra11:00
*** amitgandhinz has quit IRC11:01
*** jkilpatr has quit IRC11:01
*** thorst_ has joined #openstack-infra11:04
*** esikachev has quit IRC11:06
ianwpabelanger: ok, not much closer on this ubuntu-xenail fail.  have a look at my local edits to /usr/local/bin/disk-image-builder (search ianw)11:06
*** spzala has joined #openstack-infra11:06
flip214trying to fetch files via wget could make use of some optimizations...11:06
flip214"Last-modified header missing -- time-stamps turned off.11:07
flip214and so an interrupted transfer needs to fetch all files anew11:07
zigopabelanger: For when you wake up: repo signing worked, look here: \o/11:07
zigoHowever, how do we get the public key?11:07
*** caowei has quit IRC11:07
zigoCould you drop the public-key.gpg somewhere in the repo?11:07
ianwpabelanger: outputting the image size report fails.  the final "cat" seems to bail out when it dumps.  i've put it under strace and i'll need to output of that to keep debugging.  other builds hvae started, and it's late here, so i'll ooop back on this in the morning11:08
*** oanson has quit IRC11:09
*** spzala has quit IRC11:11
*** BobBall is now known as BobBall_AWOL11:11
openstackgerritMerged openstack-infra/groups: Custom membership fields and reporting
*** lucasagomes is now known as lucas-hungry11:13
*** dizquierdo has joined #openstack-infra11:17
*** rbuzatu has joined #openstack-infra11:22
*** spzala has joined #openstack-infra11:23
*** spzala has quit IRC11:23
*** dizquierdo has quit IRC11:24
*** asettle has quit IRC11:25
*** asettle has joined #openstack-infra11:26
*** Genek has quit IRC11:27
*** vgridnev has quit IRC11:29
azvyagintsevyolanda: ping  :) folks answer 'yes'11:29
*** berendt has quit IRC11:29
*** vgridnev has joined #openstack-infra11:29
*** Genek has joined #openstack-infra11:30
*** berendt has joined #openstack-infra11:30
*** ramishra has quit IRC11:31
*** ldnunes has joined #openstack-infra11:31
*** vgridnev has quit IRC11:32
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.
*** vgridnev has joined #openstack-infra11:33
*** ramishra has joined #openstack-infra11:33
*** nwkarsten has joined #openstack-infra11:35
*** changzhi has quit IRC11:36
*** changzhi has joined #openstack-infra11:37
*** jkilpatr has joined #openstack-infra11:38
zigoianw: Do you know if it is possible to have just *one* more ready Jessie node, waiting for incoming jobs?11:39
*** ociuhandu has joined #openstack-infra11:39
zigoThat would help us *a lot*.11:39
openstackgerritMerged openstack-infra/system-config: Set public and private IP addresses for compute002.vanilla
*** pilgrimstack has joined #openstack-infra11:43
*** changzhi has quit IRC11:43
openstackgerritRichard Theis proposed openstack-infra/project-config: Add non-voting coverage job to networking-ovn
*** kzaitsev_mb has quit IRC11:44
*** Goneri has quit IRC11:44
openstackgerritRichard Theis proposed openstack-infra/project-config: Add non-voting coverage job to networking-ovn
*** nwkarsten has quit IRC11:45
*** roxanaghe has joined #openstack-infra11:46
*** armax has quit IRC11:46
*** ayoung has quit IRC11:50
*** roxanaghe has quit IRC11:50
*** dprince has joined #openstack-infra11:52
openstackgerritEli Qiao proposed openstack-infra/project-config: Zun: Add python-zunclient to dsvm
ianwzigo: you we can up the nodes, see project-config:openstack-infra/nodepool/nodepool.yaml config11:55
*** armax has joined #openstack-infra11:55
*** tqtran has joined #openstack-infra11:56
*** Genek has quit IRC11:57
*** amitgandhinz has joined #openstack-infra11:57
*** Goneri has joined #openstack-infra12:00
*** kzaitsev_mb has joined #openstack-infra12:00
*** tqtran has quit IRC12:00
*** amitgandhinz has quit IRC12:02
*** annegentle has joined #openstack-infra12:02
*** rbuzatu has quit IRC12:09
*** e0ne has joined #openstack-infra12:09
*** woodster_ has joined #openstack-infra12:10
tobiashhi, is it possible in nodepool to use a single image with multiple flavours in different labels?12:14
tobiashas far as I can see in the docs the flavor is attached to the image itself12:14
*** baoli has joined #openstack-infra12:15
*** edmondsw has joined #openstack-infra12:15
*** ansmith has joined #openstack-infra12:15
*** baoli_ has joined #openstack-infra12:16
*** nwkarsten has joined #openstack-infra12:16
*** Julien-zte has joined #openstack-infra12:19
*** lucas-hungry is now known as lucasagomes12:19
*** baoli has quit IRC12:20
*** ihrachys has joined #openstack-infra12:20
*** berendt has quit IRC12:22
*** javeriak has quit IRC12:22
*** tpsilva has joined #openstack-infra12:22
*** nwkarsten has quit IRC12:23
*** spzala has joined #openstack-infra12:24
*** nwkarsten has joined #openstack-infra12:24
*** matthewbodkin has quit IRC12:24
mordredtobiash: that's right, we do not currently have much in the way of multi-flavor support12:25
tobiashmordred: would a patch in this direction be accepted?12:25
tobiashthen I'd try to implement this as I have a use case for this in my environment12:26
*** gouthamr has joined #openstack-infra12:26
*** gordc has joined #openstack-infra12:27
tobiashI'm currently thinking of specifying min-ram also in the labels section which could override the image min-ram12:28
sdague - internap mirror / network issues?12:28
*** nwkarsten has quit IRC12:28
*** boogibugs has quit IRC12:28
*** spzala has quit IRC12:29
*** fguillot has joined #openstack-infra12:29
sdaguerelated -
*** fguillot has quit IRC12:30
sdaguelooks like the only 2 fails in the gate right now are internap network12:30
*** mdrabe has joined #openstack-infra12:30
*** pradk has joined #openstack-infra12:31
*** ihrachys has quit IRC12:31
mordredtobiash: we're in a little bit of an awkward moment to accept a patch like that, as we're currently working on a large feature branch in service of zuul v3 ...12:32
openstackgerritMerged openstack-infra/tripleo-ci: Add TripleO RSS Feed from Juan Antonio Osorio's blog
*** ihrachys has joined #openstack-infra12:32
mordredtobiash: I think we should get jeblair's take on such a patch before you write it ... it might also be that we're going to have that use case covered in zuul v3, or it might be a thing that we need to add to the list for v3 and that we'd welcome the help implementing12:32
*** asettle has quit IRC12:34
*** asettle has joined #openstack-infra12:34
sdagueyeh, - also internap network12:35
*** thiagop has joined #openstack-infra12:36
mordredsdague: wow, it's neat when network stuff works poorly inside of a cloudregion12:37
tobiashmordred: ok, then I'll go for multiple image definitions for now as a workaround12:37
*** annegentle has quit IRC12:37
tobiashthanks for the info12:37
mordredtobiash: yah - sorry about the awkward timing part - but I think it's a worthwhile request12:38
*** rlandy has joined #openstack-infra12:39
*** jcoufal has joined #openstack-infra12:40
tobiashmordred: btw, is there a rough time schedule for zuul v3 somewhere? Didn't find much info about that topic apart from the blueprint and the patches in gerrit.12:40
*** berendt has joined #openstack-infra12:41
mordredtobiash: "as soon as we can get it ready" ... we've got some people working on it full time now, so we're hoping that it's not more than a couple of months out - but it's a big beastie, so hard to predict the first release12:42
sdaguemordred: so far from perfect because the data we have, but I think basically networking issues there starting about 8 hours ago12:43
sdaguewe get to a 40% fail rate in the gate on dsvm jobs (which do reasonable amount of network ops) starting about then12:43
*** burgerk has joined #openstack-infra12:43
openstackgerritMarton Kiss proposed openstack-infra/groups: Group contanct report csv dump filter by group status
sdaguepreviously, it was negligible12:44
mordredmgagne: ^^12:44
*** zhurong has joined #openstack-infra12:45
*** rodrigods has quit IRC12:48
*** rodrigods has joined #openstack-infra12:48
*** pilgrimstack has quit IRC12:49
*** gouthamr is now known as gouthamr_afk12:51
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: ovb: create dummy eth1
*** e0ne has quit IRC12:52
*** kgiusti has joined #openstack-infra12:52
*** julim has joined #openstack-infra12:53
*** nwkarsten has joined #openstack-infra12:53
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Remove diskimage-builder repo elements folder from ELEMENTS_PATH
openstackgerritThomas Goirand proposed openstack-infra/project-config: Up the debian-jessie min-ready
*** e0ne has joined #openstack-infra12:55
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Add vlan_id variable with value 2551
*** jtomasek has quit IRC12:57
*** amitgandhinz has joined #openstack-infra12:58
*** esikachev has joined #openstack-infra12:58
*** ihrachys has quit IRC12:59
*** raildo has joined #openstack-infra13:00
*** amotoki_ has quit IRC13:00
*** weshay_ has joined #openstack-infra13:00
*** pilgrimstack has joined #openstack-infra13:00
*** ihrachys has joined #openstack-infra13:02
asselinrcarrillocruz, is this right?
asselinshould the loop_var be item_cloud?13:02
*** amitgandhinz has quit IRC13:02
*** nwkarsten has quit IRC13:04
asselinrcarrillocruz, here's my input command:
*** rbuzatu has joined #openstack-infra13:04
*** esikache1 has joined #openstack-infra13:04
*** nwkarsten has joined #openstack-infra13:04
*** pilgrimstack has quit IRC13:06
*** matthewbodkin has joined #openstack-infra13:06
*** amotoki has joined #openstack-infra13:06
*** esikachev has quit IRC13:06
*** oanson has joined #openstack-infra13:08
*** psilvad has joined #openstack-infra13:08
kzaitsev_mbfolks, DSS Cinder CI is commenting on Murano patches (and says that our patches are in merge failed) О_о13:11
*** esikache1 has quit IRC13:11
kzaitsev_mbсan anyone point me how to debug this situation =)13:11
rcarrillocruznot sure , i haven't used the launch_node playbook in a long time, it's still in review13:11
rcarrillocruzthe include/with_items nested13:11
rcarrillocruzand loop_var stuff13:11
rcarrillocruzdoesn't work on ansible <
rcarrillocruzyou may want to check that13:12
rcarrillocruzother than that, put a debug on the var before the line it fails to see what's up13:12
*** inc0 has joined #openstack-infra13:12
rcarrillocruz- debug: var=vars13:12
asselinrcarrillocruz, ok i'm on ansible==
asselin, but yeah I upgraded cloud-launcher and still using the previous playbooks13:12
rcarrillocruz^ shows everything13:12
kzaitsev_mb this commit for example13:12
*** xyang1 has joined #openstack-infra13:12
*** ihrachys has quit IRC13:13
openstackgerritMerged openstack-infra/system-config: Set compute004.vanilla ip addresses
*** mhickey has quit IRC13:14
*** ihrachys has joined #openstack-infra13:16
*** links has quit IRC13:16
asselinkzaitsev_mb, in addition to reporting here: try e-mailing them: DSS Cinder CIbudik_06@list.ru13:17
kzaitsev_mbasselin: thanks, will do )13:17
*** ihrachys has quit IRC13:18
rcarrillocruzi just got a vote from DSS Cinder CI on the puppet-infracloud13:18
rcarrillocruzor well, a comment rather13:18
*** Julien-zte has quit IRC13:18
*** berendt has quit IRC13:18
cloudnullmornings all13:19
sdagueman in the middle error in the gate - ???13:19
*** amitgandhinz has joined #openstack-infra13:20
mordredsdague: is it possible that something in that job changed the ssh host key?13:20
sdaguemordred: don't know, it's super early13:21
*** mdrabe has quit IRC13:21
sdaguethat ansible log is basically all the logs that exist13:21
cloudnullhow has the re-enablement of trusty on OSIC been going? Anything still needing to be worked on / looked at there?13:22
odyssey4meOdd - I'm starting to see 'DSS Cinder CI' comment on some of our patches... it's clearly external CI, but I am wondering why they're subscribed to our repo. Is there a page somewhere where I can find the owner of the CI to contact?13:22
mordredcloudnull: jeblair wrote the entropy pump yesterday13:22
mordredcloudnull: I believe pabelanger has the systemd unit file for it up for review13:22
cloudnullyup, which was great work jeblair!13:23
mordredoh - actually - it looks like the disable trusty patch revert has been merged13:24
azvyagintsevFolks, can we merge ?13:24
*** mriedem has joined #openstack-infra13:25
mordredbut this isn't merged:
mordredso I'm honestly not sure what the status is13:25
odyssey4meheh, this is a pretty cool tool which I had no idea existed:
*** Julien-zte has joined #openstack-infra13:26
*** matt-borland has joined #openstack-infra13:26
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins
openstackgerritMerged openstack-infra/system-config: Set compute006.vanilla addresses
*** amotoki has quit IRC13:27
*** mhickey has joined #openstack-infra13:28
cloudnullmordred: last night it was found that the init-urandom bits were needed in fedora too. -- -- believe ianw found the issue, I just made the PR.13:28
*** vikrant has quit IRC13:30
sdaguegerrit reference problem?13:31
*** dmitryme has joined #openstack-infra13:31
sdague - I can't seem to -d that change13:31
*** mdrabe has joined #openstack-infra13:33
cloudnullwelp, no news is good news :)13:33
* cloudnull knocks on wood13:33
cloudnulli'll be around today should any news pop up13:33
openstackgerritMerged openstack/diskimage-builder: Change DIB_IPA_CERT resulting file name
mordredsdague: those change numbers are different13:35
sdaguegah, never mind13:35
*** esikache1 has joined #openstack-infra13:35
sdagueI need more coffee13:35
mordredsdague: :)13:35
mordredsdague: me too13:35
*** vgridnev has quit IRC13:36
rcarrillocruzso am I13:36
*** amitgandhinz has quit IRC13:36
EmilienMcould I have some feedback on please?13:36
*** amitgandhinz has joined #openstack-infra13:36
*** amotoki has joined #openstack-infra13:37
*** esberglu has joined #openstack-infra13:39
*** pilgrimstack has joined #openstack-infra13:39
*** BigWillie has joined #openstack-infra13:39
*** caowei has joined #openstack-infra13:40
*** hongbin has joined #openstack-infra13:41
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add scheduling thread to nodepool builder
*** dimtruck is now known as zz_dimtruck13:41
*** zz_dimtruck is now known as dimtruck13:41
*** spzala has joined #openstack-infra13:42
*** _coolsvap_ is now known as coolsvap13:42
openstackgerritMerged openstack-infra/project-config: Use ip6tables if nodepool is using IPv6
openstackgerritMerged openstack-infra/project-config: Resolve DNS soon in
sslypushenkorcarrillocruz:  Can I ask you to review and merge it if it is work for you13:43
pleia2rcarrillocruz: I can go through these patches to set the infracloud ips, but is there a document I should be matching them against?13:45
rcarrillocruzpleia2: well, i think it's fine that i autoapprove them13:45
mmedvedewznoinsk: I am around now13:45
rcarrillocruzi've done so a couple13:45
rcarrillocruzbut if you want13:45
rcarrillocruzi'm basiclly adding 6 do the computexxx number13:46
rcarrillocruzcompute000 => and
pleia2ok, easy enough13:46
rcarrillocruzreason is because first 3 IPs are reserved for HPE net folks13:46
rcarrillocruz.4 is baremetal13:46
rcarrillocruz.5 is controller13:46
*** ihrachys has joined #openstack-infra13:47
zigoHi mordred! Could you please approve this? Waiting 30 minutes between each commit is really a no-brainer for us, it's really terrible, I'm sure this will help a lot.13:47
openstackgerritMiguel Angel Ajo proposed openstack-infra/project-config: Enable QoS on neutron for python-openstacksdk.
*** ihrachys has quit IRC13:50
dmsimardJust saw a mirror error, is it a one time thing ?
*** dimtruck is now known as zz_dimtruck13:51
*** ihrachys has joined #openstack-infra13:53
*** kaisers1 has joined #openstack-infra13:53
*** asselin_ has joined #openstack-infra13:54
timrcyolanda: Definitely :)13:54
pleia2rcarrillocruz: just realized this one doesn't update to be provisioning_ipv4_address, should it?
timrcrcarrillocruz: ++ re: The Sangria fruit-to-liquid ratio.13:54
rcarrillocruzit should , thanks!13:54
rcarrillocruztimrc: you taking additional days to sightseeing?13:55
*** asselin has quit IRC13:56
timrcrcarrillocruz: I need to figure that all out.  I probably should :)13:57
rcarrillocruzyou really should :-)13:57
*** edtubill has joined #openstack-infra13:57
timrcIt'd be before the Summit though.  I don't want to miss Halloween with my kids.13:57
*** ayoung has joined #openstack-infra13:57
*** itisha has joined #openstack-infra13:57
*** xarses has quit IRC13:57
rcarrillocruzah yeah, you have to prepare another arduino/rpi project this year ;-)13:57
*** xarses has joined #openstack-infra13:58
timrcYeah! I fall pretty short when it comes to actual engineering though :P13:58
*** timello has quit IRC13:59
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute013.vanilla IP addresses
yolandatimrc, i have it very easy this time. At a driving distance :)14:00
yolandabut i will stay for 3 nights there as well14:01
*** burgerk has quit IRC14:02
sslypushenkoHi, folks! Can anybody told whom should be asked for review and merge
pleia2rcarrillocruz: Inline comments:
pleia2rcarrillocruz: Needs provisioning_ipv4_address fix:
pleia2rcarrillocruz: Both:
pleia2then we should be ok :)14:04
pleia2approved the rest14:04
*** DrifterZA has joined #openstack-infra14:04
*** sjmc7 has joined #openstack-infra14:04
*** elo has quit IRC14:04
*** caowei has quit IRC14:04
openstackgerritNathan Bohman proposed openstack-infra/shade: Don't cast image propertiess to str
openstackgerrityolanda.robla proposed openstack-infra/system-config: Temporarily add rabbit keys to hiera
*** caowei has joined #openstack-infra14:06
*** weshay_ is now known as weshay_mtg14:06
odyssey4meOK, this is going to be a problem. Every single patch we submit, or recheck we issue, is getting an immediate fail from 'DSS Cinder CI <>'. It's effectively spam. See for a sample. Is there a way we can disable this external CI or somewhere where we can try to figure out who the Ci administrator is?14:09
*** rbrndt has joined #openstack-infra14:09
asselin_rcarrillocruz, any ideas?
*** zz_dimtruck is now known as dimtruck14:10
*** xarses has quit IRC14:10
*** edtubill has quit IRC14:10
*** e0ne has quit IRC14:10
rcarrillocruzi suspect your launch playbook does not have volumes14:11
asselin_odyssey4me, infra-root can disable the account. You can try e-mail them at their e-mail address provided to gerrit. Looks like a new account. DSS Cinder CI budik_06@list.ru14:11
*** ihrachys has quit IRC14:11
rcarrillocruzotherwise you'd have an empty 'volumes' dict on what you pasted14:11
asselin_rcarrillocruz, that's true, no volumes14:11
pleia2asselin_: sure, I'll have a look14:12
pleia2er, odyssey4me :)14:12
odyssey4methanks pleia214:13
*** _nadya_ has quit IRC14:13
asselin_kzaitsev_mb reported as well ^^14:13
* pleia2 nods14:13
odyssey4meI'll mail the ML asking them to fix the scope of their tests, and forward the mail to them too. But if there's anything you can do from an infra standpoint to scope their account that'd be great.14:14
pleia2odyssey4me: that would be helpful, paper trail and all14:14
*** ihrachys has joined #openstack-infra14:14
pleia2I'll reply to your email when I disable the account (after finding it)14:14
*** e0ne has joined #openstack-infra14:14
asselin_rcarrillocruz, so how do I 'not' specify any extra volume?14:14
rcarrillocruzyou shouldn't have, make sure your launch_node.yml does contain line 1414:15
pleia2odyssey4me: ty14:15
rcarrillocruzthat line will add an empty volumes if nothing is passed as variable14:15
*** ihrachys has quit IRC14:16
*** edtubill has joined #openstack-infra14:17
*** sree has joined #openstack-infra14:17
asselin_rcarrillocruz, actually i'm out of sync a bit with your patch....using before you added volumes:
*** gildub has quit IRC14:18
*** gildub_ has quit IRC14:18
rcarrillocruzyeah, what i suspected14:18
*** Julien-zte has quit IRC14:18
*** timello has joined #openstack-infra14:18
*** ihrachys has joined #openstack-infra14:18
asselin_rcarrillocruz, any idea when this will be merged upstream?14:19
rcarrillocruznot sure, i need reviewers14:19
rcarrillocruzpabelanger said he wanted to test it out14:20
*** asettle has quit IRC14:20
rcarrillocruzthe role itself he used it to bring up the bluebox resources with success14:20
*** edtubill has quit IRC14:20
rcarrillocruzbut by feeding a resources.yml, not the playbook launch node14:20
rcarrillocruzi'll paste you14:21
*** asettle has joined #openstack-infra14:21
*** asettle has quit IRC14:21
*** rfolco has joined #openstack-infra14:21
rcarrillocruzthe idea is to have some day a mechanism that runs the launcher post-merge whenever those layouts change14:21
*** asettle has joined #openstack-infra14:21
*** vgridnev has joined #openstack-infra14:21
asselin_oh I like that14:22
rcarrillocruzbut i need to fix a couple things first to what i have in mind for doing post-resource provisioning14:22
rcarrillocruzthe os_server module does not have an idempotent return status14:22
*** valderrv has joined #openstack-infra14:22
openstackgerritMerged openstack-infra/system-config: Set compute019.vanilla IP addresses
rcarrillocruzit will return 'changed' regardless if it created anything or not14:22
rcarrillocruzi need to fix that14:22
rcarrillocruzand i also i want to return what the task did14:22
rcarrillocruzcos depending on that meta-result, we can run tasks based on it14:23
rcarrillocruzwhere i'm going with ^ is having something like:14:23
*** baoli_ has quit IRC14:23
*** baoli has joined #openstack-infra14:24
*** edtubill has joined #openstack-infra14:24
rcarrillocruzsomething along those lines14:24
rcarrillocruzdefine pre/post create/delete tasks defined in the resources model14:25
asselin_rcarrillocruz, i like the direction14:25
rcarrillocruzand for thost pre/post create/delete/update we need to get what the os_server is doing any time14:25
rcarrillocruzit will take some time, i have now being completely pulled by the infracloud14:26
rcarrillocruzasselin_: you saw the 'scaling' thing on the launcher14:26
asselin_rcarrillocruz, yes, I think it merged right? I didn't get to try  it out yet. I need to launch a single node before doing clusters14:27
rcarrillocruzyup, and it's tested on the gate too14:27
*** edtubill has quit IRC14:27
asselin_rcarrillocruz, I see you took this out. any particular reason? "- { role: set_hostname }"14:28
*** raunak has joined #openstack-infra14:28
sdaguemordred / pleia2 so, internap, 40% gate failure rate14:28
sdaguewho can investigate?14:28
rcarrillocruzhmm, i'd say no, that should be still there14:31
rcarrillocruzfeel free to push a patchset14:31
*** annegentle has joined #openstack-infra14:31
asselin_rcarrillocruz, ok, let me get it working and i'll update14:31
openstackgerritNathan Bohman proposed openstack-infra/shade: Don't cast image propertiess to str
*** edtubill has joined #openstack-infra14:34
openstackgerritZara proposed openstack-infra/storyboard-webclient: Prettify Task-Status-Changes in Recent Events
*** mhickey has quit IRC14:34
*** jed56 has quit IRC14:35
azvyagintsevFoolks , can we merge it ? sdague mordred odyssey4me yolanda14:36
*** kszukielojc has joined #openstack-infra14:39
openstackgerritMerged openstack-infra/tripleo-ci: Fail fast if a testenv couldn't be created
kszukielojcHi all can we get a +2 for ?14:39
bswartzhaving a weird gerrit problem:
*** Julien-zte has joined #openstack-infra14:40
odyssey4meazvyagintsev I think you mean to ask the infra project-config admins. I'm not one of those. :)14:40
azvyagintsevbroadcast :)14:40
*** raunak has quit IRC14:41
*** edtubill has quit IRC14:41
*** kzaitsev_mb has quit IRC14:41
*** pilgrimstack has quit IRC14:41
*** d0ugal has quit IRC14:41
*** pilgrimstack has joined #openstack-infra14:41
dansmithsdague: mordred: pleia2: at 40% fail, does it even make sense to have internap in the pool? I would think it's doing more harm than good at that point14:42
pabelangerzigo: great. I'll have to talk to fungi about the public key, but don't think it will be an issue14:42
*** xarses has joined #openstack-infra14:42
zigopabelanger: I solved it by adding the public key block within the code, but I'd prefer downloading it so you'd be free to update it.14:42
zigopabelanger: In the mean while, can you +2 this one?
*** rbuzatu has quit IRC14:43
zigopabelanger: Another issue. It looks like reprepro only accept packages that are comming with a .changes. Am I right?14:43
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Fix bridge creation in no vlan use case
fungizigo: pabelanger: if you're talking about publishing the public signing key somewhere, that's in the works but pending since we want to make sure it's served via https14:43
*** edtubill has joined #openstack-infra14:44
*** Julien-zte has quit IRC14:44
zigoFYI, I'm fighting to add a backport of dpkg so that dpkg-source ignores the .gitreview, which otherwise blocks us for building... :/14:46
*** burgerk has joined #openstack-infra14:46
dansmithsdague: pleia2: mordred: I see all kinds of network failures in those logs, ntp syncs, apt, pip, etc..14:46
pabelangerzigo: yes, only .changes right now.  Obviously that could be updated14:46
*** edtubill has quit IRC14:47
zigopabelanger: How can this be fixed, and then how will it gather the .deb files?14:47
*** mhickey has joined #openstack-infra14:47
*** rbuzatu has joined #openstack-infra14:47
*** ihrachys has quit IRC14:47
zigopabelanger: The problem is when I'm downloading from official jessie-backports and trying to push to the jessie-newton-backports repo...14:47
*** edtubill has joined #openstack-infra14:47
*** eeiden has quit IRC14:47
pabelangerzigo: you are not building them?14:47
zigopabelanger: Do you think I should attempt to rebuild?14:48
*** vinaypotluri has joined #openstack-infra14:48
zigoI'm currently not.14:48
*** wcriswell has quit IRC14:48
*** stewie925 has quit IRC14:48
*** d0ugal has joined #openstack-infra14:48
*** d0ugal has quit IRC14:48
*** d0ugal has joined #openstack-infra14:48
pabelangerYa, today the pipeline expects packages to be built by openstack14:48
*** _ari_ has joined #openstack-infra14:49
zigoIt should be trivial to rebuild, I should just change stuff in packages-list (ie: just add -d jessie-backports instead of --download-from-jessie-backports)14:49
pabelangerwe could add another jobs to just import existing packages, but there maybe some objections to that14:49
zigoI'll attempt and see if it builds correctly.14:49
*** skipp has quit IRC14:49
*** rajinir has joined #openstack-infra14:50
bswartznevermind my earlier question I discovered it's my fault14:51
*** dmellado|off is now known as dmellado14:52
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins
openstackgerritMerged openstack-infra/project-config: Up the debian-jessie min-ready
*** tonytan4ever has joined #openstack-infra14:57
*** sree has quit IRC14:58
*** edtubill has quit IRC14:58
*** nmagnezi has quit IRC14:58
*** gouthamr_afk is now known as gouthamr14:59
*** _nadya_ has joined #openstack-infra14:59
clarkbzigo: did you see that the deb-websockify repo is "broken" and needs to be corrected before we can import it into gerrit?14:59
*** kgiusti is now known as kgiusti|meeting14:59
*** edtubill has joined #openstack-infra14:59
openstackgerritMerged openstack-infra/system-config: Set compute017.vanilla IP addresses
zigoclarkb: I didn't, but shouldn't the CI catch it?15:00
zigoclarkb: In what way is it broken?15:00
*** tbarron|afk is now known as tbarron15:00
*** salv-orlando has joined #openstack-infra15:00
clarkbzigo: no, the problem is in importing it into gerrit so no CI can run on it. The problem is it uses zero padded file modes which gerrit does not allow. You can see this as a warning if you git fsck --full15:00
*** yamamoto has quit IRC15:01
clarkbzigo: describes how to fix it. If you do that and get the repo updated to a state you are happy with we can finish the import process15:01
*** sree has joined #openstack-infra15:01
*** sree has quit IRC15:02
mordredzigo: also, the example script has this line: git fast-export --all | (cd ../myFixedRepo; git fast-import)115:02
mordredthe "1" isn't part of the script - it's a footnote indicator - which I think is a bit lame15:02
*** sree has joined #openstack-infra15:02
*** hockeynut has joined #openstack-infra15:04
*** bhunter71 has joined #openstack-infra15:04
*** yamamoto has joined #openstack-infra15:04
*** devkulkarni has joined #openstack-infra15:04
electrofelixwaynr zxiiro: I think tomorrow is the last scheduled virtual sprint, but I'm not sure if we're ready with the remaining patches to review?15:05
zxiiroelectrofelix: oh shoot i totally forgot. It's been a busy week for me.15:05
zigoclarkb: It's fixed now.15:06
zxiiroelectrofelix: i'll poke at the remaining patches but ti sounds like we may need at least another session?15:06
zigomordred: Fixed.15:06
zigoHow can the import be attempted again?15:06
*** sree has quit IRC15:06
zigoWill gerrit try again by itself?15:06
electrofelixzxiiro: I think they may not have been lined up, so I was thinking we just use the same timeslot to line them up, and work on the TODO's outstanding assuming that works for waynr and we can then have a later one to focus on landing and remaining nits?15:08
*** timello has quit IRC15:08
*** d0ugal has quit IRC15:09
*** yamamoto has quit IRC15:09
*** edtubill has quit IRC15:09
zxiiroelectrofelix: yeah lets do that15:10
*** markusry has joined #openstack-infra15:10
zxiiroelectrofelix: I'm heading to LinuxCon tomorrow but I'll be around during our meeting time15:10
zxiiroelectrofelix: seems like only 13 patches remaining15:11
zxiiroelectrofelix: I'll spend the afternoon going over them and helping rebase if needed15:11
electrofelixzxiiro: cheers, I think two of the early ones have alternative architectural approaches that could be better, which is why it might be difficult to land them tomorrow, but at least we can have any immediate conflicts solved15:13
*** elo has joined #openstack-infra15:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OSA Xenial jobs to voting
*** edtubill has joined #openstack-infra15:15
*** eeiden has joined #openstack-infra15:15
*** coolsvap is now known as _coolsvap_15:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OSA Xenial jobs to voting
openstackgerritMerged openstack-infra/jenkins-job-builder: Update Delivery Pipeline plugin to use convert xml
*** csomerville has joined #openstack-infra15:18
*** cody-somerville has quit IRC15:18
*** stewie925 has joined #openstack-infra15:19
clarkbsdague: dansmith what are we looking at to get that 40% number? is this a single test failing? an aggregate of all tests? any example failures?15:19
*** skipp has joined #openstack-infra15:20
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for CodeCover plugin
dansmithclarkb: I dunno about the number, but just looking at a bunch of failures, they seem to be a variety of things.. clarkb like this non-fatal one:
dansmithand then...15:20
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for enabling empty results
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Temporary allow merges for sahara-tests-release
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Revert "Temporary allow merges for sahara-tests-release"
*** ihrachys has joined #openstack-infra15:21
*** mmedvede has quit IRC15:21
dansmithclarkb: this one gets way farther, but similar fail:
clarkbdansmith: the ntpd issue is being worked on. We need to change how that works and its different for every distro :/15:21
dansmithi.e. network stuff15:21
*** rcernin has quit IRC15:22
clarkblooks like talking to the mirror host timed out which was the actual fail15:22
dansmithclarkb: okay,  I figured that was related to this since it seems to be random network fails, but fair enough15:22
clarkbdansmith: it has to do with how and when ntp is willing to step vs skew the clock and its a mess. Probably not network related15:22
*** ihrachys has quit IRC15:22
dansmithclarkb: ntpdate first?15:23
clarkbdansmith: except ntpdate is deprecated and should not be used15:23
dansmithclarkb: this one got killed early, apt fail:
dansmithclarkb: ....riiight... :P15:23
clarkbbut yes maybe we should go back to ntpdate temporarily until we can get the other bits sorted15:23
*** pilgrimstack has quit IRC15:24
clarkbdansmith: ya it apparently was a fork of ntp code from the land before time and is buggy and insecure15:24
dansmithclarkb: yeah, but alas, I dunno how else to programmatically get the clock close enough for ntpd, but that's not important15:24
*** d0ugal has joined #openstack-infra15:24
*** mhickey has quit IRC15:24
dansmithhere's a non-grenade pip-based fail:
dansmithclarkb: so I dunno about the 40% number, but the logstash query I'm looking at shows about 15% ratio of FAILURE to SUCCESS15:26
dansmiththat's just the gate, so maybe it's higher if you factor in check15:26
clarkbok, I will have to look at mirror server logs to see if we can narrow down the case of the time outs15:26
mordredclarkb: it all seems to be internap15:26
mordredclarkb: sdague earlier was musing that there were internap network issues15:26
*** pilgrimstack has joined #openstack-infra15:28
*** vhosakot has joined #openstack-infra15:28
*** amitgandhinz has quit IRC15:28
*** sdake_ has joined #openstack-infra15:29
*** amitgandhinz has joined #openstack-infra15:29
*** pilgrimstack has quit IRC15:29
*** abregman_ has joined #openstack-infra15:30
openstackgerritMerged openstack-infra/jenkins-job-builder: Add valid_dict parameter to convert xml
*** e0ne has quit IRC15:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Rework the task list layout
*** hieulq_ has joined #openstack-infra15:31
*** jianghuaw-m has joined #openstack-infra15:31
*** abregman has quit IRC15:32
*** Swami has joined #openstack-infra15:32
openstackgerritMerged openstack-infra/jenkins-job-builder: Remove handle_entity_children for convert xml
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker
openstackgerritMerged openstack-infra/jenkins-job-builder: Update LogStash plugin to use convert xml
*** mmedvede has joined #openstack-infra15:34
*** zhurong has quit IRC15:34
toabctlsomebody there to have a look at ?15:36
*** mhickey has joined #openstack-infra15:36
clarkbmordred: did anyone manage to take a look at the mirror host logs to see if there was any more info there? (I am trying to filter through scrollback but not noticing much)15:37
mordredclarkb: no - I haven't gotten to the part of my morning where I'm useful yet15:38
dansmithclarkb: I was going to look to see if I see a bulk of similar failures for other providers15:38
dansmithclarkb: otherwise seems like the mirror logs are probably not that relevant, right?15:38
*** jianghuaw-m has quit IRC15:39
*** Apoorva has joined #openstack-infra15:39
clarkbdansmith: possibly. If something was breaking afs globally then we should still see that in internap for example but ya it would give us better data to see if it was a problem in other providers too15:39
*** esikache1 has quit IRC15:40
*** kgiusti|meeting is now known as kgiusti15:41
mordredclarkb: the logstash query from sdague earlier was:*dsvm*%20AND%20build_queue:gate15:41
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker
*** wcriswell has joined #openstack-infra15:43
dansmithI'm trying to alter that query a bit to figure out if this is more than internap, but I suck at logstash so it's taking me a bit15:43
pabelangerjeblair: mordred: clarkb: It looks like ubuntu-xenail builds in diskimage-builder are broken for the moment, ianw was looking at it last night. So until we fix it, we cannot land our current patches for osic-cloud115:43
clarkbpabelanger: have a paste of the error handy?15:44
clarkbthe internap mirror did have a hung process yesterday according to inaccurate dmesg -T output15:44
cloudnullpabelanger: if i can help in some way let me know15:45
pabelangerclarkb: a little large:
*** jerryz has joined #openstack-infra15:45
*** mmedvede has quit IRC15:45
dansmithclarkb: mordred: yeah, that pip timeout error seems to only dig up internap for me15:46
*** javeriak has joined #openstack-infra15:46
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Growl plugin
fungiyikes, rootfs utilization on is climbing rapidly15:48
fungiwe probably need to put the afs cache there on a separate volume before it asplodes15:49
clarkbis it not on a separate volume?15:49
*** pilgrimstack has joined #openstack-infra15:50
funginot that i can see in cacti. checking locally in a sec15:50
fungithough more likely this is apache logs15:50
jeblairlast time i looked, or cache use was very small15:50
fungiwe generate crazy amounts of (mostly useless?) apache log volume15:50
openstackgerritMerged openstack-infra/jenkins-job-builder: Update Build Name Setter plugin to use convert xml
jeblairit's 12g on that host15:50
*** chem has quit IRC15:50
openstackgerritMerged openstack-infra/jenkins-job-builder: Update cobertura to use convert xml
david-lylesorry for the noise, but any chance for a second +2 on it's fixes a blocking gate issue for searchlight, it's just making a job non-voting in project-config15:50
jeblairand it's on its own volume15:51
fungiyeah, apache logs are 32G there currently according to du15:51
jeblair(which is enormous)15:51
*** javeriak has quit IRC15:51
*** javeriak has joined #openstack-infra15:52
fungisnmpd has been running there since May2315:52
fungiso probably predates the cache volume and just needs a restart to pick it up15:52
fungii'll do that now15:52
*** chem has joined #openstack-infra15:52
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin
openstackgerritMerged openstack-infra/jenkins-job-builder: adding support for the xcode builder plugin
fungii wonder if we should do something to scale down either our logging or log retention for apache on our mirrors15:53
*** edtubill has quit IRC15:53
openstackgerritTimothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3
*** jerryz has quit IRC15:54
*** edtubill has joined #openstack-infra15:55
*** tqtran has joined #openstack-infra15:57
*** pcm_ has quit IRC15:58
*** weshay_mtg is now known as wesha_15:59
*** wesha_ is now known as weshay_15:59
fungiclarkb: what hung process did we have yesterday on the internap mirror? dmesg says it was >2 weeks ago, and while i don't entirely trust its timestamps i don't see similar messages in our syslog going back as far as we're retaining it (a week)15:59
*** pc_m has joined #openstack-infra16:00
fungithe most recent one i see mentioned by dmesg was wednesday, august 316:00
*** tonytan4ever has quit IRC16:00
*** edtubill has quit IRC16:01
*** tonytan4ever has joined #openstack-infra16:01
*** tqtran has quit IRC16:02
clarkboh sorry that was august 3rd nevermind16:02
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config
clarkbneed more caffeine16:02
faboclarkb: who can merge a merge request these days? I have a mr open since with a +2 since 4 months now16:03
clarkbfungi: mgagne thats a client side pip request log saying the request timed out and apache claiming the transfer succeeded but transfered not enough bytes by my count16:03
*** oanson has quit IRC16:03
faboclarkb: or do you know Darragh nick if he's online16:03
clarkbfungi: mgagne it almost looks like apache thought it was done there for some reason?16:03
fungifabo: electrofelix16:03
faboelectrofelix: follow up to ;) let me know what's the next steps to get it merged please16:04
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config
fungifabo: the main jenkins-job-builder core reviewers according to,members are hashar (not in irc at the moment), electrofelix, zaro, zxiiro and waynr16:05
*** sree has joined #openstack-infra16:06
*** yamamoto has joined #openstack-infra16:06
fungiclarkb: logs indicating a short response can, i believe, indicate an abnormally terminated transfer (like client closing the connection early for example)16:06
*** hpe-hj has joined #openstack-infra16:07
fungiseems like we dug into similar failures in the past related to our git farm16:07
clarkbfungi: mgagne ya we have seen similar in rax when the bw limitations kick in and they close connections for us. Any chance internap does similar?16:07
clarkb does show plateaued graphs though16:07
clarkb*doesn't show16:08
electrofelixfabo: I'd forgotten about that one, doesn't show up on my review dashboard since I've already +2'ed it, I'll do a quick look and hopefully zxiiro can review/approve16:08
waynrfabo fungi not sure i'll be able to get to that review today, possible tomorrow though during the JJB sprint16:08
*** vgridnev has quit IRC16:08
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add zookeeper-servers to fake config
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Set default branch to feature/zuulv3
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add an example logging.conf for development
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a fake-secure.conf
*** valderrv has quit IRC16:09
*** gyee has joined #openstack-infra16:09
fabowaynr: that's fine, it's in the queue for 2 years now :) I'm not at +/- 1 day16:10
*** sree has quit IRC16:10
fabowaynr: just by curiosity, is there's some info about JJB sprint? what are the topics and/or expected outcome?16:11
*** yamamoto has quit IRC16:11
*** cody-somerville has joined #openstack-infra16:11
faboelectrofelix: thanks!16:11
fungiclarkb: yeah, that graph doesn't look like we're seeing a rate limit kick in, more likely some piece of gear is getting overloaded or we've got a noisy neighbor chewing up host nic buffer space or something like that16:13
*** csomerville has quit IRC16:13
*** yamahata has joined #openstack-infra16:14
zxiiroelectrofelix: there's a pretty bad bug in the "disabled" job configuration. It accepts any input as "true" rather than checking if the user passed true|false :(16:14
zxiiroelectrofelix: just got burned by it. I'll look into submitting a patch later today16:14
electrofelixzxiiro: sorry which one is that?16:14
zxiirowe tried to make our job templates have the disabled field configurable and it ended up disabling all of our jobs16:14
*** shashank_hegde has joined #openstack-infra16:14
electrofelixfabo: so we've been running a sprint every 2nd week, email about it was sent to openstack-infra, although maybe it should go to openstack-dev these days as well?16:15
electrofelixfabo: see
zxiiroelectrofelix: here's the lines of code causing the issue
*** tphummel has joined #openstack-infra16:15
electrofelixzxiiro: so you didn't use '{obj:disabled}' to ensure the original object was passed through instead of a string?16:16
openstackgerritDmitry Ilyin proposed openstack-infra/project-config: Enable voting cheack for the Fuel unit tests Puppet 4.5
*** sdake_ has quit IRC16:16
*** ihrachys has joined #openstack-infra16:16
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix issue that cause JCloud instance to be set
zxiiroelectrofelix: we did but the problem is we have a defaults configured disabled: False16:16
openstackgerritMerged openstack-infra/jenkins-job-builder: Update cfp builder to use convert xml
zxiiroelectrofelix: it seems defaults area always returned as strings16:16
electrofelixzxiiro: nuts16:17
*** DrifterZA has quit IRC16:17
openstackgerritMerged openstack-infra/jenkins-job-builder: Update findbugs_settings to use convert xml
electrofelixI thought that was changed?16:17
clarkbpabelanger: ianw appears to be the relevant part of the dib log. Looks like we are du'ing things to see where our disk image size is when it traps the exit handler16:17
clarkbgreghaynes: ^ have you seen that before?16:17
faboelectrofelix: thanks for the pointers and good to know16:17
zxiiroelectrofelix: obj:disabled only works if you configure it in the project section of the config and not as a default value16:17
clarkbI wonder if du is exiting non zero?16:17
*** amitgandhinz has quit IRC16:18
zxiiroelectrofelix: so I think the right thing to do is convert bool to string always for this field and parse the string data16:18
*** amitgandhinz has joined #openstack-infra16:18
greghaynesclarkb: thats post-failure16:19
electrofelixzxiiro: if we just convert it to newer style would it do the right thing?16:19
greghaynesclarkb: is that what youre looking for? something failing in cleanup16:19
zxiiroelectrofelix: what do you mean by newer style?16:19
greghaynesclarkb: that line at the top trap_cleanup means the exit trap is starting so something exit 1'd before then16:19
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.
openstackgerritMerged openstack-infra/jenkins-job-builder: Update maven-builder to use convert xml
electrofelixzxiiro: XML.SubElement(xml, 'disabled').text = str(disabled.lower())16:19
openstackgerritMerged openstack-infra/jenkins-job-builder: Update runscope plugin to use convert xml
openstackgerritMerged openstack-infra/jenkins-job-builder: Update docker build publish to use convert xml
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.
zxiiroelectrofelix: yes that was what i was thinking16:20
electrofelixzxiiro: remove the nested if else?16:20
electrofelixleave the top level one though16:20
openstackgerritMerged openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options
openstackgerritMerged openstack-infra/jenkins-job-builder: gitlab trigger: Update Gitlab to >=1.3.0
clarkbgreghaynes: ya what was running in line one is the du to capture where file usage went16:20
zxiiroelectrofelix: we need to retain the "None" functionality though16:20
clarkbgreghaynes: there are many lines of that so I trimmed it down to one16:20
openstackgerritMerged openstack-infra/jenkins-job-builder: test non-default value of attach-build-log
zxiiroelectrofelix: because we had a request that this field should keep the existing settting if the user does not pass it16:20
electrofelixzxiiro: yeap, that's what I was thinking ;)16:20
*** mmedvede has joined #openstack-infra16:20
zxiiroelectrofelix: i'll fix it and put a comment so that we remember16:20
openstackgerritMerged openstack-infra/jenkins-job-builder: add compress-log option to compress log
*** Goneri has quit IRC16:20
*** abregman_ has quit IRC16:20
*** mtanino has joined #openstack-infra16:20
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Search for VLAN devices on config drive to create bridge on it
clarkbgreghaynes: that happens in bin/disk-image-create16:22
rcarrillocruzyolanda: ^16:22
openstackgerritMarton Kiss proposed openstack-infra/groups: Security update for Google Analytics, Panels, Panelizer
clarkbgreghaynes: eval ${du_output_show} <<< "$du_output" that is the line that appears to cause the trap to trigger16:23
clarkbgreghaynes: assuming all output got flushed16:23
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker
clarkbgreghaynes: does dib run with pipefail on or off?16:24
greghaynesclarkb: off in that script IIRC16:24
greghaynesclarkb: looks like theres grep fun16:24
greghaynesclarkb: so probably the thing to do is bash right before there and play with runnign du_output_show16:25
clarkbgreghaynes: the greps happen before I think du_output_show's command is what is likely breaking16:25
greghaynesdu_output_show is two greps? I am confused16:25
greghaynesoh, you mean the sorts and whatnot16:26
*** _nadya_ has quit IRC16:26
clarkbits sort | numfmt16:26
clarkbtehre is no grep there16:26
greghaynesah, right, the greps are in a conditional we dont set16:26
*** tesseract- has quit IRC16:26
greghaynesyea, regardless youre just going to want to bash and run that command which I think youve got16:26
greghaynessee what the actual error is16:27
*** sambetts is now known as sambetts|afk16:27
greghaynesI kind of want to remove set -e around this area too. Its really silly to fail when printing out trace info16:27
clarkbI also think we can drop the numfmt entirely and just use du -h and sort -h which may make this simpler and less flaky16:28
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Enable shared networks for te-broker
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Add nodepoolrc-openstackzuul to te-broker
*** dronshaw has joined #openstack-infra16:30
greghaynesclarkb: since these are all pipes im somewhat confused what would fail and still output partially - sort requires a full pass before it can print anything...16:30
greghaynesand it looks like we set +o in that block16:30
*** Apoorva has quit IRC16:30
greghaynesclarkb: so it must be numfmt?16:31
*** matthewbodkin has quit IRC16:31
clarkbya I am guessing that numfmt isn't liking an input to it16:31
clarkbthat it can't convert properly16:31
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add scheduling thread to nodepool builder
clarkbeg field 1 isn't a number16:32
*** dronshaw has left #openstack-infra16:32
*** pilgrimstack has quit IRC16:32
*** caowei has quit IRC16:32
*** baoli has quit IRC16:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Search for VLAN devices on config drive to create bridge on it
clarkbExit status is 0 if all input numbers were successfully converted.  By default, numfmt will stop at the first conversion error with exit status 2.16:32
greghayneshrm, we got exitval 116:33
greghaynesoh wait, I misread from before - we are going to do the greps that get appended to du_output_command16:34
greghaynesso its possible that numfmt fails then because no pipefail the grep's fail16:34
*** ccamacho has quit IRC16:34
greghaynescan you run with a bash in ther? Itll take me a while to do a real nodepool image16:35
clarkboh we do append those greps I just double checked the echo output to make sure we go into that block16:35
zxiirofabo: i'll take a look at your patch this afternoon16:35
clarkbgreghaynes: I will likely try to reproduce locally16:35
*** hieulq_ has quit IRC16:35
*** nwkarsten has quit IRC16:35
greghaynesI like the use du -h btw. I think this is ianw code so he may have thoughts16:36
*** ihrachys has quit IRC16:36
*** nwkarsten has joined #openstack-infra16:36
fungithe downside to du -h is that it's not easily sortable16:36
*** armax has quit IRC16:37
clarkbgreghaynes: check out echo "9.5MiB" | grep -v '\..MiB'16:38
clarkbfungi: it is, works fine with sort -h16:38
*** ihrachys has joined #openstack-infra16:38
openstackgerritMonty Taylor proposed openstack/os-client-config: Add support for configuring split-stack networks
clarkbgreghaynes: that returns 1 because it inverts the match. Now to see if we get that if any fail to maatch16:38
clarkbgreghaynes: I would expect 0 if any match but whatever time to test16:38
*** jheroux has joined #openstack-infra16:38
openstackgerritDmitry Ilyin proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5
mordredShrews, cloudnull, clarkb: ^^ this is needed to deal with multi-node in OSIC for proper categorization of the public quality of v4 and v6 split networks16:39
fungiclarkb: hah, i never knew about sort -h16:39
*** mhickey has quit IRC16:39
clarkbgreghaynes: ya in a stream if any match grep should give us 0 so the invert probably isn't breaking us16:39
mordredShrews, cloudnull, clarkb (I'm ^^ the occ change, not the project-config change)16:39
mordrednotmorgan: you may also be interested16:40
*** amotoki has quit IRC16:40
*** nwkarsten has quit IRC16:40
greghaynesclarkb: my bet is on numfmt fail and then grep fails with no input16:40
greghaynesclarkb: seems really easy to make happen - make a file with a newline in it16:40
*** itisha has quit IRC16:40
*** kzaitsev_mb has joined #openstack-infra16:40
greghayneser, filename16:40
clarkbgreghaynes: ha ok16:40
*** ihrachys has quit IRC16:41
clarkbgreghaynes: so maybe we want --invalid=warn passed to numfmt16:41
*** jpich has quit IRC16:41
greghaynesclarkb: Yep, and if that goes to stderr we have an easy way to debug actually16:42
*** lucasagomes is now known as lucas-dinner16:42
wznoinskhi all16:42
clarkbor use a zero byte delimter16:42
clarkbor do both16:42
* clarkb tests16:42
*** jordanP has quit IRC16:43
*** sdake has joined #openstack-infra16:43
*** ihrachys has joined #openstack-infra16:44
Shrewsmordred: reviewed. issues found16:44
mordredShrews: woot. thanks!16:45
wznoinskI have a problem with dhcp-all-interfaces element, it kicks off at the network-interface (getting up) but only then it adds the /etc/network/interfaces auto iface/dhcp details and it's doing it after static-network-up-emitted16:46
wznoinskit's causing cloud-init-nonet seeing static-network-up-emitted and progressing to init phase where not all the interfaces are yet up16:47
openstackgerritMonty Taylor proposed openstack/os-client-config: Add support for configuring split-stack networks
Shrewsmordred: and another16:47
wznoinsk(disk image builder) ^16:47
*** Na3iL has joined #openstack-infra16:47
Shrewsmordred: oh, you got it too16:47
greghayneswznoinsk: youre using cloud-init and dhcp-all-interfaces?16:48
greghayneswznoinsk: any reason youre using both? I'd expect cloud-init to dhcp for you16:49
fungiwznoinsk: interesting. i don't think we're using those elements in the minimal image types for our ci... also, there is a #openstack-dib irc channel now16:49
*** sdague has quit IRC16:49
greghaynesthat too :)16:49
wznoinskfungi: yes, I'm on that # but here's a bigger audience...16:50
fungiit still has that "new channel smell" and everything16:50
wznoinsksure, will move tto there16:50
clarkbgreghaynes: du -a -c -x --null ./ | sort -nr --zero-terminated | numfmt --to=iec-i --padding=7 --suffix=B --field=1 --from-unit=1024 --zero-terminated --invalid=warn | egrep --null-data 'MiB|GiB|TiB|PiB' | grep --null-data -v '\..MiB' | tr '\0' '\n'16:50
*** raunak has joined #openstack-infra16:50
clarkbthat seems to work just in a random dir on my local machine16:50
openstackgerritMerged openstack-infra/storyboard: Create timeline events for boards and worklists
fungiwznoinsk: i guess my point was, if people are basing the images for their ci on our nodepool configuration, they're unlikely to be using that combination of elements16:50
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Have buildZooKeeperHosts take a ConfigValue object
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a script to print the ZK tree
fungigreghaynes: you might want to add your shiny new channel to at some point16:52
greghaynesfungi: ah, good tip16:52
*** d34dh0r53 is now known as RichardLongus16:52
*** ccamacho has joined #openstack-infra16:52
*** ccamacho has left #openstack-infra16:52
wznoinskfungi: actually that's not in CI context anymore, we just finishing on creating our own dev cloud, the one I'm now working on is the baremetal one, kolla, ironic, dib, cloud-init and other flowers, I've taken the dhcp-all-elements from ironic guide I think16:53
wznoinskgreghaynes: I have to check cloud-init doing dhcp for interfaces16:54
*** eandersson_ has quit IRC16:54
fungiwznoinsk: oh, got it. then yeah, that might be relevant to what we're doing on infra-cloud but rcarrillocruz and crinkle would probably be able to say for sure16:54
wznoinskmoving to #openstack-dib if anyone interested16:54
*** shashank_hegde has quit IRC16:54
*** annegentle has quit IRC16:54
fungithough we're not using kolla or cloud-init16:55
fungiglean is being leveraged pretty heavily on our infra-cloud deployment instead of cloud-init16:55
mgagneclarkb: which flavor are you using?16:56
*** tosky has quit IRC16:56
*** vgridnev has joined #openstack-infra16:56
mgagneclarkb: I think it's 100mbps for A1 series and 50mbps for B1.1 and B1.2 and 1gbps for the rest16:56
*** annegentle has joined #openstack-infra16:57
dhellmanndoes anyone know what "DSS Cinder CI" is or why it's voting on changes in the releases repository?16:57
dhellmannor commenting, at least16:57
*** asettle has quit IRC16:58
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a notice when worklists or boards are private or archived
*** RichardLongus is now known as d34dh0r5316:58
dmsimarddhellmann: I've seen it start commenting on Packstack reviews today too, nonsense afaict .. complained about non-mergeable review while it was entirely fine16:58
clarkbmgagne: A1.216:58
*** ihrachys has quit IRC16:58
dmsimardNo clue where it comes from or why it would want to do CI on Packstack.16:58
*** bethwhite__ has quit IRC16:58
*** asettle has joined #openstack-infra16:58
mgagne100mbps then16:58
fungidhellmann: dmsimard: pleia2 disabled the account earlier today16:59
*** derekh has quit IRC16:59
dhellmannfungi, pleia2 : ok, thanks!16:59
fungiwe assume it was misconfigured and reporting on all changes in gerrit16:59
dmsimardfair assumption, it was mine as well16:59
dhellmannthat's what I figured16:59
*** yamahata has quit IRC17:00
*** tkelsey has joined #openstack-infra17:00
*** vgridnev has quit IRC17:00
*** yamahata has joined #openstack-infra17:00
clarkbmgagne: ok, any chance its getting a little overzealous and rate limiting closer to 75mbps?17:00
clarkbmgagne: we appear to peak at about 60-75mbps17:01
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Increase the quota in the OSIC cloud1
mgagneclarkb: I hope not, it's the default bw limiting feature in Nova/libvirt17:01
clarkbgreghaynes: I am waiting for a local build to hopefully reproduce the xenial issue the nI can update dib with my null and warn stuff17:01
cloudnullIDK if we're ready for that yet, or if we want to wait for the DIB issues to be sorted, but there's the change.17:01
clarkbmgagne: do you knwo if its possible to check for that in logs on the hypervisor side?17:02
*** piet has joined #openstack-infra17:02
clarkbcloudnull: I think we want to get these image builds sorted first17:02
*** jordanP has joined #openstack-infra17:02
fungiclarkb: keep in mind that those peaks in cacti are a 5-minute average. if it's hitting 100mbps for one minute and 60mbps for the other four minutes in a sample...17:02
greghaynesclarkb: has there been thought given to adding image builds in to CI for you all?17:03
*** sarob has joined #openstack-infra17:03
clarkbgreghaynes: and it looks like we can set the DIB_SHOW_IMAGE_USAGE var to work around it in the mean time17:03
*** asettle has quit IRC17:03
*** sarob has quit IRC17:03
*** valderrv_ has joined #openstack-infra17:03
clarkbgreghaynes: there has been but it would be far too slow to gate on I think17:03
clarkbfungi: good point17:03
*** electrofelix has quit IRC17:03
*** sarob has joined #openstack-infra17:03
*** tonytan_brb has joined #openstack-infra17:04
fungimy estimates were slightly off... (100+69*4)/5=75.2 but you get my point17:04
mgagneclarkb: will do some tests on my side first17:04
clarkbgreghaynes: we would need to hack it so that dib could use the local preexisting caches as much as possible17:04
clarkbmgagne: thank you17:04
greghaynesclarkb: hrm. Yea thats what I was just wondering - is that time mostly grabbing git repos?17:04
zaroyolanda: hi, i think there's some confusion with
clarkbgreghaynes: even just updating the git repos is slow17:04
greghaynesbecause its actually pretty easy to point dib at the right place for that17:04
clarkbgreghaynes: when you have >1k of them17:04
fungiclarkb: we could throw ntop or sar or something on there and then chart a more fine-grained burst rate17:04
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8
mgagneclarkb: but I suggest you upgrade to something in the B series so you get more bandwidth, I'm sure you will hit 100mbps very soon anyway and A series won't answer your bw needs17:05
clarkbgreghaynes: 1366 repos to update according ot my local build17:05
clarkbmgagne: ok17:05
*** Apoorva has joined #openstack-infra17:05
pabelangerokay, we have the new DNS for osic-cloud8 but still missing SSL, so ^ is WIP until that is fixed17:05
*** sree has joined #openstack-infra17:05
pabelangerfor ssl, mrhillsman is helping get that only17:05
openstackgerritMerged openstack-infra/system-config: Set compute009.vanilla IP addresses
clarkbmgagne: looks like its 500mbps not 50mbps for the B series? am I reading the flavor show output correctly17:06
mgagneyea, 500mbps17:06
*** sdague has joined #openstack-infra17:06
fungithat's probably plenty of overhead then17:06
mgagneclarkb: yea, typo earlier17:06
*** tonytan4ever has quit IRC17:06
mgagneclarkb: 100mbps for A1 series and 500mbps <-- for B1.1 and B1.2 and 1gbps for the rest17:06
clarkbgreat looks like B1.2 has 8GB of ram but the same disk and vcpus and 500mbps over 100mbps when compared to A1.2 so maybe we update to that17:07
sdagueclarkb: I got the 40% number as the failure rate for dsvm jobs on internap since the first failure at 1am EST17:07
sdaguefor the gate17:08
sdaguewhich should be 0.17:08
mgagneclarkb: it also has more CPU allocation (3:1 vs 1:1) IIRC17:08
*** vgridnev has joined #openstack-infra17:08
sdagueand every spot check was pip or apt related17:08
clarkbpabelanger: you redid the rax one do you also want to rebuild the internap one? I would like to get to the bottom of this dib issue befure I context switch too heavily into something else17:08
clarkbsdague: ya beginning to think its probably network bw rate limiting in internap like we had in rackspace17:08
*** timello has joined #openstack-infra17:09
*** vgridnev has quit IRC17:09
mgagneclarkb: I'm not sure libvirt logs anything related to bw limiting when it kicks17:09
*** sree has quit IRC17:10
pabelangerclarkb: just catching up, we need a new mirror in internap?17:10
*** vgridnev has joined #openstack-infra17:10
clarkbpabelanger: yes I think we need a larger flavor. We are rate limited to 100mbps on the A1.2 flavor and are seeing many failures in internap. APache logs show that transfers are completing before all bits are transfered17:10
pabelangerclarkb: sure, I can launch a replacement today17:11
clarkbpabelanger: the B1.2 flavor will give us 500mbps has the asme number of cpus and disk but has 4x the memory17:11
*** timello has quit IRC17:11
*** timello has joined #openstack-infra17:12
*** devkulkarni1 has joined #openstack-infra17:12
pabelangerclarkb: okay, need to lunch first, then will dive into it17:12
mrhillsmanyeah we need to update the endpoints to use SSL - got someone on that17:12
mrhillsmanhope is they will be done shortly17:13
*** mminesh has quit IRC17:13
clarkbgreghaynes: DIB_SHOW_IMAGE_USAGE doesn't seem to be set by nodepool. Any idea why we are executing that code at all?17:13
mgagneclarkb: we are investigating the networking bw limit issue. a coworker thinks it could be related to something else.17:13
*** abadger1999 has joined #openstack-infra17:14
clarkbmgagne: ok thank you for checking and let us know what you find17:14
clarkbpabelanger: ^17:14
*** devkulkarni has quit IRC17:14
*** mminesh has joined #openstack-infra17:14
openstackgerritMonty Taylor proposed openstack-infra/shade: Stop getting extra flavor specs where they're useless
openstackgerritMonty Taylor proposed openstack-infra/shade: Allow image and flavor by name for create_server
fungieither way, we've got 5-minute averages within 75% of the cap for that flavor, so should still upgreade17:14
*** shashank_hegde has joined #openstack-infra17:15
openstackgerritKhai Do proposed openstack-infra/system-config: Remove duplicate code to setup gerrit local replication
abadger1999hey dhellmann, I was looking at using python-reno and wondered if you could answer one usage question... If my project has a mainline (devel) and branches stable-X.Y from there. Would we want to clear out the release-notes directory in devel every time we branch a new stable-X.Y?17:17
*** esberglu has quit IRC17:17
*** mmedvede has quit IRC17:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add osic-cloud8 to openstack-infra
*** mmedvede has joined #openstack-infra17:18
*** weshay_ has quit IRC17:20
pabelangerfungi: mrhillsman: ^ adds our nodepool configuration for cloud-8, so one SSL is confirmed to be working. We can launch an AFS server, setup our DNS, then land above17:21
pabelangerI'll poke around a grafana next to make sure osic-cloud8 will be rendered17:21
*** electrofelix has joined #openstack-infra17:21
*** valderrv_ has quit IRC17:21
*** electrofelix has quit IRC17:22
*** jerryz has joined #openstack-infra17:24
*** furlongm has quit IRC17:24
*** Hal1 has quit IRC17:24
*** furlongm has joined #openstack-infra17:24
*** valderrv_ has joined #openstack-infra17:24
greghaynesclarkb: looks like it is set in nodepool builder.py17:24
*** tonytan_brb is now known as tonytan4ever17:25
*** Hal has joined #openstack-infra17:26
*** SumitNaiksatam has joined #openstack-infra17:26
dhellmannabadger1999 : why don't we talk in #openstack-release to keep this channel clear for infra17:26
fungipabelanger: mrhillsman: is an amusing domain... not using for this one? whois indicates the domain registrant is hidden17:26
abadger1999I've pinged dhellman in the openstack channels too for a question on usage.dhthanks17:26
*** abadger1999 has left #openstack-infra17:27
clarkbgreghaynes: gah that means my local test needs to be restarted to set it myself17:27
*** tosky has joined #openstack-infra17:27
greghaynesclarkb: :(17:27
mrhillsmanfungi hehe17:27
*** gyee has quit IRC17:28
*** Apoorva has quit IRC17:28
*** shashank_hegde has quit IRC17:28
mrhillsmanit is temporary, did not want to increase delay waiting on that subdomain since it may have to go through a few hands17:28
fungimrhillsman: might want to be careful with that domain... wouldn't want to frighten the foundation legal team17:28
funginot that i'll say anything17:28
mrhillsmanhopefully one day it will come to be what i envision17:29
mrhillsmannothing for profit so not worried :)17:29
*** stewie925 has quit IRC17:30
mrhillsmanand nothing to take away from openstack itself17:30
*** dprince has quit IRC17:30
*** vgridnev has quit IRC17:30
*** _nadya_ has joined #openstack-infra17:30
*** esberglu has joined #openstack-infra17:31
fungimrhillsman: cool17:32
*** stew925 has joined #openstack-infra17:33
*** valderrv_ has quit IRC17:34
*** degorenko is now known as _degorenko|afk17:35
fungiannegentle: loquacities: it seems rackspace opened a support ticket to discuss trimming down, indicating that it's got more files than any other site in cloudsites... does any of the old accumulated content get audited for removal periodically, or are you only removing files people explicitly ask to have deleted?17:35
annegentlefungi we are pretty good at trimming, but it is a big site17:35
mordredfungi: we keep having more $things than other things17:35
annegentlefungi and that's funny that it's their biggest17:36
fungi"As it stands the site uses ~594,000 files, causing the account to be at a grand total of ~853,000 files. This is the largest single site on the storage device in our environment at this time. While for our customers we don't bill for inode consumption, it is still a very finite resource in Cloud Sites and we try our best to monitor and help find ways to reduce the total consumption to keep our17:36
fungistorage devices as happy as we can.17:36
annegentlefungi FTP in and do a cursory check17:36
annegentlefungi heh. wow.17:36
*** baoli has joined #openstack-infra17:37
fungithey opened the ticket on the 12th and pinged it for an update today. i only happened to notice because i logged into that tenant to poke at dns and happened to see that there was an open support ticket according to the dashboard17:37
annegentlefungi do you have a suggestion for archiving say, if they are not willing to?17:38
annegentlefungi I would want to have archival copies other than archive.org17:39
*** bryan_att has joined #openstack-infra17:39
JayFis a cloud sites site or a cloud files + CDN site?17:39
*** tkelsey has quit IRC17:39
annegentlefungi bexar and cactus are only containing index.html17:39
* jroll wonders if openstack pricing will change with the spinout of cloud sites17:39
fungiJayF: cloudsites17:39
annegentleJayF No idea. jroll we wonder also17:39
*** tqtran has joined #openstack-infra17:39
jrollyeah, probably time to think about moving it anyway17:39
mgagneclarkb: so I'm getting 93.4MB/s when uploading to server and 100MB/s when downloading from server. I think bw limiting isn't the issue here (where it would cap at 70MB/s instead of 100MB/s)17:40
*** psilvad has quit IRC17:40
mgagneclarkb: I suspect you are hitting peaks which aren't shown in the graph.17:40
annegentlejroll yep, we want to17:40
fungiagreed, we've already been having discussions about speeding up the afs-backed docs.o.o server plan17:40
JayFfungi: perhaps moving the generated, static docs to a cloud files CDN site on a different hostname would help?17:40
jrollok, awesome17:40
clarkbmgagne: ya as fungi says its a 5 minute average so there are likely peaks in there where we need more bw17:40
mgagneThe server I tested with is hosted on the exact same compute node as the mirror. I tested with another server on a different compute node and VM so routers aren't bypassed.17:41
mgagneVM -> VLAN17:41
*** gouthamr is now known as gouthamr_afk17:42
*** jcook has joined #openstack-infra17:42
annegentleJayF that's the proposed solution17:42
mordredall it needs is someone hacking on it17:42
*** gouthamr_afk is now known as gouthamr-afk17:42
sjmc7hi all, i’d appreciate a +A on to unblock our gate if anyone’s got a second. simple review to make a job non-voting17:43
*** elo has quit IRC17:43
fungiannegentle: loquacities: here's the meat of the ticket they opened
*** hockeynut has quit IRC17:44
fungiworth mention, it echoes JayF's idea of switching to cloudfiles17:45
*** valderrv_ has joined #openstack-infra17:45
dansmithclarkb: mgagne: is this because for internap workers we use a local mirror that itself is getting maxed out?17:45
clarkbdansmith: yes that is what I think is happening17:45
JayFfungi: big downside of cloud files: no SSL support17:45
clarkbso we will replace ut with bigver instance17:46
mordredand also no rsync support17:46
annegentlefungi good to know. JayF I want https :)17:46
JayFfungi: well, SSL is supported; you just can't give it a cert to make the hostname happy17:46
dansmithclarkb: I wonder if it would make sense to do some shaping so that we provide fairness instead of starving one or two connections to the point of timeout?17:46
fungiJayF: well, bit that we'd probably do it anyway, but we don't have https on docs.o.o right now (and we will be able to add it with the afs plan)17:46
annegentlemordred and something like rsync17:46
mordredannegentle: yup17:46
odyssey4mecloudnull doesn't turbolift does rsync-like functionality for cloudfiles? cc mordred17:46
fungier, not that we'd probably do it anyway i mean17:46
*** kzaitsev_mb has quit IRC17:46
mordrednot sure. in general though we have a lovely AFS cell that we use for stuff like this that has all of the things it needs17:47
openstackgerritMerged openstack-infra/system-config: Test Gerrit CI comment regex for rel="nofollow"
odyssey4mein that case, perhaps we should implement a turbolift publisher for infra, so that docs can be properly mirrored from builds?17:47
annegentlefungi the redirects part is interesting, it might be good to get rid of .htaccess, but doesn't the proposal suggest yet more files? And still we need the archive solution.17:47
mordredso I don't really think there is much value in exploring tools that exist to publish into cloudfiles17:47
cloudnullshameless plug
fungidansmith: i doubt we have any control over the rate limiting gear in internap17:47
dansmithfungi: no, but we have control over our own nic :)17:47
fungidansmith: fair enough, and that could work for egress on that machine, which is the bulk of the traffic17:48
*** dprince has joined #openstack-infra17:48
mordred(no offense intended to any o those tools)17:48
annegentlefungi also everything in /search is from 2011 so probably safe to say we're not using that mysql database for search.17:48
jrollmordred: ++17:48
mordredannegentle: ++17:48
dansmithfungi: if you leave your nic unrestricted and it tries to send at link speed, but are limited to 100mbit, then you're just going to get the upstream router killing your connections with dropped packets17:48
fungidansmith: agreed17:48
dansmithfungi: right, if the problem is egress from the mirror, we can easily work on that17:48
*** psilvad has joined #openstack-infra17:48
*** shashank_hegde has joined #openstack-infra17:48
annegentlefungi so basically yep looks like we can trim, but give us an archive idea17:48
fungidansmith: i had originally misinterpreted your suggestion as switching internap's packet shaping to use red or similar17:49
dansmithfungi: nay17:49
jeblairfungi: should i dust off the afs docs plan and propose it for next meeting?17:49
fungijeblair: i think that would be wise17:49
mordredjeblair: I think so - it seems like the world is conspiring to make it time for that to get real work17:49
jeblairokay, i will do so17:49
fungiannegentle: i'll try to summarize in a response on that ticket17:50
odyssey4meproject-config cores, I'd appreciate reviews of asap please17:50
annegentlefungi make sure you get loquacities in though before17:50
annegentlefungi or Andreas17:51
annegentlefungi since I don't know everything :)17:51
fungiannegentle: yep, i won't get to updating it before she's awake anyway17:51
*** rcernin has joined #openstack-infra17:51
fungiannegentle: and i want to avoid bothering andreas on his much-deserved vacation ;)17:51
annegentlefungi oh for sure17:51
*** weshay has joined #openstack-infra17:52
jeblairapparently i said "there are two approaches we can take" and then listed 3 things and no one called me on it.17:53
mordredjeblair: I think the difference between 2 and 3 is within the margin of error17:53
fungijeblair: fuzzy math17:53
annegentlejeblair snerk17:53
jeblairmordred: yeah, that would have been my rejoinder :)17:53
jeblairalso, half of the future in #2 has come to pass, so it needs a rethink with v2.517:54
*** devkulkarni1 has quit IRC17:55
*** baoli has quit IRC17:56
*** Apoorva has joined #openstack-infra17:56
*** baoli has joined #openstack-infra17:56
jeblairlike -- put afs creds on launchers and modify the jjb->ansible translation to just sync right into afs on the launcher.17:56
*** kzaitsev_mb has joined #openstack-infra17:57
clarkbgreghaynes: :( I don't have enough disk space for this reproduction build /me makes a bigger VM17:58
clarkbmaybe I should start with just disabling the output in the nodepool builder17:58
clarkbthen we can get this image built17:58
fungijeblair: giving launchers afs write permission seems safe to me17:59
*** acoles is now known as acoles_17:59
*** annegentle has quit IRC17:59
fungias opposed to what we ended up needing to do with the swift publishing, granting write access to job nodes (which was way more complicated to make safe)17:59
openstackgerritClark Boylan proposed openstack-infra/nodepool: Stop running DIB usage report
clarkbpabelanger: fungi jeblair greghaynes ^ thats the temporary workaround for the xenial image build fails I think18:00
clarkbwell maybe not temporary we should just set this per image rather than always18:00
*** kzaitsev_mb has quit IRC18:01
jeblairclarkb: wfm.  also, we could just 'stat' the file and print it in nodepool.  :)18:01
clarkbjeblair: it runs a fancy formatted du thing so you get a breakdown by dir and file for anything > 10MB18:02
*** esikachev has joined #openstack-infra18:02
*** inc0 has quit IRC18:02
fungiand apparently has portability issues18:03
*** xarses has quit IRC18:03
*** valderrv_ has quit IRC18:03
rhedlindAnyone know the first version of nodepool that dropped support for snapshot images and went to exclusively using diskimage builder? I would like to downgrade to the version before that. :)18:04
clarkbrhedlind: I don't think it has happened yet18:05
*** Sukhdev has joined #openstack-infra18:05
fungirhedlind: clarkb: though i believe the intent is for nodepool 0.3.0 to be that last version you can count on for some things like snapshot build support and lack of zookeeper dependency18:06
*** elo has joined #openstack-infra18:06
fungiodds are that stuff will change before the next release18:07
* mordred agrees with fungi and clarkb18:07
*** mfedosin has joined #openstack-infra18:07
rhedlindmine is complaining that I don't have a images-dir in my config, so I figured it had already happened18:08
pabelangerclarkb: Wouldn't it make more sense to pass that in via nodepool.yaml in the future?18:08
*** jtomasek has joined #openstack-infra18:08
*** mmedvede- has joined #openstack-infra18:08
fungirhedlind: did you pin to 0.3.0 already, or are you consuming from master branch tip?18:09
clarkbpabelanger: yup thats what the commit message says18:09
clarkbpabelanger: basically we don't need to bake it into nodepool particularly while it is buggy. And you can always set it via the nodepool config18:09
pabelangerclarkb: cool, I should read that18:09
pabelangerclarkb: ++18:09
rhedlindI am at 0.3.0, not sure what dev though18:09
*** nwkarsten has joined #openstack-infra18:10
fungirhedlind: it got recommended in a few places...
*** yamahata has quit IRC18:10
mfedosinhi folks! I proposed a patch in project-config some time ago: so, could you review it if you have time?18:10
mfedosinthanks in advance18:10
fungirhedlind: we expect disruptive changes on the master branch after the 0.3.0 tag18:11
openstackgerritClark Boylan proposed openstack/diskimage-builder: Make disk usage reporting more robust
clarkbgreghaynes: ^ thats an untested start. Need to build a smaller image so that things don't break locally when I run out of disk18:12
pleia2harddrive in my laptop just died (it's been unwell for a couple days)18:12
rhedlindfungi: I did use nodepool==0.3.0 when running pip install18:12
*** rbrndt has quit IRC18:13
pleia2so I'm going to step away for the afternoon (and mostluly gone tomorrow for pre-conference anyway)18:13
jeblairpleia2: i have some spares but they probably won't fit in your laptop18:14
*** kzaitsev_mb has joined #openstack-infra18:14
pleia2jeblair: I'm in Philly anyway :)18:14
pleia2my husband is coming in tomorrow to give me a spare, woo18:14
jeblairpleia2: oh noes, they probably don't have fry's there!18:14
jeblairpleia2: oh good :)18:15
*** furlongm has quit IRC18:15
*** furlongm has joined #openstack-infra18:15
fungirhedlind: interesting. do you mind popping the full error up on and linking it in here?18:16
*** zul has quit IRC18:16
fungii'm skimming the state of the code at 0.3.0 to see why it would care, and which commit started making it care18:16
fungimy guess is the config validator started expecting it18:17
jeblairit's probably okay to just set it to /tmp if you're not going to use it18:17
fungithe only place (in 0.3.0 anyway) that i see expecting it is in builder.py18:19
fungiahh, yep18:19
fungiso nodepoold is starting a builder thread that isn't needed?18:19
dansmithfungi: pabelanger: either of you willing to run arbitrary code on the internap mirror as root from a pastebin I provide? :)18:20
fungioh, or maybe the snapshot-based builds also happen out of the builder18:20
*** annegentle has joined #openstack-infra18:20
dansmithfungi: pabelanger er I mean, I have some fairness and limiting commands that are dead simple which would be nice to test on that to avoid stalling individual connections at the peaks :)18:20
mgagnelike curl | sudo sh - ? :D18:21
dansmithmgagne: exactly :P18:21
fungidansmith: i'm happy to evaluate it at least. i've done some packet shaping on linux in the past18:21
dansmithfungi: limit outbound to 1mbit below your limit, provide fairness per flow18:21
dansmithfungi: change the DEV accordingly of course18:21
fungiit's worth considering whether we should look into implementing it on more than just that mirror18:21
*** kzaitsev_mb has quit IRC18:21
dansmithfungi: definitely if we have other places where we're limited, and we could adjust this to be specific to mirror traffic, etc but this is the simplest test18:22
dansmithfungi: and you can do this to show details about it afterwards: tc -s -d class show dev $DEV18:22
dansmithfungi: which will show you balance between all the flows, etc18:23
*** zul has joined #openstack-infra18:23
*** pvaneck has joined #openstack-infra18:23
fungiyep, i'm just refreshing my reading of the tc manpage18:23
dansmithfungi: tested locally with iperf18:23
fungithis looks sane to me. clarkb: you have any opinion on it?18:24
pabelangerfungi: rhedlind: are you doing nodepoold  --no-builder ?18:24
rhedlindfungi: looks like the exception went in on Jan 12:
rhedlindpabelanger: no, just nodepoold -d18:24
pabelangerrhedlind: if you add the switch, that will stop the builder process from spawning18:25
rhedlindpabelanger: don't I need it for snapshot images though?18:25
*** pilgrimstack has joined #openstack-infra18:25
clarkbfungi: not really other than it would be nice if libvirt was better about advertising appropriate line speeds18:25
pabelangerI should we did snapshots still in nodepoold18:26
clarkbbut virtio probably means more direct access to things rather than a fake full duplex 100Mbps interface18:26
clarkbgreghaynes: I have a build going now without a lot of infra's extra stuff to at least check my changes work properly and will see if I can reproduce with a reduced image spec18:27
greghaynesclarkb: sounds like fun ;)18:29
clarkbJournal size too big for filesystem.18:30
clarkbthats a nwe one18:30
greghaynestoo big!?18:30
*** persia has quit IRC18:30
clarkbthats what it says18:30
greghaynesI guess that means the journal is just plain larger than the fs?18:30
clarkblet me make a paste with more context18:30
*** tphummel has quit IRC18:31
greghayneswe do statically set a journal of some size IIRC, basically its a way to handle the fact that folks usually resize up the images so a larger journal is wanted18:31
*** persia has joined #openstack-infra18:32
clarkbI wonder if this is crashed due to staleness from my previous build that ran out of disk18:34
clarkblike loop0 is unhappy18:34
greghayneshrm, you can check losetup -a18:35
toskyhi, if anyone could approve this patch which temporarily allows for merges in sahara-tests by sahara-tests cores, we (from Sahara) would quickly finish a code import:
greghaynessee if a loop device is still mapped to an image.raw18:35
tosky( is the follow-up to disable that once the merges are done)18:36
*** tphummel has joined #openstack-infra18:36
clarkbI rebooted to clear out any temporary mounts or loopbacks that didn't get cleaned up18:36
greghaynesclarkb: yea, that reads like p1 is a crazy small partition18:36
clarkbbecause I am lazy and reboot is really fast when you don't wait for urandom18:36
*** hockeynut has joined #openstack-infra18:36
*** jheroux has quit IRC18:36
*** ayoung has quit IRC18:37
pabelangerrcarrillocruz: mordred: have time to review the cloud_launcher stack? Want to get that landed so I can add osic-cloud8 support18:39
pabelangerrcarrillocruz: mordred: I've tested it a few times and works18:40
fungidansmith: clarkb: the requested 99mbit htb and sfq have been applied to eth0 on mirror.nyj01.internap.o.o so let's see if further failures fitting this pattern emerge18:42
dansmithfungi: woot18:42
dansmithfungi: can you run that status command and pastebin it? just because I'm curious18:42
funginote this is temporary, but if we see positive results we should discuss puppeting it with known bw caps for all our mirrors18:42
clarkbwe should also consider increasing the bw if we are hitting the limits. Particularly at 100mbps18:43
*** mfedosin has quit IRC18:43
fungiclarkb: yep, i still think we should bump the flavor on the internap mirror regardless18:43
dansmithclarkb: we can use the backlog thing to tell how far oversubscribed we are.. it could be that the occasional peak is just disrupting enough to be a problem18:43
*** honza is now known as honza|afk18:43
pabelangerfungi: clarkb: I'm ready to do that work now, launching a new server18:44
*** sjmc7 has left #openstack-infra18:44
fungidansmith: yeah, in theory if we safely buffer through brief bursts then it may no longer have any adverse impact18:44
dansmithfungi: right, that's what I'm thinking, but we'll see if it pans out that way18:44
fungithe real risk is when your bursts are too sustained, too large, or too close together to be able to safely buffer18:45
fungibut even then, you ideally get the least disruptive requests sacrificed rather than having it obliterate a bunch of established flows18:45
*** yamahata has joined #openstack-infra18:46
fungiin my former isp/hosting life i spent way too much time tuning packet shaping rules because our owners were cheapskates who wanted to squeeze by on underprovisioned/oversubscribed hardware and links wherever possible18:46
fungiit was one of those "how far can you push it before the customers will notice" sorts of situations18:47
*** dtardivel has quit IRC18:47
*** ociuhandu has quit IRC18:47
jeblairfungi: the *remaining* customers, that is.  :)18:47
dansmithfungi: the fairness queuing is really good practice anyway, just to avoid one thing DoSing everything else18:48
fungiheck, we still had isdn customers in 2005, so some contracts you just can't shake18:48
dansmithyou want the DoSer to fail, not everything else18:48
clarkbgreghaynes: ok more info, that same du is used to set the image size :) and its 64MB which tells me I probably have a bug in my change \o/18:48
jkilpatrWho maintains the ansible linters gate?18:48
fungidansmith: yep18:48
greghaynesclarkb: in theory CI should have failed...18:48
fungiclarkb: or we're getting some awesome compression now18:49
*** rbrndt has joined #openstack-infra18:49
greghaynesok, I have to meeting for a bit18:50
*** jheroux has joined #openstack-infra18:52
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** kzaitsev_mb has joined #openstack-infra18:52
*** amitgandhinz has quit IRC18:53
fungiclarkb: dansmith: we actually missed an snmp sample at 17:50 so there is almost certainly something disturbing our network there (could of course have been a huge burst i guess)
dansmithfungi: spike on the graph below around the same time18:55
dansmithdifferent time scale hehe18:55
openstackgerritMerged openstack/os-client-config: Add support for configuring split-stack networks
fungisimilar missing sample in osic around 16:00
dansmithfungi: note the big increase in traffic right at the end of jul18:56
dansmithfungi: which would explain this becoming more of a problem18:56
fungidansmith: yeah, probably somebody repeatedly rebasing huge series of nova changes ;)18:56
dansmiththose damn nova people, I swear18:56
*** zz_ja has quit IRC18:57
fungiactually, i think the gap on the internap graph represents two missing samples back to back18:57
dansmithyeah looks like it18:58
jkilpatrpabelanger, hey what version of ansible-linters are you using? My project (browbeat) has a playbook failing that works fine when I install ansible-linters from pip18:58
*** _nadya_ has quit IRC18:58
clarkbmy vote is move forward with a B1.2 isntance with 5x the bw18:58
*** jtomasek has quit IRC18:59
clarkbjkilpatr: have a link to the log? I am hoping we use tox to run that linter...18:59
fungithough this provides a good experiment for how we might reduce the impact of brief spikes in providers where we know we're rate limited18:59
*** zz_ja has joined #openstack-infra18:59
*** jheroux has quit IRC19:00
clarkbjkilpatr: yup it uses tox -elinters and the version installed is so thats completely controlled by your project19:00
pabelangerjkilpatr: test-requirements.txt will have the version, but it is uncapped. So it will pull down the latest version19:00
*** _sarob has joined #openstack-infra19:01
*** sarob has quit IRC19:02
pabelangerclarkb: fungi: ya, using B1.219:02
jkilpatrpabelanger, clarkb ok same version local and remote, "find . -type f -regex '.*.y[a]?ml' -print0 | xargs -t -n1 -0 ansible-lint" runs with some warnings but exits successfully when I run it locally. Which seems to be all tox does19:02
clarkbjkilpatr: have you tried running tox to reproduce?19:03
pabelangertox -elinters should give the same results at the gate19:03
clarkbor -re linters if you have run it before (this will start you with a clean venv)19:04
fungimodulo any missing packages or constraints applied19:04
jkilpatrclarkb, guess that's the next step. I was hoping it was just linters. Down the rabbit hole I go19:04
openstackgerritMonty Taylor proposed openstack-infra/shade: Stop getting extra flavor specs where they're useless
openstackgerritMonty Taylor proposed openstack-infra/shade: Allow image and flavor by name for create_server
mordredok. those should work now19:05
*** fifieldt has quit IRC19:06
openstackgerritMerged openstack-infra/jenkins-job-builder: builders: add 'publish over ssh' support as a build step
*** amitgandhinz has joined #openstack-infra19:08
*** kzaitsev_mb has quit IRC19:08
*** kzaitsev_mb has joined #openstack-infra19:10
* clarkb is really confused by dib and has reverted the new patch and is trying the last release to convince himself that dib works :P19:10
clarkband of course it does work19:11
*** julim has quit IRC19:13
*** jheroux has joined #openstack-infra19:14
*** julim has joined #openstack-infra19:14
*** timello has quit IRC19:15
*** salv-orl_ has joined #openstack-infra19:15
*** fifieldt has joined #openstack-infra19:18
*** salv-orlando has quit IRC19:18
*** vhosakot has quit IRC19:18
*** vhosakot has joined #openstack-infra19:18
pabelangerclarkb: looks like we may need to update, internap is complaining:
pabelangerclarkb: I guess that mean, when I create a new volume, it already has a FS?19:21
*** timello has joined #openstack-infra19:22
*** Na3iL has quit IRC19:22
*** kzaitsev_mb has quit IRC19:22
clarkbpabelanger: the problem appears to be that is grabbing /dev/vdb before the volume mount gets it19:23
*** vhosakot has quit IRC19:23
cineramagreghaynes: we're meeting today, yes?19:24
*** vhosakot has joined #openstack-infra19:24
pabelangerclarkb: let me see why that is19:24
*** kszukielojc has quit IRC19:25
clarkbpabelanger: looks like make_swap just assumes it can use /dev/vdb19:25
*** markusry has quit IRC19:26
pabelangerclarkb: ya19:27
*** kzaitsev_mb has joined #openstack-infra19:27
pabelangerI don't think shade wait for the volume to be mounted19:27
pabelangerso we might be racing19:27
*** Hal has quit IRC19:28
clarkbpabelanger: I don't think shade will but launch node should19:29
*** _nadya_ has joined #openstack-infra19:29
clarkbpabelanger: but I think /dev/vdb is the volume we have just conflicted our two disk management scripts19:29
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-logstash: Make the java heap size configurable
pabelangerclarkb: Ah, you are right19:31
*** matt-borland has quit IRC19:31
pabelangerwe mount the volume in launch-node19:31
pabelangererr, attach19:31
clarkbpabelanger: so probably the work around here is to just remove the /dev/vdb check from make_swap.sh19:33
*** dgonzalez has quit IRC19:34
*** matt-borland has joined #openstack-infra19:34
*** kzaitsev_mb has quit IRC19:34
*** e0ne has joined #openstack-infra19:34
pabelangerclarkb: okay, trying that now19:35
*** _nadya_ has quit IRC19:37
zarozxiiro: are you by chance using the gerrit trigger plugin?19:38
*** esikachev has quit IRC19:38
*** nwkarsten has quit IRC19:38
*** kzaitsev_mb has joined #openstack-infra19:39
*** nwkarsten has joined #openstack-infra19:39
*** nwkarste_ has joined #openstack-infra19:40
*** florianf has quit IRC19:41
rcarrillocruzpabelanger: sorry, was out19:42
rcarrillocruzsorted out that change i see19:42
rcarrillocruzcould you pls review and ?19:42
rcarrillocruzshould be quick19:42
rcarrillocruzconf changes19:42
*** nwkarsten has quit IRC19:44
clarkbpabelanger: I think the underlying issue with that make_swap and make_volume interaction is that ephemeral disks (where we want to put swap and /opt) and volumes can share the same /dev device names. And not only that but they can be different in different clouds :(19:44
*** ociuhandu has joined #openstack-infra19:45
*** dgonzalez has joined #openstack-infra19:45
pabelangerclarkb: ya, I'm going to just bring up the server first19:46
pabelangersee what is happening, as I am blind right now19:47
clarkbI am not sure that nova enforces a naming scheme across hypervisors19:47
clarkbone thing low on my list is to switch to using the uuid based lable stuff with cinder volumes as its more reliable to map a volume to its device in linux19:47
pabelangerrcarrillocruz: just need a +2 on that to land the stack19:47
clarkbits possible that ephemeral devices show up in a more readily identifiable way like that too19:48
*** rbuzatu has quit IRC19:48
*** piet has quit IRC19:48
mtreinishclarkb: does look sane to you. jesusaur and I were looking at it before, but there were some open questions (which I don't remember)19:48
rcarrillocruzit would be good if in the mid-cycle we documented the various differences on how our providers show the cinder volumes19:48
*** kzaitsev_mb has quit IRC19:49
rcarrillocruzas i want to put that make swap thing on the launcher playbook19:49
jesusaurmtreinish: clarkb: iirc, the discussion before had to do with how to tell if the plugin is already installed to avoid running the install command on every puppet run19:51
clarkbmtreinish: we are trying to move away from alll that overparameterized stuff and just pass in fiels using their conf.d setup19:51
clarkbmtreinish: jesusaur so I don't think we would want to do it that way and instead just update our 99-output appropriately19:51
clarkbmtreinish: jesusaur then have an install_plugin manifest that the consumers of logstash module can use to install the necessary plugins19:52
clarkbso that looks like define logstash::install_plugin() in puppet-logstash19:52
clarkbthen in system-config use that to install mqtt plugin and edit the output file to use it19:52
jesusaurclarkb: ++19:53
*** Hal has joined #openstack-infra19:53
jesusauri had forgotten about the new conf.d setup19:53
mtreinishheh, I didn't even know that was a thing19:54
*** xarses has joined #openstack-infra19:54
openstackgerritJeremy Stanley proposed openstack-infra/puppet-askbot: Add explicit configuration support for Akismet
fungicrinkle: thanks! ^ updated per your comment19:55
zxiirozaro: yes we still use it heavily. Zuul doesn't do what we want so we haven't moved to it yet19:55
*** andrey-mp has joined #openstack-infra19:57
*** Apoorva_ has joined #openstack-infra19:59
andrey-mpHi! Is it right channel to ask about IRC? I need to add log saving for openstack-ec2api channel... Are there instructions about this?20:00
clarkbandrey-mp: should cover the details for that20:01
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute010.vanilla IP addresses and enabling it as it is functional
fungiclarkb beat me to the draw20:01
andrey-mpthank you!20:01
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute007 IP addresses
*** Apoorva has quit IRC20:03
fungiandrey-mp: what it doesn't mention there is that after dealing with the instructions in the access section, you need to add your channel to
fungito get channel logging20:03
andrey-mpfungi: thank you! this is that i looking for20:04
pabelangerclarkb: I did a 2-step process for the volume20:04
pabelangerworked with out an issue20:04
pabelangerI'll kick over the dns here in a few minutes20:05
*** _ari_ has quit IRC20:05
zarozxiiro: i'm trying out gerrit-trigger ver 2.22.1 with core 1.609.3 but i can't seem to make it work at all. which version are you using?20:05
crinklefungi: yay20:07
openstackgerritMerged openstack-infra/project-config: Add glue to get initialize-urandom installed
pabelanger#status log replacement server now online, DNS has been updated to
openstackstatuspabelanger: finished logging20:08
*** ayoung has joined #openstack-infra20:09
fungithanks pabelanger!20:09
pabelangerfungi: no problem20:09
clarkbpabelanger: thanks and ya two step is ok for things that dont immediately try to usr the volume like es20:09
clarkbes is tricky because we have to make aure volume is mounted priper before puppet runs20:09
pabelangerI'll check back in 60mins to confirm traffic has stopped on the original server20:09
clarkbotherwise sadness20:10
zxiirozaro: i got jenkins core 1.651.3 and GT 2.21.120:10
zxiirozaro: what isn't working?20:10
pabelangerclarkb: ya, it did work for the other mirror we did, so 50% right now20:10
fungipabelanger: since you seem to be really into a mirror maintenance headspace at the moment, any thoughts on what we should do about the absurd amounts of apache logs we accumulate, for example on the current osic mirror?20:10
zxiirozaro: the job's not passing in the GERRIT_* properties?20:10
*** gyee has joined #openstack-infra20:11
fungipabelanger: if we don't do something about it, we're on track to fill up the rootfs there with apache access logs sometime today or early tomorrow20:11
pabelangerjust seen20:12
fungii mean, i can delete some old logs, compress/rotate, whatever, but unless we scale back on either the log volume or retention we are going to be right back here in short order20:12
pabelangerfungi: at first glance, maybe more the 7 day log retention?20:12
openstackgerritSean Dague proposed openstack-infra/devstack-gate: Change default for FORCE_CONFIG_DRIVE to False
fungithat seems reasonable to me. i could even be convinced to scale back to 4 days access log retention (and maybe keep or increase the current error log retention)20:13
*** javeriak has quit IRC20:13
*** kzaitsev_mb has joined #openstack-infra20:13
pabelangerlet me see how it works today20:13
pabelangerya, it looks to be unmanaged right now20:16
pabelangerthe default is rotate 5220:16
*** andrey-mp has left #openstack-infra20:16
openstackgerritSean Dague proposed openstack-infra/project-config: Drop postgresql jobs in gate
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs.
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser.
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules.
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings.
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Make builder.Jenkins inherit from jenkins.Jenkins
*** bhunter71 has quit IRC20:19
*** psachin has joined #openstack-infra20:20
*** _coolsvap_ has quit IRC20:21
dansmithfungi: sdague: the one failure on internap recently was not network-related, FYI if you go back to look20:23
sdaguedansmith: ok, cool20:23
fungidansmith: excellent20:23
sdaguefungi: it does look like by about 10am it self corrected on internap20:23
*** afred312_ has joined #openstack-infra20:23
sdaguethe failure window is 1am - 10am EST20:23
fungidansmith: sdague: keep in mind as of a few minutes ago that mirror has been replaced with an instance using a flavor with a much higher bandwidth cap20:24
*** esikachev has joined #openstack-infra20:24
sdaguehowever, anything that prevents future fails would be great20:24
dansmithfungi: gah20:24
*** BigWillie has quit IRC20:24
*** kzaitsev_mb has quit IRC20:24
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added npm-docs draft build for eslint-config-openstack.
*** afred312 has quit IRC20:24
fungiso if the issues were related to rate limiting, they should cease unless we increase activity 5x over20:24
*** kgiusti has left #openstack-infra20:25
jeblairfungi: so good until next thursday?20:25
fungijeblair: bingo!20:25
prometheanfireso... ?20:26
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Created DSVM Job for NPM Projects
openstackgerritPaul Belanger proposed openstack-infra/system-config: Update logrotate for mirror server
pabelangerfungi: ^ that should give us 7 day retention20:28
jkilpatrclarkb, where can I get the tox.ini the ansible-linters test is using? its not the same as the one in the browbeat repo20:29
pabelangerclarkb: so we need to restart nodepool today to might up your patch?20:32
pabelangerjkilpatr: it should be20:32
*** xarses_ has joined #openstack-infra20:33
jkilpatrpabelanger, ignore me, need to rebase this branch, old repo sorry20:33
pabelangerjkilpatr: Ya, I just added it this week20:34
*** baoli has quit IRC20:34
clarkbpabelanger: ya but only the builder20:34
openstackgerritMerged openstack-infra/puppet-infracloud: Add vlan_id variable with value 2551
*** Jeffrey4l has quit IRC20:35
*** Jeffrey4l has joined #openstack-infra20:35
*** jcoufal_ has joined #openstack-infra20:35
fungigotta go grab some dinner, will return to continue reviewing wiki puppet in a couple hours20:35
*** xarses has quit IRC20:36
pabelangerclarkb: okay, cool Was going to say, if we need to restart nodepoold, we'd pull in a few things, like new shade20:36
*** TravT has joined #openstack-infra20:36
zarozxiiro: can't get it to trigger or report anything back to gerrit.20:37
TravThello. i'm wondering if I can get some help with approving marking a Searchlight job as non-votiing20:37
*** jcoufal has quit IRC20:37
zarozxiiro: maybe i'm setting up wrong?  i configured the server, setup a job, and added a trigger for comment-added, change-abandoned and change-restored. i see the events on gerrit side but nothing happens on jenkins side20:38
zxiirozaro: my first guess if it's not trigger might be but I think you're on an old enough version taht you're not affected20:40
*** vhosakot has quit IRC20:41
openstackgerritMerged openstack-infra/nodepool: Add an example logging.conf for development
zxiirozaro: just in case you have a bad job config you can compare with this
zarozxiiro: thanks.  will try with 1.651.3 now20:43
*** _nadya_ has joined #openstack-infra20:43
openstackgerritMerged openstack-infra/nodepool: Add a fake-secure.conf
*** baoli has joined #openstack-infra20:44
*** armax has joined #openstack-infra20:45
*** rbuzatu has joined #openstack-infra20:48
*** kzaitsev_mb has joined #openstack-infra20:48
pabelangerclarkb: going to enqueue 357393 into gate, so we can start up on xenail builds20:49
*** ansmith has quit IRC20:49
pabelangerit still hasn't passed check yet20:49
clarkbpabelanger: ok20:49
clarkbI am still fighting with dib to get a working bug fix20:50
clarkbI am learning much about bash20:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Set compute020.vanilla IP addresses
*** vhosakot has joined #openstack-infra20:52
*** rbuzatu has quit IRC20:52
ianwclarkb: did you see the strace output for the hack i put in?  i'm not sure it corresponds to the null problem20:53
ianwi'm just looking now20:53
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Fix disabled always returning true
ianw3879  write(1, "ib_tmp/dib_build.y8yDOp9f/built/"..., 11127) = -1 EAGAIN (Resource temporarily unavailable)20:53
ianwthat is from "cat"20:54
*** _nadya_ has quit IRC20:54
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Have buildZooKeeperHosts accept a config object
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a script to print the ZK tree
ianwclarkb: have a look in /opt/dib_tmp/dib_build.y8yDOp9f/du_output.tmp.strace20:55
*** SumitNaiksatam has quit IRC20:55
clarkbianw: I didn't20:56
ianwclarkb: my concern is that *maybe* this is more about a large amount of output and the way nodepool is capturing it20:56
openstackgerritMerged openstack-infra/nodepool: Stop running DIB usage report
*** spzala has quit IRC20:56
clarkbianw: the large amount of output from du? I am definitely struggling with that when trying to debug locally I can't export du_output for example20:56
ianwclarkb: so yeah, the du_output as a var was a bad choice, it's like 100mb20:56
ianwclarkb: but i proposed
ianwclarkb: but /usr/local/bin/disk-image-create on nodepool itself currently has a little hack i put in last night my time20:57
clarkbianw: cool I also think we could/should just do du -h | sort -h20:57
ianwclarkb: have a look at it, search for #ianw20:57
*** kzaitsev_mb has quit IRC20:57
armaxmordred: is this ready to roll?
clarkbianw: and avoid numfmt entirely20:58
ianwclarkb: "luckily" dib doesn't clean up properly and left behind -> /opt/dib_tmp/dib_build.y8yDOp9f/ where you can see each of the steps20:58
*** gouthamr-afk is now known as gouthamr20:58
ianwclarkb: the problem is, it still happens if you output numfmt to a file, then just "cat" that file20:59
*** raildo has quit IRC20:59
clarkbah so its the output of the data itself not in the processing20:59
mordredarmax: oh - funny story, I was just poking you in #openstack-neutron20:59
ianwclarkb: yep20:59
armaxmordred: lol20:59
armaxmordred: power of preemption21:00
clarkbianw: and likely only reproduceable when you have 1366 git repos full of files as part of the du21:00
armaxmordred: I just happened to look at my email, sheer coincendence21:00
ianwclarkb: which is yeah, why i wonder if we're in full pipes in python subprocess land, or something ...21:00
ianwclarkb: but why only xenial?  i mean it's been working fine for ages21:00
ianwthat was where i got to last night :)21:00
zarozxiiro: still not working.  did you need to do the workaround for 1.651.3?21:01
*** dprince has quit IRC21:02
clarkbianw: maybe as want to restrict the depth that du lists too?21:02
zxiirozaro nope with that version you done need a workaround.21:02
zxiirozaro actually I just remembered something21:03
ianwclarkb: well, we could do that.  i think my concern more is that if dib can kill itself by outputing a large amount of text, that's going to bite us somewhere else sooner or  later21:03
zxiirozaro: don't use the comment added and add a regex trigger just to test. see if you can make it trigger with regex21:03
clarkbianw: yup, though this could be specific to fd 1 (stdout) being a pipe attached to python21:03
*** julim has quit IRC21:04
zxiirozaro if that works it means the comment added trigger needs tweaking. if I recall the default categories don't work out of the box21:04
ianwclarkb: yeah, i'm almost certain it is :)21:04
*** e0ne has quit IRC21:04
*** beisner is now known as beisner-biab21:05
ianwclarkb: i wonder if the thread running the subprocess doesn't collect it's output into the log file for a while, how much pipe buffer we have there21:06
zarozxiiro: i already have comment added trigger setup to 'recheck' same as your jjb build job. nothing is triggering21:06
zarozxiiro: actually i've added patchset-created, comment-added, abandoned, and restored all setup.21:07
clarkbianw: it does while True: process.stdout.readline(); logline21:07
*** kzaitsev_mb has joined #openstack-infra21:08
*** Apoorva_ has quit IRC21:08
clarkbianw: and it reads until it gets empty string then does a process.wait()21:08
clarkbianw: is it possible it is somehow reading an empty string, stopping to read off the buffer and just waits until the subprocess crashes?21:08
*** Apoorva has joined #openstack-infra21:08
*** yamamoto has joined #openstack-infra21:09
*** yamamoto has quit IRC21:09
ianwclarkb: right, but if that loop doesn't get scheduled for a while, the data is backing up21:09
ianwfrom the strace, cat get EAGAIN21:09
*** aeng has joined #openstack-infra21:09
zarozxiiro: should jobs report back to gerrit if you just manually trigger the jobs?21:10
clarkbthere shouldn't be much thread contention there unless we are overlapping uplaods and builds (which weshouldn't be doing as it is build first then upload, I guess if uplaods continue into then ext day maybe we would see that)21:10
ianwclarkb: yeah, and it was totally repeatable last night when it was quiet, so i admit it's a long shot :)21:11
*** jkilpatr has quit IRC21:11
*** hockeynut has quit IRC21:12
clarkbpabelanger: the change to nodepool to stop setting the flag for the du is merged and installed. We do have a few image uplaods running not sure if you want to wait for those to finish or just go ahead and restart the builder and do xeniakl21:12
greghaynesianw: huh, does cat not handle EAGAIN?21:15
*** kzaitsev_mb has quit IRC21:16
*** thorst_ has quit IRC21:16
ianwgreghaynes: here's the strace of cat ->
clarkbI wonder if this is related to the python subprocess warning to use communicate instead of stdin/stdout/ to avoid deadlock due to fuilling buffers21:16
*** ldnunes has quit IRC21:16
greghaynesianw: looks like a no, then21:16
clarkbthough thats deadlock not eagain21:16
greghaynesso yea, looks like just full buffers21:16
greghayneswe can write du out to a tempfile and work on that?21:17
clarkbgreghaynes: thats what the cat was for, it was cating the tempfile at the end to put the data in the dib output21:17
greghaynesoh, hah it just cat'd so much?21:17
clarkbya that is what ianw is saying21:18
*** matt-borland has quit IRC21:18
clarkbbasically nodepool can't keep up with dib when it writes this data out and ofr some reason only affects the xenial builds21:18
ianwgreghaynes: yeah, sorry i was debugging it last night, i put some local edits into our disk-image-create on nodepool21:18
greghaynesclearly this isnt ready for the internet if it cant handle that many cats21:18
jesusaurcats have clogged the pipes? sounds like the internet to me...21:19
ianwgreghaynes: i was thinking the same thing, bad formatting, but i put in
pabelangerclarkb: Ya, lets restart and get ubuntu-xenial into osic-cloud1. That gives us more resources21:20
ianwand the final cat was still failing21:20
pabelangerclarkb: then we can land cloudnull change to bump max-servers21:20
clarkbpabelanger: we will also need mordreds fix for the private IP in shade/nodepool I think21:21
*** pilgrimstack has quit IRC21:21
clarkbpabelanger: so that multinode continues to work21:21
pabelangerclarkb: I thought the ip6tables was the fix for that?21:21
pabelangeror did I miss something21:21
clarkbpabelanger: it is but no new image means no fix in21:21
clarkbwell its part of the fix21:21
greghaynesianw: that was failing when run under nodepool?21:21
pabelangerclarkb: okay, we should pull that in once we build ubuntu-xenial21:22
ianwgreghaynes: yes -- only when run under nodepool.  run those commands by hand and it works21:22
pabelangerwe can also to ubuntu-trusty if needed21:22
ianw"The data read is buffered in memory, so do not use this method if the data size is large or unlimited."  <--- "large" ... that's helpful21:22
clarkbpabelanger: the new concern is that once that ip6tables fix is in then we will have ready nodes for multinode in osic and then ovs will fail to make the vxlan connections over the private network because ipv621:22
pabelangerclarkb: ah, I see what you are saying21:23
clarkbpabelanger: we can avoid that by removing osic from the 2-node labels21:23
greghaynesianw: I wonder if we should bash loop over it21:23
clarkbpabelanger: so thats a simple fix21:23
*** Goneri has joined #openstack-infra21:23
greghaynesianw: basically read echo loop21:23
pabelangerclarkb: sure, we can do that21:23
ianwgreghaynes: i think we need to fix it in nodepool, because although this triggers it, it's just going to be something else flooding output that kills it next time21:24
pabelangerclarkb: has mordred done the fix and we just need a new release of shade?21:24
greghaynesianw: yea, fair21:24
*** baoli has quit IRC21:24
mordredpabelanger: I'm still working on that fix21:24
clarkbpabelanger: looks like it was an occ fix and it merged so we just need a release?21:25
clarkbmordred: ^ is that all thats left? or do you need to bubble things up through shade and nodepool now?21:25
mordredpatch for shade coming soon21:25
*** baoli has joined #openstack-infra21:25
*** bhunter71 has joined #openstack-infra21:25
mordrednodepool shold not need any patches21:25
*** edmondsw has quit IRC21:25
mordredand I'm 3/4 of the way done with the shade patch21:25
*** mriedem has left #openstack-infra21:25
cloudnullclarkb mordred pabelanger: to unblock multi-node gates, can a tenant network be used as a secondary nic allowing the OVS magic to happen on that device?21:25
clarkbcloudnull: only if shade will report that as the private IP properly21:26
*** mriedem has joined #openstack-infra21:26
* cloudnull has no idea how the multi-node gate is built21:26
mordredI think just waiting for this patch is likely better21:26
mordredit's not a big one21:26
clarkbcloudnull: the existing instances gets a private ipv4 and public ipv6 already but list the public ipv6 as the private network which won't work because we need private v421:26
mordredI just need to finish it, which I will do as soon asI stop runnin gmy mouth here21:26
clarkbcloudnull: so as long as the ipv4 addrs there are routable between osic isntances we will be fine with mordred's fix21:27
cloudnullah ok. makes sense21:27
*** beisner-biab is now known as beisner21:28
clarkbcloudnull: the v4 addrs we do get should be routable between instances ya?21:28
cloudnullyes. they should be.21:29
* cloudnull goes to make sure.21:29
clarkbnone of this would be an issue if we could just make tunnels over v621:29
ianwclarkb / greghaynes: i wonder if is involved ...21:29
*** kzaitsev_mb has joined #openstack-infra21:30
cloudnullif shade can detect the v4 addr you should be good to go21:30
clarkbcloudnull: ty ty21:31
clarkbianw: heh resolution says fixed but they clearly did not fix it :/21:31
*** jheroux has quit IRC21:31
*** markusry has joined #openstack-infra21:32
greghaynesianw: hrm, that could cause the full buffers if something else is killing the fd21:32
greghaynesalthough wed still need to know what is killing the fd21:33
*** jcoufal_ has quit IRC21:34
*** baoli has quit IRC21:35
*** annegentle has quit IRC21:36
*** annegentle has joined #openstack-infra21:37
greghaynesianw: so for 'fixing in nodepool', we could change the stdout=PIPE to write out to a file and then read that in later?21:40
greghaynesor write out to a fifo21:40
*** psachin has quit IRC21:44
clarkbpabelanger: ok want me to go ahead and restart the nodepool builder and start a xenial build?21:45
*** pradk has quit IRC21:45
*** furlongm has quit IRC21:45
*** furlongm has joined #openstack-infra21:46
*** valderrv has joined #openstack-infra21:46
mtreinishheh, mutt 1.7.0 was installed in my daily package upgrade. But, there aren't release notes or anything posted on the official website yet. (just a tarball on the ftp server this morning)21:46
Krenairfungi, pleia2: so... gate-infra-puppet-apply-ubuntu-precise failed despite succeeding earlier?21:46
ianwgreghaynes: we want it streaming ... i think that would be the same as using communicate() and just dumping the resulting output21:46
* mtreinish is trying to figure out everything that's changed21:47
*** annegentle has quit IRC21:48
openstackgerritMerged openstack-infra/puppet-snmpd: Remove initscript
*** kzaitsev_mb has quit IRC21:52
*** xarses_ has quit IRC21:53
greghaynesianw: yea, the fifo would be equivalent. I'm actually pretty surprised that this is an issue... all the python process should be doing is reading from that one fd21:54
greghaynesand then theres the one logging thread its feeding to21:54
*** vhosakot has quit IRC21:54
clarkband the gearman thread21:54
greghaynesyea theres other threads but if ianw did a one off test they all should be relatively quiet21:54
*** jerryz has quit IRC21:55
*** annegentle has joined #openstack-infra21:55
clarkbwell I don't hear any objects so I am going to go ahead and restart the nodepool builder to get the code change then build xenial and upload it21:55
clarkbor oh wait maybe pabelanger already did that?21:56
greghaynesianw: I'm having a hard time coming up with a good way to prove that the read/log loop is what is clogging up the works21:56
clarkboh cool its running21:56
greghaynesianw: one simple fix might be to change that stdout.readline() to a read(), although I think that would be minor21:57
*** amitgandhinz has quit IRC21:57
*** kzaitsev_mb has joined #openstack-infra21:58
*** mriedem has quit IRC21:58
*** amitgandhinz has joined #openstack-infra21:58
pabelangerclarkb: ya, I kicked it just after nodepool updated21:58
clarkbpabelanger: awesome21:58
clarkbI am going to push a change to remove osic from the 2-node labels as a precaution21:59
clarkbthen when we get shade and nodepool updated we can add osic back to those labels21:59
openstackgerritPaul Belanger proposed openstack-infra/system-config: Update logrotate for mirror server
*** vinaypotluri has quit IRC22:01
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add Neutron experimental job for scenario tests running on LinuxBridge
*** nwkarste_ has quit IRC22:01
*** tbarron is now known as tbarron|afk22:02
*** nwkarsten has joined #openstack-infra22:02
*** gordc has quit IRC22:02
*** annegentle has quit IRC22:03
openstackgerritClark Boylan proposed openstack-infra/project-config: Remove OSIC from 2-node label types
clarkbpabelanger: ^ thats the precaution chnage22:05
*** thorst_ has joined #openstack-infra22:06
*** nwkarsten has quit IRC22:07
pabelangerclarkb: great22:07
*** vinaypotluri has joined #openstack-infra22:07
ianwgreghaynes: yeah, i don't know at this point.  i might try to write a small thing to try repro22:07
pabelangeralso looks like the original server has stopped getting traffic22:07
pabelangergoing to delete it22:08
*** kzaitsev_mb has quit IRC22:09
*** burgerk has quit IRC22:10
*** thorst_ has quit IRC22:10
*** thorst_ has joined #openstack-infra22:14
*** xyang1 has quit IRC22:14
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case
mordredcloudnull: congratulations - you have your own unittest in shade now :)22:15
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Do not add c-api,c-bak on subnode with Ironic
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** thiagop has quit IRC22:16
*** hpe-hj has quit IRC22:16
*** esberglu has quit IRC22:16
openstackgerritMerged openstack-infra/system-config: Also mirror source packages for debian / ubuntu
*** tpsilva has quit IRC22:18
*** mdrabe has quit IRC22:19
*** julim has joined #openstack-infra22:19
*** Goneri has quit IRC22:20
zarozxiiro: i configured it out.  i neglected to configure the "Gerrit project" section.  I just assumed that it would trigger on any project but you must specify a specific one. Also that config (from UI view) was offset to make it look like it's a subset of the Dynamic trigger config.22:23
zxiirozaro: ah right. You can make it for any project by setting your project to be "**" with the ANT pattern22:24
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
mordredclarkb, cloudnull, Shrews: ^^22:24
*** asettle has joined #openstack-infra22:24
mordredthat will not pass tests until the new occ is cut22:24
mordredbut there's the code22:25
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework
clarkbmordred: can you review while we wait for that to happen?22:25
clarkbmordred: will prevent job breakage in the interim22:25
mordredclarkb: +A22:25
*** nwkarsten has joined #openstack-infra22:26
*** erlon has quit IRC22:27
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Remove Neutron Postgres job from the Neutron check queue
*** nwkarsten has quit IRC22:30
*** thorst_ has quit IRC22:31
*** asettle has quit IRC22:32
*** thorst_ has joined #openstack-infra22:32
openstackgerritMonty Taylor proposed openstack/os-client-config: Add network entries for OSIC
pabelangerclarkb: going to be a while for ubuntu-xenial image, slow git clones22:32
pabelangeron nodepool.o.o22:32
mordredclarkb: ^^ actually, we should land that to22:32
mordredclarkb: ^^ actually, we should land that too22:32
clarkbpabelanger: ya I was noticing thati n the log tail22:33
pabelangermordred: What happens when osic-cloud8 comes online? Do the profiles in os-client-config need to change?22:34
mordredwell... at that point I should probably add an osic8 profile (and probably should have called this one osic1, but hindsight)22:34
mordredpabelanger: we can also just put the details directly into our clouds.yaml file22:34
clarkbya I am thinking that might actually be ebtter given what I now know of osics make many clouds model22:35
mordredk. I'll abandon that patch then22:35
clarkbocc probably doesn't need to track all of the variations. AIUI there are rax private cloud clouds, master built clouds, $otherbranch clouds, etc22:35
pabelangermordred: ya, that's what I did for nodepool clouds.yaml22:37
mordred <-- clarkb cloudnull pabelanger that's a server entry from osic with the shade and occ patches applied22:38
mordredyou should note that public_v4 is empty, private_v4 has an address and public_v6 has an address22:39
openstackgerritMerged openstack-infra/system-config: Set compute011.vanilla IP addresses
pabelangerlooks good22:39
mordredhrm. although interface_ip is ... ah - my local machine does not have ipv6 - so yes, interface_ip is correct to be blank :)22:39
*** thorst_ has quit IRC22:40
fungiKrenair: sounds like a nondeterministic/false negative. maybe a race condition in a test? i saw one earlier on another change of mine... i'll dig into the logs22:40
mordredby forcing the local ipv6 flag, I ran the above again and got an ipv6 in the interface_ip22:41
Krenairi've been seeing this with a tests around designate too. everything's fine until someone approves the change, the jenkins becomes unhappy :(22:41
*** yamamoto has joined #openstack-infra22:42
fungiKrenair: oh, on, what's the deal with the lines starting with "?" in the settings file? i was trying to find a php language reference to explain that syntax... seems similar to a trinary operator but that would be "?:" so i'm still confused22:43
Krenairfungi, there is no ? in that diff22:45
*** kzaitsev_mb has joined #openstack-infra22:45
fungiKrenair: weird... maybe gertty is misrepresenting the diff. i'll check the gerrit webui for comparison22:46
Krenairsure you're not looking at a tab character or something?22:46
*** rlandy has quit IRC22:46
fungiKrenair: oh! it is indeed replacing a hard tab with a ? for some reason...22:47
*** tonytan4ever has quit IRC22:47
fungijeblair: ^ not sure if that's intentional behavior in gertty or not?22:47
mtreinishfungi: heh, I've had unicode chars display as ? in my terminal in the past22:48
Krenairyeah we use tabs for indentation in mediawiki22:48
fungimtreinish: yep, if you use the replace resolution to ascii in python it definitely does that, but i've not seen it happen for hard tabs22:49
mtreinishlike I remember a very confusing experience where cdent used a unicode check mark in channel and it was a question mark on my end22:49
jeblairfungi: it is an unintentional behavior in gertty ... for some reason it doesn't come up often for us :)22:49
mordredjeblair: :)22:50
*** rbuzatu has joined #openstack-infra22:50
fungijeblair: yeah, mostly just surprised me, since i thought it was showing addition of a literal "?" character instead of it being a placeholder for another byte22:50
jeblairwe can totally fix it22:51
fungiacross teams, we (infra) actually have a higher volume of files with hard tabs in them than most, i expect22:51
jeblairme may have to figure out how many spaces equal a tab.22:51
*** pahuang has joined #openstack-infra22:51
jeblairprepare the bikeshed!22:51
fungiclearly the answer is "3"22:52
fungi(note, god uses three-space tabs)22:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add dual stack network config for OSIC
mordredpabelanger, clarkb: ^^ there's the clouds.yaml changes for us22:52
*** rbrndt has quit IRC22:53
*** rbrndt has joined #openstack-infra22:53
*** rbrndt has quit IRC22:54
pabelangermordred: thanks +222:54
*** ccamacho1 has joined #openstack-infra22:54
*** rbuzatu has quit IRC22:54
*** vikasc has quit IRC22:54
*** beagles is now known as beagles_biaw22:56
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144
*** vikasc has joined #openstack-infra22:59
*** Apoorva_ has joined #openstack-infra23:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add dual stack network config for OSIC
fungiKrenair: as for the gate-infra-puppet-apply-ubuntu-precise failure i think we hit a job timeout23:02
mordredpabelanger: ^^ just made an update - I tested that file with existing release of shade/occ23:02
mordredand we need to add a thing to not confused old code23:02
fungiKrenair: i've rechecked the failed change, but am digging deeper on why our apply jobs for precise have started doing this very recently23:03
*** Apoorva has quit IRC23:03
fungichances are this is tied into the ipv6 privacy addresses issue in osic23:04
fungithe job ran there, and definitely had an additional global v6 address according to the network diagnostics at the start of the console log, and was unable to traceroute6 to git.o.o23:05
fungiwe _suspect_ that having privacy addresses enabled is resulting in performance issues with git remote operations from osic nodes23:06
*** dimtruck is now known as zz_dimtruck23:06
fungiand our puppet apply jobs do a lot of remote git operations because of needing to clone all our puppet modules23:07
*** sshnaidm is now known as sshnaidm|afk23:07
*** hongbin has quit IRC23:09
fungiyeah, console log shows it updating repos quickly up until 21:14:01 when it started to work on puppe23:11
fungiand took it until 21:17:11 just to check the branches on the origin23:11
fungiand until 21:18:14 to decide what branch to use23:12
*** sdake_ has joined #openstack-infra23:12
fungiand then puppet-mysql_backup similarly took $4 minutes23:12
fungier, ~4 minutes23:12
fungiyeah, each repo past there took 4 minutes until the job was prematurely terminated for reaching our job timeout23:13
*** sdake has quit IRC23:13
clarkbfungi: fwiw the xenial image build is being very slow to update repos as well23:13
*** sdague has quit IRC23:14
*** ccamacho1 has quit IRC23:14
fungithis sounds consistent with the "4 minute" git remote times sdague was citing earlier23:15
clarkbdib ins't being that slow23:16
clarkbso I guess thats a good thing23:16
fungiclarkb: what's our egress bw cap supposed to be on git.o.o eth0?23:16
*** jordanP has quit IRC23:16
fungithe daily graph shows 5-minute samples reaching 500.89mbps23:17
openstackgerritMerged openstack-infra/system-config: Set compute013.vanilla IP addresses
clarkbfungi: the rxtx factor is 2500 so I think 1.25Gbe23:17
*** armax has quit IRC23:17
clarkber 1250mbps23:17
fungiokay, so still well below 50%23:17
*** chlong has quit IRC23:20
openstackgerritMerged openstack-infra/system-config: Set compute014.vanilla IP addresses
fungiindividual git servers don't seem to be spiking above 1-minute averages of 70mbps23:21
fungijust checked all 8 of them23:21
fungiso distribution is remarkably even23:21
*** fguillot has joined #openstack-infra23:26
openstackgerritMerged openstack-infra/puppet-infracloud: Remove diskimage-builder repo elements folder from ELEMENTS_PATH
fungiwow, we've been hovering arounf 1.5kjps today according to the graph23:29
fungier, around23:29
fungier, and 1.5kjph23:30
mgagnekilojoules per second23:30
fungikilojobs per hour is what i meant ;)23:30
mgagnethat's a lot of energy :D23:30
*** thorst_ has joined #openstack-infra23:31
fungiprobably uses a lot of energy23:31
mgagnethat's impressive still23:31
*** tosky has quit IRC23:31
*** rcernin has quit IRC23:32
*** adriant has joined #openstack-infra23:34
sridhar_raminfra-team: I'm looking for help in skipping a gate job in a specific branch (stable/mitaka). This is related to where relnotes jobs is now running but we didn't start adding relnotes (and support that in tox.ini) until newton (in the current master). How can skip relnotes job for tacker in stable/mitaka ?23:34
sridhar_ramI notices projects.yaml has a flag is branch-override .. is that it ?23:35
mordredfungi: wow. 1.5kjph is a lot of jobs23:35
fungisridhar_ram: that's not a flag... it's a parameter some jobs use to indicate that when running on any branch they should test the code from the overridden branch23:35
fungisridhar_ram: it would need to be supported in the job-template you're instantiating though23:36
*** markusry has quit IRC23:36
fungipabelanger: note the comment from cloudnull on 357453 patchset 1 about setting maxage and copytruncate23:37
fungipabelanger: but i'm going to go ahead and approve now because we're down to just a sliver still available on the osic mirror's rootfs23:37
tonybsridhar_ram: It's prety easy to specify the branch to run a job on .... let me find a good example23:38
sridhar_ramfungi: there is a job-template only for dsvm job .. there is nothing else23:38
sridhar_ramtonyb: thanks23:38
sridhar_ramfungi: tonyb: what baffles me is, this job showed up only for this above patchset .. another patchset that recently merged to stable/mitaka didn't pick up this job ..
fungisridhar_ram: yeah, tonyb is presumably going to get you an example of specifying in zuul/layout.yaml which branches jobs should be triggered on23:39
*** stew925 has quit IRC23:39
*** skipp has quit IRC23:39
tonybfungi: Yeah.23:39
sridhar_ramtonyb: fungi: thanks folks!23:39
fungiwhich is not at all the same thing as overriding what branches should be used in jobs when they run23:39
tonybsridhar_ram: it's possible that another project-config change altered the behaviour23:40
*** eeiden has quit IRC23:40
*** eeiden has joined #openstack-infra23:40
*** wcriswell has quit IRC23:41
* sridhar_ram is looking up23:41
tonybsridhar_ram: that basically says only run this job if the branch is NOT stable/liberty or stable/mitaka23:41
tonybsridhar_ram: I'll look at what changed to cause it.  It's possible that more thn tacker is affected23:44
sridhar_ramtonyb: thanks a bunch..23:44
sridhar_ramtonyb: meanwhile, i'll push this change to zuul/layout to guard against such in the future23:45
tonybsridhar_ram: yeah good plan23:45
*** Swami has quit IRC23:45
*** thorst_ has quit IRC23:46
*** thorst_ has joined #openstack-infra23:47
*** tonytan4ever has joined #openstack-infra23:47
*** stewie925_ has joined #openstack-infra23:50
*** kzaitsev_mb has quit IRC23:50
stewie925_I have internet connection with my ubuntu - sudo install commands keep getting failed to get new connections - even though I could ping google.com23:51
*** rbuzatu has joined #openstack-infra23:51
*** kzaitsev_mb has joined #openstack-infra23:52
*** tonytan4ever has quit IRC23:53
*** zhurong has joined #openstack-infra23:53
clarkbstewie925_: is it possible you are attempting to talk to a mirror that isn't functioning? or that you have a firewall between you and the mirror?23:53
fungipabelanger: i've removed apache logs >8 days old on the osic mirror for now to get us breathing room until logrotate kicks again. freed up ~5gb so should be plenty for now23:55
*** thorst_ has quit IRC23:55
*** mriedem has joined #openstack-infra23:55
*** rbuzatu has quit IRC23:56
clarkbxenial image build is finally past the git caching23:58
*** gildub_ has joined #openstack-infra23:59
*** gildub has joined #openstack-infra23:59
stewie925_hi clarkb:  I just tried sudo apt-get install and it was blazing fast - success23:59
openstackgerritMerged openstack-infra/project-config: Remove OSIC from 2-node label types
stewie925_maybe something with this pudb23:59

Generated by 2.14.0 by Marius Gedminas - find it at!