Monday, 2014-12-08

*** emagana has quit IRC00:00
*** zz_avozza is now known as avozza00:01
*** shashankhegde has joined #openstack-infra00:01
*** armax has quit IRC00:01
*** harlowja_at_home has joined #openstack-infra00:02
*** sandywalsh_ has quit IRC00:03
*** Masahiro has joined #openstack-infra00:07
*** e0ne has joined #openstack-infra00:08
*** alexpilotti has quit IRC00:08
*** Longgeek has joined #openstack-infra00:09
*** e0ne has quit IRC00:10
*** Masahiro has quit IRC00:11
grue_pmmm seems I'm getting invited to a lot of 3/4am meetings00:12
*** shashankhegde has quit IRC00:13
grue_pmmordred while time may be irrelevent in some cases like this it sucks. I either need to change my sleep patterns or miss stuff.00:13
*** Longgeek has quit IRC00:14
*** pc_m has quit IRC00:14
* grue_pm wishes there was more recording of meetings to listent to for information at least.00:14
*** e0ne has joined #openstack-infra00:14
grue_pmor a summary but it doesnt really help sometimess00:15
mordredgrue_pm: well  .. all of the pertinent IRC channels are logged - and we _do_ have meeting summaries and stuff00:16
grue_pmmordred: by we do you mean gozer. This is in relation to some hp coffee cloud talk I just got00:18
*** e0ne has quit IRC00:19
grue_pmmordred: cody is rotating team meeitngs a bit - 5:30-6:30 am for me is easier.00:19
*** e0ne has joined #openstack-infra00:23
nibalizergrue_pm: i've not gone to those00:24
*** e0ne has quit IRC00:26
*** ddieterly has quit IRC00:29
*** ddieterly has joined #openstack-infra00:29
*** e0ne has joined #openstack-infra00:30
grue_pmnibilizer: ah okay - I'm just trying to work out which of the volumous meetings/seminars I should be attending in order to learn00:32
*** oomichi has joined #openstack-infra00:35
fungias far as i'm concerned, all the relevant meetings are in openstack meeting channels in irc ;)00:36
fungithen again, it's been a couple years since i was in any hp-specific meetings. maybe they've added rodeo clowns or something similarly entertaining since then?00:37
grue_pmfungi: I'm trying to work out generally what is good to attend be it corporate or general openstack stuff.00:38
jrollgrue_pm: if it's openstack or has food, attend :)00:39
* jroll is not hp and you should probably not listen to me :P00:39
* fungi tries to remember how to set his fax machine to receive pizza00:39
grue_pmjroll: doesnt have to be company related as I want to learn everything and irc pizza is a bit less filling than I'd like00:40
*** stevemar has quit IRC00:40
*** alex7376 has joined #openstack-infra00:40
jrollgrue_pm: right, I just have no idea which hp meetings are valuable :P00:41
*** alex7376 is now known as asettle00:41
grue_pmjroll: I wasn't specifically meaning corporate stuff - openstack is way beyond that00:42
* grue_pm is sorry she mentioned a company name in infra now ;-p00:42
jrollah, I see, sorry00:43
grue_pmjroll: I've already been reminded to ask company specific questions in a different area. I blame the lurgy for my muddy head currently.00:44
*** Masahiro has joined #openstack-infra00:44
*** ryanpetrello has joined #openstack-infra00:44
fungigrue_pm: on a more pleasant note, seeing you in here again, i guess that means after evaluating our crazy project, you didn't run screaming, and actually decided to join in?00:45
*** penguinRaider has quit IRC00:45
grue_pmfungi: yup - I'm now officially working with Cody's bunch. They are awesome and the crazy learning curve actually attracted me.00:46
fungiin that case, welcome to openstack! stick around and we'll have ever so much fun00:46
*** e0ne has quit IRC00:49
*** emagana has joined #openstack-infra00:50
*** ryanpetrello has quit IRC00:51
*** boris-42 has quit IRC00:52
*** imcsk8 has quit IRC00:53
*** imcsk8 has joined #openstack-infra00:53
*** e0ne has joined #openstack-infra00:53
*** emagana has quit IRC00:54
*** otter768 has joined #openstack-infra00:55
*** otter768 has quit IRC00:59
*** ryanpetrello has joined #openstack-infra01:00
*** marun has quit IRC01:02
*** e0ne has quit IRC01:06
*** shashankhegde has joined #openstack-infra01:08
*** e0ne has joined #openstack-infra01:09
*** e0ne has quit IRC01:11
*** viglesias has quit IRC01:12
*** ZZelle has quit IRC01:13
*** viglesias has joined #openstack-infra01:17
*** viglesias has joined #openstack-infra01:22
*** avozza is now known as zz_avozza01:22
*** shashankhegde has quit IRC01:23
*** viglesias has joined #openstack-infra01:27
*** wuhg has joined #openstack-infra01:31
*** salv-orlando has joined #openstack-infra01:34
*** harlowja_at_home has quit IRC01:37
*** ryanpetrello has quit IRC01:37
*** emagana has joined #openstack-infra01:38
*** tnovacik has quit IRC01:42
*** yaguang has joined #openstack-infra01:50
*** ddieterly has quit IRC01:54
*** emagana has quit IRC01:57
*** Longgeek has joined #openstack-infra01:58
*** Masahiro has quit IRC02:01
*** Longgeek has quit IRC02:02
*** Masahiro has joined #openstack-infra02:04
*** stevemar has joined #openstack-infra02:05
*** fifieldt has joined #openstack-infra02:05
*** otter768 has joined #openstack-infra02:07
*** ryanpetrello has joined #openstack-infra02:10
*** viglesias has quit IRC02:20
*** viglesias has joined #openstack-infra02:20
*** xyang1 has quit IRC02:22
*** camunoz is now known as camunoz_away02:24
*** ddieterly has joined #openstack-infra02:24
*** ddieterly has quit IRC02:26
*** ddieterly has joined #openstack-infra02:26
*** xyang1 has joined #openstack-infra02:27
*** emagana has joined #openstack-infra02:28
*** ivar-lazzaro has quit IRC02:30
*** ddieterly has quit IRC02:31
*** baoli has joined #openstack-infra02:31
*** emagana has quit IRC02:33
*** ddieterly has joined #openstack-infra02:37
*** ryanpetrello has quit IRC02:44
*** dkranz has quit IRC02:45
*** baoli has quit IRC02:49
*** baoli has joined #openstack-infra02:49
*** ryanpetrello has joined #openstack-infra02:51
*** sputnik13 has joined #openstack-infra02:55
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Fix pep8 issues
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Call driver methods more dynamically
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for triggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for sources
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for reporters
*** achanda has joined #openstack-infra03:00
*** achanda has quit IRC03:02
*** sputnik13 has quit IRC03:02
*** yamamoto_ has joined #openstack-infra03:04
*** Masahiro has quit IRC03:07
*** baoli has quit IRC03:13
*** ddieterly has quit IRC03:15
*** ihrachyshka has quit IRC03:21
*** bhunter71 has joined #openstack-infra03:22
*** ryanpetrello has quit IRC03:23
*** fandi has joined #openstack-infra03:23
*** emagana has joined #openstack-infra03:23
*** camunoz_away is now known as camunoz03:29
*** Masahiro has joined #openstack-infra03:30
*** sabari is now known as zz_sabari03:31
*** david-lyle_afk has quit IRC03:31
*** zz_sabari is now known as sabari03:31
*** baoli has joined #openstack-infra03:32
*** mestery has joined #openstack-infra03:34
*** Masahiro has quit IRC03:35
*** bhunter71 has quit IRC03:36
*** sabari is now known as zz_sabari03:36
*** obondarev has joined #openstack-infra03:38
*** Masahiro has joined #openstack-infra03:40
*** miqui_ has quit IRC03:40
*** ddieterly has joined #openstack-infra03:40
*** amotoki__away is now known as amotoki03:41
*** david-lyle_afk has joined #openstack-infra03:43
*** baoli has quit IRC03:46
*** yaguang has quit IRC03:46
*** baoli has joined #openstack-infra03:46
*** Masahiro has quit IRC03:48
*** david-lyle_afk has quit IRC03:50
*** Masahiro has joined #openstack-infra03:52
*** bradjones has quit IRC03:53
*** ivar-lazzaro has joined #openstack-infra03:56
anteayaheyongli: okay great03:58
*** zz_sabari is now known as sabari04:02
*** david-lyle_afk has joined #openstack-infra04:02
*** david-lyle_afk has quit IRC04:02
*** Masahiro has quit IRC04:02
*** marun has joined #openstack-infra04:04
*** sabari is now known as zz_sabari04:07
*** otter768 has quit IRC04:09
*** Masahiro has joined #openstack-infra04:09
*** dimsum__ has quit IRC04:11
*** marun has quit IRC04:12
*** jerryz has joined #openstack-infra04:13
*** koolhead17 has joined #openstack-infra04:13
*** yaguang has joined #openstack-infra04:19
*** dkehnm_ has joined #openstack-infra04:20
*** dkehnm has quit IRC04:21
*** dkehnm_ is now known as dkehnm04:21
*** dkehnm has quit IRC04:27
*** dkehnm has joined #openstack-infra04:30
*** emagana has quit IRC04:30
*** emagana has joined #openstack-infra04:31
*** sarob has joined #openstack-infra04:32
*** craigbr has joined #openstack-infra04:34
*** emagana has quit IRC04:35
*** sarob has quit IRC04:38
*** ivar-lazzaro has quit IRC04:39
*** ivar-lazzaro has joined #openstack-infra04:39
*** chandankumar has joined #openstack-infra04:39
*** craigbr has quit IRC04:40
*** ddieterly has quit IRC04:42
*** craigbr has joined #openstack-infra04:43
*** mestery has quit IRC04:43
*** zz_sabari is now known as sabari04:45
*** Longgeek has joined #openstack-infra04:47
*** sabeen has joined #openstack-infra04:48
*** koolhead17 has quit IRC04:50
*** Longgeek has quit IRC04:52
*** koolhead_ has joined #openstack-infra04:56
*** hdd has joined #openstack-infra04:56
*** shashankhegde has joined #openstack-infra04:58
*** david-lyle_afk has joined #openstack-infra05:01
*** koolhead_ has quit IRC05:02
*** koolhead17 has joined #openstack-infra05:03
*** rushiagr_away is now known as rushiagr05:07
*** koolhead17 has quit IRC05:07
*** obondarev has quit IRC05:11
*** dimsum__ has joined #openstack-infra05:11
*** dkehnm has quit IRC05:12
*** ddieterly has joined #openstack-infra05:12
*** ddieterly has quit IRC05:14
*** ddieterl_ has joined #openstack-infra05:14
*** dimsum__ has quit IRC05:17
*** emagana has joined #openstack-infra05:18
*** ddieterl_ has quit IRC05:19
*** baoli has quit IRC05:22
*** emagana has quit IRC05:23
*** ddieterly has joined #openstack-infra05:25
*** hdd has quit IRC05:28
*** craigbr has quit IRC05:29
*** emagana has joined #openstack-infra05:29
*** ddieterly has quit IRC05:30
*** emagana_ has joined #openstack-infra05:33
*** shashankhegde has quit IRC05:33
*** sarob has joined #openstack-infra05:34
*** emagana has quit IRC05:34
*** shashankhegde has joined #openstack-infra05:36
*** koolhead17 has joined #openstack-infra05:37
*** koolhead17 has joined #openstack-infra05:37
*** serverascode___ has quit IRC05:37
*** Masahiro has quit IRC05:38
*** Masahiro has joined #openstack-infra05:38
*** sarob has quit IRC05:39
*** serverascode___ has joined #openstack-infra05:40
*** xyang1 has quit IRC05:41
*** yfried_ has quit IRC05:42
*** shashankhegde has quit IRC05:43
*** sabeen has quit IRC05:46
*** talluri has joined #openstack-infra05:57
*** k4n0 has joined #openstack-infra06:01
*** sputnik13 has joined #openstack-infra06:02
openstackgerritSabari proposed openstack/requirements: Update urllib>=1.8.3
*** otter768 has joined #openstack-infra06:10
sabariCan someone pl approve the patch to unblock VMware CI ?06:11
*** talluri has quit IRC06:12
openstackgerritMerged openstack-infra/devstack-gate: Add is_debian and uses_debs functions
*** otter768 has quit IRC06:15
openstackgerritMerged openstack-infra/os-loganalyze: Add ignored Editors to .gitignore
*** talluri has joined #openstack-infra06:18
openstackgerritMerged openstack-infra/puppet-yum: Add .gitreview Rakefile files and update puppet coding style
*** sarob has joined #openstack-infra06:23
*** zz_dimtruck is now known as dimtruck06:25
*** emagana_ has quit IRC06:25
*** ddieterly has joined #openstack-infra06:25
*** emagana has joined #openstack-infra06:25
openstackgerritMerged openstack-infra/system-config: Update ssh module to support SUSE
*** sarob has quit IRC06:27
openstackgerritMerged openstack-infra/system-config: Document rebuilding the MediaWiki search index
*** Masahiro has quit IRC06:29
*** ddieterly has quit IRC06:30
*** emagana has quit IRC06:30
*** Masahiro has joined #openstack-infra06:31
*** tylerdurden has joined #openstack-infra06:32
*** shakamunyi has quit IRC06:34
*** Longgeek has joined #openstack-infra06:36
*** sputnik13 has quit IRC06:36
*** vigneshvar has joined #openstack-infra06:38
*** asettle has quit IRC06:38
*** liusheng has joined #openstack-infra06:39
*** Longgeek has quit IRC06:40
*** mrunge has joined #openstack-infra06:47
*** mrunge has quit IRC06:47
*** yfried_ has joined #openstack-infra06:48
*** liusheng has quit IRC06:51
*** liusheng has joined #openstack-infra06:52
*** koolhead17 has quit IRC06:53
*** koolhead17 has joined #openstack-infra06:54
*** koolhead17 has joined #openstack-infra06:54
*** emagana has joined #openstack-infra06:59
*** yfried_ has quit IRC07:00
*** koolhead_ has joined #openstack-infra07:03
*** stevemar has quit IRC07:03
*** emagana has quit IRC07:04
*** koolhead17 has quit IRC07:05
*** yfried_ has joined #openstack-infra07:07
*** camunoz is now known as camunoz_gone07:09
*** dimtruck is now known as zz_dimtruck07:10
*** ddieterly has joined #openstack-infra07:25
*** jlibosva has joined #openstack-infra07:29
*** ddieterly has quit IRC07:30
*** jgjl has joined #openstack-infra07:30
heyonglianteaya,  the spec  is uploaded :
*** ivar-lazzaro has quit IRC07:33
*** talluri has quit IRC07:36
*** Longgeek has joined #openstack-infra07:37
*** belmoreira has joined #openstack-infra07:37
*** Longgeek has quit IRC07:37
*** Longgeek has joined #openstack-infra07:37
*** alex7376 has joined #openstack-infra07:40
*** mrunge has joined #openstack-infra07:41
*** jgallard has joined #openstack-infra07:48
*** miqui has quit IRC07:51
*** emagana has joined #openstack-infra07:53
*** teran has quit IRC07:55
*** rakhmerov has quit IRC07:58
*** emagana has quit IRC07:58
*** tsufiev has quit IRC07:58
*** katyafervent has quit IRC07:59
*** dteselkin has quit IRC07:59
*** HeOS_ has quit IRC07:59
*** akuznetsova has quit IRC08:00
*** IvanBerezovskiy1 has joined #openstack-infra08:01
*** ala_ has joined #openstack-infra08:02
*** liusheng has quit IRC08:02
*** liusheng has joined #openstack-infra08:03
*** talluri has joined #openstack-infra08:05
*** rakhmerov has joined #openstack-infra08:07
*** akuznetsova has joined #openstack-infra08:08
*** ihrachyshka has joined #openstack-infra08:08
*** BobBall_AWOL is now known as BobBall08:09
*** katyafervent has joined #openstack-infra08:09
*** dteselkin has joined #openstack-infra08:10
*** otter768 has joined #openstack-infra08:11
*** HeOS_ has joined #openstack-infra08:11
*** Masahiro has quit IRC08:11
*** sarob has joined #openstack-infra08:11
*** jamielennox is now known as jamielennox|away08:12
*** talluri has quit IRC08:14
*** tsufiev has joined #openstack-infra08:14
*** talluri has joined #openstack-infra08:15
*** otter768 has quit IRC08:16
*** jistr has joined #openstack-infra08:16
*** armax has joined #openstack-infra08:16
*** Masahiro has joined #openstack-infra08:16
*** arxcruz has joined #openstack-infra08:16
*** sarob has quit IRC08:16
*** ildikov_ has quit IRC08:20
*** amuller has joined #openstack-infra08:21
*** zz_avozza is now known as avozza08:25
*** ddieterly has joined #openstack-infra08:25
*** tnovacik has joined #openstack-infra08:26
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (eavesdrop)
*** ddieterly has quit IRC08:30
*** Longgeek has quit IRC08:31
*** fandi has quit IRC08:31
*** ildikov_ has joined #openstack-infra08:32
*** emagana has joined #openstack-infra08:34
*** _nadya_ has joined #openstack-infra08:36
*** _nadya_ has quit IRC08:38
*** yolanda has quit IRC08:38
*** _nadya__ has joined #openstack-infra08:38
*** Nakato_ is now known as Nakato08:38
*** emagana has quit IRC08:39
*** Longgeek has joined #openstack-infra08:40
*** ddieterly has joined #openstack-infra08:42
*** pblaho has joined #openstack-infra08:42
*** markus_z has joined #openstack-infra08:43
*** mpaolino has joined #openstack-infra08:43
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (eavesdrop)
*** ddieterly has quit IRC08:47
*** yfried_ is now known as yfried|afk08:47
*** mrmartin has joined #openstack-infra08:48
*** MaxV has joined #openstack-infra08:48
*** gothicmindfood has quit IRC08:48
*** yfried|afk is now known as yfried_08:49
*** yamamoto_ has quit IRC08:50
openstackgerritMerged openstack/requirements: Update urllib>=1.8.3
*** Longgeek has quit IRC08:51
*** jcoufal has joined #openstack-infra08:54
*** armax has quit IRC08:57
*** teran has joined #openstack-infra08:58
*** Longgeek has joined #openstack-infra08:58
odyssey4mehi there - if anyone's around, since merged our check is showing as 'NOT_REGISTERED' - does a dvsm job have to be registered somewhere?08:59
*** jlibosva has quit IRC09:00
*** jlibosva has joined #openstack-infra09:02
*** Masahiro has quit IRC09:05
*** fandi has joined #openstack-infra09:06
*** Hal_ has joined #openstack-infra09:06
*** Masahiro has joined #openstack-infra09:08
*** salv-orlando has quit IRC09:10
*** emagana has joined #openstack-infra09:13
*** dimsum__ has joined #openstack-infra09:15
*** gothicmindfood has joined #openstack-infra09:15
*** yfried_ is now known as yfried|afk09:16
*** yfried|afk is now known as yfried_09:17
*** derekh has joined #openstack-infra09:18
*** emagana has quit IRC09:18
*** yamamoto has joined #openstack-infra09:19
*** afazekas has joined #openstack-infra09:19
*** dimsum__ has quit IRC09:20
*** yamamoto has quit IRC09:22
*** oomichi has quit IRC09:22
*** jedimike has joined #openstack-infra09:24
*** yaguang has quit IRC09:31
*** jlibosva has quit IRC09:31
*** jlibosva has joined #openstack-infra09:32
*** dtantsur|afk is now known as dtantsur09:34
*** yaguang has joined #openstack-infra09:44
*** skolekonov has joined #openstack-infra09:44
*** nfedotov has joined #openstack-infra09:46
*** rlandy has joined #openstack-infra09:46
*** e0ne has joined #openstack-infra09:48
*** sabari is now known as zz_sabari09:48
*** lttrl has joined #openstack-infra09:50
*** Masahiro has quit IRC09:56
*** kingia has joined #openstack-infra09:57
*** kingia is now known as iain-king09:57
openstackgerritAntoine Musso proposed openstack-infra/zuul: Pass change and pipeline to the status_url
*** belmoreira has quit IRC09:58
*** Masahiro has joined #openstack-infra10:00
*** sarob has joined #openstack-infra10:00
*** fandi has quit IRC10:02
*** e0ne has quit IRC10:04
*** sarob has quit IRC10:05
*** e0ne has joined #openstack-infra10:06
*** yfried_ has quit IRC10:07
*** emagana has joined #openstack-infra10:07
*** MaxV has quit IRC10:08
*** jp_at_hp has joined #openstack-infra10:08
*** fandi has joined #openstack-infra10:09
*** Masahiro has quit IRC10:09
*** yaguang has quit IRC10:10
*** salv-orlando has joined #openstack-infra10:11
*** _nadya__ has quit IRC10:11
*** emagana has quit IRC10:12
*** otter768 has joined #openstack-infra10:12
*** salv-orlando has quit IRC10:16
*** otter768 has quit IRC10:16
*** cdent has joined #openstack-infra10:17
*** MaxV has joined #openstack-infra10:22
*** yamamoto has joined #openstack-infra10:23
*** bradjones has joined #openstack-infra10:25
*** yamamoto has quit IRC10:27
*** ociuhandu has joined #openstack-infra10:29
*** pelix has joined #openstack-infra10:36
*** ildikov_ has quit IRC10:40
*** sdake has quit IRC10:44
*** andreykurilin_ has joined #openstack-infra10:44
*** rediskin has joined #openstack-infra10:44
*** ldnunes has joined #openstack-infra10:47
*** jistr has quit IRC10:51
*** ildikov_ has joined #openstack-infra10:53
*** ZZelle has joined #openstack-infra10:57
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix the postbuildscript documentation
*** ildikov_ has quit IRC10:58
*** ildikov_ has joined #openstack-infra10:58
*** Hal_ has quit IRC10:58
*** emagana has joined #openstack-infra11:01
*** erlon has joined #openstack-infra11:02
*** simonmcc_ is now known as simonmcc11:03
openstackgerritMerged openstack-infra/jenkins-job-builder: include publisher.xunit test as doc
*** aysyd has joined #openstack-infra11:05
*** emagana has quit IRC11:06
*** jgjl has quit IRC11:07
*** yfried_ has joined #openstack-infra11:08
*** jistr has joined #openstack-infra11:09
openstackgerritThierry Carrez proposed openstack-infra/release-tools: spec2bp v2 -- support in-review specs
*** Masahiro has joined #openstack-infra11:10
*** mpaolino has quit IRC11:11
*** jgjl has joined #openstack-infra11:13
*** Masahiro has quit IRC11:14
*** aysyd has quit IRC11:15
*** yolanda has joined #openstack-infra11:16
*** andreaf has joined #openstack-infra11:19
*** mpaolino has joined #openstack-infra11:20
openstackgerritAntoine Musso proposed openstack-infra/jenkins-job-builder: fix typo, must be 'classifier' to appear on
*** jgallard has quit IRC11:23
*** ihrachyshka has quit IRC11:25
*** HeOS has quit IRC11:28
*** ihrachyshka has joined #openstack-infra11:29
*** HeOS has joined #openstack-infra11:30
*** andreaf has quit IRC11:32
*** mrmartin has quit IRC11:35
*** fandi has quit IRC11:41
*** HACKING-FACEBOOK has joined #openstack-infra11:44
*** andreaf has joined #openstack-infra11:47
*** dimsum__ has joined #openstack-infra11:47
*** avozza is now known as zz_avozza11:55
*** emagana has joined #openstack-infra11:56
*** salv-orlando has joined #openstack-infra12:00
*** emagana has quit IRC12:00
*** acruz_ has joined #openstack-infra12:01
*** dimsum__ is now known as dims12:05
*** salv-orlando has quit IRC12:05
*** luisjariz has joined #openstack-infra12:09
*** otter768 has joined #openstack-infra12:12
*** andreykurilin_ has quit IRC12:14
*** yamamoto has joined #openstack-infra12:15
*** mase_x200 has joined #openstack-infra12:15
*** otter768 has quit IRC12:17
*** Hal_ has joined #openstack-infra12:20
*** jpich has joined #openstack-infra12:23
*** mrmartin has joined #openstack-infra12:23
*** HACKING-FACEBOOK has quit IRC12:24
*** HACKING-FACEBOOK has joined #openstack-infra12:26
*** zz_avozza is now known as avozza12:26
*** jistr has quit IRC12:27
*** mpaolino has quit IRC12:28
*** jistr has joined #openstack-infra12:28
*** HACKING-FACEBOOK has quit IRC12:29
openstackgerritMerged openstack-infra/jenkins-job-builder: Add examples for 'predefined-parameters'
*** jgjl has quit IRC12:30
*** HACKING-FACEBOOK has joined #openstack-infra12:30
*** jamielennox|away is now known as jamielennox12:31
*** _nadya_ has joined #openstack-infra12:32
*** mrunge has quit IRC12:32
openstackgerritAntoine Musso proposed stackforge/python-jenkins: Workflow documentation is now in infra-manual
*** luisjariz has quit IRC12:32
*** iain-king has quit IRC12:33
*** mrunge has joined #openstack-infra12:35
*** jgallard has joined #openstack-infra12:36
*** adalbas has joined #openstack-infra12:36
openstackgerritMerged stackforge/python-jenkins: fix typo, must be 'classifier' to appear on
*** jamielennox is now known as jamielennox|away12:41
openstackgerritDerek Higgins proposed openstack-infra/project-config: Remove tripleo control variables
openstackgerritDerek Higgins proposed openstack-infra/project-config: Use a job-template for the tripleo jobs
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a tripleo juno overcloud jobs
*** koolhead_ has quit IRC12:42
*** Masahiro has joined #openstack-infra12:44
*** _nadya_ has quit IRC12:48
*** Masahiro has quit IRC12:49
*** emagana has joined #openstack-infra12:50
*** koolhead17 has joined #openstack-infra12:50
*** HeOS has quit IRC12:51
*** sandywalsh has joined #openstack-infra12:52
*** amotoki has quit IRC12:54
openstackgerritMerged stackforge/python-jenkins: Workflow documentation is now in infra-manual
*** emagana has quit IRC12:55
*** cnesa has joined #openstack-infra12:55
*** vigneshvar has quit IRC12:58
*** HeOS has joined #openstack-infra12:59
KiallIs Zuul is the process of exploding? "Queue lengths: 1777 events, 35 results. " with virtually nothing running (20 check, 15 gate)12:59
Kialland it suddenly cleared.. heh13:00
*** salv-orlando has joined #openstack-infra13:01
*** chandankumar has quit IRC13:02
sdaguemerge check is probably back13:02
*** _nadya_ has joined #openstack-infra13:03
*** dprince has joined #openstack-infra13:04
*** salv-orlando has quit IRC13:05
*** mbacchi has joined #openstack-infra13:06
*** mfink has quit IRC13:07
*** alexpilotti has joined #openstack-infra13:08
*** bknudson has quit IRC13:08
*** hashar has joined #openstack-infra13:10
*** koolhead17 has quit IRC13:13
*** koolhead17 has joined #openstack-infra13:13
*** koolhead17 has quit IRC13:18
*** obondarev has joined #openstack-infra13:19
*** dkehnm has joined #openstack-infra13:21
*** ihrachyshka_ has joined #openstack-infra13:22
*** salv-orlando has joined #openstack-infra13:22
*** ihrachyshka has quit IRC13:23
openstackgerritJulien Danjou proposed openstack-infra/git-review: Fix encoding header
openstackgerritJulien Danjou proposed openstack-infra/git-review: Allow to specify which commit to send
*** koolhead17 has joined #openstack-infra13:29
odyssey4meDo dvsm jobs need to be registered or something? I'm getting a 'NOT_REGISTERED' result in a new dvsm job.13:31
*** sweston_ has joined #openstack-infra13:32
odyssey4meThe commit that brought it into being is this one:
sdagueodyssey4me: what's the job that failed?13:32
*** doug-fish has joined #openstack-infra13:33
odyssey4mesdague: check-os-ansible-deployment-dvsm-aio-build13:33
sdagueodyssey4me: please provide a patch where it failed13:33
*** baoli has joined #openstack-infra13:33
odyssey4meThere are plenty of others - it's failing every time it runs.13:34
*** yfried_ is now known as yfried|afk13:36
*** yfried|afk is now known as yfried_13:36
*** kgiusti has joined #openstack-infra13:36
sdagueso probably someone better with zuul needs to step in. I suspect there is something more subtle about the structure there.13:37
*** radez_g0n3 is now known as radez13:39
odyssey4methanks sdague - they should be on in an hour or two13:41
*** koolhead17 has quit IRC13:43
*** mpaolino has joined #openstack-infra13:43
*** koolhead17 has joined #openstack-infra13:44
*** emagana has joined #openstack-infra13:44
*** sweston_ has quit IRC13:46
*** mwagner_lap has quit IRC13:47
*** emagana has quit IRC13:48
*** koolhead17 has quit IRC13:48
dstufftfungi: btw we talked about the SQLAlchemy requirement a while back, I went ahead and submitted a CR to do what I beleive the intent is in aw ay that can be consistently interpreted ->
*** mfink has joined #openstack-infra13:50
*** mase_x200 has quit IRC13:50
jd__any plan to release a new pbr version btw?13:54
*** sabeen1 has joined #openstack-infra13:55
sdaguedstufft: maybe it's time to just uplift to 0.9.7 and call it a day13:57
dstufftsdague: that's doable too!13:58
dstufftsdague: I didn't want to make that choice for Openstack so I just translated13:58
sdaguedstufft: why don't you propose that. We aren't really testing with 0.8.x at the moment anyway, and lifting that on master before the first milestone seems sane13:58
dstufftsdague: should I make it a new CR or just update the current?13:59
sdaguelet's just update the current one13:59
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static)
sdagueI'll leave a comment on the current patch to explain the direction change13:59
*** cnesa has quit IRC14:00
*** cnesa has joined #openstack-infra14:01
*** dkranz has joined #openstack-infra14:01
*** talluri has quit IRC14:02
*** _nadya_ has quit IRC14:03
*** ala_ has quit IRC14:03
openstackgerritDonald Stufft proposed openstack/requirements: Depend on any 0.9.x SQLAlchemy that is >= 0.9.7
sdaguedstufft: +2 ... thanks14:05
dstufftsdague: okies, updated, thanks14:06
*** ala_ has joined #openstack-infra14:06
*** pblaho has quit IRC14:06
fungiodyssey4me: i replied to you over the weekend. should fix it14:06
SergeyLukjanovsdague, super fast reviewing :)
jcoufalanybody from core devstack, who can help to get this patch merged? (already reviewed, needs just approvals)14:06
jcoufalit's blocking horizon patches to get in14:06
sdaguejcoufal: +A14:07
odyssey4melol, silly me - thanks fungi - I missed your reply in my scrollback14:07
jcoufalsdague: thanks14:07
*** pblaho has joined #openstack-infra14:08
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static)
sdaguejcoufal: we should probably remove that cells job from horizon anyway14:08
sdagueit doesn't test horizon14:08
*** acruz_ has quit IRC14:08
jcoufalsdague: sounds reasonable to me14:08
*** krtaylor has quit IRC14:09
fungiodyssey4me: yeah, we do tend to be a chatty bunch in here, even over weekends and holidays14:10
openstackgerritMerged openstack-infra/project-config: Correct misnamed check-{name}-dsvm-aio-build jobs
*** otter768 has joined #openstack-infra14:13
*** _nadya_ has joined #openstack-infra14:14
openstackgerritSean Dague proposed openstack-infra/project-config: Remove dsvm-cells tests from places they don't do anything
sdaguejcoufal: ^^^14:14
*** ddieterly has joined #openstack-infra14:14
jcoufalsdague: nice14:15
*** dkliban_afk is now known as dkliban14:16
*** mikedillion has joined #openstack-infra14:17
*** amitgandhinz has joined #openstack-infra14:18
*** otter768 has quit IRC14:18
*** salv-orlando has quit IRC14:21
*** IvanBerezovskiy1 has left #openstack-infra14:21
odyssey4mefungi odd, the patch has merged, but the patch I put into the repo afterwards still ran with the old name - does zuul/jenkins need some time to refresh?14:21
*** ryanpetrello has joined #openstack-infra14:22
*** koolhead17 has joined #openstack-infra14:23
*** mjturek has joined #openstack-infra14:23
fungiodyssey4me: yes, the merge at 14:12 above would get picked up by the 14:15 puppet apply pass which according to the zuul status page caused it to reload its configuration at 14:22:4414:24
*** cpowell has joined #openstack-infra14:24
fungiodyssey4me: looks like you tried about one minute early ;)14:24
*** yfried_ is now known as yfried|afk14:24
odyssey4mefungi ah, thanks - will retry now14:24
*** aysyd has joined #openstack-infra14:24
*** ayoung has joined #openstack-infra14:26
*** ayoung has quit IRC14:26
*** weshay has joined #openstack-infra14:27
*** yfried|afk is now known as yfried_14:30
*** krtaylor has joined #openstack-infra14:30
openstackgerritAntoine Musso proposed openstack-infra/zuul: cloner: support for ref-updated parameters
openstackgerritAntoine Musso proposed openstack-infra/zuul: cloner: set zuul_branch on ref-update events
*** baoli has quit IRC14:31
openstackgerritMerged openstack-infra/git-review: Fix encoding header
*** baoli has joined #openstack-infra14:32
*** Masahiro has joined #openstack-infra14:33
*** ayoung has joined #openstack-infra14:33
*** mriedem has joined #openstack-infra14:35
*** amuller_ has joined #openstack-infra14:35
*** amuller has quit IRC14:35
*** mestery has joined #openstack-infra14:36
*** mestery has quit IRC14:37
*** mestery has joined #openstack-infra14:37
*** Masahiro has quit IRC14:38
*** emagana has joined #openstack-infra14:38
*** mestery has quit IRC14:38
*** craigbr has joined #openstack-infra14:40
*** jaypipes has joined #openstack-infra14:40
*** jerryz has quit IRC14:40
*** obondarev_ has joined #openstack-infra14:43
*** tonytan4ever has joined #openstack-infra14:43
*** obondarev has quit IRC14:43
*** mpaolino has quit IRC14:43
*** emagana has quit IRC14:43
*** mpaolino has joined #openstack-infra14:44
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static)
*** yamamoto has quit IRC14:48
*** yamamoto has joined #openstack-infra14:50
*** obondarev_ has quit IRC14:51
hasharfungi: hello!  git:// seems to have some troubles14:51
*** armax has joined #openstack-infra14:51
*** eharney has quit IRC14:52
fungihashar: i'll look at the graphs for the git servers, but it could just as easily be a network problem in rackspace dfw14:52
*** yamamoto has quit IRC14:52
*** signed8bit has joined #openstack-infra14:52
*** marcusvrn has quit IRC14:52
*** dkehnm_ has joined #openstack-infra14:55
fungiseveral of the git servers look fairly busy, but none look overloaded14:55
*** dkehnm has quit IRC14:56
*** dkehnm_ is now known as dkehnm14:56
hasharfungi: seems it is solved now.14:56
hasharaccording to a logstash search for 'fatal:' :)14:56
*** armax has quit IRC14:56
pleia2good morning14:57
fungiyeah, the graphs show there was a spike in activity for 30 minutes or so, but not high enough that it looks like it should have caused that14:57
fungiprobably just the spike from our image updates14:57
*** esker has joined #openstack-infra14:58
*** obondarev has joined #openstack-infra14:58
*** ddieterly has quit IRC14:58
*** mwagner_lap has joined #openstack-infra15:00
*** esker has quit IRC15:00
*** pradk has joined #openstack-infra15:00
*** bknudson has joined #openstack-infra15:00
*** julim has joined #openstack-infra15:02
*** obondarev has quit IRC15:03
openstackgerritMerged openstack/requirements: Drop greenio dependency
*** amuller_ has quit IRC15:05
openstackgerritMerged openstack-infra/devstack-gate: Add conf to replace nova V2 endpoint with V2.1 API
*** esker has joined #openstack-infra15:06
*** rushiagr is now known as rushiagr_away15:06
*** jgjl has joined #openstack-infra15:06
*** salv-orlando has joined #openstack-infra15:08
*** obondarev has joined #openstack-infra15:08
*** pblaho_ has joined #openstack-infra15:10
*** jgjl has quit IRC15:10
*** armax has joined #openstack-infra15:11
*** salv-orlando has quit IRC15:12
*** armax has quit IRC15:13
*** pblaho has quit IRC15:14
openstackgerritTerry Howe proposed openstack/requirements: Add in python-openstacksdk to requirements.txt
*** dmsimard_away is now known as dmsimard15:15
*** hdd has joined #openstack-infra15:17
*** weshay has quit IRC15:21
*** weshay has joined #openstack-infra15:21
*** zz_dimtruck is now known as dimtruck15:22
*** sputnik13 has joined #openstack-infra15:22
*** nelsnelson has joined #openstack-infra15:23
*** sputnik13 has quit IRC15:23
*** obondarev has quit IRC15:24
*** ddieterly has joined #openstack-infra15:25
*** marcusvrn has joined #openstack-infra15:25
*** mpaolino has quit IRC15:26
*** e0ne has quit IRC15:28
*** mestery has joined #openstack-infra15:29
*** sarob has joined #openstack-infra15:29
*** baoli has quit IRC15:29
*** chandankumar has joined #openstack-infra15:29
mordredjd__: is there something in pbr master we need to get out there?15:29
mordredjd__: I'll take a look once I've chugged through morning email and whatnot15:29
*** chandankumar has quit IRC15:30
jd__mordred: I know I want to use the doc generation stuff patch I merged15:31
*** teran has quit IRC15:32
*** andreaf has quit IRC15:32
*** emagana has joined #openstack-infra15:32
*** amuller_ has joined #openstack-infra15:32
*** andreaf has joined #openstack-infra15:33
*** ihrachyshka_ has quit IRC15:33
*** teran has joined #openstack-infra15:33
*** julim has quit IRC15:33
*** sputnik13 has joined #openstack-infra15:33
mordredah. gotcha.15:34
*** dkehnm has quit IRC15:34
*** mestery has quit IRC15:35
*** sarob has quit IRC15:35
*** julim has joined #openstack-infra15:35
mordredjd__: oh jeez, we've really waited a while between releases haven't we?15:35
*** dannywilson has joined #openstack-infra15:36
*** dprince_ has joined #openstack-infra15:36
*** sdake has joined #openstack-infra15:36
mordredfungi, dhellmann: you guys know where we are on the lifeless semver patches?15:36
jd__yeah not really release early release often anymore :)15:36
mordredjd__: :)15:36
*** mestery has joined #openstack-infra15:36
*** baoli has joined #openstack-infra15:36
dhellmannmordred: I think master was broken, so we didn't release them. I don't know if he has fixes up yet.15:36
*** dprince has quit IRC15:36
mordreddhellmann: yah. I was afraid of that :(15:37
fungino clue--haven't looked back into it recently15:37
*** emagana has quit IRC15:37
dhellmannmordred: it's a shame we can't use
mordreddhellmann: yah.15:38
*** _shaps_ has joined #openstack-infra15:38
*** wenlock has joined #openstack-infra15:38
*** pc_m has joined #openstack-infra15:38
mordreddhellmann: the packagers would hate pbr even more than they already do if we vendored a library in though, and transitive deps during setup_requires brings on cthulu levels of pain15:38
dhellmannmordred: yeah15:39
*** e0ne has joined #openstack-infra15:39
*** k4n0 has quit IRC15:40
*** pc_m_ has joined #openstack-infra15:40
*** mikedillion has quit IRC15:40
*** weshay has quit IRC15:41
*** eharney has joined #openstack-infra15:41
*** weshay has joined #openstack-infra15:42
*** obondarev has joined #openstack-infra15:42
*** ala_ has quit IRC15:43
*** pc_m has quit IRC15:44
*** ala_ has joined #openstack-infra15:45
*** zz_jgrimm is now known as jgrimm15:45
*** mpaolino has joined #openstack-infra15:46
ttxfungi: about the stable teams refactoring, how would you like to proceed and when ? If you're stuck with Neutronites today, maybe not now ?15:46
*** armax has joined #openstack-infra15:47
*** sdake has quit IRC15:47
fungittx: i'll probably be free by 18:00 utc if you want to take a stab at it then. i expect we'll do something similar to what we did for the third-party testing project-specific acls/groups15:47
*** sdake has joined #openstack-infra15:48
*** sdake has quit IRC15:48
*** sdake has joined #openstack-infra15:48
ttxfungi: hmm, how about we do that on Wednesday ?15:48
openstackgerritMaish Saidel-Keesing proposed openstack-infra/project-config: Add rss.opml file to repository
fungittx: sure, sounds good15:49
ttxfungi: ok will ping you then15:50
*** amotoki has joined #openstack-infra15:51
*** HACKING-FACEBOOK has quit IRC15:52
*** yamamoto has joined #openstack-infra15:53
*** rushiagr_away is now known as rushiagr15:55
*** ihrachyshka_ has joined #openstack-infra15:57
*** yamamoto has quit IRC15:58
*** mestery has quit IRC15:59
*** jaypipes has quit IRC15:59
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static)
krotscheckStoryboard meeting in #openstack-meeting-316:01
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of Ask website
mrmartinfungi: ^^^^^^^^16:01
*** ihrachyshka_ is now known as ihrachyshka16:02
fungimrmartin: exciting--thank you!!!16:02
*** armax has quit IRC16:02
mrmartinfungi: more than exciting und wunderschön16:02
*** harlowja_at_home has joined #openstack-infra16:03
*** amuller_ is now known as amuller16:04
mrmartinanyway, we need to find a timeslot and talk about the details, it works in my vagrant, and can migrate the actual but for final deployment I suggest to setup the instance first, migrate the actual site for testing, and not change the public ip until it is passing some acceptance test.16:04
mrmartindo you have the certs for this site?16:04
*** sarob has joined #openstack-infra16:05
pleia2woo ask.o.o :)16:05
fungimrmartin: that sounds like a safe plan. let me see if i have access to the key16:05
fungimrmartin: hrm, i can't ssh into that instance16:05
mrmartinI guess I'll write an email to infra, there is a lot to do16:05
*** dkranz has quit IRC16:05
mrmartinok, I can :) we could exchange the keys and backups somehow16:05
fungimrmartin: yes, let's have an ml thread to highlight this patch16:06
mrmartindo we have some example in infra scripts for postgresql db backup?16:06
fungimrmartin: and since you have ssh access into it, i can just have you add me so i can collect the stuff we'll need to copy to the new server16:06
*** david-lyle_afk is now known as david-lyle16:06
fungimrmartin: not sure--we likely want to duplicate something for the database backup cron job we use for mysql16:07
*** amitgandhinz has quit IRC16:07
clarkbI take mysql still doesnt work? though migrating pgsql -> mysql probably not worth the trouble16:07
mrmartinok, basically in middle term, we can migrate this back to mysql, I wanted to avoid that to decrease the number of possible issues now.16:07
clarkbcan we? askbot did not work with mysql when I tried it16:08
mrmartinaskbot works well with mysql, but the actual deployment runs on pgsql.16:08
clarkband ++ to minimizing the diff16:08
mrmartinI guess the reason it was running pgsql that it used some full text search capability there, but it is replaced now with solr.16:08
*** arxcruz has quit IRC16:09
clarkbwell that and mysql just didnt work16:09
clarkbthe migrations were broken among other things iirc16:09
fungisounds like askbot has likely evolved in the year or so since that was last attempted16:09
mrmartinworked well in my vagrant anyway.16:09
clarkbthats good16:09
*** esker has quit IRC16:10
mrmartinI spent a lot of time discovering misterious python bugs :) The best one was, not to use WSGIPythonOptimize parameter in apache with mod_wsgi, because asserts won't throw an AssertException. And the site basically worked until a point :)16:11
fungier, hah16:12
*** rushiagr is now known as rushiagr_away16:12
*** salv-orlando has joined #openstack-infra16:12
*** mestery has joined #openstack-infra16:12
*** dannywilson has quit IRC16:13
*** yfried_ has quit IRC16:13
mrmartinanyway: Error: Could not install module 'puppetlabs-stdlib' (v4.3.2)16:14
mrmartinfungi: had you met with something similar before?16:14
*** otter768 has joined #openstack-infra16:14
mrmartinand this one: complains about askbot python code quality16:16
*** markmcclain has joined #openstack-infra16:16
*** amuller has quit IRC16:16
openstackgerritMerged openstack/requirements: Depend on any 0.9.x SQLAlchemy that is >= 0.9.7
*** armax has joined #openstack-infra16:17
openstackgerritAntoine Musso proposed openstack-infra/zuul: cloner: support for ref-updated parameters
*** sputnik13 has quit IRC16:17
*** ala_ has quit IRC16:18
*** armax has quit IRC16:18
fungimrmartin: i think this is where the puppet-askbot module errors are coming in
*** otter768 has quit IRC16:19
openstackgerritRoman Podoliaka proposed openstack-infra/project-config: oslo.db: make SQLAlchemy jobs voting
mrmartinGive me a sec, I'm uploading a quick patch, for some typo problems16:19
*** aysyd has quit IRC16:19
mrmartinanyway I needed to add a new puppet module, concat16:19
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of Ask website
*** dkranz has joined #openstack-infra16:20
fungimrmartin: also, as for the pep8 issues inside modules/askbot/files i wonder if we shouldn't have puppet retrieving those files somehow rather than embedding them into the module's revision control16:20
*** aysyd has joined #openstack-infra16:21
mrmartincan we copy a file with a File resource ?16:21
clarkbyup use a path for source16:21
clarkbor you can set up a symlink too16:22
*** Masahiro has joined #openstack-infra16:22
fungibasically point it to the copy on the file system installed from the askbot source16:22
*** achanda has joined #openstack-infra16:22
*** mikedillion has joined #openstack-infra16:22
mrmartinyep, I can copy from askbot's setup_templates directory16:23
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: hipchat bug fix
openstackgerritAntoine Musso proposed openstack-infra/zuul: Support stracktracing on Geard embedded server
mrmartinok, I'll try to solve those problem today, and at least reach a point where askbot puppet can pass the gating with a +116:24
fungimestery: dougwig: getting ready to help coordinate your git repo updates?16:24
dougwiganteaya: (sub vpnaas and fwaas for the other two)16:24
mesteryfungi: Yes sir16:24
*** jaypipes has joined #openstack-infra16:25
fungimrmartin: it may also be worth discussing whether this should start out as separate openstack-infra/puppet-askbot and openstack-infra/puppet-solr git repositories16:25
fungimestery: dougwig: awesome--just making sure you weren't otherwise occupied with sprint setup problems16:26
clarkbfungi +116:26
mrmartinI agree16:26
dougwigwarning: advanced services will be broken in neutron for a few days (see the ML).  additionally, the new neutron-lbaas, neutron-fwaas, and neutron-vpnaas repos will not be usable for a few days.16:26
clarkbmaybe a solr puppet module exists too?16:26
*** Masahiro has quit IRC16:26
clarkbaskbot didnt at least not way back when16:26
openstackgerritAntoine Musso proposed openstack-infra/zuul: Doc: ref links for "approval" requirement
*** emagana has joined #openstack-infra16:26
mrmartinclarkb, yep, this solr have a multi-core support. :)16:27
anteayafungi: I'm sitting beside them, right now we are working on communication plan and -2'ing patches in flight proposed to the dir that will be split16:27
fungianteaya: great--also thrilled you're there!16:28
*** armax has joined #openstack-infra16:28
mesteryfungi: Thanks, so far we're good, but we know where to find you when we have issues. And anteaya is also here to assist us :)16:28
hasharclarkb: hello! I pushed a doc change for Zuul to document yappi.  Could use your review there since you added the related code (doc:,unified  ) :]16:28
*** salv-orlando has quit IRC16:29
anteayafungi: thanks16:29
mrmartinok, I'm going to eat something and play with the children, sitting here since morning. I'll see back later, and care the askbot patch.16:29
clarkbhashar thanks will look16:30
fungimrmartin: thanks again for working on that16:30
dougwiganteaya: neutron/db, neutron/services, neutron/tests/unit/services16:30
dougwiganteaya: info on the split:
*** emagana has quit IRC16:31
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: add support for pyton and tox matrix project axis
*** mestery has quit IRC16:35
clarkbhashar: just one tiny misspelling otherwise it looks good to me16:36
*** ParsectiX has joined #openstack-infra16:36
hasharclarkb: let me know, will fix right now :]16:37
clarkbhashar: yup left a comment16:37
*** achanda has quit IRC16:37
fungiother than that typo, lgtm too16:37
*** achanda has joined #openstack-infra16:38
hasharyappi dumpes ...., reset16:38
*** jpich has quit IRC16:39
hasharclarkb:  I meant:  maybe 'reset' needs an extra s as well?16:39
hasharit dumpes the informations collected, resetS  <--- ?16:39
hasharI am terribly with the third person S16:39
clarkbhashar: if you change it to resets make sure you s/states/state/16:40
*** packet has joined #openstack-infra16:40
clarkbso that it becomes 'resets all yappi state'16:40
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin
*** mestery has joined #openstack-infra16:40
*** tnovacik has quit IRC16:40
clarkbI think reset all yappi states is correct though16:40
clarkbfungi: ^16:40
hasharclarkb: then I guess it stopS profiling?16:40
*** mpaolino has quit IRC16:40
clarkbhashar: ya16:40
openstackgerritAntoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler
hasharclarkb: :-]16:41
fungiclarkb: agreed, resets->state or reset->states depending on whether there is a singular state of yappi or it maintains multiple individual states16:42
clarkbI am going to make Administrators own the old third party groups now and mark them not visible to the world. Then remove all group members sometime this week16:42
fungihashar: additional functions16:42
clarkbanteaya: ^ do you want me to capture the current state anywhere before I start doing that?16:42
*** achanda has quit IRC16:42
fungihashar: and thread stats16:43
fungihashar: the information collected16:43
hasharfungi: well there are several threads aren't they?16:43
fungii'll add inline comments16:43
fungihashar: in that case the threads' states (possessive plural trheads)16:43
*** hdd has quit IRC16:44
fungier, threads' stats16:44
hasharI need english lessons Oo,16:44
*** jgrimm has quit IRC16:44
clarkbanteaya: I will hold off on that until I hear if you need anymore info16:44
clarkbhashar: its ok I need them too16:44
*** maurosr has quit IRC16:44
fungihashar: wouldn't help. people who grow up speaking and writing it make at least this many errors :/16:44
*** achanda has joined #openstack-infra16:46
fungihashar: okay, i think i have the correct pluralization and possessive punctuation in my latest inline comments16:46
openstackgerritRoman Podoliaka proposed openstack/requirements: Bump the minimum required version of retrying
*** nfedotov has quit IRC16:48
hasharfungi: that is nice. Amending thank you very much16:48
openstackgerritAntoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler
openstackgerritAntoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler
hasharand rebased. Should be fine now16:49
*** ParsectiX has quit IRC16:51
*** amotoki has quit IRC16:53
fungihashar: perfect--thanks!16:53
*** pblaho_ has quit IRC16:55
*** salv-orlando has joined #openstack-infra16:55
*** markus_z has quit IRC16:56
*** ParsectiX has joined #openstack-infra16:56
*** mpaolino has joined #openstack-infra16:57
*** rushiagr_away is now known as rushiagr16:57
*** Hal_ has quit IRC16:58
hasharone less patch in queue16:59
openstackgerritMerged openstack-infra/zuul: Document usage of yappi profiler
anteayaclarkb: capturing current state can't hurt17:01
anteayaand then go ahead17:02
anteayaand yay17:02
*** harlowja_at_home has quit IRC17:02
*** Hal_ has joined #openstack-infra17:03
*** ParsectiX has quit IRC17:03
*** hashar has quit IRC17:03
openstackgerritMerged openstack-infra/nodepool: Make the external links actual hyperlinks
*** krtaylor has quit IRC17:04
clarkbanteaya: ok I will list group membership for the two groups. Have a preference on where I stash that?17:05
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: add support for pyton and tox matrix project axis
*** krtaylor has joined #openstack-infra17:05
*** ParsectiX has joined #openstack-infra17:05
*** zz_sabari is now known as sabari_17:06
jeblairclarkb, fungi: want to do the nodepool swap soonish?17:07
jeblairalso good morning17:07
anteayaclarkb: paste is nice17:07
*** esker has joined #openstack-infra17:07
anteayamorning jeblair17:07
fungijeblair: good morning, and sure17:07
*** jmeridth_ has left #openstack-infra17:08
clarkbjeblair: yup definitely17:09
jeblairclarkb: what needs prepping?17:09
clarkbjeblair: was thinking after we updated neutron repos, but those things can happen concurrently so not a big deal to do it at the same time17:09
clarkbjeblair: nodepool-dev should have all firewall rules set up properly. We should maybe kick off image builds on that host manually so that we have up to date image17:09
fungithere are also several of us if we need to divide attention17:09
clarkbjeblair: and also manually update gear on nodepool-dev17:10
*** sputnik13 has joined #openstack-infra17:10
clarkbjeblair: update gear on zuul.o.o and once that is done we can restart zuul and flip nodepool over17:10
*** sabari_ is now known as zz_sabari_17:10
*** nelsnelson has quit IRC17:11
jeblairclarkb: we have images on np-dev, right?17:11
*** afazekas has quit IRC17:11
openstackgerritMerged openstack-infra/project-config: Add fwaas and vpnaas repos to #openstack-neutron
fungithey're likely outdated, but checking now17:11
clarkbjeblair: we do, but they would be older from the last time the daemon was running17:11
fungiyeah, about 1.5 weeks stale now17:11
jeblairif we wait for that, then we're talking a couple of hours, right?17:12
fungiand we have had some slave script updates go in which would make them break quite a few jobs i think17:12
clarkbjeblair: yes so maybe don't wait and have it update images concurrently17:12
clarkbfungi: oh ya17:12
clarkbfungi: so we do need new images first17:12
*** ParsectiX has quit IRC17:12
fungii can manually build images now if desired17:12
*** avozza is now known as zz_avozza17:12
clarkbya I think we should start with that17:12
jeblairfungi: in that case, i think we should probably delete the stale ones too17:12
*** mmaglana has joined #openstack-infra17:12
clarkb++ to deleting stale images too17:12
fungijeblair: clarkb: on it now17:13
jeblairokay, i'll go ahead and install gear+proposed change on nodepool-dev and zuul17:13
*** nelsnelson has joined #openstack-infra17:14
*** armax has quit IRC17:15
jeblairclarkb, fungi: gear is installed both places17:15
*** dkranz has quit IRC17:16
*** nelsnelson has quit IRC17:17
*** nelsnelson has joined #openstack-infra17:17
*** baoli has quit IRC17:18
clarkbthen for the flip over, stop nodepool on old host, delete all ready nodes there, start nodepool on new host, delete all nodes on old host in a couple hours/after jobs are done running on them17:18
clarkbwe probably want to do the deletion of ready nodes while zuul is off so that they don't transition state17:19
*** openstackgerrit has quit IRC17:19
*** openstackgerrit has joined #openstack-infra17:19
*** tylerdurden has quit IRC17:20
*** baoli has joined #openstack-infra17:22
jeblairclarkb: we need to restart zuul though...17:23
*** shashankhegde has joined #openstack-infra17:23
clarkbjeblair: right, I think we turn it off, do deletion then start zuul17:24
jeblairclarkb: so i think it's stop zuul, stop old nodepool, delete all old nodes, start new nodepool, start zuul17:24
clarkbthat works17:24
*** doug-fish has quit IRC17:24
fungithough deleting that many nodes will take probably 20 minutes or more17:24
*** TimO_ has joined #openstack-infra17:25
*** armax has joined #openstack-infra17:25
clarkbya so the optimization is we can just delete ready nodes first17:26
fungii suppose we could minimize missed events by putting zuul in graceful shutdown, stopping nodepool, and then removing ready/building nodes in parallel with the running jobs wrapping up17:26
*** MaxV has quit IRC17:26
*** marun has joined #openstack-infra17:26
*** armax has quit IRC17:27
fungiand possibly then deleting straggler used/delete state nodes in parallel with starting nodepoold on nodepool-dev17:27
clarkboh I see why zuul restart is important means we don't ahve to wait on the other nodes. So ya delete ready, start new nodepool, start zuul, delete all other old nodepool nodes would potentially be quicker17:27
jeblairyeah, i think that would work.  but then we're committed to whatever timetable zuul has17:27
jeblairthat was for fungi ^17:27
*** nelsnelson is now known as nels|afk17:28
TimO_As of Friday, my ssh gerrit stream events stopped (ssh -p 29418 gerrit stream events) for my OS Jenkins testing17:28
fungitrue. weighing missing more events in the stream vs letting jobs pile up for longer17:28
*** tnovacik has joined #openstack-infra17:28
*** tonytan4ever has quit IRC17:29
anteayaTimO_: you are testing operating systems?17:29
*** dkranz has joined #openstack-infra17:29
fungiTimO_: when you run that you see no events right now?17:29
TimO_yes as of a few min. ago...I tried it.  I will attempt again right now17:30
clarkbwe restarted gerrit over the weekend which would kill existing connections17:30
fungiTimO_: also it's stream-events (there's a hyphen)17:31
fungiTimO_: and i'm getting events on the stream17:31
TimO_my typo...i always put it in (just not this time...typing it in here :-) thks...17:31
fungiall old images on nodepool-dev are now deleted. working on rebuilding them now17:32
*** nels|afk has quit IRC17:33
TimO_will try again via jenkins automation17:33
*** wuhg has quit IRC17:33
*** zz_avozza is now known as avozza17:35
anteayaclarkb: thank you17:36
clarkbzaro: good morning17:36
anteayamorning zaro17:36
*** Hal_ has quit IRC17:36
anteayafungi: I hope you are not waiting for us, we are a ways away17:36
clarkbI have put the old third party groups into "inactive" mode but not removed any members just yet17:36
anteayaclarkb: awesome17:37
fungianteaya: nope--working on nodepool stuff17:37
anteayafungi: awesome17:37
*** jgallard has quit IRC17:37
anteayafungi: I will give you a heads up when I feel we are close enough to schedule some of your time17:37
fungianteaya: thanks!17:37
TimO_$ ssh -p 29418 gerrit stream-events <-- is hanging now, no events17:38
*** yamamoto has joined #openstack-infra17:39
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add support for new patchset-created exclude fields
clarkbTimO_: you do need to waitfor events to happen though our gerrit should be pretty active17:41
clarkbTimO_: I would give it at least a few minutes17:41
openstackgerritMerged openstack-infra/jenkins-job-builder: fix typo, must be 'classifier' to appear on
clarkbok I am going to step out for a short bit to eat some breakfast17:43
*** pcrews has joined #openstack-infra17:44
*** shashankhegde has quit IRC17:46
openstackgerritQin Zhao proposed openstack/requirements: Move retrying dependency to version 1.2.3
*** e0ne has quit IRC17:46
fungiall snapshot-based images updating on nodepool-dev now... working on dib-based next17:47
*** SumitNaiksatam has joined #openstack-infra17:47
*** yamamoto has quit IRC17:49
*** doug-fish has joined #openstack-infra17:52
*** stevemar has joined #openstack-infra17:52
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/devstack-gate: Add the new oslo.context
funginodepool-dev diskimages updating now too17:54
*** dannywilson has joined #openstack-infra17:56
*** dannywilson has quit IRC17:56
clarkbdib is less critical as we dont run any non test dib jobs on them but may as well17:59
*** nuritv has quit IRC17:59
lifelessmordred: Its a tiny patch, I've just been holidaying hard17:59
*** timrc is now known as timrc-afk18:00
mordredlifeless: yah, I know you're holidaying18:00
mordredlifeless: as well you should18:00
*** Ryan_Lane has joined #openstack-infra18:01
fungiclarkb: 2014-12-08 17:56:21,837 INFO fatal: destination path '/root/.cache/image-create/source-repositories/devstack_9383d75c34a24bf0cd2ce50da1f4b74274dfb57b.tmp' already exists and is not an empty directory.18:01
*** achanda has quit IRC18:01
fungiclarkb: devstack-trusty-dib died at the same spot18:01
jeblairyolanda: ^18:01
*** derekh has quit IRC18:02
fungiclarkb: but devstack-centos7-dib is still running fine18:02
*** ParsectiX has joined #openstack-infra18:02
*** achanda has joined #openstack-infra18:02
fungimakes me think we have a race on some assumption that there won't be parallel builds using the same tempspace?18:02
fungier, cachespace18:02
fungimy theory is that devstack-centos7-dib created that and so devstack-precise-dib and devstack-trusty-dib found it and failed trying to cache the same18:03
fungiooh, we've had an image go ready in nodepool-dev now18:04
*** patrickeast has joined #openstack-infra18:04
fungi(77 more snapshot images to go!)18:04
clarkbfungi: ya that may be a race in dib caching18:04
*** cpowell has quit IRC18:05
fungi3 images ready18:06
*** achanda has quit IRC18:06
fungifalling like dominoes now18:06
*** armax has joined #openstack-infra18:06
*** ivar-lazzaro has joined #openstack-infra18:06
*** amcrn has joined #openstack-infra18:07
clarkbfungi: if you restart trusty/precise-dib image builds now do they fail again?18:07
fungiclarkb: not so far any way18:07
clarkbif they don't then I think it is likely that dib caching is not thread safe18:07
fungithe error message alone makes me think this is likely18:07
*** melwitt has joined #openstack-infra18:07
clarkbwe can probably make it so that all jobs run without updating the cache then have a separate thing keep the cache up to date18:07
clarkbmordred: ^18:08
*** mattfarina has joined #openstack-infra18:08
fungiperhaps concatenating the pid or some random nonce in the cache tempfile string would solve it18:08
*** ivar-lazzaro has quit IRC18:08
*** MaxV has joined #openstack-infra18:08
clarkbfungi: perhaps but then we will have X copies of the thing when we want 118:08
fungias long as the cache updates are atomic per file (which i expect they are or it wouldn't be updating to a tempfile)18:08
*** ivar-lazzaro has joined #openstack-infra18:09
* mordred reads18:09
*** Parsecti_ has joined #openstack-infra18:10
clarkbianw: when you wake up has some comments from me. Would like to see that move forward so let me know what you think18:10
*** ParsectiX has quit IRC18:10
mordredSpamapS: ^^18:10
mordredclarkb: I'll dig in to it18:10
*** Masahiro has joined #openstack-infra18:11
fungialso, when nodepool diskimage-build fails, it ends with a message of "image blah is ready" which is sorta misleading18:11
*** jistr has quit IRC18:11
SpamapSI believe we only ever do one dib build per VM right now, so it's possible the cache has grown some non-atomic bits18:11
fungias it comes right after the message about "image blah deleted"18:11
SpamapSWe'd certainly want it to stay coherent through parallel builds.18:12
*** jp_at_hp has quit IRC18:12
SpamapSsource-repositories is certainly the most complex part of the cache.18:12
*** emagana has joined #openstack-infra18:13
*** cpowell has joined #openstack-infra18:13
SpamapSah yeah that looks entirely non parallelizable.18:14
*** Masahiro has quit IRC18:15
*** otter768 has joined #openstack-infra18:15
SpamapSWe just go ahead and skip the git command if the dir exists18:15
*** jlibosva has quit IRC18:16
SpamapSso then the second encounter leads to assuming this empty dir is the git dir.18:16
clarkboh is this fixed but not in mordred fork?18:16
SpamapSthis is broken period18:16
*** _nadya_ has quit IRC18:16
SpamapSfurther down18:16
*** tonytan4ever has joined #openstack-infra18:17
SpamapSso we probably need to use an flock to make the second one wait18:17
*** mikedillion has quit IRC18:19
*** otter768 has quit IRC18:20
*** harlowja_away is now known as harlowja18:20
*** mikedillion has joined #openstack-infra18:20
fungiopened!/story/2000067 for the misleading failure message "image foo is built"18:20
clarkbfungi: thank you, I will take a look at that now18:21
clarkbkrotscheck: is there a reason I typically have to refresh opening a bug like ^ to see the contents?18:21
TimO_My ssh -p 29418 timed out with this error:  "ssh_exchange_identification: read: Connection timed out"18:22
TimO_any ideas here? thks...18:22
*** mikedillion has quit IRC18:22
fungii also think we should add a convenience redirect for[0-9]+)$ to!/story/$118:22
*** nfedotov has joined #openstack-infra18:22
* SpamapS almost done w/ flock fix18:22
*** shashankhegde has joined #openstack-infra18:22
*** e0ne has joined #openstack-infra18:23
fungiTimO_: perhaps your firewall administrator went rogue and blocked egress for that?18:23
fungiSpamapS: i have a similar parallelism-related issue i think on a subsequent retry18:23
*** Parsecti_ has quit IRC18:23
*** skolekonov has quit IRC18:23
fungiSpamapS: looking over the log now to see if i can figure out where it went off the rails18:23
fungiSpamapS: "Did not get /opt/dib_cache/source-repositories/repositories_flock"18:24
*** tonytan4ever has quit IRC18:25
SpamapSoh there may be an flock at a higher level18:25
TimO_possible!  thks will check out here18:26
SpamapSdoh forgot to look at that18:26
fungiSpamapS: yeah, i had two dib runs die on that one this most recent pass, with a third still running18:26
SpamapSoh well thats dumb18:26
fungii'm going to retry them both for a third time18:26
SpamapSthey take exclusive on all runs18:26
SpamapSlooks like I need to have a chat about locking w/ our review team. :)18:27
SpamapSfungi: ok so the problem is, dib tries to serialize _all_ dib runs basically18:28
*** mriedem has quit IRC18:28
fungiSpamapS: wonderful! and we don't...18:28
fungiso, er, yeah failcity18:28
jeblairfungi, mordred: we may want to have manage-projects make one of these:
jeblairfungi, mordred:
SpamapSwhat should be happening is "if you're populating a cache, exclusive, otherwise, shared until populating is done"18:29
SpamapSbut because it's a massive coarse grained lock..18:29
SpamapSit's just a massive serializer18:29
fungijeblair: oh, neat. a single locked pull request saying we don't take pull requests is a great idea18:29
SpamapSfungi: ok so basically all clones have to finish in 10 minutes or the flock fails.. seems like what might be happening18:30
jeblairwill file story18:30
*** amitgandhinz has joined #openstack-infra18:30
jeblairkrotscheck: i thought storyboard copied the title into the initial task?  that does not seem to be happening18:31
jeblair(i seem to have to type it twice)18:31
fungii noticed the same when i opened the cosmetic logging story for nodepool just now18:31
krotscheckjeblair: Huhn. It used to.18:31
openstackgerritClark Boylan proposed openstack-infra/nodepool: Fix error handling when dib image builds fail
clarkbfungi: ^18:32
*** shashankhegde has quit IRC18:32
jeblair!/story/2000068 filed18:32
fungiclarkb: Story: #2000067 in the commit message? i'll go ahead and update the status18:33
clarkbfungi: ya I need to figure out what the proper string is or that18:33
clarkbis it still closes-bug?18:33
*** mrunge has quit IRC18:33
*** Ryan_Lane has quit IRC18:33
*** andreykurilin_ has joined #openstack-infra18:33
fungioh, maybe. at one point we were doing Story: for the stuff in storyboard, but maybe we normalize to make it the same as for lp since we're importing anyway?18:33
fungithere are likely downsides to either approach18:34
jeblairit's premature to fix that18:34
jeblairwe need to determine how to address a task first18:34
clarkbgit grep -i story doesn't show much in jeepyb so must still be closes-bug18:34
jeblairuntil then, it's basically just informative18:34
jeblairthere's no automation around it anyway18:35
clarkbjeblair: fungi: in that case do people have a preference for what I use in the commit?18:35
jeblairi would stick with "Story:" for now18:35
openstackgerritClark Boylan proposed openstack-infra/nodepool: Fix error handling when dib image builds fail
*** Ryan_Lane has joined #openstack-infra18:36
*** tonytan4ever has joined #openstack-infra18:36
armaxanteaya: ping18:37
anteayahi there armax18:38
*** sputnik13 has quit IRC18:38
*** avozza is now known as zz_avozza18:39
anteayaif a job has has a provision in zuul/layout.yaml to not test stable branches, do we also need to specify icehouse and juno?18:39
fungiclarkb: in act, i just noticed the issue you're addressing in 140106 is significantly worse than i thought. they're being logged as built because nodepool is adding them to the database in a ready state according to nodepool dib-image-list18:39
anteayafor instance, should I request that juno is specified?18:39
clarkbfungi: yup18:39
*** pelix has quit IRC18:39
fungiclarkb: yikes. glad we weren't relying on this yet :/18:39
*** yfried_ has joined #openstack-infra18:40
*** eharney has quit IRC18:40
*** eharney has joined #openstack-infra18:40
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: First task tracks story title.
krotscheckjeblair: ^^18:42
SpamapSfungi: can you open a bug report against dib with the exact error you're getting?18:42
SpamapSfungi: it will help with the review process. :)18:42
fungiSpamapS: sure18:42
SpamapSfungi: ty18:42
*** e0ne has quit IRC18:43
fungiclarkb: jeblair: all but 6 snapshot images are completed now, the stragglers are retries (mostly in rax) because of instance boot/ssh connectivity failures18:43
jeblairkrotscheck: opened based on something fungi said a moment ago:!/story/200006918:43
*** SumitNaiksatam has quit IRC18:43
jeblairkrotscheck: and thx18:43
*** sputnik13 has joined #openstack-infra18:44
*** shashankhegde has joined #openstack-infra18:44
clarkbis there a plan for addressing tasks yet? maybe something similar to gerrit patchsets?18:45
clarkbeg 123456,318:45
jeblairclarkb: the thing we run into is that they can be removed and reordered18:46
clarkbthough that may imply order which isn't the same as reality in the task list18:46
clarkbya that18:46
clarkbUUID all the things?18:46
anteayamtreinish: can you look at this? this editing job runs non-voting on tempest:
jeblairotoh, 123456,0f24beb72b340628a564ce8d755341eca70a4c40 is not very friendly :)18:46
jeblairbut maybe would actually be okay...18:47
clarkbjeblair: and if the uuid is actually unique we can drop the story too18:47
clarkbjeblair: and directly address the task18:47
*** sdake has quit IRC18:47
anteayamtreinish: edited job18:47
*** sdake has joined #openstack-infra18:47
*** sdake has quit IRC18:47
*** sdake has joined #openstack-infra18:47
*** teran has quit IRC18:47
jeblairmaybe you refer to stories by friendly number, and tasks by scary uuid, but in a commit message maybe that's fine, and yeah, it could just be the uuid which implies the story so is sufficient for a link.18:47
SpamapSok so this is kind of interesting18:48
fungiSpamapS: bug 140045018:49
uvirtbotLaunchpad bug 1400450 in diskimage-builder "source-repositories caching is not concurrency-safe" [Undecided,New]
*** e0ne has joined #openstack-infra18:49
fungiclarkb: jeblair: perhaps allow task uuids to be abbreviated to arbitrary length like git does with shas?18:50
SpamapSwe have a cache per cloned URL, but it manipulates HEAD ... so we have to hold the exclusive lock through the whole process.18:50
*** yamamoto has joined #openstack-infra18:50
*** sputnik13 has quit IRC18:50
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add support for new patchset-created exclude fields
jeblairkrotscheck: have we had a conversation about when i type "jeepyb<enter>" into the search box and end up with "jee" as the search string?  because i apparently type faster than a computer can read.  (?)18:50
*** sputnik13 has joined #openstack-infra18:51
SpamapSuntil we do the second local clone to the REPO_DEST18:51
krotscheckjeblair: WE have not.18:51
*** MaxV has quit IRC18:51
jeblairclarkb, fungi, anteaya: added [LHF] to title of this story!/story/2000068 to indicate 'low hanging fruit' since we don't have tags18:51
jeblairkrotscheck: is that a known thing?  file story or no?18:52
krotscheckjeblair: That’s a usability discussion.18:52
fungijeblair: like typing on an acoustically coupled 110-baud modem and watching the echo catch up with your keystrokes?18:52
*** vigneshvar has joined #openstack-infra18:52
jeblairkrotscheck: are you saying that it is not clearly a bug?18:52
krotscheckYep. it is not clearly a bug, because it’s a combination of your typing speed, the speed at which the criteria are resolved, and the fact that you still have to hit the down arrow to select one of the returned criteria to be able to navigate to it.18:53
krotscheckor use it.18:53
anteayajeblair: thank you18:54
anteayaand a nice [LHF] story it is18:54
*** rediskin has left #openstack-infra18:54
jeblairkrotscheck: i don't have to hit the down arrow.  if i type "jeepyb<enter>" very slowly it searches for jeepyb.  if i type "jeepyb<enter>" at the speed at which a first year typing student might type it, i get "jee".18:55
*** yamamoto has quit IRC18:55
jeblairkrotscheck: i think there's only one variable there.18:55
krotscheckOh, different issue.18:55
krotscheckYour bug seemed to indicate the latter.18:55
anteayafungi: status update on force push, we have -2'd any patch we fill might be close to merging and creating havoc durng the split, dougwig has fresh repos created and has shas for all three repos, mestery has reviewed the neutron patch to remove the split out code from neutron/ we are waiting on carl_baldwin for his additional review on the netron code removal patch before we move aheahd18:56
jeblairok, i'll file a story then18:56
openstackgerritSteven Weston proposed openstack-infra/system-config: Adds puppet system-config module split script
dougwigrepo info:18:56
anteayafungi: -dev and -ops mls have been notified of the split and being told to expect new repos won't work until next week at the earliest18:57
swestonasselin: ^ there you go18:57
anteayaanything we have left out?18:57
krotscheckjeblair: How many milliseconds do you have between keystrokes?18:57
* krotscheck can’t believe he just asked that question18:57
* jeblair can't believe he doesn't know off the top of his head ;)18:58
asselinsweston, got it18:58
*** jedimike has quit IRC18:58
swestonasselin: k :-)18:58
*** dannywilson has joined #openstack-infra18:58
* jeblair is writing a program to measuer the speed he types 'jeepyb<enter>'18:58
SpamapSfungi: ok so the one about .tmp existing and not being empty is actually very weird and should not happen given the flocking in use.18:59
*** emagana has quit IRC19:00
SpamapSfungi: the other one makes all fetching of all remote things a serial operation.. and if any of them takes longer than 10 minutes.. all waiting builds fail.19:00
fungiSpamapS: yeah, either there's a race engaging the flock or that's happening outside the flock?19:00
*** davideagnello has joined #openstack-infra19:00
*** emagana has joined #openstack-infra19:00
*** armax has quit IRC19:01
SpamapSfungi: basically nothing in that script happens outside the flock except making the cache base if it doesn't exist, and some manifest stuff.19:01
SpamapSI wonder...19:01
fungianteaya: dougwig: thanks for the update19:01
SpamapSmaybe something is already using fd 919:01
SpamapSand closes it later on19:02
clarkbfungi: anteaya dougwig not sure what the context of those shas is19:02
*** dannywil_ has joined #openstack-infra19:02
*** dannywilson has quit IRC19:02
fungiclarkb: i'm assuming they're related to whatever carl_baldwin (not in channel?) is supposed to be checking19:03
*** emagana_ has joined #openstack-infra19:03
fungiSpamapS: could there be two flocks fighting over one file descriptor?19:03
jeblairkrotscheck: i think i was in the range of 180-220 ms per keystroke at the time.  i am now, for some reason, much faster at typing 'jeepyb' (down to 120!)19:04
anteayasorry the shas are the directories that are sitting in github which fungi will soon force push to the newly created (empty) neutron repos splitting out advanced services19:04
krotscheckjeblair: :-P19:04
clarkbanteaya: I am not sure why the shas are relevant htough19:04
clarkbanteaya: we push everything into the repos19:04
*** emagana has quit IRC19:04
*** doude_ has quit IRC19:04
anteayabeing overly cautious19:05
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of Ask website
ashphey I just wanted to swing by and say a huge thank you to the amazingly helpful people of this channel19:05
ashpmy migration was (eventually) a success and everyone is happy19:05
krotscheckjeblair: So the issue I think is that the typebox doesn’t kill outstanding criteria searches as you’re typing, which quickly exhausts the browser’s query pool. So the solution will require adding both a ms delay to the typeahead resolution (to reduce the number of queries) and to kill pending queries.19:05
fungiashp: you got it all working? congrats!19:05
krotscheckI can do one very quickly.19:05
clarkbashp: woot19:05
krotscheckThe other will take tiem.19:06
ashpNow we can actually have serious conversations about what to do next and where to go from here :)19:06
ashpNow we're not stuck on gerrit 2.4 forever19:06
fungiwaiting for one last snapshot update to finish, and then they're all done building19:06
clarkbfungi: yay19:06
*** timrc-afk is now known as timrc19:06
mordredjeblair: yes on manage-project pull request issue thing19:07
*** dtantsur is now known as dtantsur|afk19:07
SpamapSfungi: seems unlikely.. bash will wait for processes that it forks and execs, and I'd assume the sub-shell opening 9 will be waited on as well.19:07
SpamapSfungi: what _MIGHT_ be possible is flock gets killed or dies.19:08
*** sputnik13 has quit IRC19:08
SpamapSactually no19:09
SpamapSit isn't left running, the process has the flock19:09
*** sputnik13 has joined #openstack-infra19:09
* SpamapS will continue to investigate a bit later19:10
fungiclarkb: also dib images are done building and in the process of being uploaded19:10
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Added millisecond delay to reduce the number of queries.
krotscheckjeblair: ^^19:10
anteayafungi: carl_baldwin is +2 on, we are ready for you to do the force push, mestery is standing by to +A 139901 once we have confirmed the last commits on all three repos19:10
anteayafungi: do you have what you need to force push19:11
clarkbfungi: you going to do it then? remember to do what jeepyb does if we need to grab all the branches and tags and things19:11
*** cnesa has quit IRC19:11
*** dprince_ has quit IRC19:11
fungiclarkb: yep19:11
fungianteaya: this should be sufficient yes. looking over it now19:11
clarkbI think swapping it out directly on disk will require a reindex which is less desireable19:11
mesterythanks fungi and anteaya19:12
anteayafungi: great, let me know if you need anything more19:12
fungiclarkb: i'm going to push --force in as a project bootstrapper19:12
dougwigno branches or tags to worry about.19:12
clarkbfungi: sounds good19:12
clarkbdougwig: there should be though19:12
*** SumitNaiksatam has joined #openstack-infra19:12
clarkbdougwig: you should have stable/* and release tags19:12
mordredclarkb, jeblair: I want to put fully fixing rax-dib on hold for a week. I'll keep poking - but they're rolling out the config-drive fix for us19:12
*** sputnik13 has quit IRC19:12
fungilooking now to see if i need to care about additional tags and non-master branches19:12
mordredand I'd prefer to base what we're doing on that, rather than on nova-agent19:12
*** sputnik13 has joined #openstack-infra19:12
clarkbmordred: thats fine, we have a lot of other dib things to fix first19:12
mordredso it seems like finalizing the current code is less useful19:12
anteayafungi clarkb do say if we have more work do to here19:13
clarkbanteaya: well I expect branches and tags19:13
anteayadon't force push until you are happy19:13
mordredyah. I can hack on the parallel thing19:13
*** mwagner_lap has quit IRC19:13
clarkbbecause you are taking a repo with branches and tags and splitting some files out of that repo. Splitting should preserve those branches and tags19:13
*** pc_m_ has quit IRC19:13
mesteryclarkb: In our case, we're not interested in preserving the branches/tags in the new repos, they will still be there in the old ones.19:14
*** pc_m has joined #openstack-infra19:14
fungiclarkb: looks like no tags, and no branches besides master19:14
fungijust double-checked all three19:14
clarkbmestery: why wouldn't you be interested? that is histor...19:14
*** cnesa has joined #openstack-infra19:14
fungiclarkb: well, the branches are likely irrelevant if this is only being integrated into kilo neutron19:15
mesteryfungi: ++19:15
clarkbfungi: its relevant as history19:15
clarkbfungi: git diff master stable/foo19:15
clarkbfor example19:15
fungiclarkb: relevant until we delete those branches?19:15
clarkbfungi: they will be relaced with tags so still relevant19:15
fungiclarkb: though i guess we do tag the eventual branch state19:15
fungiclarkb: does git filter-branch handle that though?19:16
fungii've not done a filter-branch on multiple branches and preserved tags before19:16
clarkbfungi: I am pretty sure it can19:16
dougwigfyi, this is what created those repos:
mtreinishanteaya: makes sense to me, although I think that's still just a neutron job at this point19:17
clarkbfungi: -- --all passed to filter branch apparently does branches and tags19:18
fungiokay, all snapshot images and diskimages on nodepool-dev are refreshed successfully. we should be set to press forward om that front19:19
clarkbit proably increases time to run but I know one of the things that frustrates folks like sdague is when we make large moves like this and don't preserve enough history to do repo operations against important points in time19:19
*** mwagner_lap has joined #openstack-infra19:19
dougwigjust so we're all on the same page, neutron itself is *not* getting its history pruned.19:20
clarkbdougwig: understood19:20
clarkbdougwig: but if I have nuetron-foo cloned and not neutron and am on an airplane I won't have any of that info19:20
clarkbdougwig: and doing diffs across repos is harder than within a repo19:20
fungidougwig: yeah, but clarkb's right that it's useful to see what state of these files were covered in what branches/tags prior to split without having to dig that out of the neutron repo19:21
dougwigcan someone tell me exactly how to tweak this?19:23
*** marun has quit IRC19:23
anteayamtreinish: great thank you19:23
clarkbdougwig: I believe you remove HEAD and replace it with '-- --all'19:23
dougwigok, let me run that right now.19:24
*** zul has quit IRC19:25
dougwigrunning.  used to take 21 minutes.  i'll let you know.19:25
*** sarob has quit IRC19:25
*** Ryan_Lane has quit IRC19:25
anteayathank you19:25
*** esker has quit IRC19:26
*** dprince has joined #openstack-infra19:27
clarkbjeblair: fungi: I am ready to start working on nodepool switch whenever you guys are19:27
openstackgerritClark Boylan proposed openstack-infra/nodepool: Log ipv6 addresses for nodes
jeblairclarkb: ready19:30
clarkbjeblair: do we want an etherpad?19:30
jeblairclarkb: nah, we only have a few steps19:30
*** vigneshvar has quit IRC19:31
clarkband we are going with: stop zuul, stop old nodepool, delete all old nodes, start new nodepool, start zuul ?19:31
jeblairif we go with fungi's low-impact way, we need to wait at least 1 hour, possibly more, and would need to watch it the whole time19:32
fungido we want to stop old nodepool first and get ready/building node deletion underway?19:32
*** liusheng has quit IRC19:32
*** liusheng has joined #openstack-infra19:32
clarkbfungi: ya I Think first delete pass should be ready/building19:32
fungiand then delete used nodes on the old nodepool in parallel with new nodepool running?19:32
clarkbthat at least gives us the option to start new nodepool earlier19:32
clarkbjeblair: ya I am not a fan of being on zuuls schedule19:33
jeblairand also stop zuul later19:33
*** zz_jgrimm has joined #openstack-infra19:33
*** maurosr has joined #openstack-infra19:33
fungia similar alternative might be to quiesce jenkins masters instead?19:33
jeblairand even though we throw out those results, zuul continues queueing so we don't miss events19:33
clarkbI can stop old nodepool now and kick off the ready/building deletes19:33
*** andreykurilin_ has quit IRC19:33
jeblairfungi: if we do that, new nodepool will not attach new nodes to them19:33
fungifair point19:33
*** fandi has joined #openstack-infra19:34
*** amcrn has quit IRC19:34
fungithough we could do that just long enough to delete the used nodes, then start them again right after a zuul restart19:34
fungiprobably at least as messy in the end19:34
*** ociuhandu has quit IRC19:35
*** hdd has joined #openstack-infra19:35
jeblairso i think current plan is: stop old nodepool, delete ready/building, stop zuul, start new nodepool, delete all old nodes, start zuul19:35
fungithat works and minimizes zuul downtime without too much complexity19:35
clarkbjeblair: sounds good. I think we don't have to wait for al old nodes to be deleted before we start zuul. we will just hit quota issues in the worst case19:35
clarkbbut I like that plan19:35
clarkbshould I stop old nodepool and start deleting things?19:36
fungii say go for it19:36
jeblairjust a sec19:36
*** cpowell has quit IRC19:36
* nibalizer having deja vu :P19:36
jeblairstop old nodepool, delete ready/building, stop zuul, start new nodepool, wait for representative set of nodes, start zuul, delete all old nodes19:36
jeblairclarkb, fungi: revised ^ ?19:36
fungijeblair: boh, right. otherwise the dreaded not_registered19:37
clarkbhuge +2 :)19:37
openstackgerritMerged openstack-infra/project-config: Make Neutron DVR job run on the full tempest suite
jeblairokay, sounds like we're good; clarkb: ready for you to proceed19:38
jeblairi'm standing by on zuul19:38
clarkbstopping old nodepool now19:38
*** shakamunyi has joined #openstack-infra19:38
*** zul has joined #openstack-infra19:38
jeblairalso, we're currently at approx 1/2 capacity, so not the worst time to do this :)19:39
mrmartinfungi, clarkb: so the askbot.o.o almost passing the gate now, have a single problem only: I needed to add puppetlabs-concat, and it have a dependency on puppetlabs-stdlibs, but we are pinning stdlibs to 4.3.2 and puppetlabs-concat deploys a 4.4.0 as a dependency (works well anyway with 4.3.2) too. Any idea how to enforce this concat module to install after 4.3.2 stdlibs? :)
clarkbits stopped I am going to start deletions now19:39
clarkbstarting with building so that the ready nodes have a chance to change state19:39
fungimrmartin: i thought we had dealt with this once already... nibalizer: ^ ?19:39
*** krtaylor has quit IRC19:40
mrmartinthis is the full url for wrong puppet module order:
*** baoli has quit IRC19:40
*** amcrn has joined #openstack-infra19:40
jeblairSlickNik: gate-trove-functional-dsvm-mysql seems like a very slow job19:40
*** zz_jgrimm is now known as jgrimm19:40
*** cpowell has joined #openstack-infra19:41
pleia2oof, it's very clear to me now that this zanata "ansible to puppet translation" really is "write puppet stuff, use RedHat's ansible playbooks as a reference" - so much Red Hat and JBoss (we're using Widlfly) specific stuff in here19:42
*** weshay has quit IRC19:42
pleia2fortunately there's an open source wildfly puppet module I'm testing now, so that helps19:42
*** vigneshvar has joined #openstack-infra19:42
*** baoli has joined #openstack-infra19:43
*** cpowell_ has joined #openstack-infra19:43
*** KurtMartin has joined #openstack-infra19:43
nibalizerfungi: mrmartin hrm19:43
anteayapleia2: :(19:44
*** jcoufal has quit IRC19:44
nibalizerso i dont think concat enforces specific stdlib versions19:44
*** cdent has quit IRC19:44
openstackgerritSean M. Collins proposed openstack-infra/devstack-gate: Introduce support for running IPv6 networking in Neutron
nibalizer      "name": "puppetlabs/stdlib",19:45
nibalizer      "version_requirement": ">= 3.2.0 < 5.0.0"19:45
mrmartinnibalizer: seems to be that real dependency of modules not enforced here, and add to MODULES array keep the modules in ascending order by their names.19:45
*** Rockyg has joined #openstack-infra19:45
pleia2anteaya: it's ok, I kind of expected this, these were wrestled from internal RedHat, so they are bound to be a bit messy :)19:45
*** emagana_ has quit IRC19:45
pleia2only really written for internal consumption19:45
mrmartinnibalizer, you are right it is not, but puppetlabs-concat < puppetlabs-stdlibs, and puppetlabs-concat tries to deploy his dependency19:45
mrmartinso it will collide with a specific stdlibs version.19:46
*** emagana has joined #openstack-infra19:46
*** cpowell has quit IRC19:46
*** kmartin has quit IRC19:47
nibalizerso concat and stdlib are both being pulled in by other modules19:48
*** _nadya_ has joined #openstack-infra19:48
nibalizeri think19:48
nibalizerprobably postgresql is pulling in concat19:48
anteayapleia2: glad this isn't a surprise for you then19:48
clarkbdeletions seem to have slowed down19:48
mrmartinnibilizer: I specified concat there, do you think it is pulled by other modules also?
mrmartinoh nice, I checking that.19:49
*** sputnik13 has quit IRC19:49
nibalizercan you link the exact error that you're seeing in the console.html from above?19:49
clarkblooks like we may be waiting on clouds to tell us they have deleted things19:50
mrmartin2014-12-08 19:11:12.011 | [1;31mError: Could not install module 'puppetlabs-stdlib' (v4.3.2)19:50
mrmartin2014-12-08 19:11:12.011 |   Module 'puppetlabs-stdlib' (v4.4.0) is already installed19:50
*** emagana has quit IRC19:50
mrmartinthis is the dependency line of pgsql: {"name":"puppetlabs/concat","version_requirement":">= 1.1.0 <2.0.0"}19:50
*** Ryan_Lane has joined #openstack-infra19:50
dougwiganteaya, fungi: fresh repos:19:51
fungidougwig: thanks--checking over them now19:51
mrmartinnibalizer: so if I remove the concat definition from modules.env, it must work, because pgsql pulls in the puppetlabs-concat19:52
*** sputnik13 has joined #openstack-infra19:52
*** sputnik13 has quit IRC19:52
nibalizerwell 'it must work' is somethig thats hard to agree to19:52
*** julim has quit IRC19:53
nibalizerbut if your asking if concat is a transitivie dependency that you can depend  on then yes, it will be there19:53
*** tkelsey has joined #openstack-infra19:53
fungidougwig: the url for x-neutron-lbaas 404s for me, and the other two seems to still have no branches or tags. did you push all up to github or just your worktree's current active branch?19:53
dougwigyep, messed up the lbaas push.19:54
mrmartinnibalizer: this will work until we are not removing the puppetlabs-postgresql module :)19:54
*** radez is now known as radez_g0n319:54
dougwigsec, i think i pushed wrong.  standby.19:54
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of Ask website
*** armax has joined #openstack-infra19:55
clarkbjeblair: fungi according to nodepool list the ready nodes are gone but my delete command has not returned19:55
clarkbjeblair: still waiting on ~20 building nodes to delete19:55
*** krtaylor has joined #openstack-infra19:55
mrmartinso the conclusion here, it is really a good idea to move the askbot and solr modules into puppetforge, where they can precisely define their module dependencies.19:55
jeblairclarkb: think it's run down enough to make it worth continuing?19:56
*** radez_g0n3 is now known as radez19:57
clarkbjeblair: ya I think the ready nodes were the most problematic and with them gone we should be safe to stop zuul and start new nodepool19:57
*** weshay has joined #openstack-infra19:57
jeblairwho's standing buy to start new nodepool?19:58
jeblairstanding by19:58
clarkbI can start it19:58
jeblairack.  i will stop zuul now...19:58
jeblairer, hold a min19:58
nibalizermrmartin: +119:59
*** cpowell_ has quit IRC19:59
jeblairchange at top may be about to report... checking on it19:59
*** cpowell has joined #openstack-infra19:59
*** Masahiro has joined #openstack-infra19:59
*** amitgandhinz has quit IRC19:59
jeblairmaybe not.  proceeding.20:00
*** amitgandhinz has joined #openstack-infra20:00
dougwigi think i need to pull all the remote branches local before running the split, else they don't get pulled along.  standby.20:00
jeblairclarkb: zuul is stopped, ready for you to proceed20:00
clarkbjeblair: starting new nodepool now20:00
clarkbjeblair: do you have out of band geard running?20:00
jeblairclarkb: is now20:00
clarkbnew nodepool is starting20:01
clarkbit just did allocation calculations /me checks the building list20:01
*** baoli has quit IRC20:01
clarkblooks like it is building all the things20:02
mtreinishmordred: if you get a sec can you take a look at the mysql-proxy patch, if the approach doesn't make sense I can respin it to use something else.20:04
*** Masahiro has quit IRC20:04
fungidougwig: yes, you need a full clone for this to work properly20:04
dougwigi did a checkout on each branch to get it local.  is there more that needs to happen?20:04
fungidougwig: it's easier to do this with a bare clone, but your script looks like it wants to be able to inspect files20:05
*** ZZelle_ has joined #openstack-infra20:05
*** emagana has joined #openstack-infra20:05
dougwigi stole that from oslo-incubator.  if there's a better way, i can do that.20:06
clarkbwe haev devstack-trusty, bare-centos6, devstack-centos7, and bare-f20 nodes ready20:06
clarkbwaiting on bare-trusty, bare-precise and devstack-precise20:06
clarkblooks like we have those and py3k-precise now20:06
clarkbanyone else want to double check that list of ready nodes before we start zuul?20:06
fungiclarkb: suggestions on how to do a full clone which is also non-bare?20:06
fungiclarkb: yeah, checking now20:07
clarkbfungi: dougwig: for branch in git branch -a | grep origin | do some sed | checkout -b branchname origin/branchname ?20:07
*** mpaolino has quit IRC20:08
dougwigok, that's what i did.20:08
fungiclarkb: also checkout the list of tags?20:08
jeblairi stopped oob geard since i don't think we need it anymore; waiting for fungi's check before starting zuul20:08
dougwigthose came along already.20:08
dougwigsplit is running again.20:08
clarkbfungi: ya tags should magically get pulled in by the branches20:08
clarkbjeblair: ya we shouldn't need it anymore20:08
fungiclarkb: jeblair: looks like we now have bare-centos6, bare-f20, bare-precise, bare-trusty, devstack-centos7, devstack-f20, devstack-f20-virt-preview, devstack-precise, devstack-trusty, devstack-trusty-2-node, py3k-precise20:10
clarkbok building and ready node deletions are completely done on old nodepool. I am starting deletion of the other nodes on old nodepool now20:10
fungii think that's everything we care about, unless we want to wait until tripleo jobs won't not_registered too20:10
fungiwe're also waiting on devstack-precise-dib and devstack-trusty-dib but those aren't running anything critical20:11
clarkbdib should be fine, we may consider waiting for tripleo20:11
*** Longgeek has quit IRC20:11
*** dizquierdo has joined #openstack-infra20:11
jeblairtripleo are building?20:11
*** rushiagr is now known as rushiagr_away20:11
*** koolhead17 has quit IRC20:11
*** markmcclain has quit IRC20:12
fungino idea how successfully. they may be quota-constrained there20:12
fungichecking logs now20:12
clarkbI am deleting those nodes on the old side by hand now in hopes that it helps buils20:13
*** gyee has joined #openstack-infra20:13
*** subscope_ has joined #openstack-infra20:14
jeblairhrm, i think we should proceed20:14
clarkbjeblair: fine with me20:14
jeblairfungi: ?20:14
fungiyeah, go for it20:15
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: deprecate postbuildscript onsuccess and onfailure parameter names
fungithey can catch up if they need to20:15
jeblairstarting zuul20:15
clarkbwe are under 300 nodes on old nodepool side now20:15
*** ldnunes has quit IRC20:16
fungitripleo-f20 just went ready20:16
*** otter768 has joined #openstack-infra20:16
fungistill waiting for tripleo-precise20:16
fungii see used nodes already20:17
fungigood sign20:17
*** mriedem has joined #openstack-infra20:17
*** markmcclain has joined #openstack-infra20:18
clarkbwe are hitting some quota limits but that should go away as old nodepool removes nodes20:18
clarkbwhich it is doing at a reasonable rate20:18
jeblairenqueue is complete; merger queue is empty20:18
jeblair(which i think is a good thing)20:19
*** hdd has quit IRC20:20
*** cnesa has quit IRC20:20
clarkbunder 250 old nodes now20:21
*** tonytan4ever has quit IRC20:21
*** otter768 has quit IRC20:21
SpamapSfungi: going back to the dib problems... is it possible the filesystem that the flock's are running on are on AFS?20:22
SpamapSfungi: if so.... I think I know what's up20:22
fungiSpamapS: nope! (though that would be pretty cool)20:22
clarkbSpamapS: no its an ext4 ephemeral volume20:22
SpamapSOk, and also is it possible it's on top of a union filesystem?20:23
SpamapSchroot.. etc.?20:23
clarkbSpamapS: rax gives the volume to us ot could be hosted on all osrts of weirdness :)20:23
SpamapSBasically, as long as fd#9 remains open the flocks should see eachother.20:23
SpamapSclarkb: no not at this level20:24
SpamapSI mean, on the box is it an aufs or something similar.20:24
SpamapSsounds like no20:24
clarkbno its just ext420:24
*** nfedotov has quit IRC20:24
clarkbno lvm either20:24
*** radez is now known as radez_g0n320:24
SpamapSnothing below the filesystem layer matters.. flock, however, is an in-box process-namespaced thing.20:25
SpamapSso another possibility.. are these running inside containers?20:25
clarkbdib doesn't like running in a container20:25
SpamapSThere are patches up to fix that.20:25
SpamapSNot sure why they haven't landed it's been months actually. :-/20:26
clarkbSpamapS: ya I am still skeptical because you have to let it do things like mount20:26
clarkbwhich for dib is valuable beacuse you can build as if you were in a different OS and you can likely deal with isolation in oither ways20:26
clarkbbut it always seems to me like a failure if we have to unisolate isolated things20:26
clarkbunder 200 ndoes now20:26
*** alexpilotti has quit IRC20:27
SpamapSmount is something that can be done by users given the appopriate capabilities.20:27
SpamapSbut that is neither here nor there.20:27
clarkbSpamapS: yes and it completely undoes any isolation you might have :)20:27
SpamapSthe mounting is entirely so we can build the end product efficiently and produce an actual filesystem.20:27
SpamapSIf you have a way to do that without a VM, I'm all ears. :)20:28
*** radez_g0n3 is now known as radez20:28
clarkbSpamapS: I think dib should be mounting under the container then "booting" the container in the mount20:28
*** salv-orlando has quit IRC20:28
clarkbSpamapS: and it should do that regardless of the mount because my local machine spends most of its time CPing stuff into the mounted location20:28
clarkber regardless of containers or not20:28
SpamapSthats like, 'rewrite the whole thing to always use containers'. A fine idea, but not an easy implementation to swallow. :)20:29
clarkbtechnically its a mv but since it crosses FS boundaries it is slow like a cp20:29
clarkbSpamapS: no its not20:29
*** timrc is now known as timrc-afk20:29
clarkbdib should be doing this anyways regardless of containers. I just haven't gotte naround to fixing it yet20:29
SpamapSoh I know what you're saying20:29
SpamapSyeah we tried that. it's slow.20:29
SpamapSRunning against tmpfs is about 2x faster.20:30
SpamapSdpkg is to blame.20:30
clarkboh we don't tmpfs so I don't care20:30
clarkbso my interest is making not tmpfs not suck20:30
SpamapSand $(all the other things that use sync())20:30
mordredSpamapS: did you see my "make dpkg not use sync()" patch?20:30
clarkbSpamapS: something like if not tmpfs then do what clarkb says20:30
SpamapSclarkb: we have these other users that do use it.. so.. please don't regress their experience.20:31
SpamapSmordred: yeah I think that would probably get 80% of the speedup tmpfs did.20:31
SpamapSwhich might be enough to ditch the tmpfs and cp method.20:31
*** shashankhegde has quit IRC20:31
SpamapSwe also tried eatmydata btw.. but that broke a bunch of stuff20:32
fungii have just wasted far too much of my day arguing by e-mail with a "security researcher" (it must be true because he put it in his .signature!) about how ~all in our spf records isn't a massive security vulnerability20:32
*** dkranz has quit IRC20:32
*** e0ne has quit IRC20:32
*** tonytan4ever has joined #openstack-infra20:32
mordredSpamapS: there is a non-eatmydata dpkg option20:32
fungihe even went to the trouble to provide a php (php?!?) proof of concept exploit and sent a spoofed e-mail to me. oh the horror20:33
SpamapSmordred: yes I'm quite aware, the ubuntu installers use it.20:33
SpamapS(both the fast one for applying updates and the d-i based one)20:33
fungialso he kept repeatedly misspelling smtp "smpt" and insisted for several iterations that it wasn't a typo before i finally convinced him20:33
SpamapSAnyway, back to concurrency issues... I'm reminded that Helion's CI does lots of concurrent builds and doesn't hit this. I wonder if there is just some poor error handling going on that is letting a failed lock get through as a working lock.20:34
*** weshay has quit IRC20:34
mordredSpamapS: in case you want to review ... (clarkb you too)20:35
clarkbjust about at 100 nodes to go20:35
jeblairfungi: i.  wow.20:36
openstackgerritKhai Do proposed openstack-infra/system-config: add a turbo hipster worker for testing
fungijeblair: apparently he's going to post this all over some "security" webboard and we should expect lots of people taking advantage of our customers by spoofing e-mail20:37
fungii'm so afraid20:37
clarkbjeblair: so we did hit a TimeOut error listing functions according to the log at 12:08 which is when you killed the standalone geard20:37
clarkbjeblair: other than that log looks clean so far20:37
fungiSpamapS: clarkb: i really still think having virtual machine dib workers backending nodepool's image updates is inevitable, particularly for some unusual image types we may eventually end up wanting20:38
clarkbfungi: yup20:38
clarkbfungi: I just think we can make dib do the container thing better too :)20:38
clarkbbut it requires changing a pretty fundamental piece of how dib works20:39
fungiwe simply won't be able to guarantee we can vbuild all the things we want from one single starting system20:39
clarkbor at least that was my impression when I first looked at it and decided to leave that for later20:39
*** dizquierdo has quit IRC20:39
* mordred also has some container-based patches he wants to make for dib20:39
fungii think the centos 7 tar on ubuntu precise issue is just the tip of that iceberg20:39
mordredand also wants to be able to use dib to create containers20:39
jeblairmerger queue is still empty20:40
clarkber wait no TimeoutErrors are more recent than that. I need more caffeine20:41
clarkbwe haev had two20:41
SpamapSclarkb: it's not that fundamental. There's one output from the setup process.. "Where to write things you want in the image". And then that is the input to the "make an image" process.20:41
clarkb2014-12-08 20:37:16,852 and 2014-12-08 20:25:29,69920:41
clarkbSpamapS: right and it should just start as chroot/container on dest of image20:41
clarkbSpamapS: at least as an option20:42
*** MaxV has joined #openstack-infra20:42
jeblairclarkb: no timeouterrors in zuul20:42
clarkbjeblair: I think you may be right about nodepool starving itself at times20:43
mordredsdague, morganfainberg: /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/ SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See for details.)20:43
mordred  SecurityWarning20:43
morganfainbergmordred, pong, looking20:43
clarkbjeblair: which should be fine if it generally doesn't and the server keeps running without losing track20:43
mordredsdague, morganfainberg: I get this with most recent client libs installed - seems to be a requests thing - is it a thing we're aware of/care about?20:43
mordredmorganfainberg: OR -is that telling me that the target cert I'm connecting to is yuck?20:44
morganfainbergnot sure.20:44
sdagueso... I thought we weren't using their vendored urllib3 any more20:44
morganfainbergsdague, i.. lost track of that conversation20:44
morganfainbergand i *hate* that vendored lib thing20:44
sdagueI think it's that your cert is crap20:44
mordredjroll: it's possible that there is a rax cert issue ^^ OR that there is simply a bonghits client lib on our side. no idea which20:44
morganfainbergbut i think that is a cert issue20:44
fungimordred: morganfainberg: that sounds backwards... it's complaining that there's no subjectAltName and that it has to fall back on commonName?20:44
*** _nadya_ has quit IRC20:45
morganfainbergfungi, yeah that *sounds* like a bad cert.20:45
morganfainbergand it's complaining that it's trying to fallback?20:45
mordredmorganfainberg: I think maybe falling back is the thign that's deprecated?20:45
morganfainbergmordred, ++20:45
morganfainbergthat sounds right20:46
clarkbin general it appeas to be listing demand and allocating resources so Iam not worried yet20:46
*** baoli has joined #openstack-infra20:46
fungimorganfainberg: but, i mean, shouldn't using only commonName be perfectly safe? why would that be warning-fodder20:46
clarkbjeblair: and the launch all the nodes at start may contribute20:46
mordredjroll, JayF: so, as you guys are my current funnels into rackspace - there may be a cert issue there that might want to be raised with someone20:46
morganfainbergfungi, yeaaaah i don't know the details on RFC281820:46
mordredjroll, JayF: its not a problem for us - just being friendly20:46
*** baoli has quit IRC20:47
dougwigfungi, anteaya - new repos, with branches and tags:20:47
*** weshay has joined #openstack-infra20:47
fungimorganfainberg: it strikes me as being like an mta that warns that it had to fall back on an a rr because the destination had no corresponding mx rr20:47
*** baoli has joined #openstack-infra20:47
jeblairclarkb, fungi: i'm starting full packet dumps on zuul and nodepool (filtering for each other, so not all traffic) to correlate with the next timeouterror20:47
fungilike, you should just do that. it's a legitimate configuration20:47
*** shashankhegde has joined #openstack-infra20:47
fungidougwig: thanks--looking at latest iteration now20:47
morganfainbergfungi, sure. but if $SECURITY_PEOPLE have determined that that behavior in certs can be abused [maybe thats the issue], that fallback is not sane?20:48
morganfainbergfungi, i think this is something i want to know more about now.20:48
*** Sukhdev has joined #openstack-infra20:49
fungijeblair: sounds good... nodepool on the precise host exhibited the same issue a couple times right after we started it, last time we switched back, but then didn't continue to do that with any regularity20:49
fungimorganfainberg: agreed. i'm looking for explanations as to why a lack of subjectAltName would be a risk20:50
jeblairfungi: yeah, i think thread starvation as clarkb suggested is not inconceivable20:50
clarkbwe are down to 15 nodes on old nodepool. they are being stubborn20:50
fungiin which case not a regression in behavior, just something else which needs fixing20:50
jeblairfungi: this should help tell us if the problem is server or client side if it happens again.  and if it doesn't happen again, it helps support that idea :)20:50
morganfainbergfungi, ok... odd: If a subjectAltName extension of type dNSName is present, that MUST20:50
morganfainberg   be used as the identity. Otherwise, the (most specific) Common Name20:50
morganfainberg   field in the Subject field of the certificate MUST be used. Although20:50
morganfainberg   the use of the Common Name is existing practice, it is deprecated and20:50
morganfainberg   Certification Authorities are encouraged to use the dNSName instead.20:50
jrollmordred: lovely, thenks20:50
* jroll looks into this20:50
mordredmorganfainberg, jroll: for context, I get that warning several times now on every operation against rackspce cloud using python-*client20:50
morganfainbergwow, that was not expected to be multiple lines...20:50
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Add oslo.context
jrollmordred: do you have a gist or something that's more comprehensible than scrollback?20:51
mordredjroll: I will make you one20:51
morganfainbergoh interesting, this might be something-something TLS with wildcards20:51
fungimorganfainberg: interesting--so that's quoted from 2818?20:51
morganfainbergfungi, yeah20:51
morganfainbergand chromium deprecated/removed it20:52
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Add oslo.context
*** jamespage_ has joined #openstack-infra20:52
*** marun has joined #openstack-infra20:52
morganfainbergso urllib3 is20:52
morganfainbergfollowing in suit20:52
fungimorganfainberg: right, subjectAltName is there to handle name-based virtual hosting, which is why it seemed odd it would be viewed as a preferred field already20:52
mordredjroll: there's the command and output to reproduce20:52
morganfainbergfungi, yep20:52
morganfainbergfungi, well. apparantly 10 yrs ago20:53
jrollmordred: thanks20:53
* fungi suddenly feels very old20:53
morganfainberg"preferred" according to the urllib3 bug20:53
morganfainbergfungi, eyah...20:53
clarkbjeblair: we just had a timeout20:53
jrollmordred: is this new today? new version of urllib3 or?20:53
mordredjroll: looks like changes in tls/ssl related to RFC2818 are producing deprecating warnings with the certs on some elements of public RAX api endpoints20:53
clarkbjeblair 2014-12-08 20:53:14,979 ERROR nodepool.GearmanClient: Exception while listing function20:53
morganfainbergfungi, so - uh - yeah i think it's a fair warning now20:53
jrollmordred: word, ty20:53
mordredjroll: this is just most recent python-glanceclient and dependencies - so it might be new urllib3 and/or new requests lib20:53
fungidougwig: all three lgtm now--i see the branches and tags i'd expect... did you need anyone else to bless it before i cram it into gerrit?20:54
mordredjroll: but it happens with python-novaclient too - so I think it's anything using requests, tbh20:54
dougwigno, good to go.20:54
*** Rockyg has quit IRC20:54
morganfainbergmordred, yeah this is a cert issue20:54
dougwigi'm sitting next to mestery and anteaya20:54
morganfainbergmordred, with a new-ish requests with a new urllib320:54
*** mrmartin has quit IRC20:54
fungimorganfainberg: at any rate, i'm going to be paying much closer attention to the subjectAltName in my certs now ;)20:54
*** dprince has quit IRC20:55
morganfainbergfungi, ack20:55
fungii used to mostly ignore it because i was doing per-ip-address exclusively20:55
*** yamamoto has joined #openstack-infra20:56
clarkbjeblair: and again. there were a lot of ssh timeouts too20:56
clarkbit does seem to recover quickly from those timeouts though. not too long later we get successful listings20:57
jeblairclarkb: yeah, which also supports either thread starvation or network hiccup20:57
*** MaxV has quit IRC20:57
fungimorganfainberg: yep, on closer inspection, all my certs have my X509v3 Subject Alternative Name DNS matching the Subject CN, and othername marked unsupported20:57
*** e0ne has joined #openstack-infra20:59
*** jamespage_ has quit IRC21:00
*** MaxV has joined #openstack-infra21:01
*** e0ne has quit IRC21:02
*** sputnik13 has joined #openstack-infra21:04
*** annegent_ has joined #openstack-infra21:04
*** sputnik13 has quit IRC21:06
*** jamespage_ has joined #openstack-infra21:06
*** amcrn has quit IRC21:06
*** e0ne has joined #openstack-infra21:06
*** craigbr has quit IRC21:07
clarkbthe ready and used node numbers are climbing21:07
lbragstadhi all, quick question on this project. There doesn't seem to be anyone listed as the core maintainers (,members). Outside of the normal core proposal process, is there a way to 'boot strap' a core group?21:07
clarkbwe appear to be chewing through the backlog and I don't see oscilations yet21:07
clarkblbragstad: the people that create the project request an initial group member. That initial member can then fill out the remaining members21:08
*** rkukura has joined #openstack-infra21:08
*** kgiusti has left #openstack-infra21:09
*** sputnik13 has joined #openstack-infra21:09
*** annegent_ has quit IRC21:11
lbragstadclarkb: ok, that makes sense, I don't believe the original proposer is still active on the project ( Is there another way to do it?21:12
clarkblooks like the project came out of rackerlabs? maybe someone from rax wants to have ownership?21:13
clarkblbragstad: I would start by contacting the original proposer21:13
lbragstadI have a co-worker that wants to pick it up?
*** salv-orlando has joined #openstack-infra21:14
*** sputnik13 has quit IRC21:14
lbragstadclarkb: I'll do a little more digging and see what i can come up with. Thank you for the help!21:14
clarkbI guess my only concern is handing it off to someone to have the original interested party go "huh? /me is still here"21:14
*** armax has quit IRC21:14
*** mriedem1 has joined #openstack-infra21:15
*** fifieldt_ has joined #openstack-infra21:16
lbragstadclarkb: yeah, I use to work with the original maintainer but he moved to a new project a couple months ago21:16
lbragstadI'll have the two of them work it out21:17
*** mriedem has quit IRC21:17
clarkbmattoliverau: good morning21:18
*** julim has joined #openstack-infra21:19
mordredlook. australia has awokened21:19
mordredmattoliverau: congrats. you are now all of oz21:19
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Converted application to use utcnow()
*** fifieldt has quit IRC21:19
mattoliverauI always suspected it :p21:19
* krotscheck is rather embarrassed that ^^ was a thing.21:19
*** salv-orlando has quit IRC21:20
grue_pmmordred no he's not21:20
jeblairkrotscheck: did you mean 'prescribe' instead of 'proscribe' in that commit msg?21:21
* grue_pm turns out the lights and glares21:21
mordredgrue_pm: fair21:21
*** marun has quit IRC21:21
*** baoli has quit IRC21:21
mordredkrotscheck: or maybe circumscribe ... or circumnavigate!21:21
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Converted application to use utcnow()
*** andreykurilin_ has joined #openstack-infra21:22
clarkbwoot we are down to 5 building nodes and have a ton used and a few ready I think nodepool is mostly beyond the shock of being moved21:22
krotscheckmordred: circumstantiated?21:22
clarkbjeblair: fungi ^ from about now on is where we should look for that oscillation to happen21:22
*** MaxV has quit IRC21:23
mtreinishmordred: heh, we need to circumnavigate more in commits...21:24
*** TimO_ has quit IRC21:26
*** tonytan4ever has quit IRC21:26
*** tonytan_brb has joined #openstack-infra21:26
fungithis is all very circumspect21:26
*** MaxV has joined #openstack-infra21:27
mordredclarkb, fungi, jeblair: I believe I have end-to-end dib-rax working - although as I mentioned I'd really liek to land it using cloud-init instead21:28
mordredbut I have now created an image, uploaded it, created a vm based on it and ssh'd in to that vm21:28
*** e0ne has quit IRC21:29
fungimestery: dougwig: anteaya: i have the pushes going in now, though it's pretty slow-going (lots of history for it to deal with). i'll give you a heads-up when it completes21:30
mesteryfungi: Thanks man!21:30
dougwigfungi: sweet, thank you21:30
clarkbhrm not so sure the Beginning node launch calculation and Finished node launch calculations are happening as quickly as I expected21:32
clarkbjeblair: fungi ^21:32
*** gyee has quit IRC21:32
jeblairclarkb: will look21:33
*** e0ne has joined #openstack-infra21:34
jeblairclarkb: last begin->end took 4 seconds, is that what you mean?21:34
jeblairclarkb: or do you mean end->begin ?21:34
clarkbjeblair: end -> begin21:34
jeblairah k21:34
clarkbjeblair: beacuse when we timeout we end then don't start up again quickly21:34
fungimestery: dougwig: anteaya: is showing the branches and tags replicated there now21:35
dimsfungi: sdague: clarkb: can one of you please take a peek at when you get a chance? (Update to Zuul Status page, has +2 from Sergey)21:36
*** tkelsey has quit IRC21:36
jeblairclarkb: the main loop has a delay of 10 seconds, so i would expect timeout -> start to be ~ 10 secs; but we haven't seen a timeout recently, right?21:36
clarkbjeblair: we did at 21:33 UTC21:36
clarkbjeblair: 2014-12-08 21:33:14,836 ERROR nodepool.GearmanClient: Exception while listing functions21:37
mesteryfungi: Looking sharp!21:37
*** radez is now known as radez_g0n321:37
jeblairclarkb: gah, was tailing log on wrong host :(21:37
clarkbjeblair: oh :)21:37
jeblairclarkb: fortunately, was tcpdumping on correct one21:38
*** MaxV has quit IRC21:38
clarkbI am not sure its super happy right now but yay to tcpdumps21:38
*** tonytan_brb is now known as tonytan4ever21:38
jeblairclarkb: what do you think is unhappy?21:38
*** armax has joined #openstack-infra21:38
clarkbjeblair: the number of times we calculate demand seems to be low21:39
clarkbjeblair: especiall when you factor in timeouts21:39
*** marun has joined #openstack-infra21:39
fungimestery: dougwig: anteaya: looks correct to me now too21:40
*** nelsnelson has joined #openstack-infra21:41
clarkbzuul geard seems super responsive to me, so not sure why nodepool is so unhappy unless it is related to starvation within that one process21:41
*** aysyd has quit IRC21:41
jeblairi'm going to analyze the tcp dumps now21:42
fungiclarkb: yeah, wondering whether maybe nodepool is waiting on itself, not on teh gearman server?21:42
clarkbjeblair: fungi: if you change the node graph from zuul status page to -4 hours you can see it oscilate21:42
clarkbthe oscilations are definitely not as far apart as they were though so I think we sort of made it better21:42
*** nelsnelson has quit IRC21:43
fungimestery: dougwig: anteaya: just finished now too. let me know if anything looks awry21:43
dougwigfungi: checking21:43
mesteryfungi: This is fantastic, thanks again for the help! Thanks to anteaya for the guidance in person as well. :)21:44
*** nelsnelson has joined #openstack-infra21:44
*** nelsnelson has quit IRC21:45
*** nelsnelson has joined #openstack-infra21:45
*** sarob has joined #openstack-infra21:46
*** nelsnelson has quit IRC21:48
*** subscope_ has quit IRC21:48
clarkbtop does say that nodepoold is basically using an entire CPU at all times21:48
*** Masahiro has joined #openstack-infra21:48
clarkbso maybe the way we tackle this is to make nodepoold more multi CPU aware21:48
*** cpowell has quit IRC21:49
clarkblooking at graphs more. I think it may be related to the number of building nodes21:49
*** yamamoto has quit IRC21:49
*** mpaolino has joined #openstack-infra21:49
clarkbbuilding nodes will steadily drop then spike and so on21:49
clarkbhrm same with deleting21:50
*** teran has joined #openstack-infra21:50
openstackgerritKyle Mestery proposed openstack-infra/reviewstats: Update neutron core
*** Masahiro has quit IRC21:53
*** nelsnelson has joined #openstack-infra21:53
*** armax has quit IRC21:54
*** MaxV has joined #openstack-infra21:54
*** markmcclain has quit IRC21:54
*** rkukura has quit IRC21:55
mesteryfungi: Relocating, but had this for you:21:55
mesteryfungi: Can we force merge this one (, it's waiting for a tempest test to merge to disable services tests to pass the gate.21:55
mesteryfungi: This is from anteaya and her advice too.21:56
mesteryfungi: OK, moving, think that over and I'll be back21:56
dougwighold up 2 mins21:56
*** jgrimm is now known as zz_jgrimm21:57
fungimestery: dougwig: anteaya: merging that will definitely mean that no neutron changes are able to pass those jobs and will be completely broken until the corresponding test changes merge21:57
*** obondarev has quit IRC21:58
clarkbCPU use is low right now and nodepool is calculating launch nodes21:58
clarkbso I think it must be that21:58
*** zz_jgrimm is now known as jgrimm21:58
*** marun has quit IRC21:58
fungimestery: dougwig: anteaya: if you wait until lands you should be able to self-test 13990121:59
*** Sukhdev has quit IRC22:00
fungialso, looks like we passed the 140k changes mark today22:00
*** boris-42 has joined #openstack-infra22:00
mriedem1what happened to the nice nodepool graph on zuul.o.o?22:00
mriedem1or can someone point me at docs for what the current limit for nodepool is? trying to put together some education slides.22:01
clarkbmriedem1: it should still be there22:01
fungimriedem1: you want status.o.o22:01
*** shashankhegde has quit IRC22:01
mriedem1fungi: ah yea22:01
mriedem1so looks like 800 is about the max for nodepool?22:02
clarkbmriedem1: I think ~90022:02
fungimriedem1: just add up all the max-servers values in if you want a more exact number, but that's fairly close yeah22:02
*** teran_ has joined #openstack-infra22:03
clarkbya we definitely timeout as nodepoold related load climbs22:03
mriedem1fungi: ok, cool, that helps me22:03
fungimriedem1: clarkb: i get 88422:04
*** pradk has quit IRC22:04
clarkbalso that number is somewhat artificial due to hpcloud node sizestuff22:04
*** dkranz has joined #openstack-infra22:05
*** jamespage_ has quit IRC22:05
mriedem1ok, ~900 it is though, i'm just doing a lunch and learn for january, very high level intro on openstack22:05
*** teran has quit IRC22:05
*** Sukhdev has joined #openstack-infra22:05
mordred[    3.799143] input: PC Speaker as /devices/platform/pcspkr/input/input622:06
fungicould be higher come january ;)22:06
fungimordred: did you find an emulated pc speaker on a vm?22:06
mordredfungi: yup22:06
mordredfungi: connected as an input device22:06
fungimordred: floppy drives too?22:06
fungioh, as an input!22:06
*** marun has joined #openstack-infra22:06
jeblairclarkb, fungi: network traffic for 21:33:14,836 looks good.  entire request/response happened betwen 21:31:44 and 21:31:4722:07
mordredalso, there's a QEMU USB Tablet too22:07
jeblairclarkb, fungi: network traffic for 21:33:14,836 TimeoutError looks good.  entire request/response happened betwen 21:31:44 and 21:31:4722:07
* mordred boggles22:07
fungijeblair: yeah, this does sound like it's all on the nodepool end then22:07
clarkbyup I am almost positive it is22:08
*** armax has joined #openstack-infra22:08
clarkbI have roughly correlated timeouts to periods of high load  and successes to periods of low load22:08
fungijeblair: so it sent the request, had the full response within 3 seconds, then spun on its heels for several minutes22:08
*** mbacchi has quit IRC22:08
dtroyermordred: in case you haven't seen this yet:
*** dkliban is now known as dkliban_Afk22:08
*** dkliban_Afk is now known as dkliban_afk22:08
*** shashankhegde has joined #openstack-infra22:08
clarkbfungi: ya22:08
clarkbI wonder if either python or our kernel is doing something ufnny to us when scheduling22:09
*** markmcclain has joined #openstack-infra22:10
jeblairclarkb, fungi: the timeout error is generated by a thread condition, and probably ultimately is generated by the kernel, so it makes sense that it could fire before the thread we are interested in has gotten around to reading from the socket and processing the data.22:10
*** oomichi has joined #openstack-infra22:11
clarkbI think the server side changes have made nodepool timeouts less terrible though22:11
anteayafungi: thank you22:11
clarkbWe could try using not pvhvm image for this node22:12
*** obondarev has joined #openstack-infra22:12
jeblairclarkb: yeah, i think at this point we've cleared server-side problems; if new geard continues to hold up, we're in a much better place to work from22:12
*** mestery has quit IRC22:12
jeblair(much easier to change nodepool/gear client than zuul/gear server)22:12
*** teran has joined #openstack-infra22:13
mordreddtroyer: magic22:14
dougwigfungi: concur on waiting on 14009122:15
*** reed has joined #openstack-infra22:16
*** teran_ has quit IRC22:17
*** otter768 has joined #openstack-infra22:17
fungidougwig: it passed check22:17
dougwigfungi: i just pinged QA.22:18
dougwigthat plus a quick db unit fix and i think the neutron patch will go in clean.22:18
fungiexcellent. the gate's not busy, so shouldn't have to wait too terribly long22:18
oomichianteaya: hi22:18
*** rkukura has joined #openstack-infra22:19
anteayaoomichi: hello22:19
oomichianteaya: re:
anteayaoomichi: what are you asking from me?22:19
clarkbjeblair: fungi: other than making nodepool more multi cpu aware I could build another host on a non pvhvm image22:19
clarkbbut I think those are probably our two options22:19
oomichianteaya: we can get +1s from nova-drivers and it seems enough to go.22:20
anteayaI'm at netron sprnt and kind of focused22:20
*** MaxV has quit IRC22:20
fungiclarkb: what's the expectation, that pvhvm is making context switches more expensive or something?22:20
jeblairclarkb: since we think this is a python threading thing, we should be able to reproduce on test machines using nodepool's fake mode22:20
oomichianteaya: ok, will see you later.22:21
clarkbfungi: possibly or potentially tickling the newer kernel in unexpected ways22:21
jeblairclarkb: overall load average is about the same as on the old host22:21
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for running Neutron with IPv6 Networking
*** otter768 has quit IRC22:21
openstackgerritDoug Wiegley proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas
*** tonytan4ever has quit IRC22:23
openstackgerritMichael Pratt proposed openstack-infra/git-review: Add reviewers on upload
*** salv-orlando has joined #openstack-infra22:24
fungidougwig: what prompted 140164?22:25
openstackgerritKhai Do proposed openstack-infra/system-config: add a turbo hipster worker for testing
*** mfink has quit IRC22:25
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
dougwigfungi: mtreinish didn't like the other patch, even though it was a temp fix.  so i was trying an alternate, which i don't think will actually work.22:25
jeblairclarkb, fungi:
mtreinishdougwig: if you give me a couple min, I'll add onto that patch which will also skip the tempest tests...22:26
jeblairclarkb, fungi: on the new one, i think hour 20 has some time with no load, and hour 22 is only half over22:26
dougwigmtreinish: sweet, was just researching the right way.  will definitely be happy to wait.  thank you!22:26
mtreinishdougwig: I just need to spin up devstack to generate the complete ext list for master22:27
*** salv-orlando has quit IRC22:27
*** mestery has joined #openstack-infra22:28
*** rkukura has quit IRC22:29
*** andreykurilin_ has quit IRC22:30
*** mfink has joined #openstack-infra22:30
*** andreykurilin_ has joined #openstack-infra22:31
*** mpaolino has quit IRC22:31
*** rkukura has joined #openstack-infra22:31
ZZelle_fungi, hi22:32
*** ChuckC has quit IRC22:32
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add elements for Infra servers
jeblairclarkb, fungi: i'm going to spin up a test node and see if i can produce the bad behavior with fakes22:32
clarkbjeblair: ok let me know if I can help22:33
*** ociuhandu has joined #openstack-infra22:34
*** kumartin has joined #openstack-infra22:35
*** mestery has quit IRC22:35
jgriffithanteaya: ping22:36
jgriffithanteaya: I *really* hate to bug you with this.... but :(22:36
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add debootstrap to the needed packages for dib
openstackgerritMonty Taylor proposed openstack-infra/system-config: Make apt skip grabbing translations
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add elements for Infra servers
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add rinse to nodepool
clarkbjeblair: I did use the pvhvm trusty image from rax when I built this node. So you likely want to start with that image22:36
*** shashankhegde has quit IRC22:36
jgriffithanteaya: do we have anything in mind for the overwhelming email spam from all the Cinder 3'rd paty CI systems?22:36
jeblairclarkb: yep, building that now.22:37
jgriffithanteaya: or should I just create a filter in my inbox22:37
fungiZZelle_: hey there, what's up?22:37
clarkbjgriffith: I think most of us bitbucket gerrit email fwiw. Then use tools like gertty to better poll gerrit for things you are interested in22:38
jgriffithclarkb: yeah... I seem to have made the mistake of not doing that with Cinder reviews22:38
jgriffithclarkb: time to evolve :)22:38
jgriffithclarkb: fair answer, thanks22:38
jeblairjgriffith: that, but also we did make it a requirement to have "^.*CI" as a name, so if you do need to match on that, you should be able to22:39
jeblairjgriffith: "^.* CI$" even22:39
ZZelle_fungi, if you have some times, could have a look to ? which is a great feature for lazy people (=>me)22:39
jgriffithjeblair: yeah.. I've been meaning to try that.  Ok, thanks folks22:40
fungijgriffith: what i did was set up e-mail filters on the gerrit commenter full e-mail address with a pattern match on " CI <"22:40
anteayajgriffith: clarkb's and jeblair's suggestions sound like the way forward22:40
jeblairactually "^.*? CI$".  i love regexes.22:40
anteayaand fungi's22:40
fungier, what jeblair just said22:40
* fungi is juggling too many conversations22:40
fungijgriffith: but that was the main impetus originally for standardizing the account display names22:40
*** ihrachyshka has quit IRC22:40
fungito make them easier to filter (in e-mail as well as the webui)22:41
*** timrc-afk is now known as timrc22:41
*** mestery has joined #openstack-infra22:41
*** shashankhegde has joined #openstack-infra22:42
jgriffithfungi: now it all becomes clear22:43
openstackgerritMerged openstack-infra/project-config: Add non-voting docs job for git-review
*** teran_ has joined #openstack-infra22:44
*** rkukura has quit IRC22:44
fungiZZelle_: looking22:44
*** mwagner_lap has quit IRC22:45
*** teran has quit IRC22:45
ZZelle_fungi, i added a question about using git ls-remote --get-url when available and disabling the feature otherwise (which seems more bulletproof)22:46
jeblair./ --flavor "8 GB Performance" --cert --image "Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)"22:46
jeblairclarkb, fungi: ^ any idea why that would not work?22:46
*** packet has quit IRC22:46
jeblairclarkb, fungi:
jeblairoh, ci-puppetmaster?22:47
*** alexpilotti has joined #openstack-infra22:47
jeblairlaunch-node is old22:47
fungijeblair: current is doing that, or you didn't update to tip of master first?22:48
jeblairfungi: i used what was in ~root which was not updated22:48
fungiaha, i usually run it non-root from my homedir22:48
jeblairperhaps i should just remove that  checkout :)22:48
fungihence hadn't run into that22:48
*** mattfarina has quit IRC22:48
fungiand i always update to tip of master plus whatever patches to the launch tooling i'm testing if needed22:49
*** yamamoto has joined #openstack-infra22:50
*** eharney has quit IRC22:50
clarkbya I run it from my homedir too22:51
*** rkukura has joined #openstack-infra22:53
clarkbjeblair: so we are not running on new nodepool best I can tell22:54
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas
*** yamamoto has quit IRC22:54
mtreinishdougwig: ^^^22:54
dougwigmtreinish: ty, looking22:54
clarkbjeblair: but I am not sure that that matters since the server should disconnect us22:54
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Users may now issue their own access tokens.
krotscheckzaro: ^^22:55
*** bknudson has quit IRC22:55
krotscheckActually, lemme rebase that on master.22:55
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Users may now issue their own access tokens.
krotscheckThere we go22:55
jeblairclarkb: we are probably not running it; it's probably not urgent, but we should get that in so that if something goes wrong, we don't get stuck unable to query gearman22:55
jeblairclarkb: the server may not disconnect us22:56
* grue_pm wonders why she consistently reads mtrenish as mctrenish22:56
jeblairor "mac t" as we will all now call him22:57
clarkbjeblair: oh right as the server may not see that as an error having occurred22:57
clarkbjeblair: should I go ahead and approve that change to nodepool now?22:57
jeblairclarkb: prolly a good idea22:57
clarkbor do you want to monkeypatch it in first22:58
anteayajeblair: +22:58
*** rlandy has quit IRC22:58
*** vigneshvar has quit IRC22:58
clarkbok approving now22:59
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
mtreinishdougwig, anteaya: heh, we probably should make sure that tempest skips all the approriate things before we approve it. Because I didn't test it locally :)22:59
mordredgrue_pm: that's the best new name for mtreinish ever - mtreinish, you have a new casual nick friday nick I think23:00
*** weshay has quit IRC23:01
anteayamtreinish: oh23:01
anteayagood to know23:01
openstackgerritMerged openstack-infra/reviewstats: Updating horizon core
anteayamtreinish: hyphen or underscore to bring your t along with you, mac t?23:02
openstackgerritMerged openstack-infra/nodepool: Reconnect to gearman on error
*** ayoung has quit IRC23:03
mtreinishmordred: heh, it will definitely cause less confusion than the last one I used, mriedem-dplgnr23:03
openstackgerritMerged openstack-infra/reviewstats: Adjust wording for new Workflow URLs
jeblairclarkb: did you have a problem compiling zmq?23:03
*** rkukura has quit IRC23:04
grue_pmmordred: well it's in line with aussie nicknames - we shorten everything and add y where possible ;-p23:04
anteayamtreinish: ha a ha23:04
*** alex7376 is now known as asettle23:04
anteayagrue_pm: mtreinish is half-aussie, the other aussie's picked up on it23:04
anteayagrue_pm: mtreinish he still tries to deny it though23:05
clarkbjeblair: ya, you need
clarkbjeblair: sorry forgot about that until you asked. we should probabl merge that change23:06
*** salv-orlando has joined #openstack-infra23:06
zarokrotscheck: thanks. i'm currently reviewing the crontab one23:06
mtreinishanteaya: heh hardly, I'm pretty sure that I'm the first person in my family who's ever gone to AU...23:07
jeblairclarkb: thx23:07
krotscheckzaro: Mind you, that patch only makes it easy for users to issue tokens for themselves. Admins can’t yet issue arbitrary tokens for users.23:07
zaronibalizer: can you pls take a look at again?23:08
*** Sukhdev has quit IRC23:08
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
*** armax has quit IRC23:09
jeblairclarkb: aprvd23:09
*** kumartin has quit IRC23:09
*** alexpilotti has quit IRC23:11
*** armax has joined #openstack-infra23:11
*** doug-fish has left #openstack-infra23:11
anteayamtreinish: they all acknowledge you though, didn't for the rest of us :D23:11
*** ryanpetrello has quit IRC23:12
jeblairImportError: No module named Crypto.PublicKey23:12
jeblairclarkb: ^ seen that?23:12
jeblairclarkb: i think pycrypto may have failed to install but pip didn't notice23:13
*** Longgeek has joined #openstack-infra23:13
fungijeblair: oh, yes, it does that23:13
mordredbecause, you know, why fail on failure?23:14
*** ChuckC has joined #openstack-infra23:14
fungipycrypto encounters some fatal condition and then returns 023:14
openstackgerritMerged openstack-infra/system-config: Install build deps for nodepool installation
clarkbjeblair: ya, I think its easier if you just rebuild nodepool from scratch either b booting new node or installing all the things again after deleting them23:14
clarkbjeblair: that said I think I may see the bug23:14
clarkbjeblair: maybe23:14
*** ryanpetrello has joined #openstack-infra23:14
fungiand pip goes, oh! exited successfully, this must be installed now. i'll make note of that23:14
jeblairpip install pycripto; pip install ecdsa ...23:14
jeblairpycrypto even23:14
mriedem1fungi: clarkb: just to confirm my understanding, this means the rax-dfw provider provides at most 184 VMs, right?
*** teran has joined #openstack-infra23:15
mriedem1however many compute nodes it takes to service that is up to the provider i'm assuming23:15
fungimriedem1: yep23:15
*** jamielennox|away is now known as jamielennox23:15
mriedem1what does max-servers: 0 mean?23:15
fungimriedem1: we don't see compute nodes--that's all the provider's internal business23:15
fungimriedem1: 0 means exactly 023:16
*** hdd has joined #openstack-infra23:16
fungimriedem1: as in don't build any here23:16
mriedem1ok, off limits23:16
fungimriedem1: a convenient means of disabling a provider23:16
mriedem1yup, cool23:16
*** MaxV has joined #openstack-infra23:16
*** mestery has quit IRC23:16
*** marun has quit IRC23:16
clarkbjeblair: actually maybe not /me is reading code23:17
clarkbbecomes much more familiar with gear23:17
mriedem1is there anything that describes the flavor requirements? i thought we run 8 GB RAM and 8 VCPU flavors, but wasn't sure about disk, or if that's universal23:17
clarkbmriedem1: its not universal, we get different disk from rax and hpcloud23:17
mriedem1clarkb: but ram/vcpu is the same?23:17
mriedem1or depends on the job?23:18
*** mestery has joined #openstack-infra23:18
clarkbmriedem1: ram/vcpu/disk are the same per provider23:18
*** Longgeek has quit IRC23:18
*** rkukura has joined #openstack-infra23:18
mriedem1ah i see min-ram in here23:18
mriedem1but ok23:18
*** teran_ has quit IRC23:18
clarkbmriedem1: and when min-ram is > 8GB we artificially limit to 8GB with kernel boot params23:18
*** ryanpetrello has quit IRC23:20
dougwigfungi: can we get a force merge on this and the two sequentially after it?  i had botched the gitreview in the three repos, and don't want anyone to clone them that way:
fungi(like in hpcloud where we need to boot 32gb flavors to get the same vcpu count as 8gb has in rax)23:20
mriedem1fungi: i imagine managing those mappings is a real treat23:21
mriedem1or relatively trivial?23:21
dougwig(140156, 140157, 140158)23:21
fungimriedem1: you're welcome to part of that treat if you like!23:21
mriedem1so it's not tasty23:22
mtreinishfungi: you can't go recruiting mriedem1 for infra yet, he's got to get his tempest review count up first...23:23
mriedem1mtreinish: and that sql testr pet project of yours23:24
*** marun has joined #openstack-infra23:24
mtreinishmriedem1: all you need are 2-5 real reviews for that and I'm comfortable giving you +2 :)23:25
mtreinishor 1 real patch...23:25
*** salv-orlando has quit IRC23:26
*** annegent_ has joined #openstack-infra23:26
fungidougwig: i'm convinced the gating bypass on those relatively trivial patches is warranted so people don't end up with potentially disasterous git remotes while you're iterating on the packaging fixes23:27
mriedem1that's more than i'm available for right now :)23:27
dougwigfungi: thanks.23:27
clarkbjeblair: you know I wonder if its possibly related to calling conn.recv with a 1 byte buffer over and over and over again23:27
*** salv-orlando has joined #openstack-infra23:27
jeblairclarkb: talking about gear?23:27
clarkbjeblair: ya23:27
nibalizerzaro: sure23:27
nibalizeralso idunno if you've been to ##openstack-robots yet23:28
nibalizerbut i got that hubot up for you :P23:28
clarkbjeblair: since its possible that is a yield point and we could then be yielding a lot23:28
nibalizerif you give me an ssh key ill give you total access23:28
jeblairnibalizer: hubot?23:28
zaronibalizer: i haven't but that's great.  need to check it out23:28
*** Sukhdev has joined #openstack-infra23:29
zarojeblair: nibalizer and i talked about using it for the purpose of replacing all of the other bots.23:30
*** MaxV has quit IRC23:30
zaronibalizer: my weechat died and i lost that channel, cann you please send me another invite?23:31
jeblairzaro: excellent idea, which is why we were looking into errbot23:31
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas
mtreinishdougwig: ^^^ closing " are important...23:32
dougwigmtreinish: ty, i hate when i do that.23:33
fungilooks like hubot is using mit-expat license23:33
sc68caljeblair: Hey, I'm sitting next to Anita and working on IPv6 testing in DevStack, at the gate - could I bother you for a review? It's a devstack-gate repo review -
*** amitgandhinz has quit IRC23:33
nibalizerzaro: its not password protected or anything23:33
zarojeblair: ohh cool, python version of it.23:33
nibalizeryou can just hop it23:33
*** Sukhdev has quit IRC23:34
fungithough hubot is coffeescript23:34
morganfainbergfungi, nodejs23:34
morganfainbergor is it coffee?23:34
fungimorganfainberg: yep23:34
*** Sukhdev has joined #openstack-infra23:34
morganfainbergor are those... inter...changable23:34
*** craigbr has joined #openstack-infra23:34
* morganfainberg knows little about nodejs.23:34
jeblairyeah, tbh i'm much more interested in using something like errbot because of the python.23:35
fungimorganfainberg: right, so the bot engine seems to be in javascript/node and then its extensibility is via coffeescript23:36
jeblairafter all, our existing bots are in python23:36
fungiagreed on the "something in python" preference23:36
*** Ryan_Lane has quit IRC23:36
fungijavascript seems like an odd language choice for an irc bot23:36
*** Ryan_Lane has joined #openstack-infra23:37
*** Masahiro has joined #openstack-infra23:37
fungithen again, probably no more maddening than eggdrop being written in tcl (though i guess it must have descended from an expect script so maybe not surprising)23:37
grue_pmfungi: as long as it doesnt have a SLAP database backend ;-p23:38
*** shashankhegde has quit IRC23:39
jeblairclarkb: i think the fundamental problem is that the thread is starving, and i think that is happening outside of the gearman calls too23:40
jeblairclarkb: after all, we've seen up to 10 minutes between cycles, and only 90 seconds of that, at most, could be gearman recv's23:40
jeblairclarkb: so it could be exacerbating the problem, but i don't think it's the only source of delay23:41
clarkbjeblair: yes, I think that is correct. But Ithink that it is possible that recv(1) is contributing to that23:41
clarkbjeblair: no it could be >90 seconds beacuse you have to read all the bytes off of the receive queue before you get to the next admin request23:41
*** Masahiro has quit IRC23:41
*** jgrimm is now known as zz_jgrimm23:41
clarkbthe exception should be thrown within ~90 seconds though23:42
openstackgerritKyle Mestery proposed openstack-infra/reviewstats: Update neutron core
jeblairclarkb: it's never going to spend more than 90 seconds during each cycle recving though23:43
clarkbjeblair: wouldn't it have to in order to remove the now useless bytes from the queue?23:44
jeblairclarkb: oh, actually, recving is a different thread23:44
jeblairlemme look at something23:45
*** salv-orlando has quit IRC23:45
jeblairclarkb: yeah, so it's the client poll loop that deals with recving, while it's the main nodepool thread that's waiting on a complete event to be set by that thread; and it's the main nodepool thread that gets the 90 second timeout23:46
*** zz_avozza is now known as avozza23:47
jeblairclarkb: so we don't really know how often the gearman io thread is being serviced23:47
clarkbyup, so what I think may be happening is the client poll loop thread isn't getting enough cpu time to completely read in the entire status packet23:47
jeblairit could be way behind since it's not subject to the timeout23:47
clarkbthen the nodepool loop timesout23:47
clarkband since recv() is a likely yield point calling it a bunch is giving up that threads control of the cpu23:48
jeblairclarkb: having said that, my test server is using 116% cpu and appears to be idle.23:49
jeblairi think there's something to this pathological scheduling theory23:49
*** shashankhegde has joined #openstack-infra23:50
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
jeblairclarkb: it's only created 100 out of the 1000 servers it scheduled so far23:50
jeblairclarkb: note that the fake drivers spawn a thread that sleeps, so this is actually much worse from a # of python threads perspective (approx 2x while building)23:51
clarkbjeblair: but the fake status getting is working?23:52
*** dmsimard is now known as dmsimard_away23:52
clarkbjeblair: maybe make that status result large?23:52
jeblairclarkb: yeah, i have not made the status result large yet; i feel like there's already some weird behavior going on, and i want to directly compare it to precise23:52
jeblairclarkb: do you think we should roll back to precise or continue on trusty?23:53
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
clarkbjeblair: the way the test load has been today trusty has been ok23:54
jeblairyeah, though we could be a bit unhappy if it increases significantly23:54
clarkbjeblair: I guess it depends on whether or not we are worried about the load picking up and being caught in a bad spot.23:54
openstackgerritSean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking
clarkbI am happy to rollback if we want to avoid future potential trouble23:55
clarkbshould I start that process?23:55
*** dimtruck is now known as zz_dimtruck23:56
jeblairomg, python recv does a select each time23:57
*** gyee has joined #openstack-infra23:58
jeblairrecv -> {release gil; select, recv; acquire gil}  (broadly speaking)23:59

Generated by 2.14.0 by Marius Gedminas - find it at!