Monday, 2016-08-15

*** roxanaghe has quit IRC00:00
*** thorst_ has joined #openstack-infra00:00
*** dingyichen has joined #openstack-infra00:02
*** zhurong has quit IRC00:03
*** raunak has quit IRC00:03
*** baoli has quit IRC00:04
*** raunak has joined #openstack-infra00:05
*** thorst_ has quit IRC00:09
*** thorst_ has joined #openstack-infra00:10
*** baoli has joined #openstack-infra00:12
*** fguillot_ has quit IRC00:13
*** fguillot_ has joined #openstack-infra00:13
*** thorst_ has quit IRC00:19
*** sflanigan has joined #openstack-infra00:22
*** tqtran has joined #openstack-infra00:24
*** tqtran has quit IRC00:28
*** thorst_ has joined #openstack-infra00:34
*** pahuang has joined #openstack-infra00:35
*** raunak has quit IRC00:37
*** thorst_ has quit IRC00:39
*** fguillot_ has quit IRC00:42
*** fguillot_ has joined #openstack-infra00:43
*** sarob has joined #openstack-infra00:45
*** greghaynes has quit IRC00:48
*** greghaynes has joined #openstack-infra00:49
*** sarob has quit IRC00:49
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** amitgandhinz has joined #openstack-infra00:51
*** thorst_ has joined #openstack-infra00:53
*** thorst_ has quit IRC00:53
*** amitgandhinz has quit IRC00:56
*** mriedem has quit IRC00:58
*** Hal has quit IRC00:59
*** fguillot_ has quit IRC01:02
*** bswartz has joined #openstack-infra01:02
*** tphummel has joined #openstack-infra01:10
*** adrian_otto has joined #openstack-infra01:12
*** bswartz has quit IRC01:13
*** roxanaghe has joined #openstack-infra01:25
*** aeng has quit IRC01:26
*** roxanaghe has quit IRC01:29
*** gildub has joined #openstack-infra01:32
*** adrian_otto1 has joined #openstack-infra01:33
*** adrian_otto has quit IRC01:34
*** sflanigan has quit IRC01:34
*** yamahata has joined #openstack-infra01:41
*** tonytan4ever has joined #openstack-infra01:42
*** aeng has joined #openstack-infra01:42
*** yanyanhu has joined #openstack-infra01:44
*** sflanigan has joined #openstack-infra01:46
*** tonytan4ever has quit IRC01:47
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio
*** amitgandhinz has joined #openstack-infra01:52
*** thorst_ has joined #openstack-infra01:52
*** amitgandhinz has quit IRC01:56
*** baoli has quit IRC01:56
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Vagrant files for puppet-phabricator
*** aeng has quit IRC01:59
*** adrian_otto1 has quit IRC01:59
*** sflanigan has quit IRC02:00
*** tonytan4ever has joined #openstack-infra02:01
*** thorst_ has quit IRC02:07
*** thorst_ has joined #openstack-infra02:08
*** jamielennox is now known as jamielennox|away02:10
*** aeng has joined #openstack-infra02:12
*** sflanigan has joined #openstack-infra02:12
*** sflanigan has joined #openstack-infra02:12
*** thorst_ has quit IRC02:16
*** jamielennox|away is now known as jamielennox02:30
*** gongysh has joined #openstack-infra02:30
openstackgerritkyle liu proposed openstack-infra/project-config: Add new project networking-zte
*** nwkarsten has joined #openstack-infra02:36
*** gongysh has quit IRC02:37
*** apetrich has joined #openstack-infra02:37
*** gothicmindfood has quit IRC02:40
*** nwkarsten has quit IRC02:40
*** sarob has joined #openstack-infra02:46
*** sarob has quit IRC02:50
*** gildub has quit IRC02:51
*** amitgandhinz has joined #openstack-infra02:52
*** amotoki has quit IRC02:54
*** amotoki has joined #openstack-infra02:55
*** amotoki has quit IRC02:56
*** amitgandhinz has quit IRC02:57
*** tphummel has quit IRC02:58
*** zhurong has joined #openstack-infra02:59
*** nwkarsten has joined #openstack-infra03:04
*** nwkarsten has quit IRC03:08
*** thorst_ has joined #openstack-infra03:15
*** thorst_ has quit IRC03:21
*** yamamoto has joined #openstack-infra03:22
*** baoli has joined #openstack-infra03:24
*** gothicmindfood has joined #openstack-infra03:37
*** gothicmindfood has quit IRC03:37
*** baoli has quit IRC03:39
*** amotoki has joined #openstack-infra03:39
*** vikrant has joined #openstack-infra03:40
*** tonytan4ever has quit IRC03:46
*** ramishra has quit IRC03:49
*** vikrant has quit IRC03:51
*** ramishra has joined #openstack-infra03:51
*** vikrant has joined #openstack-infra03:52
*** amitgandhinz has joined #openstack-infra03:53
*** kzaitsev_mb has quit IRC03:53
*** amitgandhinz has quit IRC03:57
*** amotoki has quit IRC04:01
*** aeng has quit IRC04:03
*** sflanigan has quit IRC04:05
*** vikrant is now known as vikrant|brb04:08
*** amotoki has joined #openstack-infra04:09
*** amotoki has quit IRC04:13
*** amotoki has joined #openstack-infra04:17
*** thorst_ has joined #openstack-infra04:18
*** aeng has joined #openstack-infra04:20
*** vikrant|brb is now known as vikrant04:20
*** chlong has joined #openstack-infra04:21
*** esberglu has joined #openstack-infra04:22
*** armax has joined #openstack-infra04:24
*** thorst_ has quit IRC04:26
*** tqtran has joined #openstack-infra04:26
Jeffrey4l_any guys can review this?
*** armax has quit IRC04:29
*** tqtran has quit IRC04:30
Jeffrey4l_fungi, anteaya this totally blocked kolla-kube project.04:31
*** gouthamr has quit IRC04:33
*** shashank_hegde has joined #openstack-infra04:40
*** _nadya_ has joined #openstack-infra04:42
*** zhurong has quit IRC04:43
*** zhurong has joined #openstack-infra04:44
*** sarob has joined #openstack-infra04:46
*** jaosorior has joined #openstack-infra04:47
*** roxanaghe has joined #openstack-infra04:49
*** kzaitsev_mb has joined #openstack-infra04:50
*** sarob has quit IRC04:51
*** roxanaghe has quit IRC04:52
*** amitgandhinz has joined #openstack-infra04:54
*** psachin has joined #openstack-infra04:56
*** amitgandhinz has quit IRC04:58
*** gildub has joined #openstack-infra04:58
*** aeng has quit IRC05:06
*** bswartz has joined #openstack-infra05:07
*** gildub has quit IRC05:10
*** esberglu has quit IRC05:11
*** _nadya_ has quit IRC05:13
*** pabelanger has quit IRC05:14
*** wfoster has quit IRC05:15
*** pabelanger has joined #openstack-infra05:15
*** lucas-dinner has quit IRC05:15
*** senk_ has joined #openstack-infra05:16
*** aeng has joined #openstack-infra05:18
*** wfoster has joined #openstack-infra05:19
*** lucasagomes has joined #openstack-infra05:19
*** rbergeron has quit IRC05:24
*** rbergeron has joined #openstack-infra05:24
*** dmsimard has quit IRC05:25
*** rcernin has joined #openstack-infra05:25
*** thorst_ has joined #openstack-infra05:25
*** dmsimard has joined #openstack-infra05:26
*** thorst_ has quit IRC05:31
*** gildub has joined #openstack-infra05:36
*** kzaitsev_mb has quit IRC05:40
*** rbuzatu has quit IRC05:45
*** r-mibu has quit IRC05:46
*** r-mibu has joined #openstack-infra05:46
*** baoli has joined #openstack-infra05:51
*** roxanaghe has joined #openstack-infra05:52
*** amitgandhinz has joined #openstack-infra05:55
*** baoli has quit IRC05:56
*** roxanaghe has quit IRC05:57
*** amitgandhinz has quit IRC05:59
*** florianf has joined #openstack-infra06:02
*** rbuzatu has joined #openstack-infra06:03
*** jmccrory is now known as jmccrory_away06:12
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image
*** zhurong has quit IRC06:15
*** zhurong has joined #openstack-infra06:16
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
*** yamamoto has quit IRC06:18
*** _nadya_ has joined #openstack-infra06:20
*** aviau has quit IRC06:22
*** aviau has joined #openstack-infra06:22
*** rbuzatu has quit IRC06:26
*** armax has joined #openstack-infra06:26
*** arif-ali has quit IRC06:27
*** thorst_ has joined #openstack-infra06:29
*** armax has quit IRC06:31
*** arif-ali has joined #openstack-infra06:31
*** vinaypotluri has quit IRC06:31
*** shashank_hegde has quit IRC06:34
*** rbuzatu has joined #openstack-infra06:34
*** senk_ has quit IRC06:35
*** david-lyle has joined #openstack-infra06:36
*** kzaitsev_mb has joined #openstack-infra06:36
*** pcaruana has joined #openstack-infra06:36
*** thorst_ has quit IRC06:36
*** shashank_hegde has joined #openstack-infra06:37
*** david-lyle_ has quit IRC06:39
*** kzaitsev_mb has quit IRC06:41
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
*** javeriak has joined #openstack-infra06:43
*** rwsu has joined #openstack-infra06:44
*** _nadya_ has quit IRC06:44
*** martinkopec has joined #openstack-infra06:46
*** tonytan4ever has joined #openstack-infra06:47
openstackgerritMerged openstack-infra/zuul: Move other-requirements.txt to bindep.txt
*** esikachev has joined #openstack-infra06:48
*** tonytan4ever has quit IRC06:52
*** rbuzatu has quit IRC06:53
*** amitgandhinz has joined #openstack-infra06:55
*** nmagnezi has joined #openstack-infra06:56
*** skraynev has joined #openstack-infra06:56
*** rbuzatu has joined #openstack-infra06:58
*** yamamoto has joined #openstack-infra06:59
*** amitgandhinz has quit IRC07:00
*** javeriak has quit IRC07:00
*** Thelo has joined #openstack-infra07:02
*** chlong has quit IRC07:08
*** Thelo has quit IRC07:10
*** senk_ has joined #openstack-infra07:11
*** Thelo has joined #openstack-infra07:12
*** Thelo has left #openstack-infra07:17
*** shashank_hegde has quit IRC07:20
*** _nadya_ has joined #openstack-infra07:20
*** jpich has joined #openstack-infra07:21
*** savihou has joined #openstack-infra07:21
*** skraynev has quit IRC07:24
*** skraynev has joined #openstack-infra07:25
*** chlong has joined #openstack-infra07:25
*** ifarkas_afk is now known as ifarkas07:26
openstackgerritJakub Libosvar proposed openstack-infra/project-config: Replace DVR multinode full nv job with DVR scenario tests
*** shashank_hegde has joined #openstack-infra07:26
*** matrohon has joined #openstack-infra07:28
*** ihrachys has joined #openstack-infra07:31
*** amotoki_ has joined #openstack-infra07:32
*** thorst_ has joined #openstack-infra07:34
*** amotoki has quit IRC07:35
*** yamahata has quit IRC07:36
*** kzaitsev_mb has joined #openstack-infra07:37
*** javeriak has joined #openstack-infra07:41
*** roxanaghe has joined #openstack-infra07:41
*** thorst_ has quit IRC07:41
*** kzaitsev_mb has quit IRC07:42
*** matthewbodkin has joined #openstack-infra07:42
*** roxanaghe has quit IRC07:45
*** bkero-pto is now known as bkero07:56
*** amitgandhinz has joined #openstack-infra07:56
*** dimtruck is now known as zz_dimtruck07:59
*** zzzeek has quit IRC08:00
*** amitgandhinz has quit IRC08:01
*** sdake has joined #openstack-infra08:01
*** zzzeek has joined #openstack-infra08:02
*** tonytan4ever has joined #openstack-infra08:03
*** savihou has quit IRC08:04
*** chlong has quit IRC08:07
*** tonytan4ever has quit IRC08:08
*** gildub has quit IRC08:11
*** e0ne has joined #openstack-infra08:15
*** sdake has quit IRC08:15
*** kzaitsev_mb has joined #openstack-infra08:16
*** dtantsur|afk is now known as dtantsur08:18
*** dingyichen has quit IRC08:24
*** dingyichen has joined #openstack-infra08:25
*** strigazi is now known as strigazi_AFK08:26
*** ccamacho has joined #openstack-infra08:26
*** ccamacho has quit IRC08:26
*** ccamacho has joined #openstack-infra08:26
*** tqtran has joined #openstack-infra08:27
*** Na3iL has joined #openstack-infra08:28
*** Na3iL has quit IRC08:28
*** dingyichen has quit IRC08:31
*** javeriak_ has joined #openstack-infra08:31
*** tqtran has quit IRC08:31
*** esikachev has quit IRC08:31
*** javeriak has quit IRC08:32
*** javeriak has joined #openstack-infra08:32
*** esikachev has joined #openstack-infra08:33
*** Thelo has joined #openstack-infra08:34
*** e0ne has quit IRC08:35
*** sdake has joined #openstack-infra08:35
*** javeriak_ has quit IRC08:36
*** e0ne has joined #openstack-infra08:37
*** thorst_ has joined #openstack-infra08:39
*** yaume has joined #openstack-infra08:40
*** markvoelker has joined #openstack-infra08:41
*** sdake has quit IRC08:43
*** javeriak has quit IRC08:44
*** javeriak has joined #openstack-infra08:44
*** markvoelker has quit IRC08:45
*** thorst_ has quit IRC08:47
*** Thelo has quit IRC08:52
*** Gibi has joined #openstack-infra08:53
*** Goneri has joined #openstack-infra08:55
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
*** amitgandhinz has joined #openstack-infra08:57
*** electrofelix has joined #openstack-infra08:58
*** rbuzatu has quit IRC08:58
*** tkelsey has joined #openstack-infra09:00
*** yaume has quit IRC09:01
*** amitgandhinz has quit IRC09:02
*** kzaitsev_mb has quit IRC09:02
*** acoles_ is now known as acoles09:03
*** kzaitsev_mb has joined #openstack-infra09:06
*** Hal has joined #openstack-infra09:07
*** kzaitsev_mb has quit IRC09:11
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: stable-mu branches for maintenance and stable for upgrades
*** shashank_hegde has quit IRC09:18
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Fix type in test_attr_list handling
*** yaume has joined #openstack-infra09:22
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Fix typo in test_attr_list handling
*** savihou has joined #openstack-infra09:24
*** oanson has joined #openstack-infra09:28
*** armax has joined #openstack-infra09:28
*** roxanaghe has joined #openstack-infra09:29
*** sshnaidm has quit IRC09:29
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image
*** armax has quit IRC09:33
*** roxanaghe has quit IRC09:33
*** jaosorior is now known as jaosorior_brb09:38
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting
*** asselin has joined #openstack-infra09:45
*** thorst_ has joined #openstack-infra09:45
*** sdake has joined #openstack-infra09:46
*** yaume has quit IRC09:46
*** asselin_ has quit IRC09:48
Jeffrey4l_any guys can review this?
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting
*** yamamoto has quit IRC09:50
*** thorst_ has quit IRC09:52
*** tosky has joined #openstack-infra09:55
openstackgerritamrith proposed openstack-infra/project-config: Revert "[trove] Promote scenario tests to voting and gating"
*** gildub has joined #openstack-infra09:57
*** oanson has quit IRC09:58
*** amitgandhinz has joined #openstack-infra09:58
*** rbuzatu has joined #openstack-infra09:58
*** amitgandhinz has quit IRC10:02
*** rbuzatu has quit IRC10:03
*** vgridnev has joined #openstack-infra10:03
vgridnevhello team, could you please review ?10:04
odyssey4meWe have a bot called 'ops-bot' in #openstack-ansible. Before I kick it for being super annoying (it repeats every link pasted in the channel) I'd like to know if the bot is an infra test of some sort?10:04
*** tonytan4ever has joined #openstack-infra10:04
openstackgerritMerged openstack-infra/release-tools: if we fail to send mail, the job should fail
openstackgerritMerged openstack-infra/release-tools: Move other-requirements.txt to bindep.txt
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting
*** tonytan4ever has quit IRC10:08
*** zz_dimtruck is now known as dimtruck10:18
*** kzaitsev_mb has joined #openstack-infra10:18
*** rbuzatu has joined #openstack-infra10:19
*** yanyanhu has quit IRC10:22
openstackgerritNadya Shakhat proposed openstack-infra/project-config: Add fuel-plugin-openstack-telemetry
*** Na3iL has joined #openstack-infra10:25
*** asettle has joined #openstack-infra10:25
*** dimtruck is now known as zz_dimtruck10:28
*** jaosorior_brb is now known as jaosorior10:31
*** sdague has joined #openstack-infra10:31
*** yamamoto_ has joined #openstack-infra10:32
*** javeriak has quit IRC10:36
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image
*** markvoelker has joined #openstack-infra10:42
openstackgerritThomas Bechtold proposed openstack-infra/project-config: designate: Add a non-voting job with postgres as DB backend
*** pblaho has joined #openstack-infra10:44
*** markvoelker has quit IRC10:47
*** Hal has quit IRC10:47
*** amitgandhinz has joined #openstack-infra10:58
*** javeriak has joined #openstack-infra11:00
*** amitgandhinz has quit IRC11:03
*** thorst_ has joined #openstack-infra11:04
*** zhurong has quit IRC11:05
*** skraynev is now known as skraynev__11:05
*** lucasagomes is now known as lucas-hungry11:16
*** roxanaghe has joined #openstack-infra11:17
*** roxanaghe has quit IRC11:22
*** ldnunes has joined #openstack-infra11:25
*** armax has joined #openstack-infra11:30
*** amotoki_ has quit IRC11:31
*** jkilpatr has joined #openstack-infra11:32
*** amotoki has joined #openstack-infra11:34
*** jaosorior has quit IRC11:34
*** armax has quit IRC11:35
*** jaosorior has joined #openstack-infra11:35
*** asettle has quit IRC11:36
*** amotoki has quit IRC11:37
*** rodrigods has quit IRC11:38
*** rodrigods has joined #openstack-infra11:38
*** rhallisey has joined #openstack-infra11:38
*** rhallisey_ has joined #openstack-infra11:39
*** vgridnev has quit IRC11:40
DuncanTCan anybody help me understand why the gate-cinder-python27-db-ubuntu-xenial job is marked as failed on please? All the tests seem to list status 'ok'11:46
*** amotoki has joined #openstack-infra11:48
*** weshay_afk is now known as weshay11:48
*** asettle has joined #openstack-infra11:50
*** furlongm has joined #openstack-infra11:51
*** furlongm_ has quit IRC11:51
*** rhallisey has quit IRC11:51
*** rfolco has joined #openstack-infra11:51
*** dprince has joined #openstack-infra11:52
*** Na3iL has quit IRC11:54
*** amitgandhinz has joined #openstack-infra11:59
*** baoli_ has joined #openstack-infra12:00
*** vgridnev has joined #openstack-infra12:01
*** sdake has quit IRC12:02
*** apetrich has quit IRC12:02
*** rhallisey has joined #openstack-infra12:03
*** kgiusti has joined #openstack-infra12:03
*** amitgandhinz has quit IRC12:04
*** edmondsw has joined #openstack-infra12:04
*** tonytan4ever has joined #openstack-infra12:05
*** apetrich has joined #openstack-infra12:05
*** yamamoto_ has quit IRC12:06
*** sdake has joined #openstack-infra12:06
*** yamamoto has joined #openstack-infra12:07
*** yamamoto has quit IRC12:07
*** yamamoto has joined #openstack-infra12:07
*** bethwhite has quit IRC12:08
*** tonytan4ever has quit IRC12:09
*** gouthamr has joined #openstack-infra12:10
*** sigmavirus|away is now known as sigmavirus12:10
*** asettle has quit IRC12:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement LXD hypervisor experimental check
*** tonytan4ever has joined #openstack-infra12:13
*** moravec has quit IRC12:13
*** moravec has joined #openstack-infra12:14
mordredodyssey4me: nope. none of our bots, or our trolls, are named ops-bot12:14
odyssey4memordred ok awesome, I've banned it for a week - hopefully it won't return12:15
mordredodyssey4me: you don't find repeating links to be useful?12:15
*** amotoki has quit IRC12:16
*** bethwhite- has quit IRC12:17
*** roxanaghe has joined #openstack-infra12:18
*** jkilpatr has quit IRC12:18
*** moravec has quit IRC12:18
*** zz_dimtruck is now known as dimtruck12:18
*** gordc has joined #openstack-infra12:19
*** Zara has quit IRC12:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/diskimage-builder: Change DIB_IPA_CERT resulting file name
*** SotK has quit IRC12:20
*** moravec has joined #openstack-infra12:20
*** amotoki has joined #openstack-infra12:23
*** roxanaghe has quit IRC12:23
*** bethwhite has joined #openstack-infra12:24
*** sshnaidm|afk has joined #openstack-infra12:26
*** lucas-hungry is now known as lucasagomes12:27
*** psilvad has joined #openstack-infra12:28
*** xarses has quit IRC12:28
*** amotoki has quit IRC12:28
*** dimtruck is now known as zz_dimtruck12:28
pleia2good morning12:29
pleia2(east coast again this week)12:29
*** markvoelker has joined #openstack-infra12:30
mordredpleia2: enjoy philly!12:31
pleia2mordred: thanks :)12:31
pleia2it was really hot yesterday, but looks like the heat wave broke last night12:31
*** woodster_ has joined #openstack-infra12:32
*** pradk has joined #openstack-infra12:33
*** jkilpatr has joined #openstack-infra12:33
*** rhallisey_ has quit IRC12:34
*** mtanino has joined #openstack-infra12:35
mordredpleia2: good to know it's hot somewhere - it's gotten chilly here - it's only 83 right now!12:36
* mordred shivers12:36
*** rlandy has joined #openstack-infra12:36
*** sdake has quit IRC12:38
*** amitgandhinz has joined #openstack-infra12:39
mordredpleia2: if you're bored/waking up - wanna poke an easy project-config patch?
pleia2mordred: sure12:40
pleia2easy indeed, lgtm12:41
* mordred likes to be friendly early in the morning12:42
*** kzaitsev_mb has quit IRC12:43
*** julim has joined #openstack-infra12:44
*** vikrant has quit IRC12:46
*** raildo has joined #openstack-infra12:46
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove python3 jobs from nodepool
*** SotK has joined #openstack-infra12:47
*** esberglu has joined #openstack-infra12:47
*** Zara has joined #openstack-infra12:47
*** amoralej|off has quit IRC12:50
*** amoralej has joined #openstack-infra12:53
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton
*** xyang1 has joined #openstack-infra12:56
*** devkulkarni has joined #openstack-infra13:00
*** julim has quit IRC13:01
*** kzaitsev_mb has joined #openstack-infra13:01
*** zz_dimtruck is now known as dimtruck13:04
*** julim has joined #openstack-infra13:05
*** ccamacho has quit IRC13:05
*** esberglu has quit IRC13:05
*** esberglu has joined #openstack-infra13:05
*** mdrabe has joined #openstack-infra13:08
*** moravec has quit IRC13:08
*** moravec has joined #openstack-infra13:09
*** esberglu has quit IRC13:09
*** jcoufal has joined #openstack-infra13:13
*** vgridnev has quit IRC13:14
*** asettle has joined #openstack-infra13:17
*** gildub has quit IRC13:18
*** javeriak has quit IRC13:18
*** javeriak has joined #openstack-infra13:19
*** amotoki has joined #openstack-infra13:19
*** devkulkarni has quit IRC13:20
*** javeriak has quit IRC13:24
*** kaisers1 has left #openstack-infra13:24
*** _ari_ has joined #openstack-infra13:24
*** esberglu has joined #openstack-infra13:25
*** matt-borland has joined #openstack-infra13:27
openstackgerritThierry Carrez proposed openstack-infra/release-tools: aclmanager: Reuse releasetools.governance code
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Remove Release Managers from post-release groups
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Authenticate before doing group membership tests
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Use os.path functions instead of string slices
*** signed8bit has joined #openstack-infra13:33
*** tonytan4ever has quit IRC13:33
*** mriedem has joined #openstack-infra13:34
*** _ari_ has quit IRC13:36
odyssey4meAre the DNS resolvers for nodepool nodes set by something in infra? We're seeing configurations like in failed jobs, and
odyssey4me20-15-16.log in successful ones.13:37
*** _nadya_ has quit IRC13:37
*** amitgandhinz has quit IRC13:37
*** amitgandhinz has joined #openstack-infra13:38
*** Shrews has quit IRC13:38
*** rbergeron has quit IRC13:43
*** rbergeron has joined #openstack-infra13:43
*** nwkarsten has joined #openstack-infra13:44
*** bethwhite_ has joined #openstack-infra13:44
dansmithso I think I just saw zuul's dependency thing do something wrong13:45
*** dotplus has joined #openstack-infra13:45
dansmithif you look at 354265, it depends-on something that was un-merged.. I rechecked it and now it's in check and gate at the same time,13:46
openstackgerritJakub Libosvar proposed openstack-infra/project-config: Add scenarios from Neutron to multinode dvr full job
dansmithwait, nevermind13:46
dansmiththe bottom one just got +Wd, so nevermind :)13:46
openstackgerritMerged openstack-infra/nodepool: Add ZooKeeper connection listener
cloudnulljust as a note to what odyssey4me said, the failed job was OSIC and the success was RAX. so this may be something related to the V6 network update however it looks like the resolvers are getting setup correctly on a new instance built w/ that network --
*** martinkopec has quit IRC13:47
odyssey4mepabelanger mordred ^13:48
*** camunoz has joined #openstack-infra13:49
*** martinkopec has joined #openstack-infra13:50
*** gothicmindfood has joined #openstack-infra13:50
*** devkulkarni has joined #openstack-infra13:51
*** jheroux has joined #openstack-infra13:51
*** yamamoto has quit IRC13:52
*** chlong has joined #openstack-infra13:52
*** valderrv_ has joined #openstack-infra13:52
*** mikeym has joined #openstack-infra13:52
*** yamamoto has joined #openstack-infra13:52
*** ayoung has joined #openstack-infra13:55
*** dprince has quit IRC13:55
*** yamamoto has quit IRC13:57
*** dims has quit IRC13:59
*** adrian_otto has joined #openstack-infra14:01
*** oanson has joined #openstack-infra14:01
*** rvasilets_ has left #openstack-infra14:01
openstackgerritTimothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter
*** mhickey has joined #openstack-infra14:04
*** dims has joined #openstack-infra14:05
*** roxanaghe has joined #openstack-infra14:06
*** rbrndt has joined #openstack-infra14:07
*** Julien-zte has joined #openstack-infra14:08
openstackgerritMerged openstack-infra/devstack-gate: Add osc-lib and os-client-config to PROJECTS
*** mtanino has quit IRC14:09
jrollhi, does someone mind looking at a one line project-config change to unbreak ironic stable jobs?
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove EPEL usage
*** yamamoto has joined #openstack-infra14:10
*** roxanaghe has quit IRC14:10
*** sdake has joined #openstack-infra14:10
*** devkulkarni has quit IRC14:10
*** adrian_otto has quit IRC14:13
anteayamorning zaro14:13
anteayajroll: +214:13
anteayajroll: I was ircing in my dream last night14:13
cloudnullfungi anteaya: too RE: timeouts in the OSIC and the resolvers being set to "".14:13
anteayajroll: and I was cleaning up a channel for something but you were still using it14:14
cloudnullive been hunting aroung however I don't see where the resolvers are being written14:14
anteayajroll: it is interesting to remember ircing and seeing your username in my dream last night14:14
cloudnullmaybe something in the instance setup scripts that I'm just not seeing14:14
anteayacloudnull: I don't know14:14
cloudnullthat makes two of us :)14:14
*** annegentle has joined #openstack-infra14:14
anteayamordred: was up earlier so was pleia2 ^^14:14
cloudnullgood morning btw :)14:14
anteayacloudnull: at least you are not alone14:15
anteayagood morning to you14:15
anteayathanks for being so attentive to osic cloud14:15
openstackgerritMerged openstack-infra/elastic-recheck: Move other-requirements.txt to bindep.txt
anteayamuch appreciation to you14:15
*** savihou has quit IRC14:15
*** adrian_otto has joined #openstack-infra14:15
jrollanteaya: heh. I was editing terrible release notes in a dream last night :|14:15
cloudnullits been fun. now to make it even better.14:15
jrollanteaya: also, thanks for the review :)14:16
anteayahi DuncanT all the -db tests on that patch are failing14:17
anteayaplease ask huyang to stop rechecking that patch14:17
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
anteayajroll: ha ha, you must have been online in the dream time same time as me14:18
anteayacloudnull: awesome14:18
DuncanTanteaya: Sure, but if you read the log, there's no indication at all of what failed or why14:18
anteayajroll: I wonder if I am more efficent working online in the dreamtime than I am when I'm awake14:18
DuncanTanteaya: Every test reports 'ok'14:18
anteayaDuncanT: okay I am looking14:18
anteayaDuncanT: sure sometimes the job fails if something goes wrong in test teardown14:19
anteayathe job and the test are two different things14:19
openstackgerritEmilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra
anteayathe job runs the test14:19
anteayaand the job does other things14:19
anteayathe job has to succeed for jenkins to report success, not just the test14:19
DuncanTanteaya: And outputs logs. None of which appear to tell me why jenkins is unhappy14:19
anteayaright, looking14:20
*** adrian_otto has quit IRC14:20
anteayabut let's stop rechecking in the meantime14:20
anteaya2016-08-15 05:42:28.552244 | [Zuul] Job complete, result: FAILURE14:20
anteayaDuncanT: the last ansible command run was TASK [copy]
anteayawhich recieved no output14:21
*** jtomasek is now known as jtomasek|biab14:23
*** burgerk has joined #openstack-infra14:23
*** tonytan4ever has joined #openstack-infra14:23
anteayaDuncanT: from what I can tell it was able to successfully install python
openstackgerritSam Betts proposed openstack-infra/project-config: Fix syntax error in ironic-python-agent post job
DuncanTanteaya: It must have been able to, or the tests passing in console.log would not be passing14:24
sbezverkinfra team, please review, this issue completely blocking development on kolla-kubernetes project14:26
*** dprince has joined #openstack-infra14:27
*** annegentle has quit IRC14:27
anteayaDuncanT: sure, I am justing pointing out the place in the log where that is documented14:28
anteayamordred: can you explain what is happening here in the ansible log? 2016-08-15 05:42:28,386 p=28725 u=zuul |  fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362"14:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement Swift pypy experimental check
anteayamordred: my sense is the words failed and fatal aren't terrific but it is in the middle of the log14:28
anteayaso how fatal is it?14:29
*** _nadya_ has joined #openstack-infra14:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement Swift pypy experimental check
*** dimtruck is now known as zz_dimtruck14:29
*** tqtran has joined #openstack-infra14:30
*** oanson has quit IRC14:30
*** pcaruana has quit IRC14:30
*** _nadya_ has quit IRC14:30
*** devkulkarni has joined #openstack-infra14:30
*** thiagop has joined #openstack-infra14:31
*** _nadya_ has joined #openstack-infra14:31
anteayaDuncanT: from what I can see something in the build didn't do what it was supposed to do for zuul to finish with status success14:32
anteayaDuncanT: also I have not been able to find in the logs, with confidence, anything that shows what that thing was14:33
*** armax has joined #openstack-infra14:33
anteayawe may have to wait for fungi14:33
DuncanTanteaya: That's pretty much where I was too. Thanks.14:33
anteayaDuncanT: thank you, and the person doing rechecks is asleep right now I take it?14:33
*** zz_dimtruck is now known as dimtruck14:34
DuncanTGiven his timezone, probably, yes14:34
*** tqtran has quit IRC14:34
*** admcleod_ has joined #openstack-infra14:35
anteayado you know them, or should I comment on the patch that rechecking over and over isn't the best approach?14:35
anteayaI don't know if they know that14:35
*** admcleod has quit IRC14:35
*** _nadya_ has quit IRC14:35
*** cody-somerville has joined #openstack-infra14:35
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.
*** armax has quit IRC14:37
*** zhurong has joined #openstack-infra14:37
*** mdrabe has quit IRC14:37
anteayaDuncanT: thank you14:38
DuncanTanteaya: For those who don't want to dig into the guts of infra stuff, it actually is actually the best method to get a patch through in practice in cases like this, though it's usually best to give it twelve hours or so between runs14:39
*** zhurong has quit IRC14:39
*** bethwhite_ has quit IRC14:39
anteayaDuncanT: or at least look at the logs and comment saying the logs don't show me a failure14:40
*** Julien-zte has quit IRC14:40
anteayaare programmers unwilling to at least open a log?14:40
anteayastdout is sharing some input in that logfile14:41
anteayaI'm uncertain if what it is saying is enough to result in failure14:42
*** devkulkarni has quit IRC14:42
DuncanTHe did look at console.log, saw all successes and emailed me confused14:42
*** krtaylor has quit IRC14:42
*** bethwhite_ has joined #openstack-infra14:42
mordred2016-08-15 05:42:28,386 p=28725 u=zuul |  fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid": "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"}14:43
DuncanTI looked at console.log, saw all success and was confused too. I looked at the end of the other logs, didn't see anything suggesting they failed, was even more confused and so came here14:43
mordredis where the problem is14:43
*** devkulkarni has joined #openstack-infra14:43
anteayamordred: what is the problem14:43
anteayaI saw that but am unable to understand what it is trying to convey to me14:44
mordredwell - there is the problem and then the problem that's causing the problem14:44
anteayamordred: do expand14:44
* anteaya draws up a chair14:44
anteayaDuncanT: I'm glad you looked, thank you14:44
mordredthe direct problem is that the task execution bit of ansible, in this case we use the async task runner, has received additional output from something that is supposed to be only json14:44
*** mdrabe has joined #openstack-infra14:45
mordredthe indirect problem is whatever is causing that additional output14:45
anteayashould a test fail in this scenario?14:45
mordredI do not konw the current state of investigation into when this happens14:45
*** jdennis1 has quit IRC14:45
anteayamordred: so far this is the first incident I have seen14:45
*** kzaitsev_mb has quit IRC14:46
anteayabut I will say I am not up on all the backscroll14:46
mordredwell, it is to - as ansible essentially has broken in its ability to read the return code from the test process14:46
mordredso from ansibe's pov all it knows is "something broke"14:46
jeblairmordred: i think investigation needs to be re-opened14:46
mordredjeblair: I agree14:46
mordredjeblair: I thought I remembered pabelanger saying something about library warnings and had a hypothesis14:46
mordredjeblair: luckily - it seems we have a change that consistently fails :)14:47
jeblairmordred: i think you and Shrews landed 2 changes in ansible to address this, right?  i don't remember what they were supposed to do though14:47
*** sdake has quit IRC14:48
stevemarcan someone help me in getting me added to keystone-release? apparently we're going to be using them for the upcoming release, but i'm not in the group (am in stable release fwiw)14:48
*** hongbin has joined #openstack-infra14:50
*** nwkarsten has quit IRC14:50
*** nwkarsten has joined #openstack-infra14:51
DuncanTmordred: There doesn't seem to be any logging anywhere that lets anybody not intimately familiar with all this debug what actually output the wrong thing though... or even for that matter for somebody who is familiar. Might there be some benefit to adding -vvv to the ansible execution? It gets sent to a separate log, so it won't be noise in the normal case14:52
*** jbernard1 has joined #openstack-infra14:52
*** xarses has joined #openstack-infra14:53
*** ociuhandu has joined #openstack-infra14:55
*** nwkarsten has quit IRC14:55
mordredDuncanT: well, unfortunately in this particular case there is no additional information available that would be any more useful to anyone than the error message that's there... and it's a bug that such a behavior is surfacing to the user at all14:56
jeblairmordred: i found 229d8f6b21109e4180e457d95765379d07af384e14:57
jeblairmordred: wasn't there another one?14:57
mordredDuncanT: I say that not to discount the idea, which is good - but more to give you a sense of where we're at with debugging when this happens - as soon as we can characterize what's actually happening (which we don't know) we'll be able to respond and be resilient - and/or add user facing messages that would help a user deal with it14:57
*** dtantsur is now known as dtantsur|mtg14:57
jeblairmordred: was there a change to actually output what it's failing to parse?14:58
*** Julien-zte has joined #openstack-infra14:58
DuncanTmordred: Fair enough. I'm no kind of ansible expert, but running -vvv is my usual first port of call in debugging things14:58
mordredjeblair: looking14:59
jeblairDuncanT: does running with -vvv output the data that it fails to parse?14:59
*** jimbaker has joined #openstack-infra14:59
*** xarses has quit IRC14:59
*** apetrich has quit IRC14:59
*** bin_ has joined #openstack-infra15:00
DuncanTjeblair: In this specific case, I don't know - I don't know how to repro this environment to find out. In many cases, it prints the command line being run and any unexpected stderr15:00
jeblairDuncanT: the issue in this case is that there is an internal error parsing the json that the ansible async module passes around15:01
jeblairDuncanT: in previous versions of ansible, there was no way to see the data that caused the parse error15:01
jeblairDuncanT: *that* is what we need to proceed in debugging15:01
*** spzala has joined #openstack-infra15:01
jeblairDuncanT: i'm trying now to ascertain if there is such a way in the newly released version15:01
DuncanTjeblair: I know precisely nothing about the async module, but I'd be surprised if it has changed. It's just python code though, right, so we could, in theory, patch it (or, more sensibly, create our own async module with better verbose output and send the patch upstream in the hope we can drop it in future)15:02
*** elo has quit IRC15:02
jeblairDuncanT: well, it was *supposed* to have changed with some patches from mordred and Shrews to address this problem15:03
jeblairmordred: was the other one 4e239f6ce0d8ed96d734ef6ca75fa745c3925045 ?15:03
DuncanTjeblair: Ah, got you. Ok, I'll sit back and wait for a while, clearly I don't have enough history to be more than noise at this point, since I've not got any time to really dig in15:03
mordredjeblair: I do not see that commit?15:03
Jeffrey4l_any guys can review this?
Jeffrey4l_it block kolla-kubernets project now.15:04
openstackgerritMerged openstack/os-testr: Remove discover from test-requirements
openstackgerritMerged openstack/os-testr: Delete openstack/common in flake8 exclude list
*** devkulkarni has quit IRC15:05
*** devkulkarni has joined #openstack-infra15:06
*** devkulkarni has quit IRC15:06
*** senk_ has quit IRC15:06
anteayaDuncanT: thanks for bringing this to our attention15:06
*** rcernin has quit IRC15:06
mordredjeblair: it looks like we do return "async_result" ... but that does not include the raw thing15:06
jeblairDuncanT: well, if you know anything about how the async module works or have suggestions on how to debug this, that would be great.  however, at our volume of work, we can't afford to run with -vvv except for just a few minutes, so we need to know it's going to help.15:06
jeblairmordred: second commit was in modules/core15:07
*** nwkarsten has joined #openstack-infra15:07
jeblairmordred: next step: have ansible log the failed-to-parse data?15:08
mordredjeblair: see it15:08
*** Shrews has joined #openstack-infra15:09
mordredjeblair: so - it didn't fail in utilities/logic/ best I can tell15:11
*** dprince has quit IRC15:12
mordredjeblair: becuase both of the json parsing exception handlers there set failed=1 in the result dict15:12
mordred2016-08-15 05:42:28,386 p=28725 u=zuul |  fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid": "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"}15:12
mordredoh. wait. gah15:12
*** karthik__ has joined #openstack-infra15:12
*** nwkarsten has quit IRC15:13
*** yamamoto has quit IRC15:14
*** yamamoto has joined #openstack-infra15:14
*** kzaitsev_mb has joined #openstack-infra15:15
sbezverkjeblair: Curios why infra team keeps ignoring the issue we are trying to bring to your team attention for past three days?15:17
*** nwkarste_ has joined #openstack-infra15:18
mordredsbezverk: well, the last two days were the weekend15:18
*** javeriak has joined #openstack-infra15:18
sbezverkmordred: It sounds like a reply I would get from IT if 1990's15:18
jeblairsbezverk: back when people didn't work on weekends?15:19
jeblairsbezverk: i don't work on weekends15:19
jeblairsbezverk: you are welcome to join the infra team and work on weekends if you like15:19
*** nwkarst__ has joined #openstack-infra15:19
*** nwkarst__ has quit IRC15:19
sbezverkif I could +2 I would15:19
sbezverkbut I cannot and I have to rely on existing cores15:20
jeblairsbezverk: start by +1ing15:20
jeblairsbezverk: eventually it turns into +215:20
*** yamamoto has quit IRC15:20
*** jtomasek|biab is now known as jtomasek15:20
jeblairsbezverk: though the -1 is more important for that15:20
*** phschwartz has quit IRC15:20
*** nwkarsten has joined #openstack-infra15:20
sbezverkjeblair: we have kolla-kube projecy on hold because of the gate15:20
*** nwkarsten has quit IRC15:20
sbezverkit is surprising to see this being ignored..15:21
anteayasbezverk: I reviewed the patch15:21
anteayasbezverk: have you read my review yet?15:21
*** nwkarst__ has joined #openstack-infra15:22
odyssey4meanteaya if you have a moment, reviews of & would be appreciated15:22
*** nwkarste_ has quit IRC15:22
*** nwkarst__ has quit IRC15:22
anteayaodyssey4me: sure I'm chairing a meeting15:22
anteayaif I get a moment after that I will look, thank you15:23
odyssey4mesure, once you're done of course15:23
*** nwkarste_ has joined #openstack-infra15:23
*** nwkarste_ has quit IRC15:23
anteayathank you15:23
sbezverkanteaya: done, I posted15:23
*** nwkarste_ has joined #openstack-infra15:24
*** nwkarste_ has quit IRC15:24
anteayathank you15:24
anteayaonce I finish chairing this meeting I will look again15:24
sbezverkanteaya: thank you15:24
anteayayour welcome15:25
*** nwkarste_ has joined #openstack-infra15:25
anteayaand if you want to start reviewing infra patches, let me know if you want any guidance on that15:25
anteayahappy to have more reviewers15:25
* mordred waves at Shrews and hopes he's excited about this morning's ansible issue15:26
ShrewsSO excited15:27
Shrewsand i just can't hide it15:27
*** nwkarst__ has joined #openstack-infra15:27
*** nmagnezi has quit IRC15:28
*** nwkarst__ has quit IRC15:29
Shrewsjeblair: 354419 seems to have a silly typo in the string format  :)15:29
*** nwkarst__ has joined #openstack-infra15:29
*** esikachev has quit IRC15:29
*** nwkarste_ has quit IRC15:29
*** nwkarste_ has joined #openstack-infra15:30
*** nwkarste_ has quit IRC15:30
jeblairShrews: %i is a thing :)
jeblairShrews: the problem we're looking at today is:15:31
jeblair15:12 < mordred> 2016-08-15 05:42:28,386 p=28725 u=zuul |  fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid":  "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"}15:31
Shrewsjeblair: ugh, my bad. i was going off of this:
*** nwkarsten has joined #openstack-infra15:31
*** nwkarsten has quit IRC15:32
jeblairthough i almost never use %i because %s is easier than thinking.15:32
*** nwkarsten has joined #openstack-infra15:33
Shrewsit's neat that they have that spec in 2 different places and they aren't the same15:33
*** armax has joined #openstack-infra15:33
fungisbezverk: i was working quite a lot through the weekend (much to my wife's annoyance) but it wasn't obvious to me what was going on there either, nor that it was imminently urgent for you. i'm sorry about that, but i also have to say i find your choice of words rather offensive so please keep discussion constructive in here in the future15:33
*** nwkarsten has quit IRC15:33
*** Julien-zte has quit IRC15:33
*** edtubill has joined #openstack-infra15:33
*** nwkarsten has joined #openstack-infra15:34
*** nwkarst__ has quit IRC15:34
*** mdrabe has quit IRC15:35
*** thiagop has quit IRC15:35
*** mdrabe has joined #openstack-infra15:35
*** baoli_ has quit IRC15:35
Shrewsjeblair: yeah, mordred's already brought that to my attention. when i've seen that in the past (i think), it was b/c the results were nothing (not something and that something couldn't be parsed)15:35
*** nwkarste_ has joined #openstack-infra15:35
*** nwkarste_ has quit IRC15:36
jeblairShrews, mordred: i think the weakness here is that they are not logged.15:36
jeblairso we're still just guessing15:36
*** thiagop has joined #openstack-infra15:36
*** nwkarste_ has joined #openstack-infra15:36
mordredShrews: could that be another race? like, when you say "results were nothing" - you mean the async result file was empty, yeah?15:36
Shrewsmordred: yeah15:36
Shrewsmordred: dunno if it's another race though. just stating what i saw before15:37
anteayafungi: mordred I just skimmed backscroll but thank you for the weekend gerrit reindex15:37
*** armax has quit IRC15:37
anteayaI haven't had a chance to read why it was required yet15:38
*** nwkarst__ has joined #openstack-infra15:38
*** nwkarst__ has quit IRC15:38
anteayaand zaro thanks for your help on the weekend15:38
anteayaand if I missed anyone else15:38
*** nwkarsten has quit IRC15:38
*** nwkarsten has joined #openstack-infra15:38
fungianteaya: because i missed that puppet was going to care about the ownership on the gerrit build i downloaded and treat correcting it as a gerrit upgrade (and the manifest still tells it to restart gerrit and run an offline reindex when that happens)15:39
*** nwkarst__ has joined #openstack-infra15:39
anteayaah :(15:39
zaroactually we all forgot about that15:39
anteayaoh my15:39
fungianteaya: now that has merged, it shouldn't happen again15:39
sbezverkfungi: Appologies for being offensive, but please understand my frustration too, since friday none of the ready patches got merged and bunch of new PS are all failing and we need to produce working demo in 1 week15:40
*** edtubill has quit IRC15:40
*** edtubill has joined #openstack-infra15:40
*** harlowja_at_home has joined #openstack-infra15:40
anteayaI'm glad you fixed the problem, fungi and zaro and mordred15:40
*** nwkarste_ has quit IRC15:40
*** elo has joined #openstack-infra15:40
anteayathank you15:40
*** nwkarste_ has joined #openstack-infra15:41
*** nwkarste_ has quit IRC15:41
*** devkulkarni has joined #openstack-infra15:41
anteayasbezverk: what you describe is typical of every group we work with15:41
anteayawe do our very best, every day, all day long15:41
*** nwkarste_ has joined #openstack-infra15:42
anteayaand some folks past that15:42
*** nwkarste_ has quit IRC15:42
anteayawe didn't agree to your timeline, that is your doing15:42
anteayawe have over 100 projects to address and deal with15:42
anteayaour work can be very draining and tiring15:42
anteayawe do best fueled by gratitude15:42
anteayawhich is always appreciated15:42
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add blurb about communication to docs landing page
fungiluckily this doesn't feel like a job to me, so i don't mind doing it 60-80 hours a week but we all need to sleep sometime ;)15:43
*** nwkarsten has quit IRC15:43
anteayanow if you need more from us that what we provide on a daily basis, please discuss this with us in advance15:43
anteayafungi: or mow the lawn as the case may be :)15:43
*** nwkarst__ has quit IRC15:43
wznoinsklennyb: lennyb: the urllib problem was caused us using wrong version of devstack (hence devstack/lib/tempest)15:43
*** nwkarste_ has joined #openstack-infra15:44
*** nwkarste_ has quit IRC15:44
*** nwkarste_ has joined #openstack-infra15:44
*** nwkarste_ has quit IRC15:44
anteayawznoinsk: lennyb ah you found the issue?15:45
*** weshay is now known as weshay_brb15:45
wznoinskwznoinsk: yes, not sure whether that's the same as for others because our issue was caused by using wrong devstack commit15:45
*** nwkarste_ has joined #openstack-infra15:46
*** nwkarste_ has quit IRC15:46
wznoinskwhich shouldn't happen when you follow master or stable branches for devstack15:46
*** asettle has quit IRC15:46
lennybwznoinsk, urllib3==1.14 I guess so, also I've got answer that tempest uses virtual environment I am not sure how/if it works15:46
*** vhosakot has joined #openstack-infra15:47
*** nwkarste_ has joined #openstack-infra15:47
*** asettle has joined #openstack-infra15:47
*** nwkarste_ has quit IRC15:47
pabelangercloudnull: odyssey4me: We setup a local unbound service, which then forwards to google ipv4:
lennybwznoinsk,logs  #link
*** nwkarste_ has joined #openstack-infra15:48
jeblairShrews, mordred: how should we proceed?15:48
pabelangercloudnull: odyssey4me: I _think_ we can add forward-addr: google ipv6 dns just about the existing statement on line 29 and hope unbound does the right thing15:48
*** elo has quit IRC15:48
pabelangerI just need to test it on both ipv4 / ipv6 networks15:48
lennybwznoinsk, I export .._BRANCH=stable/mitaka , also for devstack branch15:48
fungilennyb: wznoinsk: right, tempest in our jobs is run from within a virtualenv so that it can depend on different versions of things than openstack services/libraries15:48
lennybfungi, when I've updated urllib3 after devtsack install it worked15:49
fungiparticularly necessary since it's branchless so would be hard to make work with teh constrained versions of any of its dependencies from every supported stable branch15:49
mtreinishfungi: well in most cases, there is one job that uses a venv with system site-packages. But there is no reason to use that at all since you can install the plugin very easily in a real venv15:49
lennybfungi, what do you mean in your jobs? how do you run it?15:49
fungilennyb: from devstack-gate's default gate_hook15:50
lennybfungi in our CI, I just cd /opt/tempest;  testr run15:50
*** nwkarsten has joined #openstack-infra15:50
cloudnullpabelanger:  is that something new-ish ?15:50
mordredjeblair: I do not yet have a good idea - still reading through things15:50
*** nwkarst__ has joined #openstack-infra15:51
cloudnullbecause we're seeing nameservers set in the resolv.conf file of instances built on other providers, like rax15:51
jeblairmordred: okay, i'll wait till you're done15:51
*** Apoorva has joined #openstack-infra15:51
pabelangercloudnull: nameserver should be on all images15:52
openstackgerritgreghaynes proposed openstack-infra/project-config: Add notifications for dib changes to openstack-dib
*** nwkarste_ has quit IRC15:52
pabelangercloudnull: been that way for a while, IIRC15:52
mordredcloudnull: but the interaction between the unbound and the ipv6-only instances is new ... so that might be where something is going strange?15:52
*** vhosakot has quit IRC15:52
*** weshay_brb is now known as weshay15:52
mordredit might not be - just mainly pointing out that that's the thing that changed15:52
*** nwkarste_ has joined #openstack-infra15:52
*** nwkarste_ has quit IRC15:53
anteaya<-- offline for a bit15:53
fungilennyb: under tox as the tempest user, from the look of things (sudo -H -u tempest tox)
*** vhosakot has joined #openstack-infra15:53
*** asettle has quit IRC15:53
*** Sukhdev has joined #openstack-infra15:53
*** nwkarste_ has joined #openstack-infra15:53
*** asettle has joined #openstack-infra15:54
fungilennyb: followed by options defining which set of tests to run based on the name of the tox env invoked15:54
*** martinkopec has quit IRC15:54
Shrewsjeblair: mordred: we're running the most recent ansible, yeah?15:54
*** nwkarsten has quit IRC15:55
jeblairpabelanger, cloudnull, mordred: if resolv.conf in rax nodes is being set to rax ns, that is unexpected behavior15:55
jeblairShrews: yes15:55
lennybfungi: thanks, I will check this approach15:55
pabelangerjeblair: agreed15:55
*** nwkarsten has joined #openstack-infra15:55
pabelangerconfirming that now15:55
jeblairpabelanger, cloudnull, mordred: a spot check of an idle rax instance confirms they are being set to rax ns15:55
*** nwkarsten has quit IRC15:55
cloudnullfailure in OSIC:
cloudnullsuccess in RAX:
pabelangerjeblair: cloudnull: mordred: confirmed. I can work on a patch15:56
*** nwkarst__ has quit IRC15:56
*** Na3iL has joined #openstack-infra15:56
*** nwkarsten has joined #openstack-infra15:56
fungiouch. i wonder what's updating our resolvers in rax? i thought we were explicitly overriding that because their resolvers were unreliable for our use case15:56
jeblairpabelanger: thanks15:56
jeblairfungi: yes, i am very interested to see how that slipped through.  again.15:56
fungimaybe a new glean feature or something15:57
*** nwkarsten has quit IRC15:57
*** Apoorva has quit IRC15:57
openstackgerritMerged openstack-infra/jenkins-job-builder: Update M2 Release plugin to use convert xml
*** xarses has joined #openstack-infra15:57
pabelangerfungi: let me check that first15:57
*** mtanino has joined #openstack-infra15:57
cloudnullfungi: I can also poke about at other regions too to see whats being set. its not total bust for us, we're just seeing some slow gates in the osic and that was a change that stood out.15:57
*** vhosakot has quit IRC15:57
*** nwkarsten has joined #openstack-infra15:58
*** nwkarste_ has quit IRC15:58
mordredfungi: we're fully on glean in rax now, right?15:58
*** gothicmindfood has quit IRC15:58
*** vhosakot has joined #openstack-infra15:58
openstackgerritMerged openstack-infra/project-config: Make the kolla-kubernetes relate jobs non-voting
fungimordred: everywhere, right15:58
*** nwkarste_ has joined #openstack-infra15:59
fungimordred: and they don't have dhcp and we're not installing nova-agent, so the only likely answer is that glean is setting it based on resolver info in the configdrive metadata15:59
pabelangerI can see DNS servers in network_data.json for rax, check what glean is doing now15:59
cloudnullbut to pabelanger point if we could get ipv6 DNS in the unbound forwarder that would be fantastic too15:59
fungicloudnull: yes, we need to do that16:00
*** nwkarst__ has joined #openstack-infra16:00
pabelangercloudnull: yup16:00
pabelangerI think we could use glean for that too16:00
*** nwkarst__ has quit IRC16:00
pabelangeroh wai16:00
*** jpich has quit IRC16:00
pabelangerno, that is not correct16:00
lennybwznoinsk, thanks, I will update you if/when I have more data16:00
fungiclarkb mentioned late last week (friday?) that we were likely hardcoding ipv4 resolvers in our unbound forwarders16:00
*** nwkarst__ has joined #openstack-infra16:00
jeblairyes, probably so16:00
*** elo has joined #openstack-infra16:01
fungiwhich would need adjusting for v6-only environmentsd16:01
*** yamahata has joined #openstack-infra16:01
*** vinaypotluri has joined #openstack-infra16:01
jeblair2001:4860:4860::8888 2001:4860:4860::884416:01
*** ifarkas is now known as ifarkas_afk16:01
*** mat128 is now known as mat128|afk16:02
jeblairare the v6 addrs for google16:02
*** dtantsur|mtg is now known as dtantsur16:02
*** adrian_otto has joined #openstack-infra16:02
*** nwkarsten has quit IRC16:02
*** apetrich has joined #openstack-infra16:02
*** nwkarsten has joined #openstack-infra16:03
*** nwkarsten has quit IRC16:03
fungiyeah, i think the suggestions were we could check at boot whether we have a global route for ipv6 and if so default to the v6 resolver addresses falling back to the current configuration for v4-only servers, or that we could set it in our nodepool ready scripts based on the provider where we're booting (though that would mean services starting at boot have no working name resolution i guess)16:03
*** nwkarste_ has quit IRC16:03
*** nwkarsten has joined #openstack-infra16:04
*** martinkopec has joined #openstack-infra16:04
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar
wznoinsklennyb: it looks like a different issue, you get the urllib 3.16 installed for tempest at 2016-08-11 13:48:49.91616:04
*** wznoinsk has left #openstack-infra16:04
*** wznoinsk has joined #openstack-infra16:04
*** nwkarste_ has joined #openstack-infra16:05
*** elo has quit IRC16:05
fungii think things like time synchronization would probably be broken unfortunately so that second idea is probably not viable16:05
*** nwkarste_ has quit IRC16:05
*** nwkarst__ has quit IRC16:06
*** nwkarste_ has joined #openstack-infra16:06
pabelangerjeblair: fungi: mordred: yes, looks like glean is setting up DNS in rax:
lennybwznoinsk but pip freeze shows urllib3=1.14
*** martinkopec has quit IRC16:07
wznoinsk3.16 was in a venv, if you're running tempest out of venv youre using 3.16, if not then 3.1416:07
*** nwkarst__ has joined #openstack-infra16:07
*** tqtran has joined #openstack-infra16:08
openstackgerritMerged openstack-infra/system-config: Add firehose.o.o to cacti
*** nwkarst__ has quit IRC16:08
mordredpabelanger: yah. are we not re-overwriting that in our ready scripts?16:08
jeblairmordred: we don't set up networking in the ready script16:08
jeblairmordred: we set up dns in the image16:08
*** nwkarsten has quit IRC16:08
*** nwkarst__ has joined #openstack-infra16:08
jeblairso glean is undoing that16:08
lennybwznoinsk, I am running tempest from the shell. so I guess it uses 3.1416:09
jeblairwe need to tell glean not to touch resolv.conf16:09
*** nwkarst__ has quit IRC16:09
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Fingerprint plugin
pabelangerit looks like can parse something and set switches on glean16:09
pabelangermaybe a /etc/defaults/glean file?16:10
*** jtomasek is now known as jtomasek|afk16:10
*** nwkarsten has joined #openstack-infra16:10
fungithere has been some back and forth on whether glean should have a config file16:10
*** ihrachys has quit IRC16:10
*** nwkarsten has quit IRC16:10
*** dims has quit IRC16:10
mordredyah - and how it should know that a thing like resolv.conf should not be modified16:10
*** nwkarsten has joined #openstack-infra16:11
fungichattr +i? ;)16:11
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar
jeblairsadly, we figured out how to convince dhclient and friends not to modify it16:11
pabelangerfungi: nice16:11
jeblairfungi: basically, yes, we did that16:11
jeblairnow we have to figure it out again with glean16:11
*** nwkarste_ has quit IRC16:11
jeblairor write a new network bootstrapping system with even *fewer* features16:11
wznoinsklennyb: tempest log would tell you that, i.e.:
*** gyee has joined #openstack-infra16:12
pabelangerfungi: I lol'd more then I should have on that16:12
jeblairor, i dunno just give up16:13
jeblairand use the cloud dns systems16:13
jeblairi admit, i'm frustrated16:13
*** nwkarste_ has joined #openstack-infra16:13
jeblairbecause we spent so long getting this right before glean, and then we undid it.16:13
*** admcleod has joined #openstack-infra16:13
*** admcleod has joined #openstack-infra16:13
*** nwkarste_ has quit IRC16:13
*** nwkarste_ has joined #openstack-infra16:14
*** admcleod_ has quit IRC16:14
jeblairi want to say it took us a few months16:14
lennybwznoinsk, ok, I've got it, I will run tempest from virt env16:14
jeblairbecause each change is an image rebuild16:14
pabelangerjeblair: what are you thoughts on setting up google ipv6 dns for osic-cloud1?  When should we write them to /etc/unbound/forwarding.conf?16:14
*** gothicmindfood has joined #openstack-infra16:15
pabelangerstruggling to find the right solution16:15
mordredwell, I am going to go back to thinking about the other problem, because I'm finding the tone of dealing with this problem to be quite unpleasant and unproductive16:15
jeblairmordred: thanks16:15
*** nwkarst__ has joined #openstack-infra16:15
*** nwkars___ has joined #openstack-infra16:16
*** nwkarsten has quit IRC16:16
*** e0ne has quit IRC16:17
openstackgerritMerged openstack-infra/jenkins-job-builder: Update xvnc to use convert xml
*** nwkarsten has joined #openstack-infra16:17
*** vhosakot has quit IRC16:17
greghayneschattr +i seems like a great idea IMO16:17
*** nwkarsten has quit IRC16:17
jeblairpabelanger: i would say we should add the v6 addrs in the same place we add the v4.  maybe we can add them both16:17
*** lucasagomes is now known as lucas-afk16:17
greghaynesand if glean doesnt handle chattr +i I'd consider it a glean bug16:18
*** nwkarsten has joined #openstack-infra16:18
*** vhosakot has joined #openstack-infra16:18
*** nwkarsten has quit IRC16:18
pabelangerjeblair: okay, that is DIB element today. I'll continue testing that path, thanks16:18
*** nwkarste_ has quit IRC16:18
*** nwkarsten has joined #openstack-infra16:18
*** nwkarst__ has quit IRC16:19
*** karthik__ has quit IRC16:19
*** nwkarste_ has joined #openstack-infra16:20
*** nwkars___ has quit IRC16:20
*** _nadya_ has joined #openstack-infra16:21
jeblairgreghaynes: if we go with chattr we will have *literally* gone full circle with glean:
jeblairso, i mean, if that's the interface we want to go with, we do have the patches already written.16:22
*** nwkarst__ has joined #openstack-infra16:22
krotscheckOh wow, AJaeger isn't around. Is the world ending?16:23
*** nwkarst__ has quit IRC16:23
pabelangerkrotscheck: PTO for the next 10days I believe16:23
jeblairgreghaynes: otoh, as a user, i don't think it's a good interface, and istr we had many problems with it.16:23
krotscheckpabelanger: Aaaah16:23
krotscheckpabelanger: Smart man to sign out of IRC :)16:23
krotscheckpabelanger: Did the new bindep images get uploaded?16:23
*** nwkarst__ has joined #openstack-infra16:23
pabelangerkrotscheck: I believe all clouds are using it16:23
*** nwkarsten has quit IRC16:24
greghaynesjeblair: Do you remember any of the issues with it? My thinking is simply that its a lot less complexity for glean to detect failure when writing to the file as opposed to config parsing16:24
*** dprince has joined #openstack-infra16:24
*** lbeliveau has quit IRC16:24
greghaynesand regardless glean should handle a failure there16:24
greghaynesI lack the context on why you all switched off chattr +i, I thought you always wanted to override resolv.conf to be nameserver because of unbound, so no matter what glean wouldnt be writing the correct thing there16:25
jeblairgreghaynes: one of the issues is highlighted in the comments:  "Of course this means Puppet won't be able to update it either after this, but we don't plan on changing it."16:25
mordredgreghaynes: I think I'm leaning more towards a glean config16:26
*** nwkarste_ has quit IRC16:26
*** nwkarste_ has joined #openstack-infra16:26
fungi don't see a problem with glean supporting its own dedicated configuration, so long as it has sane default behaviors when there is no glean config present16:26
mordredbecause honestly, inferring whether or not the resolv.conf that came in the image is more valid thatn the metadata provided by the cloud via config-drive or dhcp ... is likely never going to happen16:26
mordredfungi: yah16:27
mordredconfig should be highly optional16:27
Shrewsjeblair: so, i *think* i have identified an issue with the code we added to ansible before for this16:27
* mordred concurs with Shrews theory16:27
*** nwkarst__ has quit IRC16:28
*** lbeliveau has joined #openstack-infra16:28
*** harlowja_at_home has quit IRC16:28
*** nwkarst__ has joined #openstack-infra16:28
fungii mean, you could do it with envvars passed from the calling startup script or command-line options/arguments, but those are basically just configuration supplied in different ways16:28
*** nwkarst__ has quit IRC16:28
greghaynesYea, I'd go configuration over env vars. I'm not super opposed to env vars, jsut trying to find the path of least resistence16:28
greghaynessounds like config file might be that16:28
*** hockeynut has joined #openstack-infra16:28
greghayneser, sorry, not super opposed to a config file16:29
fungicloud-init and dhclient support configuration files on disk that can tell them to leave resolv.conf alone16:29
*** nwkarst__ has joined #openstack-infra16:29
Shrewsjeblair: we should be doing async_result.get('parsed', False) there16:29
*** nwkarst__ has quit IRC16:29
*** yamahata has quit IRC16:29
jeblairgreghaynes: it looks like using chattr had a cascading failure effect and broke everything in rackspace, which is why we reverted it over the weekend.  it was probably rackspace-specific stuff which doesn't apply now.  but i think it caused me to think of the approach as fragile.
Shrewsjeblair: that job that failed seemed to be taking longer than normal (based on past jobs i looked at) which likely gave it more time to fail in that way we are trying to catch there16:30
*** nwkarsten has joined #openstack-infra16:30
*** nwkarsten has quit IRC16:30
Shrewsbut we failed at catching it16:30
*** nwkarste_ has quit IRC16:31
*** dkehn_ has quit IRC16:31
krotscheckpabelanger: Excellent!16:31
*** _nadya_ has quit IRC16:31
greghaynesok, so the world just hates readonly files it sounds like16:31
*** tqtran has quit IRC16:31
*** dkehn has quit IRC16:31
greghaynesseems plausible given that glean probably explodes right now with them due to the same reasons, its not somthing most folks consider16:31
jeblairShrews: thinking...16:31
krotscheckinfra-core: according to pabelanger, Ajaeger's comment on has now been resolved - can anyone step in and give it the missing +A? (already has 2x+2)16:31
*** Apoorva has joined #openstack-infra16:31
*** nwkarste_ has joined #openstack-infra16:31
greghaynesmordred: config file SGTM - theres another TODO of making glean support not re-asserting state based on machine-id which needs some similar code I think16:32
Shrewsjeblair: tl;dr, if 'parsed' isn't in the response, we don't want to quit16:32
*** xarses has quit IRC16:32
*** dkehn has joined #openstack-infra16:32
*** Jeffrey4l_ has quit IRC16:33
clarkbgood morning16:33
cloudnullo/ clarkb16:33
*** nwkarst__ has joined #openstack-infra16:33
*** nwkarst__ has quit IRC16:33
*** nwkarst__ has joined #openstack-infra16:34
*** Hal has joined #openstack-infra16:34
openstackgerritMerged openstack-infra/git-review: Clarify that submitting multiple commits is OK
*** nwkars___ has joined #openstack-infra16:35
jeblairShrews, mordred: i agree, defaulting the getter to true does not match the behavior in the comment.  we have probably effectively changed many of the "timeout" errors to "unparseable" errors with that change.16:35
mordredjeblair: ++16:35
jeblairShrews, mordred: could probably just be "async_result.get('parsed')"16:35
mordredjeblair:  we want to bail from the loop when we have parsed a failure16:35
*** nwkarsten has joined #openstack-infra16:36
*** nwkarsten has quit IRC16:36
jeblair(to match the rest of the getters)16:36
Shrewsmordred: trying to find your original PR for that... you remember it?16:36
mordredShrews: I can look16:36
clarkbsbezverk: fwiw you could have made those tox targets return success if things are really that pressing16:36
jeblairShrews, mordred:
mordredbeat me16:37
*** nwkarsten has joined #openstack-infra16:37
clarkbsbezverk: its not typically a great idea to have your tests unconditionally pass... but if you are indeed in such a place where you need stuff changed over the weekend that is an option available to you16:37
*** nwkarste_ has quit IRC16:37
*** tosky has quit IRC16:37
jeblairhave we ever said that's okay?16:38
jeblairi guess we have now16:38
*** nwkarst__ has quit IRC16:38
*** nwkarste_ has joined #openstack-infra16:38
anteayamorning clarkb16:39
*** nwkars___ has quit IRC16:39
zarowould any infra-core be wiling to help enable gerrit/storyboard integration today? referenced instructions are referenced in commit message:
*** sputnik13 has joined #openstack-infra16:40
anteayaclarkb: I hope you had a wonderful time mostly offline16:40
*** tqtran has joined #openstack-infra16:40
clarkbjeblair: I think it would be nice to have projects use the PTI and not set things nonvoting and just stub stuff out if they have to. Not sure if that is the situation that the kolla folks are in16:40
clarkbjeblair: there is a ton of project-config chrun just to handle "we don't have tests yet" when its trivial to have a single tests that passes16:40
clarkbsame thing with docs and pep816:40
*** florianf has quit IRC16:40
*** nwkarsten has quit IRC16:41
*** devkulkarni1 has joined #openstack-infra16:41
*** nwkarst__ has joined #openstack-infra16:41
*** dims has joined #openstack-infra16:42
Zara(:D I'm distracted with storyboard js meetup but still excitedly watching gerrit things)16:42
*** dkehn has quit IRC16:43
*** nwkarsten has joined #openstack-infra16:43
*** rbuzatu has quit IRC16:44
*** devkulkarni has quit IRC16:44
*** tqtran has quit IRC16:44
*** nwkarste_ has quit IRC16:44
*** rbuzatu has joined #openstack-infra16:44
*** nwkarste_ has joined #openstack-infra16:45
sdaguehmmm... even with wheels, it seems like it's taking us 5 minutes to do pip installs on standard runs, possibly because of bw to our mirrors?16:46
pmalikHello dear Infra cores. We (Trove/DBaaS) are looking to gather more data on some of our other supported datastores. It would be really helpful to see the tests as 'nv'. Could you possibly review at your discretion: Thanks.16:46
sdague - that looks like the download is happening at 4Mbps for numpy16:46
*** nwkarst__ has quit IRC16:46
*** nwkarst__ has joined #openstack-infra16:46
*** nwkarst__ has quit IRC16:46
*** dims has quit IRC16:47
*** nwkarst__ has joined #openstack-infra16:47
*** dkehn has joined #openstack-infra16:48
*** nwkarsten has quit IRC16:48
fungisdague: does it seem worse in rackspace than elsewhere? we get some pretty terrible network behaviors in their ord region in particular16:48
openstackgerritSean Dague proposed openstack-infra/project-config: increase novaclient functional timeout.
sdaguefungi: I don't have a systemic view here16:49
*** nwkarsten has joined #openstack-infra16:49
*** nwkarsten has quit IRC16:49
fungiyeah, that's not an easy thing to query for16:49
*** dkehn_ has joined #openstack-infra16:49
sdaguebut I've failed on job timouts twice on rax ord and was blown away by the pip_install time16:49
sdaguepip_install           37516:50
sdagueon that job16:50
*** nwkarsten has joined #openstack-infra16:50
*** nwkarsten has quit IRC16:50
*** nwkarste_ has quit IRC16:50
fungisdague: i think we need bigger servers...
fungii bet that flavor has a 100mbps bw cap16:50
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar
sdaguevs. pip_install           184 on different provider16:50
sdaguefungi: ah, yeh, probably16:51
*** nwkarsten has joined #openstack-infra16:51
*** beagles has joined #openstack-infra16:51
sdagueand there are a bunch more nodes in that region than others, right?16:51
jeblairwow, we only recently hit that16:51
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests
sdagueyeh, 195 servers in rax-ord16:52
fungisdague: yep, so other rackspace regions are likely not seeing this due to lower instance quotas, but also other providers likely don't enforce the same bw limit16:52
*** tphummel has joined #openstack-infra16:52
*** nwkarste_ has joined #openstack-infra16:52
*** nwkarste_ has quit IRC16:52
fungias jeblair points out, we only just started hitting it in the past few weeks ourselves16:52
sdagueso 1/3 of our capacity is hitting that mirror16:52
*** nwkarst__ has quit IRC16:53
*** dims has joined #openstack-infra16:53
*** nwkarste_ has joined #openstack-infra16:53
*** adrian_otto has quit IRC16:54
sdagueyeh, though the trend line was there for a while, so it was inevitable16:54
*** tonytan4ever has quit IRC16:54
jeblairit is a disturbing trend line16:54
*** nwkarst__ has joined #openstack-infra16:55
*** nwkarst__ has quit IRC16:55
jeblairto double in 3 months16:55
*** nwkarsten has quit IRC16:56
sdagueyeh, it's also the slamming portion of the cycle16:56
*** nwkarsten has joined #openstack-infra16:56
*** nwkarsten has quit IRC16:56
sdagueanyway, I guess the question is, are there sensible relief valves here?16:56
jeblairyeah, we can spin up a new mirror16:56
*** nwkarsten has joined #openstack-infra16:56
jeblairinfra-root: ^ any volunteers?16:56
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Create timeline events for boards and worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API
sdaguehow hard would it be to pre cache into the images ? do a pip install upper-constraints.txt in a venv, then delete venv? Then we'd skip a lot of the downloads from the mirror.16:57
*** adrian_otto has joined #openstack-infra16:57
anteayaif the social contracts in a given project are such that a contributor offers a patch to project-config to change tests and noone else in the project was aware of the patch, I think the solution is better socialization within the project regarding change, not changing the behaviour of tests to report false status16:58
*** nwkarste_ has quit IRC16:58
*** lucas-afk is now known as lucasagomes16:58
odyssey4mesdague IIRC you can actually just tell pip to download, not install16:58
*** jerryz has joined #openstack-infra16:59
mordredso ...16:59
mordredwe started the entire pre-cache/mirror game with doing caching downloads into the images16:59
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add IPv6 DNS support
mordredit has almost never worked like expected16:59
sdaguemordred: because?17:00
*** signed8bit is now known as signed8bit_Zzz17:00
pabelangerjeblair: clarkb: cloudnull: ^ Some testing on both ipv4 / ipv6 clouds shows that should work for unbound^17:00
*** nwkarste_ has joined #openstack-infra17:00
mordredsdague: the reasons are varied and I have forgotten many of them - but it was consistently bad enough that we built mirrors instead17:00
*** sdake has joined #openstack-infra17:00
sdaguebecause during a devstack run, we only ever download things once, just through pip's internal cache17:01
clarkbanteaya: definitely. I just know that for many projects in this situation the only reason they fail is tehy haven't configured tox17:01
sdagueso if that was already populated with "recently" then it would at least relieve preasure17:01
*** nwkarst__ has joined #openstack-infra17:01
clarkbanteaya: so if we can just get them to do that instead they have tests that work and its less burden on project-config17:01
jeblairclarkb: that wasn't this situation at all17:02
sdagueupper-constraints changes would still go through17:02
clarkbjeblair: ok17:02
*** hrubi has quit IRC17:02
*** nwkars___ has joined #openstack-infra17:03
*** nwkars___ has quit IRC17:03
*** nwkars___ has joined #openstack-infra17:03
*** _nadya_ has joined #openstack-infra17:03
*** nwkars___ has quit IRC17:04
sdagueanyway, slightly related to that, we need a timeout bump on novaclient functional tests - - which is how I discovered this mirror constraint17:04
*** nwkarsten has quit IRC17:04
*** nwkarste_ has quit IRC17:04
*** nwkars___ has joined #openstack-infra17:04
*** signed8bit_Zzz is now known as signed8bit17:04
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Make the kolla-kubernetes jobs non-voting and experimental
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Refactor Isolated Env to use in unit tests
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Set author and committer explicitly
mordredsdague, jeblair: so - it might be worth re-trying. the pip caching code has gotten much better. and we also have the constraints files - so doing a "pip install -d . -c upper-constraints.txt global-requirements.txt" in the image build might work better now than it did a few years ago17:05
sdaguemordred: yeh, I wouldn't want to do anything more complicated that pip itself17:06
mordredwhen we did it last time, the newer pip download cache had not yet been implemented17:06
*** jaosorior has quit IRC17:06
*** nwkarst__ has quit IRC17:06
clarkbmordred: sdague is that still per user?17:06
sdagueclarkb: yeh17:06
sdagueso just do it as the stack user17:06
jeblairsdague: stack user does not exist17:06
fungioh, right, the last time we tried there was no such thing as a pip cache or a wheelhouse17:06
clarkbwhich doesn't actually exist there17:06
mordredand stack user would not help non-devstack changes17:06
jeblairsdague: jenkins/zuul is the only user17:06
jeblairsdague: so you'd need to sudo move the cache17:07
sdaguemordred: it would not, however devstack changes are probably the biggest consumers17:07
*** Goneri has quit IRC17:07
jeblair(which i believe we also did)17:07
mordredjeblair: ++17:07
mordredjeblair: I agree with you17:07
fungipresumably devstack-gate could mv/cp/rsync the cache from ~jenkins to ~stack17:07
odyssey4mecan the cache path be configured in the global pip.conf perhaps?17:07
*** Hal has quit IRC17:07
fungiodyssey4me: not easily since pip wants it writeable17:07
*** mat128|afk is now known as mat12817:07
*** Hal has joined #openstack-infra17:08
odyssey4mefungi something like /opt/pip_cache - and just make it writable for anyone/everyone?17:08
fungiand if memory serves it also checks ownership of the cachedir directly, so globally-writeable is probably not a solution17:08
*** harlowja has joined #openstack-infra17:08
electrofelixYorikSar: I wonder if you might review the response I left on a while back and see if it's acceptable for you?17:08
mordredyah. it does check ownership17:08
*** kzaitsev_mb has quit IRC17:08
sdagueok, I guess this is why we can't have nice things :)17:09
jeblairmordred, sdague: if someone wants to give that a shot, i'm not opposed.  it will increase our image sizes of course and consume root filesystem space.  it is also probably worth doing a quick test against an unsaturated mirror to find out how much faster we're actually talking about.17:09
sdaguenever mind then17:09
odyssey4meperhaps an extension of z-c then, which can move the folder appropriately and set the appropriate rights?17:09
*** dprince has quit IRC17:09
*** nwkars___ has quit IRC17:09
jeblairoh, well, never mind then17:09
mordredis bandwidth cached on the private network? and if not, is it viable to try to do config to use private network to hit mirror instead of public?17:09
*** yamahata has joined #openstack-infra17:09
fungithough having devstack-gate rsync ~jenkins/.cache/pip into ~stack/.cache and ~tempest/.cache when it's also rsync'ing git repos from /opt/git to ~stack/new may make sense?17:10
sdagueso the numbers I've got just by poking is that internap is doing the pip installs in < 1/2 the time of rax-ord17:10
odyssey4memordred that sounds like a nifty idea - it should also kill the L3 interaction which should speed it up17:10
sdagueand I think internap nodes are otherwise slower17:10
clarkbodyssey4me: its still L3ing on private net iirc17:10
sdagueso back of the envelope, we're probably adding 3 - 4 minutes to every rax job because of the bw constriction17:10
clarkbglean gets a list of nets to route through that interface17:11
sdaguerax dsvm job17:11
jeblairsdague: i would like to discount the rax-ord times because the solution to that is easy, get a new server17:11
fungiodyssey4me: mordred: that would also be a fairly rax-centric choice, since we're relying on their rfc-1918 flat net spanning tenants/projects17:11
jeblairsdague: the reason to use a local pip cache, in my mind, is if it's faster than our best-case times on an unsaturated mirror17:11
odyssey4mebah, this is why we can't have nice things :p17:12
*** rajinir has joined #openstack-infra17:12
mgagnesdague: I don't know about RAX but we have a lower number of instances and therefore nodes dedicated (not shared) for ci infra. At this point, you could be your own noisy neighbours. but I didn't fully read backlog =)17:12
clarkbmordred: rereading the bw details for rax the private net can do 2x the public net17:12
*** ihrachys has joined #openstack-infra17:12
clarkbsince public net can only utilize 50% of total bandwidth allocation17:12
fungimgagne: in this specific case it's rackspace's flavor-based bandwidth rate limits17:12
* anteaya buys many things at the thrift store as she has accepted she can't have nice things17:13
mordredclarkb: this: says there is no charge for traffic on servicenet - but it does not indicate if there are bandwidth caps17:13
clarkband 200mbps is the limit for the 2GB flavor and 50% of that is 100mbps which we are seeing17:13
fungimgagne: the flavor we used for mirror.ord.rax..o.o only gets 100mbps bw, and we're topping out there under load17:13
clarkbmordred: footnote 417:13
jeblairare we seriously thinking that we should try to work around this rather than just launch a new server?17:14
fungiyeah, their "200mbps" is 100mbps egress + 100mbps ingress if memory serves17:14
clarkbjeblair: no I think we should make an 8GB instance with 800mbps17:14
fungijeblair: i think we should just boot a replacement mirror.ord.rax..o.o but i don't personally have time to do it for a few more hours17:14
jeblairclarkb: not a 4g with 400?17:14
mgagnesdague: "I think internap nodes are otherwise slower" are we talking about jobs execution time? (not network) ?17:15
clarkbjeblair: maybe start there and go bigger if necessary17:15
pabelangerfungi: jeblair: I can boot the replacement if needed.17:15
fungii can get to it later today if we settle on a preferred flavor to replace the current one17:15
fungipabelanger: oh, thank you!17:15
odyssey4methe simplest solution is certainly the best, although the creative exercise of looking at alternative solutions is also interesting and can sometimes spawn unrelated ideas17:15
*** asettle has quit IRC17:15
*** vhosakot has quit IRC17:15
*** sarob has joined #openstack-infra17:16
mordredodyssey4me: agree. in this case, I think it served to underscore why booting a new server is absolutely the right choice17:16
*** Na3iL has quit IRC17:16
*** vhosakot has joined #openstack-infra17:16
*** oanson has joined #openstack-infra17:16
fungias to sdague's other request, pre-warming the new afs cache before putting it into production, i don't think we've done that before. it seems probably doable, but it would also be very quickly self-correcting anyway17:16
*** sarob has quit IRC17:17
clarkbfungi: should be as easy as pip installing constraints against the ip addr of the new host17:17
pabelangerso, performance1-4 or performance1-8? Sounds like that is up for debate currently17:17
*** sarob has joined #openstack-infra17:17
clarkb4GB is fine with me17:17
fungiseems like performance1-4 should be fine17:17
*** _sarob has joined #openstack-infra17:18
fungiwe've only started hitting 100mbps egress a few weeks ago, so doubling that to 200mbps egress should satisfy us for a while (perhaps indefinitely unless we get a quota bump there)17:18
pabelangerjust ord for now?17:18
mordredI don't even see performance1-4 on the pricing list17:18
*** matthewbodkin has quit IRC17:18
clarkbpabelanger: you can probably check the other cacti graphs to see if other instances exhibit the same capped bw behavior17:19
fungithough since we have so much more quota in ord, it's unlikely we're hitting it elsewhere17:20
clarkbdfw and iad don't come close to 100mbps according to cacti17:20
fungibut i agree it deserves being checked17:20
*** bethwhite_ has quit IRC17:20
cloudnullmordred: performance.* flavors are now general.* i believe17:21
clarkbOVH and internap look fine too17:21
clarkbso yes, I think just ord for now17:21
* cloudnull assuming your talking about rax17:21
*** tqtran has joined #openstack-infra17:21
sdaguejeblair: ok, I believe that it is, though you'd have to instrument pip maybe to figure out17:21
sdagueor add up the size of .pip/cache and do some back of the envelope there17:21
*** sarob has quit IRC17:22
*** krtaylor has joined #openstack-infra17:22
mordredcloudnull: yah17:22
clarkbcloudnull: are there bw limits in osic?17:23
fungipabelanger: i just reviewed all our mirrors, and while some ( exceed the volume in rax-ord, none of the graphs besides that one show an envelope indicative of a bandwidth cap getting hit17:24
pabelangerfungi: great, thanks17:24
pabelangernew server launching now17:24
fungioh, though while it sounds like osic is probably fine, it's not in cacti right now17:24
*** lucasagomes is now known as lucas-dinner17:25
zarofungi: forgot about this one.  identified another duplicate cron job for gerrit git gc
*** ayoung has quit IRC17:25
*** asettle has joined #openstack-infra17:26
*** dtantsur is now known as dtantsur|afk17:26
cloudnullclarkb: do you want / need bw limits setup? we could do qos'ing via neutron or setup "tc" rule if needed.17:26
*** kzaitsev_mb has joined #openstack-infra17:26
cloudnullbut we're not doing anything as of now17:27
clarkbcloudnull: no I don't think we do :) just double checking we don't need to be aware of that like we have to be in rax17:27
anteayazaro: does root own that cron job?17:27
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Shut down gearman client in tests
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Remove testresources
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Make ZK fixture more robust
*** cody-somerville has quit IRC17:28
zaroanteaya: it looks like it to me.17:29
*** ihrachys has quit IRC17:29
*** tqtran has quit IRC17:29
anteayazaro: where are you looking, review-dev?17:29
*** cody-somerville has joined #openstack-infra17:29
fungizaro: to anteaya's point, it said user=>'gerrit2' before, and that needs to be retained when doing ensure=>absent17:29
mordredclarkb: have a sec and feel like +A on 355131 there? (it makes tests not be flaky)17:29
*** rbuzatu has quit IRC17:29
*** vhosakot has quit IRC17:30
clarkbmordred: trying to catch up on email but I can take a look17:30
mordredclarkb: email is the worst17:30
zarofungi: ohh right. will fix that.17:30
anteayazaro: fungi the code says owner gerrit2 on line 37417:30
*** degorenko is now known as _degorenko|afk17:30
*** rbuzatu has joined #openstack-infra17:30
dstufftnew pip cache is awesome, but make sure you have Etags and Cache-Control headers17:30
anteayais that enough for the cron jobs?17:31
dstufftit needs those17:31
fungizaro: however i think we also aren't using that particular cronjob in production as it's wrapped in if (!defined(File[$local_git_dir]))17:31
*** esikachev has joined #openstack-infra17:31
*** adrian_otto1 has joined #openstack-infra17:31
*** shashank_hegde has joined #openstack-infra17:31
bkerofungi: So the gerritbot2 work we discussed last week is a bit troublesome. The way that the gerritbot puppet class is made makes it a singleton-per-host. We can switch it from a class to a defined type, but that's going to be nasty to merge -- each bot would have it's own init script, logging config, channel config, maybe ssh keys.17:32
mordredjeblair, pabelanger, Shrews, DuncanT: the presumptive fix for the ansible async issue has been merged upstream17:32
bkeroAlternatively we might be able to run it on a different host without modifying it.17:32
mordredof course, for us to pick it up, we'll need to go back to running from git instead of a release17:32
anteayamordred: wonderful17:32
zarofungi: local_git_dir is the local replication correct?17:32
*** tqtran has joined #openstack-infra17:32
fungibkero: shouldn't need separate ssh keys, but it will need separate versions of the rest of that yes. i figured a lot of it would have to become erb templates17:33
jeblairmordred: should we look into running it locally like we did before?17:33
zaroaren't we repicating to local_git_dir on review.o.o?17:33
bkerofungi: Yeah, I have a ~200 line patch to do that17:33
bkeroI just don't know how it's ever going to get merged17:33
mordredit has been applied to the stable branch as well - so we could just run off of the upstream stable branch instead of Shrews branch which is based off of tip of devel17:33
mordredjeblair: ^^17:33
fungizaro: yes, so that's saying if the local replication directory is not defined then add this cron resource. but as we have local replication set up that file resource already exists so the cron resource never gets added17:34
DuncanTmordred: thanks for the update17:34
*** vhosakot has joined #openstack-infra17:34
fungizaro: i don't know why it's written that way (looks to me like someone put a } in the wrong place, but there are no comments explaining so maybe it's intentional and i'm just not able to come up with the reasoning)17:34
*** sambetts is now known as sambetts|afk17:35
*** adrian_otto has quit IRC17:35
jeblairpabelanger: can you refresh ?17:36
*** dprince has joined #openstack-infra17:36
jeblairmordred, pabelanger: let's land that, then we can manually install the ansible upstream stable branch and restart launchers to pick up both changes17:36
*** rcernin has joined #openstack-infra17:37
pabelangerjeblair: looking17:38
mordredjeblair: agree17:38
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix link to findbugs minimal example
*** e0ne has joined #openstack-infra17:38
mordredjeblair: it is confirmed to be in stable-2.1 branch17:38
openstackgerritMerged openstack-infra/jenkins-job-builder: Update HTML Publisher plugin to use convert xml
openstackgerritPaul Belanger proposed openstack-infra/zuul: Simplify zuul_console port binding logic
*** edtubill has quit IRC17:40
pabelangerjeblair: updated per your comments17:40
*** ayoung has joined #openstack-infra17:41
clarkbjeblair: mordred for 355131 I wonder if we can tell it to bind on port 0 then get the actual port back sanely (using /proc maybe?)17:42
*** tonytan4ever has joined #openstack-infra17:45
*** senk_ has joined #openstack-infra17:45
zarofungi: just took a closer look and it seems to me that whole section is just duplicating cron.pp in puppet-gerrit.  i think it should be completely removed17:46
*** rbrndt has quit IRC17:46
fungizaro: i agree. i think its vestigial dead code17:46
pabelangerclarkb: jeblair: fungi: mordred: Can we land so we can use non-root permissions for
*** inc0 has joined #openstack-infra17:47
fungipabelanger: was that the only missing piece?17:48
*** raunak has joined #openstack-infra17:48
pabelangerfungi: I believe so17:48
clarkbpabelanger: fungi it will also update the ansible cache iirc. So that needs to be writeable too17:48
zarofungi: i'm surprised that puppet lint didn't pick up that missing }17:48
pabelangerclarkb: ah, yes.17:49
Shrewsmordred: jeblair: so, this is new in stable-2.1 ( but i don't immediately see any issues with it. just FYI17:49
pabelangerOS_CLOUD=openstackci-rax OS_REGION=ORD openstack server list is not returning servers from ORD, but DFW17:50
fungizaro: it's not missing, just several resources after the file resource. in retrospect, i think that was probably added when we moved local mirror handling to the gerrit module and just never cleaned up after17:50
Shrewsit also has the fix for the temp dir race17:50
pabelangerI don't know why atm17:50
mordredpabelanger: on puppetmaster?17:50
pabelangermordred: yes17:50
Shrewsjeblair: i think you found that one ^^^^ (re: tmp dir race)17:50
mordredpabelanger: looking17:50
clarkbI always use the openstack flags not the env vars17:50
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add mirror.regionone.osic-cloud1.o.o to cacti
mordredpabelanger: OS_REGION_NAME17:51
mordrednot OS_REGION17:51
pabelangerlaunch/README is wrong17:51
pabelangermordred: thanks17:51
*** _nadya_ has quit IRC17:51
fungipabelanger: launch/README isn't "wrong" per se. it's just using different envvars in its example shell script than what openstackclient would use17:53
jeblairclarkb: i'm not sure -- i didn't think to ask proc.  however, i'm just about convinced that with the zookeeper chroot option, we can drop the per-test fixture and just expect a locally running zk...17:54
fungiit's not passing those to osc17:54
*** vhosakot has quit IRC17:54
pabelangerfungi: Ah, right. That explains it17:55
beaglespabelanger, got some weird stuff happening in some puppet-neutron CI for mitaka where a bunch of ubuntu jobs are failing (see
beaglespabelanger, who should I bug about that? :)17:56
*** _nadya_ has joined #openstack-infra17:56
*** _nadya_ has quit IRC17:57
*** kzaitsev_mb has quit IRC17:58
*** Sukhdev has quit IRC17:58
anteayabeagles: EmilienM is the ptl for puppet-openstacklib:
anteayahe might be able to help17:59
EmilienMdon't bug me at every bug in puppet modules :)17:59
beaglesanteaya, actually thanks for the correction - that's puppet-openstacklib17:59
fungibeagles: those look like they all hit a one-hour timeout running in osic, which i believe is related to the ipv6 dns discussion which was going on in here earlier17:59
beaglesanteaya, I was sent in this direction17:59
pabelangerasync task produced unparseable results17:59
beaglesfungi, interesting17:59
beaglespabelanger, yup18:00
pabelangerlooks like ansible is failing18:00
pabelangerI think we are working on patching zuul18:00
mordredpabelanger: we just landed a patch upstream for that18:00
pabelangermordred: ++18:00
sdaguejeblair: as a data point, with a primed cache on my NUC the pip_install time is 72s. So even in our best cases, my guess is that 2/3rds of the pip install time is spent on network18:00
mordredand will roll out the fix to infra at the same time as your other patch18:00
fungiwere the job timeouts in osic directly related to the ansible json parsing errors?18:00
sdaguebasically we've got a fixed cost of ~ 1 minute to install for dsvm runs, and 2 - 6 minutes of network time18:01
pabelangermordred: great18:01
pabelangerbeagles: sounds like fix is in progress18:01
beaglespabelanger, thanks man!18:01
*** inc0 has quit IRC18:01
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags
openstackgerritKhai Do proposed openstack-infra/system-config: Remove duplicate code to setup gerrit local replication
openstackgerritBen Kero proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots
bkerogreghaynes: ^18:08
bkerofungi: ^18:08
bkeroThat's also going to need a transition plan :/18:08
jeblairbkero: quick thought experiment -- how hard to make gerritbot support 2 connections?18:09
openstackgerritHenry Gessau proposed openstack-infra/project-config: Use python-db-jobs for networking-sfc
bkerojeblair: the gerritbot project itself? I have no idea, never looked at the source18:10
*** elo has joined #openstack-infra18:10
*** ihrachys has joined #openstack-infra18:13
bkerojeblair: You'd have to do some multiprocess/threaded python, since these just run/spin by themselves:
*** e0ne has quit IRC18:15
*** nwkarsten has joined #openstack-infra18:15
jeblairbkero: yeah, i'd imagine it would just end up looking a lot like running 2 bots inside of one process.  running 2 processes is probably the better way, just wanted to throw that out there in case it looked too gnarley18:15
*** vhosakot has joined #openstack-infra18:15
bkerojeblair: It's going to look gnarly either way. The easiest way would be to run on a different host.18:16
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Allow using lockfile per jenkins master
bkerobut I'm sure that's also fraught with inheritance nightmares18:16
*** e0ne has joined #openstack-infra18:17
*** apetrich has quit IRC18:18
jeblairit also has other drawbacks :)18:18
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Output additional info when exceptions occur
*** Apoorva_ has joined #openstack-infra18:20
*** bknudson has joined #openstack-infra18:21
*** inc0 has joined #openstack-infra18:21
greghaynesbkero: nice18:21
*** xyang1 has quit IRC18:22
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Refactor base test classes inheritance for reuse
sdaguecould I get some reviews on to increase timeouts on novaclient jobs?18:24
*** Apoorva has quit IRC18:24
*** vhosakot_ has joined #openstack-infra18:24
openstackgerritBen Kero proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Improve logger output for expanding templates
*** vhosakot has quit IRC18:25
*** xyang1 has joined #openstack-infra18:26
*** electrofelix has quit IRC18:27
beaglespabelanger, mordred: what should I be watching for a heads up that the expected fixes are in?18:27
*** senk_ has quit IRC18:27
mordredbeagles: we'll just ping you18:28
beaglesmordred, thanks!18:28
*** apetrich has joined #openstack-infra18:29
*** acoles is now known as acoles_18:29
*** csomerville has joined #openstack-infra18:30
*** rbrndt has joined #openstack-infra18:30
*** cody-somerville has quit IRC18:33
*** vhosakot_ has quit IRC18:33
*** ayoung has quit IRC18:37
clarkbsdague: is that related to the pip bw thing? where are we spending the other 40 minutes?18:37
rajinirGate seems to be broken. No hosts found to map to cell, exiting. Any ETA?18:37
sdagueclarkb: well, there is 7 minutes not in any log files before setup workspace, no idea why18:38
sdagueclarkb: but regardless, we've been pushing up towards our time alotment18:38
clarkbrajinir: is there more context for that? like a log file?18:38
clarkbsdague: ya I am ok with bumping it just want to amke sure we don't focus on 4 mintues of extra pip time when we have 40 minutes of setup elsewhere that may actually be the problem18:39
sdagueand we need to land code before freeze otherwise basically the nova cli will stop working18:39
sdagueclarkb: well, that also impacts dpkg installs18:39
pabelangerrajinir: where did you see that?18:39
clarkbsdague: those are cached though18:39
*** tqtran has quit IRC18:39
sdagueclarkb: ok, the biggest mystery to me right now is the missing 7 minutes here -
sdaguebecause the setupworkspace first log entry is at 27 and change18:40
*** asettle has quit IRC18:41
*** jimbaker has quit IRC18:41
*** rcernin has quit IRC18:42
rajinirpabelanger: I was watching the gate on my thirdparty CI18:42
clarkblooks like we lost time due to ntp18:43
* clarkb grumps that ntp isn't more sane18:43
jeblairclarkb: how can you tell?18:44
clarkbjeblair: due to that line I am ssuming that the logs don't jump forward in time due to a time update but instaed actually took that long18:45
*** jimbaker has joined #openstack-infra18:46
jeblairclarkb: so you're thinking because it said it failed to sync earlier, it jumped later?18:46
*** jimbaker has quit IRC18:46
*** jimbaker has joined #openstack-infra18:46
pabelangerrajinir: I cannot comment on that, but the gate is not broken. As other projects are passing properly18:46
*** Apoorva_ has quit IRC18:46
clarkbjeblair: ya thats one possibility18:46
*** karthik__ has joined #openstack-infra18:46
*** Apoorva has joined #openstack-infra18:47
*** vhosakot has joined #openstack-infra18:47
*** vhosakot has quit IRC18:47
jeblairwhat's the 10 minutes before ntp-wait?18:47
clarkbjeblair: I think tahts the 10 minutes of ntp-wait waiting18:47
rajinirpabelanger>: On the ironic channel, a couple of folks are also seeing it18:47
jeblairclarkb: oh, all output at the ned18:47
*** vhosakot has joined #openstack-infra18:48
*** spzala has quit IRC18:48
*** spzala has joined #openstack-infra18:48
clarkbrajinir: pabelanger I don't see an error in that paste either? looks like just debug logs?18:48
*** tqtran has joined #openstack-infra18:49
jeblairclarkb, sdague: ianw and pabelanger have been looking into ntp issues18:49
jeblairsdague: so if we're spending 10 real minutes waiting for ntp to sync, failing, and then losing 7 fake minutes when it eventually comes around, that's going to have an impact.  :)18:51
clarkbrajinir: it looks like it is trying to configure cells but the config doesn't exist. You might just be able to run without cells?18:51
*** spzala has quit IRC18:51
*** hockeynut has quit IRC18:51
*** spzala has joined #openstack-infra18:52
*** javeriak has quit IRC18:52
*** bstinson has quit IRC18:53
mordredclarkb: while we're looking at timing things - this one might be related to bandwidth caps and stuff ...18:53
mordredclarkb: but:
*** bstinson has joined #openstack-infra18:54
*** javeriak has joined #openstack-infra18:54
mordredclarkb: if you scan the log, it looks like every time that tries to touch git.o.o it takes 4 minutes18:54
clarkbI doubt that is related to ntp if it happens more than once. Probably something related to the git mirrors and/or networking and/or git18:55
mordredit's always a remote update  - and it's always roughly 4 minutes18:55
sdagueclarkb: I don't think it's ntp18:56
fungibandwidth utilization on git.o.o seems to be nearing/reaching 400mbps egress traffic at times
sdaguesyslog has regular logging through the whole window18:56
sdagueansible is doing something that's not logging18:56
sdagueoh, it's the filesystem rebuilds18:57
sdaguedo we still need to do that on nodes?18:57
fungiwhat's the bw cap for rax's 30gb performance flavor?18:57
*** pt_15 has joined #openstack-infra18:57
*** rbuzatu has quit IRC18:57
*** itisha has joined #openstack-infra18:58
clarkbsdague: we do need swap, and the / is tiny iirc so likely yes we need to make /opt large there18:58
sdagueclarkb: ok, that takes 7 minutes18:58
rajinirclarb: This could be something to with ironic plugin. Discussion happening in ironic channel to revert. thanks18:58
jeblairperhaps it's copying the git repos to the new device that is slow?18:59
openstackgerritEddie Ramirez proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin)
*** _nadya_ has joined #openstack-infra18:59
sdaguejeblair: no, this is the mkfs18:59
clarkbjeblair: looking at sdague's log links it is the mkfs18:59
clarkbsince it doesn't mount until 7 minutes later18:59
mordredI concur with sdague18:59
sdagueand there are no other logs in this window18:59
clarkbit is possible we want to not utilize the full disk there and make a smaller but large enough fs18:59
*** tkelsey has quit IRC18:59
*** tqtran has quit IRC18:59
*** e0ne has quit IRC18:59
jeblairthat is a very long mkfs18:59
sdagueI agree, that seems super long18:59
fungicould try -E lazy_itable_init ?19:00
jeblairwhere's the mkfs command?19:00
jeblairfungi: rxtx_factor is 2500.019:00
clarkboh wait it mounts twice19:00
clarkbthe first mount is fast so I don't think it is the mkfs19:00
openstackgerritEddie Ramirez proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin)
mordredactually - it seems to be the mount19:00
mordredyah - what clarkb said19:00
clarkbjeblair: I think you are right, it mounts first in other location, copies, then chagnes mount19:00
clarkbthe copy being the slow bit?19:00
sdagueclarkb: oh, yeh, could be19:01
fungijeblair: okay, so we're nowhere near the bw cap there i guess19:01
mordredhow horrible would it be to do this dance in the ready-script rather than in d-g?19:01
jeblairfungi: i forget the rax math needed to get to 'upstream bandwidth' though19:01
jeblairmordred: not every job needs it19:02
clarkbjeblair: fungi its divide that number by 2 and thats your mbps iirc19:02
mordredjeblair: bother19:02
clarkbso 1250mbps for public interface19:02
jeblairclarkb: that means we have 200mbit for our 2gb mirror?19:02
*** _sarob has quit IRC19:02
sdagueclarkb: yeh, with the 2 mounts I agree19:02
mordredfungi, clarkb: the "disk" optimized flavors at rackspace have a much higher bandwidth number19:02
sdaguethis is the find / copy19:02
*** sarob has joined #openstack-infra19:03
*** psachin has quit IRC19:03
sdague - is the line that seems to take ~7 minutes19:03
clarkbjeblair: hrm ya it should be 200mbps but thats not what we are seeing there. Weird.19:03
mordredscuse me - "I/O Optimized"19:03
fungiso i'm guessing the umount is flushing the write cache19:03
fungihow about we mv the contents of /opt somewhere else on the rootfs, mount the ephemeral disk at /opt, then mv the files into it?19:04
mordredoh - but nevermind- those have huge amounts of cpu and are way more pricey - just a bigger general would meet expanded needs much simpler19:04
fungithen we don't umount and mount it again19:04
jeblairmordred: not always -- io1-30==performance2-30==2500.019:04
*** sarob has quit IRC19:04
mordredjeblair: yah - sorry, I was looking at the first table entry and missing the fact that it was a 15G instance19:05
clarkbfungi: would be easy enough to push a patch that does that and compare times19:05
mordredthat seems like a mildly strange definitoin of the smallest "I/O Optmized" flavor19:05
clarkbalso need to figure out why ntp-wait is so cranky19:05
mordredclarkb: sync with ianw/pabelanger on that19:05
sdaguefungi: ok, while that is going on, anyone want to +A - so we can make forward progress with novaclient? :)19:05
mordredthere was a bunch of stuff on that topic towards the end of last week19:05
jeblairmordred: i already mentioned that :)19:06
fungiclarkb: same-fs mv should be atomic and basically instantaneous, so i expect it's a performance improvement to not umount and mount again regardless... just a question of how much19:06
mordredjeblair: yup - it's just been chatty so didn't want clarkb to miss it :)19:06
jeblairclarkb, mordred: some more background reading: bug 1361382 in ntp "ntp-wait hangs after boot for a long time, unless ntpd is restarted" [Unspecified,Closed: notabug] - Assigned to mlichvar19:06
jeblairsdague: ^19:06
*** fifieldt has quit IRC19:07
*** edtubill has joined #openstack-infra19:08
*** asselin_ has joined #openstack-infra19:08
jeblairsdague, clarkb, fungi: is it the case that we need to move the data off of / in order to free up space there for all the installs?19:08
*** sarob has joined #openstack-infra19:09
clarkbjeblair: yes I think so19:09
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend
fungijeblair: right, that's why we mv rather than cp19:09
clarkbjeblair: VMs and mysql and friends all need disk19:09
sdagueright, but wasn't a bunch of that for hp pathelogical flavors?19:10
*** Swami has joined #openstack-infra19:10
clarkbreading this seems like we could use ntpd -qg ?19:10
* fungi wishes's ntpd worked like openntpd at startup19:10
clarkbsdague: rax and hp were basicaly the same19:10
jeblairmordred: when i clone python-aodhclient locally from our git mirrors, it takes 1 second; so i don't know what's taking 4 minutes in that job you linked.19:10
clarkbsdague: tiny / huge ephemeral disk19:10
*** asselin has quit IRC19:10
mordredjeblair: me either - it was the consistency of it across multiple invocations that had me the most concerned19:10
fungiclarkb: i'd have to reread, but it sounds like ntpd -qg can still take 10+ minutes to stabilize19:11
clarkbfungi: -g says "This option allows the time to be set to any value without restriction"19:11
sdagueactually, I'm super confused, that log line is here - ?19:11
sdagueis ansible just buffering this whole thing and throwing away all the useful timestamp info?19:12
*** tqtran has joined #openstack-infra19:12
clarkbsdague: no I think we do that timestamping outside of ansible19:13
clarkbsdague: with the tooling pulled out of devstack19:13
sdaguewell, those mount timestamps don't line up with the ones in syslog19:13
sdagueand would state that the mv took 0.003s19:13
openstackgerritgreghaynes proposed openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs
jeblairi have to run to lunch now. bbl19:14
clarkbit wouldn't surprise me if it is a buffering issue in the timestamping, just not related to ansible I don't think19:14
*** elo has quit IRC19:14
openstackgerritgreghaynes proposed openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs
openstackgerritMerged openstack-infra/project-config: increase novaclient functional timeout.
*** asettle has joined #openstack-infra19:16
*** devkulkarni has joined #openstack-infra19:17
*** devkulkarni1 has quit IRC19:17
sdagueok, well, for right now, we need to get these novaclient bits sorted. So I'm going to switch gears back over to that.19:18
clarkbfungi: looks like we could also just start ntpd with the -g flag19:19
*** fifieldt has joined #openstack-infra19:19
*** asettle has quit IRC19:19
fungiclarkb: yep, i'm trying to see if i can figure out why that's not configurable for the initscript/systemd unit19:19
fungibecause if that were generally useful, you'd think it would be a startup option19:20
clarkbntpd -g for the normal daemon should work if we start within +/-68 years of current time from my reading of docs19:21
clarkb1970 is only 46 years ago so we should be fine even if we start at the epoch19:21
*** Sukhdev has joined #openstack-infra19:25
*** asselin__ has joined #openstack-infra19:25
clarkbfungi: my tumbleweed system uses -g, but it also has a force set option that will run sntp first19:25
clarkbso -g may not be sufficient?19:25
clarkbfungi: trusty has it set to -g in /etc/default ntp too19:27
clarkbbut trusty has no sntp option19:27
*** asselin_ has quit IRC19:28
*** asettle has joined #openstack-infra19:29
*** signed8bit is now known as signed8bit_Zzz19:30
*** asettle has quit IRC19:31
*** adrian_otto1 has quit IRC19:31
*** sean-k-mooney has joined #openstack-infra19:33
*** oanson has quit IRC19:34
fungiclarkb: indeed, my debian systems have NTPD_OPTS='-g' too19:36
clarkbfungi: I think we could do a quick survey of our images and see if they use -g by default and if they do try removing any ntp machinery from our jobs?19:38
fungi"When the initial offset is larger than 0.128s, ntpd will step the clock and then it will wait for at least 900 seconds (in default configuration) before it reports it's in the synchronized state."19:38
clarkbthe ntp machinery in our jobs was there to calculate job timeouts, but since ntpd can't skew things drastically those timeouts shouldnb't be terribly affected adn the -g should get us fairly close19:38
*** signed8bit_Zzz is now known as signed8bit19:38
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells v2 setup bug 1613417
openstackbug 1613417 in devstack "gate-tempest-dsvm-cells broken with cell v2 setup: "No hosts found to map to cell, exiting."" [Undecided,In progress]
clarkbI think we would have to worry about scheduling jobs on insatnces fast enough that -g isn't done doing its thing but I don't expect it to take a ton of time since its supposed to ignore all those pesky limits19:40
*** coreyob has quit IRC19:40
fungii'm hunting for code or documentation to back up the assertion in that bug report19:41
*** jimbaker has quit IRC19:41
*** tonytan4ever has quit IRC19:41
fungithe implication is that -g will avoid ntpd freaking out and exiting if the initial offset is significant, but won't actually cause it to synchronize to that new time any faster19:43
*** amitgandhinz has quit IRC19:43
*** amitgandhinz has joined #openstack-infra19:44
clarkbwe could stop ntpd, run sntp, start ntpd19:45
clarkbwhich is similar to how the old ntpdate stuff worked19:45
*** jimbaker has joined #openstack-infra19:45
*** jimbaker has quit IRC19:45
*** jimbaker has joined #openstack-infra19:45
*** kzaitsev_mb has joined #openstack-infra19:46
fungi"Under conditions of extreme network congestion, the roundtrip delay jitter can exceed three seconds and the synchronization distance, which is equal to one-half the roundtrip delay plus error budget terms, can become very large. The ntpd algorithms discard sample offsets exceeding 128 ms, unless the interval during which no sample offset is less than 128 ms exceeds 900s. The first sample after that,19:46
fungino matter what the offset, steps the clock to the indicated time."19:46
fungiso i think that means that even at start, if the local time is off by more than 128ms, ntpd won't actually synchronize the clock for 900s19:47
clarkbwhich is certainly long enough to race job starts19:47
fungiand -g simply keeps ntpd from freaking out at startup if that >128ms skew is large enough to be >1000s19:48
*** ayoung has joined #openstack-infra19:48
*** yamahata has quit IRC19:49
*** senk_ has joined #openstack-infra19:49
*** yamahata has joined #openstack-infra19:49
fungiso, i agree, this seems to be the reason for suggesting sntp19:50
*** senk_ has quit IRC19:50
fungiand centos 7 still has an "ntpdate" service which ntpd depends on for taking acre of that, but in more recent fedora releases they seem to have replaced it with an sntp "service" to do basically the same19:52
*** rbuzatu has joined #openstack-infra19:52
clarkbfungi: are they enabled by default or opt in?19:54
clarkbon suse I Have to set some flag to force sntp19:54
pabelangerfungi: clarkb: jeblair: Took longer then expected, but new mirror server in ord is online:
mordredpabelanger: woot!19:55
pabelangerfungi: clarkb: jeblair: going to enroll into ansible and update DNS19:55
clarkbfungi: I am thinking the simplest thing is to undo ntp-wait and replace ntpdate with sntp19:55
clarkbfungi: in d-g19:55
clarkbfungi: or possibly make sntp part of the ready script19:56
clarkbso that all jobs have sane ntp19:56
clarkbpabelanger: great thank you for getting that up19:56
*** rbuzatu has quit IRC19:56
fungiclarkb: it got discussed in last week's meeting. maybe skim the minutes from here to the end of the topic
fungiclarkb: basically ntpd is no longer the default time sync solution on rh-based platforms, so we likely want to go with each distro's default implementations19:58
*** tonytan4ever has joined #openstack-infra19:58
jeblairfungi: the new info for me is that apparently ntp-wait is hanging on ubuntu test nodes19:58
mordredsame here19:59
fungiwhich to me means we could add an sntp call in debian/ubuntu, but switch centos/fedora to chrony19:59
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Fix template filename
fungiand probably just drop ntp-wait from d-g altogether?19:59
clarkbfungi: a simple which sntp || which chrony type switch would be fine19:59
*** Goneri has joined #openstack-infra20:00
fungibasically rely on time sync to become a forced part of node bootup, and let jobs just assume that is a solved problem20:00
pabelangerdns updated, will take 60mins20:00
clarkbfungi: ya, we might also want to talk to debian and ubuntu about supporting a forced thing out of the box20:01
clarkbsince from what I can see that doesn't exist currently (but I may be missing some pacakge that adds it)20:01
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Add gate-novaclient-dsvm-functional-neutron-nv job
fungiback to my earlier wistfulness of having something akin to openntpd's -s option20:02
*** oomichi_ has joined #openstack-infra20:02
clarkbhrm ubuntu says they have a thing called timedatectl20:02
fungi"-s: Try to set the time immediately at startup, as opposed to slowly adjusting the clock. ntpd will stay in the foreground for up to 15 seconds waiting for one of the configured NTP servers to reply."20:02
clarkbso now we have ntpdate, sntp, chrony, and timedatectl20:03
*** sigmavirus is now known as sigmavirus|away20:03
*** oomichi_ is now known as oomichi20:03
fungiopenntpd is packaged on debian/ubuntu as well if you're making a list ;)20:03
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Raised max instance in the OSIC
clarkbbut timedatectl won't run if you ahve ntp installed20:03
mordredof course it won't20:03
clarkbI wonder if we just removed our ntp setup completely if things would just work (tm)20:03
mordredwhy would you ever make a utility that would run if you asked it to run20:04
cloudnull^ idk if infra core folks want to let my max-instance change in quite yet but i figured i'd put it up.20:04
fungimordred: clearly they think they've put a safety on their foot-cannon20:04
*** coreyob has joined #openstack-infra20:05
fungicloudnull: you won't find me disagreeing20:05
mordredfungi: I'm pretty sure that the piece of paper tape across the opening on the front of the cannon that says "danger" will keep me from shooting myself20:05
* clarkb noms on more tasty VMs20:05
anteayacloudnull: what might we be waiting for?20:05
mordredcloudnull: we like your max-instance change20:06
*** Apoorva has quit IRC20:06
pabelangershould we land IPv6 dns first?20:06
cloudnullIDK if there was need to wait on DNS things or what now20:06
jeblairwe might wait on the zuul telnet fix, or dns20:06
anteayathe crowd hath spoken20:06
clarkboh I approved it, I can remove the approval20:06
jeblairi don't know that we should, just saying those are the things to consider20:06
fungicloudnull: what did the dns solution end up being? are our queries to ipv4 resolver addresses going through a pat?20:06
cloudnullIDK if my cloud will cry, but i have a name to live up to.20:06
anteayaha ha ha20:06
clarkbfungi: they are NAT'd by the neutron router20:07
anteayacloudnull: and we will help you get there20:07
jlvillalFor 'gertty'. When looking at a diff. Is there a search the diff feature?20:07
cloudnullfungi: what clarkb said20:07
fungijlvillal: ctrl-s20:07
pabelangercloudnull: we are seeing some failures to launch in osic-cloud1: but I was going to wait until we landed dns patch to start looking why20:07
mordredcloudnull: we can always increase the level of pain we inflict on your cloud any time you feel like you need to prove your skills as a leet operator20:08
fungijlvillal: at least by default, but as with any keybindings in gertty you can set that to something else20:08
*** nmagnezi has joined #openstack-infra20:08
* cloudnull enjoys pain20:08
jlvillalfungi: Thanks. Strange a few moments ago on some diff it was showing it searching for a patch. But now it works. Odd.20:08
*** _sarob has joined #openstack-infra20:08
fungii expect dns, while needing to get solved, may be fine through pat for now. zuul-launcher ipv6 console streaming support on the other hand could be something we want to solve quickly20:08
pabelangerfungi: 355570 was my attempt at fixing dns20:09
mordredpabelanger: btw20:09
fungiis there a zuul console patch for ipv6 url support?20:09
cloudnullpabelanger: I've been monitoring / watching the logs and such. IDK what is causing the "Error Node Launch Attempts" as neutron || nova aren't stacking or really throwing any errors.20:09
cloudnullbut i'm actively trying to hunt things down.20:09
fungiaha, thanks20:10
jlvillalfungi: That search is a bit odd. It doesn't move the page down if the search result is outside the view.20:10
cloudnullit may simply be an issue with neutron programing th einterface in time. but i've not proven that at this point20:10
mordredcloudnull: oh - also, I don't know if you saw, but one of the things I was considering a problem with ipv6/shade/nodepool on osic is now at least understood ... but i don't think it's generally fixable at the moment20:10
fungijlvillal: keep hitting ctrl-s to advance20:10
jlvillalfungi: Ah sweet :) Thanks.20:11
jeblairjlvillal: ah, yeah, it doesn't look like it jumps to the initial match if outside the view.  however, repeated ctrl-s will get it there20:11
*** sarob has quit IRC20:11
cloudnullmordred: i had not seen that. something we might be able to help out with ?20:11
jlvillaljeblair: Thanks20:11
pabelangermordred: Great, +1 since I haven't done much shade yet20:11
jeblairprobably it should jump to the first one20:11
mordredcloudnull: the basic jist is that the single network with a public ipv6 and a private ipv4 is confusing to shade's concept of inferring what you want to do with your networks ... but it's not preventing us from launching nodes or using them so I'm not going to fix it until we find a way in which it breaks and can imagine a general solution20:11
jlvillaljeblair: I would vote for that behavior :)20:11
mordredcloudnull: I think it's just a deficiency in the neutron data model, and if we try to work around it TOO much in this case I think it'll lead to more not less confusion20:12
*** vhosakot has quit IRC20:12
pabelangercloudnull: So, It think we are not resolving DNS in our nodepool ready-script, we do host, and if that fails we delete the server and launch again20:12
clarkbjeblair: mordred is there a reason that that zuul patch hasn't been approved yet? can I go ahead and approve it?20:12
*** amitgandhinz has quit IRC20:12
mordredclarkb: nope. just waiting on a second +220:12
jeblairclarkb: no reason i know of.  i think pabelanger local-tested it.20:12
*** amitgandhinz has joined #openstack-infra20:12
*** _sarob has quit IRC20:13
pabelangerjeblair: clarkb: Yes, I tested it locally with a simple python app20:13
clarkbjeblair: mordred pabelanger though thinking about it, does that work if for some reaosn a host doesn't have a working ipv6 stack? do we care about such hosts?20:13
cloudnullmordred: :'( at least things are still working20:13
mordredclarkb: I do not personally care about such hosts at the moment20:13
fungiit does of course mean that people without ipv6 connectivity can't get to some of the log streams, but... join us in the new era. hurricane electric tunnels for everyone!20:13
jeblairclarkb: fungi assured us that should be fine for any linux post 1997 or something.20:13
*** rbuzatu has joined #openstack-infra20:14
pabelangerfungi: Yes! my lack of ipv6 at home is becoming a problem now20:14
clarkbfungi: thats me now that I changed ISPs20:14
clarkbI should fix that20:14
clarkbpabelanger: one trick is to ssh tunnel20:14
* sc68cal wishes FiOS would get their shit together20:14
mordredsc68cal: ++20:14
clarkbyou can v6 to v4 or v4 to v6 pretty easily with ssh20:14
clarkbsc68cal: ya thats who I changed to20:14
mordredsc68cal: oh - that reminds me - I need to call frontier to see if their Gig service is available for me20:14
pabelangerclarkb: cool, I haven't looked how to yet20:15
jeblairclarkb: that is, it should listen on v4 and v6 for dual stack hosts, which is all of them.  of course some of our nodes now are not *routable* over v4.20:15
sc68calmordred: lol humblebrag20:15
jeblairfungi: right^20:15
clarkbjeblair: yup20:15
*** sarob has joined #openstack-infra20:15
*** Goneri has quit IRC20:15
clarkbjeblair: and even if you don't have a global ipv6 addr you should have a link local addr and loopback to listen on for v620:15
anteayasc68cal: it is nice to see you, have a frowny face20:15
* sc68cal thinks he needs a REST API to POST things he needs downloaded, and ship hard drives to mordred :)20:15
fungiclarkb: pabelanger: my home ipv6 is via an he tunnel from my firewall. even have a /48 and reverse dns delegated for it20:15
clarkbjeblair: so not a problem on the bind side I don't think unless running ancient linux as fungi said20:15
clarkbfungi: ya I just know that after having native v6 with comcast very little stuff functions properly with it. Thought that may be related to the giant bitbuckets in seattel and denver in comcast land and HE is happier20:16
clarkbI have approved the zuul change20:16
clarkbfungi: I have had to disable v6 in order to get working internets more than once20:17
*** Apoorva has joined #openstack-infra20:17
fungiclarkb: the ancient behavior isn't so much lack of linklocal addressing, as older system-wide "v6only" socket behavior (which you can still set via sysctl or explicit socketopts)20:17
openstackgerritMerged openstack-infra/nodepool: Make ZK fixture more robust
clarkbfungi: I want to say ubuntu of the 2005 ish era didn't have v6 enabled at all? but thats ancient so meh20:18
fungibasically, binding a socket on :: used to only listen on all ipv6 addresses, not any ipv4 addresses20:18
mordredrobust test fixtures are great20:18
*** rbuzatu has quit IRC20:18
jeblairi'm a fan20:19
jeblairmordred, pabelanger, Shrews: i'll work on getting ansible manually installed on launchers20:19
jeblairsee if i can find my old playbooks for that20:19
*** javeriak has quit IRC20:19
*** inc0 has quit IRC20:19
Shrewsalrighty then20:20
mordredjeblair: cool20:20
mordredjeblair, Shrews: next time you're bored: ... I added tests and a release note even20:20
Shrewsmordred: i could of swore i reviewed that already. perhaps i forgot to vote20:21
fungiclarkb: controllable through the IPV6_V6ONLY sockopt (since Linux 2.4.21 and 2.6) and /proc/sys/net/ipv6/bindv6only system default20:21
fungijeblair: ^20:21
*** valderrv_ has quit IRC20:21
*** karthik__ has quit IRC20:21
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
fungior net.ipv6.bindv6only via sysctl20:22
openstackgerritMerged openstack-infra/zuul: Simplify zuul_console port binding logic
fungiit was somewhat hotly debated on debian-devel ~7 years ago
fungiso that's what i mean by "relatively modern"20:23
mordredfungi: so as a cya, we could set net.ipv6.bindv6only to false with sysctl20:23
mordredmaybe in the zuul puppet20:23
fungii think we add that if someone complains that their 7-year-old server isn't running our experimental zuul-launcher correctly?20:24
fungithe one we haven't documented much nor encouraged others to switch to?20:24
jeblairit's on the test nodes too20:24
jeblairso 'testing on a 7 year old platform'20:24
fungiahh, yeah. i'll check centos 720:24
pabelangermordred: fungi: if you have time today, would not object to a review of 354818.  Start mirroring source packages for debian / ubuntu for zigo20:24
* jeblair hopes it's not called centos '7' because it's 7 years old20:25
funginet.ipv6.bindv6only = 0 already on centos 720:25
sean-k-mooneymmedvede: are you about?20:25
fungigood thing we're not still on centos 10!20:25
jeblairif so, i'm not sure about 'upgrading'20:25
fungialso net.ipv6.bindv6only = 0 on ubuntu precise20:26
*** valderrv has joined #openstack-infra20:26
fungiso we should be fine20:26
mmedvedesean-k-mooney: I am here20:26
clarkbfungi: ianw pabelanger ok I think I have caught up on the ntp meeting discussion. From my reading of that and ubuntu docs I think we might be ok to completely drop ntp packages and services from our test images20:27
sean-k-mooneymmedvede: i tried to set up my own instance of ciwatch but the ci_id are always null so it does not render correctly20:27
*** kgiusti has left #openstack-infra20:27
sean-k-mooneymmedvede: is the most uptoday code in the gitub?20:27
clarkbfungi: ianw pabelanger we just have to make sure that the distro defaults of chrony and timedatectl end up in place20:27
*** tonytan4ever has quit IRC20:27
clarkbfungi: ianw pabelanger I can try booting some ubuntu-minimal and fedora-minimal images once I am otherwise caught up on post vacation things to see if those just work20:28
cloudnullalso, just a shout out: thanks everyone for helping the OSIC get to gating on IPv6! its really quite awesome to see all of this getting done and rolling into production.20:28
cloudnullat the next ops-meetup/summit: beers on me :)20:28
clarkbcloudnull: its pretty neat on our end too (we have long said ipv6 should mostly work and it looks like it does \o/)20:28
clarkbcloudnull: thank you !20:28
*** asselin__ has quit IRC20:28
*** asselin has joined #openstack-infra20:29
*** xyang1 has quit IRC20:29
mmedvedesean-k-mooney: yes. I have a script I can share that should setup ciwatch for you (using puppet-ciwatch module)20:29
*** xyang1 has joined #openstack-infra20:29
sean-k-mooneymmedvede: well i have it running in a docker container
sean-k-mooneybut it looks like i missed something20:30
*** devkulkarni has quit IRC20:31
clarkboh heh it looks like timedatectl may be a systemd realted thing that configures chronyd?20:31
*** ociuhandu has quit IRC20:31
clarkbthis isn't convoluted and confusing at all20:31
clarkband may not be part of precise but is available on trusty looks like20:31
*** sdake has quit IRC20:32
*** asselin_ has joined #openstack-infra20:32
mtreinishclarkb: yeah that's a systemd thing20:32
sean-k-mooneymmedvede: if you can point me towrad the script though i would be happy to compare and see what i missed20:32
mtreinishclarkb: or at least I think it is, because that's what I've had to use for time settings on my arch boxes for a while20:33
clarkbmtreinish: on ubuntu the systemd/systemd-services packages provide it20:33
*** _nadya_ has quit IRC20:33
jeblair#status log Installed ansible stable-2.1 branch on zuul launchers to pick up
openstackstatusjeblair: finished logging20:34
*** asselin has quit IRC20:35
mmedvedesean-k-mooney: it is pretty much just using puppet module to deploy it
*** nmagnezi has quit IRC20:37
sean-k-mooneymmedvede: thanks the only real difference i can see between the  puppet deployment and my manually deployment is i used the default sqlite conenction string instead of useing mysql20:38
sean-k-mooneymmedvede: ill give the puppet aproch a shot though and see if that works form me. thanks for the help20:39
pabelangereep, 200 nodes just got deleted by nodepool20:39
pabelangerchecking why now20:39
mmedvedesean-k-mooney: ok. I'll try deploying from scratch myself when I get some free time. I'll let you know if I see the same problem you are seeing20:40
fungipabelanger: gate reset?20:40
pabelangerfungi: I think because ansible was reinstalled20:41
pabelanger is a new failure20:41
pabelangerlets see if it happens again20:41
fungilooks like devstack-gate cells jobs are probably hitting the same problem rajinir was seeing in a third-party ci20:41
jeblairpabelanger: yes, i just reinstalled ansible20:42
*** esikachev has quit IRC20:43
clarkbfungi: yup they pushed a fix20:43
jeblairpabelanger: probably should have incorporated it into a graceful shutdown/reinstall/start playbook20:43
pabelangerjeblair: Ya, failures line up with that.  replacement nodes back online20:43
pabelangerjeblair: np20:43
pabelangerjeblair: I think you said you have a potential fix for inplace upgrades for ansible a while back?20:44
jeblairpabelanger: in-place upgrades of zuul, and that's there20:44
*** adrian_otto has joined #openstack-infra20:44
jeblairpabelanger: not ansible though.  we need to stop/upgrade/start for ansible20:44
jeblairbut that doesn't happen often20:44
jeblairi hope20:44
*** tonytan4ever has joined #openstack-infra20:45
pabelangerStarting to see traffic on the new server20:47
jeblairon a (perhaps related) note, i enqueued 355628 into the gate20:47
pabelangercacti.o.o confirms too20:47
clarkbfungi: ianw pabelanger looks like newer ubuntu may run by default20:48
clarkbunfortuantelky the docs for that don't say anything about how it handles skew20:48
anteayajeblair: so DuncanT should be able to recheck that patch?20:48
anteayaand beagles too?20:48
jeblairanteaya: yep20:49
anteayathank you20:49
anteayathank you mordred20:49
jeblair#status log gracefully restarting all zuul-launchers20:49
openstackstatusjeblair: finished logging20:50
openstackgerritMerged openstack-infra/project-config: Raised max instance in the OSIC
*** gouthamr has quit IRC20:51
jeblairin a few hours, we should have v6 telnet links working there20:51
clarkbjeblair: does that depend on new images in osic?20:51
clarkbI can babysit that if you think it will help20:51
jeblairclarkb: no, it's zuul-console component copied over by ansible from zuul-launcher20:51
jeblairclarkb: the few hours is the zuul-launcher global graceful restart i just kicked off20:52
pabelangerfungi: already up to 140 Mbps
jeblair(we *can* hard-restart the launchers, but it would burn more nodes)20:52
fungipabelanger: that's a great indication of how terrible things were before, if we were wanting 40% more than our bw cap there20:53
pabelangerfungi: indeed cc sdague ^20:53
fungiwe probably need to keep an eye on it and maybe replace it again with an even bigger flavor if we get closer to 200mbps20:53
pabelangeror setup load balanceers20:54
*** raildo has quit IRC20:54
mordredbigger vms20:54
mordredmore power20:54
clarkbload balancers have a similar problem20:54
clarkbsince they are restricted to the same bw constraints20:54
sdaguemordred: max powers!20:54
pabelangerclarkb: that is true20:54
clarkbso in this case its simpler to just go bigger20:54
mordredsdague: so much powers20:54
pabelangergo big or go home20:55
sdaguesimpsons ^^^20:55
fungione of my favorite episodes20:56
fungii've thought from time to time it would have made an amusing online handle/pseudonym20:57
mordredsdague: so - if I could bother you for a sec ... - I added an experimental grenade job to neutronclient so that we can show that the combo of the latest os-client-config and the patch I wrote appropriately works20:57
*** sdake has joined #openstack-infra20:57
fungiclarkb: if only rackspace had a network-heavy flavor. we don't need more ram/cpu/disk but we end up with it anyway to get more bandwidth20:58
mordredsdague: BUT - it has the sads20:58
sdaguemordred: ok, you have about 3 minutes to explain the sads before I call it a day.20:58
sdaguebut you should do that, because I'll look first thing in the morning20:58
anteayasdague: thank you20:58
mordredsdague: it complains about xenial20:58
mordredsdague: which makes me think job config issue20:58
mordredsdague: but I thought I copied all of the goo from other people20:59
sdagueright, grenade doesn't run on xenial20:59
sdaguebecause comes from mitaka20:59
sdaguewhich shipped before xenial20:59
*** dprince has quit IRC20:59
sdagueand we typically don't backport that support change21:00
mordredhrm. ok, then I think my original version of that patch was potentially more correcter21:00
*** jkilpatr has quit IRC21:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Bump tempest version to latest
sdagueso... you should probably just move this to run on trusty21:00
sdaguewe talked about just doing the backport, but clarkb didn't think it was needed when he was rolling jobs over21:01
clarkbright we decided to run mitaka to newton/master on trusty21:01
* mordred grumps21:01
mordredsdague: cool. thnaks. super helpful21:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Bump tempest version to latest
sdaguemordred: ok, great. If you need other things, feel free to send an email. Heading out for the day.21:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING
*** mhickey has quit IRC21:03
fungii think grenade pull-up jobs for newton make more sense on trusty since mitaka was only tested on trusty and newton was tested on trusty for ~half of its development21:03
*** hrubi has joined #openstack-infra21:03
clarkbyup. Thought I also think that grenade should not be so forceful about what platform I run it on21:03
openstackgerritMonty Taylor proposed openstack-infra/project-config: Run  neutronclient experimental grenade job on trusty
clarkbif I want to run it on tumbleweed please let me ...21:03
*** julim has quit IRC21:03
mordredclarkb, anteaya: ^^ per the conversation just now21:04
anteayamordred: okey dokey21:05
openstackgerritMerged openstack-infra/elastic-recheck: Add query for cells v2 setup bug 1613417
openstackbug 1613417 in devstack "gate-tempest-dsvm-cells broken with cell v2 setup: "No hosts found to map to cell, exiting."" [Undecided,In progress]
*** spzala has quit IRC21:07
*** spzala has joined #openstack-infra21:07
*** ihrachys has quit IRC21:08
pabelangerfungi: big spike now, 175Mbps21:08
pabelangerhow high will it go21:08
pabelangernobody knows21:08
*** sdake has quit IRC21:08
mordredwow. we produce some traffic!21:09
fungiand that's just local mirror access for rax-ord21:09
fungimakes me wonder if we have very stale caches on images there21:09
*** yamamoto has joined #openstack-infra21:10
fungior whether that's something other than distro packages21:10
*** yamamoto has quit IRC21:11
fungile sigh. i'm getting "yaml.reader.ReaderError: unacceptable character #x009b: special characters are not allowed" trying to parse openstack/governance:reference/projects.yaml21:11
pabelangerfungi: so, I created a new volume for the new server since I didn't want to break jobs running. So, cache does need to warm up there21:11
anteayafungi: :(21:11
fungipabelanger: yep, but that would show up as ingress not egress21:12
jeblairwe just had a logistical discussion in #zuul which ended up with the idea that we should make a feature branch for nodepool for the zk work.  even though we want to land that and start using it soon, it will take multiple changes to implement, and coexistence with the current builder design is difficult.21:12
pabelangerfungi: Yes, that is true21:12
fungijeblair: seems reasonable to me21:12
jeblairclarkb: i guess we should have gone with your pick for server size.  :)21:12
pabelangercloudnull: your patch just went live21:12
*** adrian_otto has quit IRC21:12
pabelangerincoming 250 nodes on osic-cloud121:12
mordredfungi: how ar eyou reading it?21:13
* cloudnull goes home for the day21:13
fungimordred: yaml.safe_load(requests.get(PROJECTS_LIST % ref).text)21:13
jeblaircloudnull: oh wait, you almost forgot your pager!21:13
* cloudnull runs21:13
*** spzala has quit IRC21:13
mordredfungi: ah - a=yaml.safe_load(open('reference/projects.yaml', 'r').read()) works for me21:13
mordredI will try your version21:13
*** rhallisey has quit IRC21:14
mordredfungi: you don't have an expansion of PROJECTS_LIST % ref handy do you?21:14
fungimordred: i think it may be with how/where i'm retrieving it from. digging deeper21:14
mordreda=yaml.safe_load(requests.get('').text) works for me21:15
*** ldnunes has quit IRC21:15
*** rbuzatu has joined #openstack-infra21:15
*** tqtran has quit IRC21:15
mordredfungi: I can re-create your error with the review.o.o url21:16
openstackgerritMerged openstack-infra/elastic-recheck: Fix template filename
fungimordred: yep. i'm finding what position 25352 is next21:16
mordredfungi: fun!21:16
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add tripleo-test-clouds AFS mirrors to cacti.o.o
mordredfungi: I blame jgit21:16
*** sdake has joined #openstack-infra21:17
* Shrews blames j<anything>21:17
*** _ari_ has joined #openstack-infra21:17
*** fitoduarte has joined #openstack-infra21:17
*** adduarte has joined #openstack-infra21:17
*** _ari_ has quit IRC21:17
fungiu'    - name: Zbyn\xc4\x9bk Schwarz\n'21:18
*** adduarte has quit IRC21:18
mordredfungi: from git.o.o I get: u' name: Zbyn\u011bk Schwarz\n    ' right around there21:18
mordredfungi: I wonder if you need a header for the requests.get to set a language or something21:19
mordredor encoding I mean21:19
fungimordred: possibly21:19
clarkbyaml is utf8 by default iirc21:20
clarkbso if you are somehow getting the bits in not utf8 that may make it mad21:21
mordredyah - but gitweb might be encoding over the wire21:21
mordredor decoding21:21
mordredor something21:21
*** elo has joined #openstack-infra21:21
*** rbuzatu has quit IRC21:21
mordredmy browser tells me that that link is being served as "Western (Windows-1252)"21:22
fungimordred: requests.get(blah).encoding indeed says 'ISO-8859-1'21:23
fungilooks like it might be a gitweb fallback behavior21:24
pabelangerokay, stepping away to run some family errands.  I think our original can be deleted now. Last hit to apache logs in 15/Aug/2016:20:47:05 +0000.  I'll do that when I get back this evening just to be safe21:25
*** matt-borland has quit IRC21:25
clarkbpabelanger: thanks again21:25
clarkbfungi: mordred fallback for when you don't set an accepts encoding?21:26
clarkbok ntp is making me go blind21:26
clarkbianw: pabelanger: any other feedback on not using our ntp mdoule on the test images at all?21:26
*** adrian_otto has joined #openstack-infra21:27
fungiclarkb: likely. i'm just reading through requests docs now21:28
*** edtubill has quit IRC21:28
*** jkilpatr has joined #openstack-infra21:28
*** apetrich has quit IRC21:29
mordredfungi: ok. SO ...21:30
mordredresponse = requests.get(';a=blob_plain;f=reference/projects.yaml;hb=master')21:31
mordredresponse.encoding = 'utf-8'21:31
mordredfungi: that ^^ works21:31
fungimordred: yep, found that gem21:31
mordredfungi: so I think what may be happening is that gitweb is returning utf8 data but setting the header wrong21:31
fungiso requests is assuming the response is in latin1 when it's actually utf8 all along21:31
mordredyah: 'Content-Type': 'text/plain; charset=ISO-8859-1'21:32
mordredthat's in the respnse headers from gitweb21:32
fungi.headers does indeed sat that21:32
fungier, say21:32
fungithat's where i just went as well21:32
mordredI think we can consider that to be independently verified results then! :)21:32
fungimordred: i think it's actually not setting an encoding, which rfc 2616 says means latin121:34
mordredfungi: lovely21:34
*** adrian_otto has quit IRC21:34
fungii'd need to use a packet sniffer to confirm whether requests is faking that in the headers dict, or apache is actually passing it21:35
*** jheroux has quit IRC21:35
*** jcoufal has quit IRC21:35
fungicould be we need apache on review.o.o configured differently21:35
fungithere's "AddDefaultCharset UTF-8" as one possibility21:36
*** yamahata has quit IRC21:38
*** sdake has quit IRC21:39
karthikp_clarkb: Hi21:40
openstackgerritClark Boylan proposed openstack-infra/system-config: Disable ntp services on single use test instances
clarkbfungi: ianw pabelanger ^ I am going to WIP that until I can do more testing of the distro boot time defaults to make sure tehy do set something sane (manpages for ubuntu claim they do and I think its all systemd related so the other distros should too)21:40
openstackgerritJulia Kreger proposed openstack-infra/project-config: Rename bifrost integration test job
clarkbkarthikp_: hello21:41
*** edtubill has joined #openstack-infra21:41
karthikp_clarkb: got a question for you regarding grenade.... any idea why this step is necessary?21:42
openstackgerritMerged openstack-infra/elastic-recheck: Fix template links for split uncategorized
clarkbkarthikp_: I don't know for sure but guessing that if the services really fail to start then the lgos won't exist21:43
clarkbkarthikp_: you might have better luck checking the git logs for that line21:43
anteayaTheJulia: does bifrost have more than one job for ipa?21:46
anteayaTheJulia: if not, how about removing the adjective and going with ipa21:46
TheJuliaanteaya: two, we build IPA with debian as well21:46
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects
anteayajust trying to see if we can avoid needing to rename the job when you change the image21:47
fungimordred: i've investigated some apache-side workarounds, but on deeper investigation it seems that gitweb has a history of returning incorrect content types, including for blob_plain
TheJuliaanteaya: renaming the job was like the last thing I wanted to do though :\21:47
fungimordred: so i'll just set the encoding in our script as a workaround21:47
anteayaTheJulia: yeah, how about ipa-cirros21:47
anteayathat is different from ipa-debian, yeah?21:48
TheJulianot as descriptive, yeah, it also fires up debian in that job, so in theory that still works21:48
*** amotoki has quit IRC21:48
* TheJulia likes it21:48
anteayawell the description should be in the log, right?21:48
anteayayay you like it21:48
*** burgerk has quit IRC21:48
TheJulia:)  I'll update it in a little bit21:49
anteayayup, thanks21:49
anteayagoing offline soon, I'll look tomorrow21:49
*** adrian_otto has joined #openstack-infra21:51
clarkbipa works on cirros?21:52
* clarkb wonders if thats another image in the ironic ramdisk image list21:52
karthikp_clarkb: i see .. git logs?21:52
clarkbkarthikp_: the revision control history for that repo may tell you why that line was added21:52
*** devkulkarni has joined #openstack-infra21:52
*** sdake has joined #openstack-infra21:54
anteayaI have to assume the answer to your question is yes, based on the existing name of the job21:55
*** tqtran has joined #openstack-infra21:55
*** harlowja has quit IRC21:55
TheJuliaclarkb: more like we deploy cirros as fast lightweight reliable test21:55
anteayaI'm that patch is all I am using for my assertion21:55
clarkbah you boot cirros using tinycore ramdisk21:56
karthikp_clarkb: Oh ya that was added for all the projects by sdague..i iwlll chekc with him21:56
karthikp_clarkb: thanks21:56
anteayaJayF: is thinking like me21:56
*** tonytan4ever has quit IRC21:57
* TheJulia lets there be a little chatter and goes to start cooking dinner :)21:57
JayFanteaya: that's the nicest thing you've ever said to me \o/ :)21:57
JayFMy thought was just, if I were graphing this job, I'd wanna see how the default changed it w/o having to change the name21:58
JayFif you have >1 of something, sure, specify, but maybe leave it out if it's only 121:58
*** tkelsey has joined #openstack-infra21:58
anteayaJayF: ha ha ha :)21:59
anteayaJayF: I agree with you thinking21:59
anteayaI'm so glad I could math in school, the spelling gods never looked my way22:00
*** amotoki has joined #openstack-infra22:00
*** matrohon has quit IRC22:00
*** tqtran has quit IRC22:00
openstackgerritMerged openstack-infra/project-config: Run  neutronclient experimental grenade job on trusty
*** gordc has quit IRC22:00
*** amotoki has quit IRC22:00
*** yamahata has joined #openstack-infra22:01
*** xarses has joined #openstack-infra22:01
anteaya<-- offline22:02
*** camunoz has quit IRC22:02
*** mtanino has quit IRC22:03
*** onovy has quit IRC22:03
*** tkelsey has quit IRC22:03
openstackgerritMerged openstack-infra/nodepool: Shut down gearman client in tests
openstackgerritMerged openstack-infra/nodepool: Remove testresources
*** peterlisak has quit IRC22:05
*** esberglu has quit IRC22:06
*** thorst_ has quit IRC22:08
*** tqtran has joined #openstack-infra22:08
mmedvedesean-k-mooney: around? I tested a fresh install of ciwatch, it works fine22:08
*** mdrabe has quit IRC22:09
*** mriedem has quit IRC22:09
* clarkb is working on building dib -minimal images without an explicit ntp install to see what we end up with22:09
clarkbianw: pabelanger ^ hopefully that shows us we can get away with just not doing stuff on the sinlge use nodes22:09
*** valderrv has quit IRC22:10
*** edtubill has quit IRC22:10
*** beagles is now known as beagles_brb22:11
*** thorst_ has joined #openstack-infra22:13
*** tqtran has quit IRC22:15
*** bswartz has quit IRC22:15
*** tqtran has joined #openstack-infra22:16
scottdayolanda: Would you re-approve when you have a chance? The dependent patch has merged an it needed a rebase.22:16
*** thorst_ has quit IRC22:18
*** jistr has quit IRC22:18
*** peterlisak has joined #openstack-infra22:19
*** jistr has joined #openstack-infra22:19
*** onovy has joined #openstack-infra22:19
*** sdake has quit IRC22:20
*** vhosakot has joined #openstack-infra22:25
*** netsin has quit IRC22:25
*** yamamoto has joined #openstack-infra22:26
*** hockeynut has joined #openstack-infra22:29
*** jkilpatr has quit IRC22:29
*** weshay has quit IRC22:32
*** nwkarsten has quit IRC22:32
*** nwkarsten has joined #openstack-infra22:32
*** signed8bit is now known as signed8bit_Zzz22:34
*** fguillot_ has joined #openstack-infra22:34
*** sdake has joined #openstack-infra22:35
*** nwkarsten has quit IRC22:37
*** krtaylor has quit IRC22:38
*** rbuzatu has joined #openstack-infra22:38
*** netsin has joined #openstack-infra22:38
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects
pabelangerclarkb: ack22:39
pabelangerclarkb: I haven't really been following the ntp issues from today. Will try and catch up on backscroll here in a bit22:40
clarkbpabelanger: tl;dr is after seeing the meeting notes from last week I saw you all mentioned just using the defaults on the distros. and on further investigation I think at least for systemd distros it may just work if we stop explicuitly installing ntp22:40
clarkbpabelanger: so building images locally to test that theory22:41
pabelangerclarkb: Ah, yes. I remember that22:41
clarkbpabelanger: since systemd has some built in time syncing stuff that should update the time on boot if I am reading things correctly22:42
clarkbbut want to test that first22:42
clarkband figure out what trusty and precise do22:42
*** beagles_brb is now known as beagles22:42
JayFtimesyncd is pretty nuts though. it just does a tls connection to something and steals the timestamp iirc22:42
JayFlike if that's good enough, it's good enough, just a strange way of doing things22:43
JayFah it apparently talks to real ntp servers now, that's an improvement22:43
pabelanger#status log upgraded to performance1-4 to address network bandwidth cap.22:43
pabelangerand original server now deleted22:44
*** weshay has joined #openstack-infra22:44
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow
*** rbuzatu has quit IRC22:44
*** pabelanger has quit IRC22:45
*** pabelanger has joined #openstack-infra22:45
pabelanger#status log upgraded to performance1-4 to address network bandwidth cap.22:45
openstackstatuspabelanger: finished logging22:45
*** signed8bit_Zzz is now known as signed8bit22:46
*** fguillot_ has quit IRC22:47
clarkbJayF: ya for our long lived servers we will probably continue to ntp or similar22:47
clarkbJayF: but on the test instances we really just need a mostly correct timestamps in logs that won't jump halfway through a job22:48
pabelangerokay, just starting to look into osic-cloud1 lauch node errors, first issue:
clarkbpabelanger: that needs to use iptables6 I think22:50
* mordred lookie22:50
pabelangerI think so too22:50
pabelangerOh, can we land
pabelangerhelp reduce debug logs in nodepool22:51
pabelangerwhen we cannot host git.o.o22:51
mordredoh piddle22:51
*** edmondsw has quit IRC22:51
cloudnullpabelanger: anything you need from me  ?22:51
cloudnullor any way I can help ?22:51
mordredthe 'bug' in shade (it currently doesn't do enough magic WRT IPv4/IPv6 addresses) _may_ bite us with multi-node22:52
pabelangercloudnull: I don't think so. We just need to update some nodepool scripts I think22:52
* mordred goes to look through nodepool real quick22:52
pabelangermordred: oh?22:52
*** vhosakot has quit IRC22:53
mordredyeah. blast22:53
mordredthat means I _am_ going to have to fix that22:53
* mordred cries22:53
mordredactually ...22:53
*** fguillot_ has joined #openstack-infra22:53
pabelangercloudnull: actually, I do see an SSH timeout for osic-cloud122:53
pabelangercloudnull: let me see if I can get the instance ID22:53
mordredclarkb: multinode testing networking ...22:53
clarkbya thats the setup for allowing all traffic between multinode right?22:54
mordredclarkb: we don't actually need subnodes_private to have things in it, right? because we have clouds with only public?22:54
clarkbshould be simple to just check the ip and use the right iptables command22:54
pabelangercloudnull: timeout waiting for ssh access22:55
clarkbmordred: last time I tried to use public only on clouds with both priovate and and public openstack didn't work22:55
clarkbmordred: clouds like osic when fip and bluebox22:55
clarkbI think NAT is or was creating problems for us there22:55
mordredclarkb: k. so - what if one of the things in subnodes_public was a 10. address22:55
clarkbthen other random stuff wouldn't work I would expect22:56
cloudnullpabelanger: looking22:56
mordredclarkb: the tl;dr here is that on osic we detect the 10. ipv4 address as being "public"22:56
clarkbmordred: we should put the ipv6 addr in there no?22:56
cloudnullmordred: does it make your life easier if i change that to a 192 address ?22:56
mordredcloudnull: nope22:56
mordredclarkb: well, we put the ipv6 address into interface_ip and will use it correctly for most things22:56
*** nwkarsten has joined #openstack-infra22:57
mordredclarkb: but nodepool multi-node is the one place where we might look explicitly for public/private and expect themto be correct22:57
*** sdake has quit IRC22:57
mordred(most of the rest of the cases it all just works because interface_ip has the ipv6 address and everything is happy)22:57
*** mriedem has joined #openstack-infra22:57
*** tonytan4ever has joined #openstack-infra22:58
clarkbmordred: multinode d-g wants to use the private addrs for most stuff (I think everything) due to the presumed nat issues22:58
mordredclarkb: ok. I'll work on a fix then22:58
clarkbmordred: so I wouldn't expect that to break with 10 net addr in public22:58
clarkbbut you need to have it in private list too22:58
mordredclarkb: well, the 10. will not be in private22:59
mordredonly in public22:59
clarkbI think nodepool puts it in both22:59
clarkbif there is no private addr then it writes the public to private22:59
mordredI'll go read through that code more22:59
clarkbso that things relying on "private" continue to work22:59
mordredoh good22:59
mordred(this is me really not wanting to try to solve the problem right now)22:59
mordredclarkb: for slightly more wordy context- the underlying problem is that we currently determine "does this route packets off the cloud" with the Network object. (and to be fair, that's where the router:external property which does not mean routes externally sits)23:01
*** nwkarsten has quit IRC23:01
mordredclarkb: but it turns out you can have a subnet that routes externally and a subnet that does not route externally both attached to the same Network23:01
mordredclarkb: so the _real_ question that needs to be asked is "is the port that provides this IP address attached to a subnet that can route externally"23:02
mordredbut that's a bunch more data model trolling to get consistent and right every time - and most of the time it's an extra level of complexity that doesn't show up23:02
*** rbrndt has quit IRC23:02
pabelangerclarkb: did we want to land 355570 now? So we can have the dns fix for tomorrows image builds23:02
clarkbmordred: fun23:03
mordredclarkb: yah.23:03
*** tonytan4ever has quit IRC23:03
pabelangermordred: can we remove the autohold for Automatically held after failing gate-shade-dsvm-functional-neutron ?23:04
pabelangeror is that still needed23:04
mordredpabelanger: yes. absolutely can remove23:04
clarkbpabelanger: does that work in clouds with no v6? does unbound know to do the right thing in that situation?23:04
*** devkulkarni has quit IRC23:04
*** asettle has joined #openstack-infra23:04
mordredthat's a good question23:04
pabelangerclarkb: I tested with both ovh and osic and it worked.23:05
pabelangerI can confirm with each other cloud too23:05
clarkbpabelanger: and you made sure that it was using unbound not the cloud provided resolvers?23:06
clarkbI am not sure if that happens in ovh like in rax23:06
*** markvoelker has quit IRC23:06
pabelangerclarkb: yup, nslookup used
pabelangersame with dig +trace23:07
mtreinishfungi, pabelanger, clarkb: hmm did I miss a step in adding firehose.o.o to cacti: is all blank23:07
pabelangerwelp, internap is also using DNS from cloud provider23:07
openstackgerritAbhishek Raut proposed openstack-infra/project-config: Use python-db-jobs for tap-as-a-service
jeblairmtreinish: for starters, isn't the server ''?23:08
*** xyang1 has quit IRC23:08
*** Goneri has joined #openstack-infra23:08
mtreinishjeblair: ah, yep that'd probably do it23:08
fungiahh, yep, need to fix that at
fungii missed that23:09
jeblairi'm not sure if that's the actual cause, but i'm not certain it's not.23:09
jeblaircacti says 'udp ping success / snmp error'23:09
*** hongbin has quit IRC23:09
*** asettle has quit IRC23:09
jeblair(but i think our convention is actual hostnames in cacti, so i think we should change it regardless)23:10
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Fix firehose hostname on cacti hiera
mtreinishjeblair, fungi: ^^^23:10
fungias for why it's not showing up, i don't see snmpd running on the server23:10
fungiActive: active (exited) since Mon 2016-08-01 15:48:50 UTC; 2 weeks 0 days ago23:11
fungisayeth `service snmpd status`23:11
jeblairthat would do it fer shure23:11
clarkbpabelanger: approved23:12
clarkbmy local xenial host without ntp is definitely running the systemd thing23:12
clarkbtrusty doesnt' seem to do much with time though23:12
fungii'll refrain from restarting snmpd on it until the hiera change makes it onto the cacti host23:12
jeblairfungi: good plan23:12
jeblairless to delete that way23:12
fungilaziness is next to godliness23:12
fungior something like that23:13
mtreinishpleia2: it looks like puppet updated the stuff, but the cron job is still not happy:
clarkbya I think older non systemd distros are going to be a problem here23:14
clarkbdefinitely doesn't do anything on trusty23:14
clarkbthere goes that idea :P23:14
*** asselin has joined #openstack-infra23:14
pabelangerclarkb: I think we are going to be good, all clouds appear to have inet6 address on eth0 and lo0.  And unbound seems to do the right think if ipv6 entry is not accessible, fails to the next entry which is ipv423:15
*** asselin_ has quit IRC23:15
*** asselin_ has joined #openstack-infra23:15
clarkbpabelanger: most of those clouds just have link local addrs though23:16
*** baoli has joined #openstack-infra23:16
cloudnullpabelanger: interestingly I'm seeing this on the compute node where that instance was spawned.
clarkbpabelanger: which won't get them to gogole dns. The exceptions are osic, rax, and vexxhost23:16
cloudnullhowever no other errirs23:16
clarkbin any case if it falls back to ipv4 without ridiculously long timeouts we should be fine23:16
pabelangercloudnull: right, I've tested on both bluebox and ovh, if I force ipv6 dns, it fails.  If I add both, ipv6 and ipv4, dns works as expected23:17
pabelangerclarkb: Ya, it is pretty fast23:17
cloudnullpabelanger: was talking about that instance you noted as having ssh timeouts23:18
* clarkb is beginning to wonder if the simplest thing would be to install our own init script for sntp and jsut run that once at boot on all platforms23:18
clarkbprobably going to run into dependency hell with the existing distro stuff though23:18
*** asselin has quit IRC23:19
pabelangercloudnull: Oh, neat. So you are seeing something23:19
cloudnullyea i may need to do some iptables munging or neutron tweaking to make that happier.23:20
cloudnullidk quite yet23:20
* pabelanger nods23:20
cloudnullbut yes.23:20
openstackgerritMerged openstack-infra/project-config: Add IPv6 DNS support
*** xarses has quit IRC23:22
cloudnullfound a bug that was patched but it looks like its just a warning:
openstackLaunchpad bug 1565705 in neutron "iptables duplicate rule warning on ports with multiple security groups" [Medium,Fix released] - Assigned to Kevin Benton (kevinbenton)23:23
*** shashank_hegde has quit IRC23:23
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics
fungianteaya: zaro: ^ latest gerrit upgrade allowed some serious simplification there on multiple fronts23:24
zarofungi: ahh nice!23:25
zarofungi: i'm testing online index but sorta hit a snag.  not enough memory on review-dev now!23:26
fungidropped more than 50 loc23:26
fungizaro: oh, ouch!23:26
fungiwe can rebuild it bigger if needed23:26
clarkband it looks like on fedora and centos we would have to explicitly install something to set the time so they are more like ubuntu trusty23:27
zarofungi: yeah may need to if we want to test multiple users hitting it while it's reindexing.23:27
clarkbI will need to fiddle with these VMs a bit more when its not almost the end of the day23:28
clarkbfigure out what magic is needed to make things happen23:28
zarofungi: on the bright side it seems to be working great with just me poking at it.23:28
*** devkulkarni has joined #openstack-infra23:29
*** gyee has quit IRC23:30
fungianteaya: dhellmann: you _should_ be able to use on your own to generate the electoral rolls now, though with the coming round of technical elections i think i should generate a set too and then election officials can confirm the lists they have match mine just to be on the safe side. if it works out though, our gerrit admins can get completely out of involvement in23:33
*** kzaitsev_mb has quit IRC23:33
fungifuture elections unless troubleshooting becomes necessary23:33
*** xarses has joined #openstack-infra23:34
dhellmannfungi : excellent23:34
dhellmannthough I won't be an election official since I'll be up for election23:35
fungiahh, yup ;)23:35
*** harlowja has joined #openstack-infra23:36
*** devkulkarni has quit IRC23:40
clarkbthe mroe I dig the more I think we might have to do our own equivalent to ntpdate at boot using system appropriate tools23:40
clarkbsince everything seems to do the gentle update to avoid making processes unhappy23:40
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.
*** hockeynut has quit IRC23:43
*** sdague has quit IRC23:44
*** pahuang has quit IRC23:45
*** jerryz has quit IRC23:46
mordredit's a jhesketh !23:47
mordredclarkb: yah - when what we want is "MAKE IT GOOD NOW"23:47
*** sarob has quit IRC23:48
* clarkb is happy his suse system already comes with this feature23:48
clarkbbut I can't find anything like ti on ubuntu23:48
*** zhurong has joined #openstack-infra23:49
*** gyee has joined #openstack-infra23:49
jheskethmordred: indeed :-)23:50
*** dingyichen has joined #openstack-infra23:51
*** baoli has quit IRC23:54
cloudnullpabelanger: sadly, yet again, I can't find anything specifically wront with the environment that would produce an ingress ssh timeout. If we can identify one of these instances and keep it online I can troubleshoot it further.23:55
cloudnullNow that I have LOTS of IPs to play with I'll try to reproduce it on my own but for now, IDK :'(23:56
*** asselin_ has quit IRC23:56
clarkbreading chrony init scripts for ubuntu it will do a burst on interface startup but not a step23:56
*** jimbaker has quit IRC23:57
*** zhurong has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!