Monday, 2016-05-02

*** baoli has quit IRC00:02
*** Hal has joined #openstack-infra00:04
*** mylu has quit IRC00:04
*** Hal is now known as Guest768700:04
*** heyongli_ has quit IRC00:05
*** heyongli_ has joined #openstack-infra00:05
*** baoli has joined #openstack-infra00:07
*** mylu has joined #openstack-infra00:07
*** mylu has quit IRC00:10
*** mylu has joined #openstack-infra00:11
*** zeih has joined #openstack-infra00:12
*** salv-orlando has joined #openstack-infra00:12
*** mylu has quit IRC00:13
*** mylu has joined #openstack-infra00:13
*** heyongli_ has quit IRC00:15
*** heyongli_ has joined #openstack-infra00:15
*** sdake has quit IRC00:15
*** baoli has quit IRC00:16
*** zeih has quit IRC00:17
*** baoli has joined #openstack-infra00:17
*** salv-orlando has quit IRC00:17
*** apoorvad has joined #openstack-infra00:18
*** Guest7687 has quit IRC00:19
*** MarkAtwood has quit IRC00:20
*** keedya_ has quit IRC00:21
*** heyongli_ has quit IRC00:25
*** heyongli_ has joined #openstack-infra00:25
*** sdake has joined #openstack-infra00:32
openstackgerritMerged openstack-infra/project-config: python-troveclient didn't have release notes till mitaka  https://review.openstack.org/31151700:34
*** heyongli_ has quit IRC00:35
*** heyongli_ has joined #openstack-infra00:36
*** sdake_ has joined #openstack-infra00:42
*** kengo_sakai has joined #openstack-infra00:42
*** sdake has quit IRC00:44
*** heyongli_ has quit IRC00:46
*** heyongli_ has joined #openstack-infra00:46
*** kengo_sakai has quit IRC00:46
*** markvoelker has joined #openstack-infra00:51
*** markvoelker has quit IRC00:56
*** heyongli_ has quit IRC00:56
*** heyongli_ has joined #openstack-infra00:56
*** Hal has joined #openstack-infra00:57
*** Hal is now known as Guest8175500:58
*** baoli has quit IRC01:02
openstackgerritSteve Martinelli proposed openstack/requirements: Update keystoneclient lower bound  https://review.openstack.org/29959301:02
*** asselin__ has quit IRC01:06
*** heyongli_ has quit IRC01:06
*** zz_dimtruck is now known as dimtruck01:07
*** heyongli_ has joined #openstack-infra01:09
*** salv-orlando has joined #openstack-infra01:13
*** heyongli_ has quit IRC01:17
*** heyongli_ has joined #openstack-infra01:17
*** salv-orlando has quit IRC01:18
*** EricGonczer_ has quit IRC01:19
*** kengo_sakai has joined #openstack-infra01:24
*** heyongli_ has quit IRC01:27
*** heyongli_ has joined #openstack-infra01:27
*** jefrite has joined #openstack-infra01:30
*** ekarlso has quit IRC01:33
*** kengo_sakai has quit IRC01:37
*** heyongli_ has quit IRC01:37
*** heyongli_ has joined #openstack-infra01:37
*** kengo_sakai has joined #openstack-infra01:37
timrcfungi: Ah yard work.  I know this thing all too well.  What's sad is that you'd never be able to tell how much effort I've put into my yard.01:38
*** fedexo has joined #openstack-infra01:43
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add install-bin element  https://review.openstack.org/31164601:46
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tinycorelinux element  https://review.openstack.org/31164701:46
*** xarses has quit IRC01:46
*** xarses_ has joined #openstack-infra01:48
*** heyongli_ has quit IRC01:48
*** heyongli_ has joined #openstack-infra01:48
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tinycorelinux element  https://review.openstack.org/31164701:49
*** zqfan has joined #openstack-infra01:51
*** markvoelker has joined #openstack-infra01:52
*** ekarlso has joined #openstack-infra01:53
*** thorst has quit IRC01:54
*** thorst has joined #openstack-infra01:55
*** markvoelker has quit IRC01:57
*** heyongli_ has quit IRC01:57
*** heyongli_ has joined #openstack-infra01:58
*** mylu has quit IRC01:59
*** mylu has joined #openstack-infra02:02
*** thorst has quit IRC02:03
*** mylu has quit IRC02:03
*** mylu has joined #openstack-infra02:04
*** heyongli_ has quit IRC02:08
*** heyongli_ has joined #openstack-infra02:10
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tinycorelinux element  https://review.openstack.org/31164702:11
*** zeih has joined #openstack-infra02:13
morganpersia: lol02:13
*** hdd has joined #openstack-infra02:13
*** salv-orlando has joined #openstack-infra02:14
*** morgan is now known as notmorgan02:17
*** zeih has quit IRC02:17
*** heyongli_ has quit IRC02:18
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tinycorelinux element  https://review.openstack.org/31164702:18
*** salv-orlando has quit IRC02:19
*** kengo_sa_ has joined #openstack-infra02:22
openstackgerritgreghaynes proposed openstack/diskimage-builder: Split pip out of pip-and-virtualenv  https://review.openstack.org/31165102:23
*** heyongli_ has joined #openstack-infra02:23
*** kengo_sakai has quit IRC02:25
*** mylu has quit IRC02:25
*** denisra has quit IRC02:27
*** denisra has joined #openstack-infra02:28
*** heyongli_ has quit IRC02:28
*** heyongli_ has joined #openstack-infra02:28
*** heyongli_ has quit IRC02:38
*** heyongli_ has joined #openstack-infra02:41
*** mylu has joined #openstack-infra02:44
*** kengo_sa_ has quit IRC02:47
*** heyongli_ has quit IRC02:49
*** heyongli_ has joined #openstack-infra02:49
*** markvoelker has joined #openstack-infra02:49
*** heyongli_ has quit IRC02:59
*** heyongli_ has joined #openstack-infra02:59
*** thorst has joined #openstack-infra03:01
*** dimtruck is now known as zz_dimtruck03:07
*** mriedem1 has quit IRC03:08
*** thorst has quit IRC03:08
*** heyongli_ has quit IRC03:09
*** heyongli_ has joined #openstack-infra03:10
*** apoorvad has quit IRC03:11
*** nadya has joined #openstack-infra03:12
*** baoli has joined #openstack-infra03:14
*** salv-orlando has joined #openstack-infra03:15
*** obondarev has quit IRC03:18
*** nadya has quit IRC03:19
*** baoli has quit IRC03:19
*** obondarev has joined #openstack-infra03:19
*** nadya has joined #openstack-infra03:19
*** heyongli_ has quit IRC03:19
*** salv-orlando has quit IRC03:19
*** heyongli_ has joined #openstack-infra03:20
*** yamamoto has quit IRC03:22
*** nadya has quit IRC03:25
*** mylu has quit IRC03:25
*** mylu has joined #openstack-infra03:26
*** raginbaji is now known as raginbajin03:28
*** Adri2000 has quit IRC03:28
*** heyongli_ has quit IRC03:30
*** heyongli_ has joined #openstack-infra03:30
*** flepied has quit IRC03:31
*** mylu has quit IRC03:39
*** heyongli_ has quit IRC03:40
*** mylu has joined #openstack-infra03:40
*** heyongli_ has joined #openstack-infra03:40
*** mylu has quit IRC03:42
*** Guest81755 has quit IRC03:42
*** Adri2000 has joined #openstack-infra03:43
*** jpeeler has joined #openstack-infra03:44
*** jpeeler has quit IRC03:44
*** jpeeler has joined #openstack-infra03:44
*** kengo_sakai has joined #openstack-infra03:48
*** heyongli_ has quit IRC03:50
*** heyongli_ has joined #openstack-infra03:50
*** mylu has joined #openstack-infra03:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove translations from operations-guide  https://review.openstack.org/31166003:55
*** kengo_sakai has quit IRC03:56
*** mylu has quit IRC03:58
*** zz_dimtruck is now known as dimtruck04:00
*** heyongli_ has quit IRC04:00
*** mylu has joined #openstack-infra04:02
*** heyongli_ has joined #openstack-infra04:03
*** thorst has joined #openstack-infra04:06
*** links has joined #openstack-infra04:07
*** heyongli_ has quit IRC04:11
*** heyongli_ has joined #openstack-infra04:11
*** hdd has quit IRC04:11
*** apoorvad has joined #openstack-infra04:12
*** thorst has quit IRC04:13
*** yamamoto has joined #openstack-infra04:14
*** zeih has joined #openstack-infra04:14
*** salv-orlando has joined #openstack-infra04:16
*** mylu has quit IRC04:17
*** apoorvad has quit IRC04:17
*** zeih has quit IRC04:18
*** mylu has joined #openstack-infra04:19
*** salv-orlando has quit IRC04:20
*** mylu has quit IRC04:21
*** bnemec has joined #openstack-infra04:21
*** heyongli__ has joined #openstack-infra04:21
*** heyongli_ has quit IRC04:25
*** psachin has joined #openstack-infra04:27
*** heyongli__ has quit IRC04:31
*** heyongli__ has joined #openstack-infra04:31
*** mylu has joined #openstack-infra04:34
*** asettle has joined #openstack-infra04:35
*** asettle has quit IRC04:41
*** heyongli__ has quit IRC04:41
*** asettle has joined #openstack-infra04:41
*** heyongli__ has joined #openstack-infra04:42
*** mylu has quit IRC04:45
*** mylu has joined #openstack-infra04:45
*** mylu has quit IRC04:47
*** mylu has joined #openstack-infra04:48
*** markvoelker has quit IRC04:49
*** asettle has quit IRC04:51
*** asettle has joined #openstack-infra04:52
*** heyongli__ has quit IRC04:52
*** heyongli__ has joined #openstack-infra04:52
*** gildub has joined #openstack-infra04:54
*** asettle has quit IRC04:55
*** oanson has joined #openstack-infra05:01
*** heyongli__ has quit IRC05:02
*** heyongli__ has joined #openstack-infra05:02
*** rcernin has joined #openstack-infra05:05
*** thorst has joined #openstack-infra05:11
*** heyongli__ has quit IRC05:12
*** mylu has quit IRC05:12
*** heyongli__ has joined #openstack-infra05:12
*** Sukhdev has joined #openstack-infra05:14
*** salv-orlando has joined #openstack-infra05:16
*** thorst has quit IRC05:19
*** mylu has joined #openstack-infra05:19
*** abregman has joined #openstack-infra05:21
*** salv-orlando has quit IRC05:21
*** mylu has quit IRC05:22
*** heyongli__ has quit IRC05:22
*** heyongli__ has joined #openstack-infra05:23
*** mylu has joined #openstack-infra05:27
*** mylu has quit IRC05:32
*** yolanda has joined #openstack-infra05:32
*** heyongli__ has quit IRC05:33
*** heyongli__ has joined #openstack-infra05:33
*** apoorvad has joined #openstack-infra05:33
*** yolanda has quit IRC05:34
*** yolanda has joined #openstack-infra05:34
*** heyongli__ has quit IRC05:43
*** heyongli__ has joined #openstack-infra05:43
*** sdake_ has quit IRC05:47
*** markvoelker has joined #openstack-infra05:50
*** sdake has joined #openstack-infra05:52
*** kengo_sakai has joined #openstack-infra05:53
*** heyongli__ has quit IRC05:53
*** heyongli__ has joined #openstack-infra05:53
*** mylu has joined #openstack-infra05:54
*** markvoelker has quit IRC05:55
*** Sukhdev has quit IRC05:55
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Add a way to edit local.conf  https://review.openstack.org/31166805:55
*** kengo_sakai has quit IRC05:57
*** salv-orlando has joined #openstack-infra05:58
*** heyongli__ has quit IRC06:03
*** heyongli_ has joined #openstack-infra06:06
*** apoorvad has quit IRC06:08
*** zeih has joined #openstack-infra06:09
*** mylu has quit IRC06:11
*** zeih_ has joined #openstack-infra06:12
*** jtomasek has joined #openstack-infra06:12
*** zeih has quit IRC06:12
*** heyongli_ has quit IRC06:14
*** heyongli_ has joined #openstack-infra06:14
*** thorst has joined #openstack-infra06:16
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/30927306:22
*** thorst has quit IRC06:23
*** dimtruck is now known as zz_dimtruck06:24
openstackgerritMoshe Levi proposed openstack-infra/irc-meetings: Add Moshe Levi as sriov meeting chair  https://review.openstack.org/31147206:24
*** heyongli_ has quit IRC06:24
*** heyongli_ has joined #openstack-infra06:24
*** ramishra has joined #openstack-infra06:26
*** roxanaghe has joined #openstack-infra06:28
*** roxanaghe has quit IRC06:29
openstackgerritBen Nemec proposed openstack/diskimage-builder: Add documentation for dib-lint  https://review.openstack.org/31167106:29
*** heyongli_ has quit IRC06:34
*** heyongli_ has joined #openstack-infra06:35
yolandagood morning06:36
AJaegermorning, yolanda!06:42
openstackgerritMerged openstack-infra/project-config: Retire kolla-mesos project  https://review.openstack.org/31157206:43
yolandai still have not been able to solve the afs issue, seems it gets stuck when doing a release06:43
yolandaso all jobs consuming new pypi packages since 29 april will fail06:44
yolandai will need help from other infra-root to debug the issue06:44
*** heyongli_ has quit IRC06:44
*** heyongli_ has joined #openstack-infra06:45
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: openstack-manuals: Adjust jobs  https://review.openstack.org/31167506:50
AJaegeryolanda: hope everybody is refreshed again after travels today...06:50
*** markvoelker has joined #openstack-infra06:51
AJaegeryolanda: why not make a #status log to mention this, so that others know about it when reading https://wiki.openstack.org/wiki/Infrastructure_Status ?06:52
yolanda++06:52
yolanda#status log AFS mirrors not publishing, they get suck on vos release since 29th April06:53
openstackstatusyolanda: finished logging06:53
*** heyongli_ has quit IRC06:55
*** markvoelker has quit IRC06:55
*** heyongli_ has joined #openstack-infra06:57
*** heyongli_ has quit IRC07:05
*** fedexo has quit IRC07:05
*** heyongli_ has joined #openstack-infra07:05
*** HeOS has joined #openstack-infra07:05
*** daemontool has joined #openstack-infra07:11
*** mylu has joined #openstack-infra07:12
*** yolanda has quit IRC07:12
*** ifarkas has joined #openstack-infra07:12
*** heyongli_ has quit IRC07:15
*** heyongli_ has joined #openstack-infra07:15
*** arxcruz has joined #openstack-infra07:16
*** mylu has quit IRC07:18
*** thorst has joined #openstack-infra07:21
*** nmagnezi has joined #openstack-infra07:21
*** daemontool has quit IRC07:21
*** daemontool has joined #openstack-infra07:21
*** jed56 has joined #openstack-infra07:23
*** heyongli_ has quit IRC07:25
*** heyongli_ has joined #openstack-infra07:26
*** acabot has joined #openstack-infra07:27
*** thorst has quit IRC07:28
*** salv-orlando has quit IRC07:28
*** roxanaghe has joined #openstack-infra07:29
*** vincentll has joined #openstack-infra07:32
*** daemontool has quit IRC07:33
*** tesseract has joined #openstack-infra07:33
*** tesseract is now known as Guest3759007:34
*** abregman_ has joined #openstack-infra07:34
*** roxanaghe has quit IRC07:35
*** abregman has quit IRC07:35
*** heyongli_ has quit IRC07:36
*** heyongli_ has joined #openstack-infra07:36
*** yolanda has joined #openstack-infra07:36
*** daemontool has joined #openstack-infra07:37
*** heyongli_ has quit IRC07:46
*** heyongli_ has joined #openstack-infra07:46
*** gildub has quit IRC07:47
*** markvoelker has joined #openstack-infra07:51
*** abregman_ has quit IRC07:52
*** kengo_sakai has joined #openstack-infra07:54
*** markvoelker has quit IRC07:56
*** heyongli_ has quit IRC07:56
*** heyongli_ has joined #openstack-infra07:56
*** derekh has joined #openstack-infra07:57
*** hashar has joined #openstack-infra07:58
*** kengo_sakai has quit IRC07:59
*** zzzeek has quit IRC08:00
*** pblaho has joined #openstack-infra08:00
*** zzzeek has joined #openstack-infra08:00
*** yamahata has joined #openstack-infra08:01
*** abregman_ has joined #openstack-infra08:05
*** dmk0202 has joined #openstack-infra08:06
*** heyongli_ has quit IRC08:06
*** heyongli_ has joined #openstack-infra08:07
*** lezbar has joined #openstack-infra08:09
*** lezbar has quit IRC08:10
*** lezbar has joined #openstack-infra08:10
*** permalac has joined #openstack-infra08:12
*** heyongli_ has quit IRC08:17
*** heyongli_ has joined #openstack-infra08:17
*** kushal has joined #openstack-infra08:17
*** derekh has quit IRC08:19
AJaegerproject-config cores, could you review https://review.openstack.org/311675 , please? That's a change for openstack-manuals that the docs team would love to see merged soonish. Thanks!08:20
*** abregman_ has quit IRC08:22
*** jistr has joined #openstack-infra08:26
*** thorst has joined #openstack-infra08:26
*** heyongli_ has quit IRC08:27
*** heyongli_ has joined #openstack-infra08:27
*** |-paul-| has joined #openstack-infra08:30
*** thorst has quit IRC08:33
*** jordanP has joined #openstack-infra08:33
*** abregman has joined #openstack-infra08:33
*** heyongli_ has quit IRC08:37
*** heyongli_ has joined #openstack-infra08:37
*** jordanP has quit IRC08:39
*** asettle has joined #openstack-infra08:41
*** tojuvone has quit IRC08:42
*** tojuvone has joined #openstack-infra08:43
openstackgerritNisha Agarwal proposed openstack/diskimage-builder: Install proliantutils in IPA's virtualenv  https://review.openstack.org/29911608:47
*** heyongli_ has quit IRC08:47
*** heyongli_ has joined #openstack-infra08:48
*** kengo_sakai has joined #openstack-infra08:52
*** markvoelker has joined #openstack-infra08:52
*** daemontool has quit IRC08:54
*** nadya has joined #openstack-infra08:54
*** kengo_sakai has quit IRC08:56
*** markvoelker has quit IRC08:57
*** heyongli_ has quit IRC08:58
*** heyongli_ has joined #openstack-infra08:58
*** asettle has quit IRC09:00
*** vincentll has quit IRC09:01
*** xnox has quit IRC09:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Do not run release notes for openstack-manuals/liberty  https://review.openstack.org/31170109:05
*** yaume has joined #openstack-infra09:05
*** jordanP has joined #openstack-infra09:07
*** heyongli_ has quit IRC09:08
*** heyongli_ has joined #openstack-infra09:08
*** nadya has quit IRC09:08
*** kengo_sakai has joined #openstack-infra09:09
*** vincentll has joined #openstack-infra09:16
*** heyongli_ has quit IRC09:18
*** heyongli_ has joined #openstack-infra09:18
*** ccamacho has joined #openstack-infra09:20
jistrHi, can someone please point us to where should we put the change for libffi-devel and openssl-devel installation? I wasn't able to find where the mentioned jenkins_slave.pp file is. https://review.openstack.org/#/c/311217/09:22
*** xnox has joined #openstack-infra09:27
*** tosky has joined #openstack-infra09:28
*** heyongli__ has joined #openstack-infra09:29
*** nadya has joined #openstack-infra09:30
*** sshnaidm has quit IRC09:31
*** heyongli_ has quit IRC09:33
*** thorst has joined #openstack-infra09:33
*** thorst has quit IRC09:38
*** heyongli__ has quit IRC09:39
*** heyongli__ has joined #openstack-infra09:39
*** sshnaidm has joined #openstack-infra09:42
*** nadya has quit IRC09:43
*** chem has joined #openstack-infra09:46
*** heyongli__ has quit IRC09:49
*** heyongli__ has joined #openstack-infra09:49
*** yamamoto has quit IRC09:53
*** markvoelker has joined #openstack-infra09:53
*** pfallenop has quit IRC09:55
*** hashar has quit IRC09:55
*** hashar has joined #openstack-infra09:56
*** markvoelker has quit IRC09:58
*** heyongli__ has quit IRC09:59
*** heyongli__ has joined #openstack-infra09:59
*** pfallenop has joined #openstack-infra10:01
*** HeOS has quit IRC10:04
openstackgerritCarlos Camacho proposed openstack-infra/project-config: TipleO CI dep missing  https://review.openstack.org/31171110:05
*** heyongli__ has quit IRC10:09
*** ramishra has quit IRC10:10
*** heyongli__ has joined #openstack-infra10:10
*** pfallenop has quit IRC10:10
*** kengo_sakai has quit IRC10:11
*** ramishra has joined #openstack-infra10:15
*** vincentll has quit IRC10:15
*** heyongli__ has quit IRC10:20
openstackgerritCarlos Camacho proposed openstack-infra/project-config: TipleO CI dep missing  https://review.openstack.org/31171110:20
*** heyongli__ has joined #openstack-infra10:20
openstackgerritCarlos Camacho proposed openstack-infra/project-config: TipleO CI dep missing  https://review.openstack.org/31171110:22
*** pfallenop has joined #openstack-infra10:22
*** yamahata has quit IRC10:22
*** pfallenop has quit IRC10:28
*** vincentll has joined #openstack-infra10:29
*** heyongli__ has quit IRC10:30
*** heyongli__ has joined #openstack-infra10:30
*** aorourke__ has joined #openstack-infra10:30
*** hashar_ has joined #openstack-infra10:32
*** eliqiao_ has joined #openstack-infra10:33
*** sflanigan_ has joined #openstack-infra10:34
*** sflanigan has quit IRC10:34
*** vincentll has quit IRC10:34
*** pfallenop has joined #openstack-infra10:34
*** vipul- has joined #openstack-infra10:35
*** megm_ has joined #openstack-infra10:35
*** thorst has joined #openstack-infra10:36
*** vincentll has joined #openstack-infra10:36
ccamachoajaeger, hello Andreas, we are getting errors in our CI due to a dep missing error, you have recommended to update the other-requirements file10:36
ccamachobut not able to get it10:36
*** kushal has quit IRC10:36
ccamachocan you elaborate more on that?10:36
*** mnencia_ has joined #openstack-infra10:37
*** aorourke_ has quit IRC10:37
*** eliqiao has quit IRC10:37
*** oanson has quit IRC10:37
*** megm has quit IRC10:37
*** vipuls has quit IRC10:37
*** hashar has quit IRC10:37
*** Guest68119 has quit IRC10:37
*** mnencia has quit IRC10:37
*** mnencia_ is now known as mnencia10:37
*** jefrite has quit IRC10:37
*** oanson has joined #openstack-infra10:38
*** zeus- has joined #openstack-infra10:38
*** chlong has joined #openstack-infra10:38
AJaegerccamacho: other-requirements.txt is documented here http://docs.openstack.org/infra/manual/drivers.html#package-requirements10:39
AJaegerccamacho: if you fix a bug, please add that to the commit message, so that we have the full background on *why* you do what you do.10:39
AJaegerIs this for paramiko, ccamacho?10:40
*** heyongli__ has quit IRC10:40
ccamachoajaeger, yes for paramiko10:40
openstackgerrityolanda.robla proposed openstack-infra/shade: WIP: Add baymodel list call for shade+magnum  https://review.openstack.org/30800910:40
AJaegerI think fungi, pabelanger, and others need to discuss what's the best solution is for this problem.10:40
*** heyongli__ has joined #openstack-infra10:40
*** nadya has joined #openstack-infra10:41
AJaegerif this fails in devstack, you need to add the package requirements for devstack, see the files directory in devstack for list of dependencies10:42
*** jefrite has joined #openstack-infra10:43
*** thorst has quit IRC10:43
ccamachoajaeger, thanks10:45
*** mylu has joined #openstack-infra10:47
*** yolanda has quit IRC10:47
*** vincentll has quit IRC10:49
*** vincentll has joined #openstack-infra10:50
*** heyongli__ has quit IRC10:50
*** heyongli__ has joined #openstack-infra10:51
*** yolanda has joined #openstack-infra10:51
*** markvoelker has joined #openstack-infra10:54
*** rhallisey has joined #openstack-infra10:54
*** zeus- is now known as zeus10:58
*** zeus is now known as Guest2591510:59
*** markvoelker has quit IRC10:59
*** heyongli__ has quit IRC11:01
*** heyongli__ has joined #openstack-infra11:01
* rcarrillocruz waves11:02
* rcarrillocruz starting to rise from dead after 2 stops and 6h layover yesterday at Madrid11:02
*** hashar_ has quit IRC11:02
*** roxanaghe has joined #openstack-infra11:06
* AJaeger waves11:10
rcarrillocruzhope everyone had safe travels11:10
*** roxanaghe has quit IRC11:10
rcarrillocruzwill try to catch up later in the day with HP NOC for the infra cloud status, got access to the vpn but still awaiting for iLO IPs11:10
rcarrillocruzi'll be on and off, today is bank holiday here11:11
*** Na3iL has joined #openstack-infra11:11
*** heyongli__ has quit IRC11:11
*** heyongli__ has joined #openstack-infra11:11
AJaegerrcarrillocruz: relax and recover ;)11:12
rcarrillocruz:P11:13
*** mwagner_lap has joined #openstack-infra11:14
*** yolanda has quit IRC11:16
*** yolanda has joined #openstack-infra11:16
*** thorst has joined #openstack-infra11:19
*** thorst has quit IRC11:21
*** thorst has joined #openstack-infra11:21
*** heyongli__ has quit IRC11:21
*** heyongli__ has joined #openstack-infra11:21
*** thorst has quit IRC11:26
*** yolanda has quit IRC11:29
*** yolanda has joined #openstack-infra11:30
pabelangeryolanda: pretty sure, we are still doing a restore between ord and dfw. we're xfering 12Mb/s between the 211:31
*** thorst has joined #openstack-infra11:31
*** heyongli__ has quit IRC11:31
*** heyongli__ has joined #openstack-infra11:32
*** ldnunes has joined #openstack-infra11:32
*** anteaya has joined #openstack-infra11:37
*** nadya has quit IRC11:37
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add centos-7 to tripleo cloud  https://review.openstack.org/31172111:41
*** heyongli__ has quit IRC11:42
*** heyongli__ has joined #openstack-infra11:42
*** trown|outtypewww is now known as trown11:43
*** e0ne has joined #openstack-infra11:44
yolandapabelanger, so do you think the long time to complete is normal?11:47
yolandaa vos status shows the forwardmulti transaction has been inactive for hours11:47
anteayarcarrillocruz: thanks for the update11:48
*** flepied has joined #openstack-infra11:49
*** woodster_ has joined #openstack-infra11:49
*** raginbajin has quit IRC11:50
pabelangeryolanda: Ya, we have almost 300GB of data to xfter between mirrors. Using 12/Mbps that's about 2.2 days to xfer that11:50
*** heyongli__ has quit IRC11:52
*** heyongli__ has joined #openstack-infra11:53
*** kzaitsev_mb has joined #openstack-infra11:53
*** markvoelker has joined #openstack-infra11:55
*** mylu has quit IRC11:56
*** markvoelker has quit IRC11:59
*** amrith is now known as _amrith_11:59
*** EricGonczer_ has joined #openstack-infra11:59
anteayafor the record I am absolutely exhausted, will be a few days before I am able to get back in a rhythm, last time I felt this exhausted was my sister's wedding12:01
openstackgerritPaul Belanger proposed openstack-infra/zuul: Add usernname configuration option for launcher  https://review.openstack.org/31172512:02
*** heyongli__ has quit IRC12:02
*** heyongli__ has joined #openstack-infra12:02
openstackgerritCarlos Camacho proposed openstack-infra/infra-ansible: TripleO CI ansible deps  https://review.openstack.org/31172612:03
*** markvoelker has joined #openstack-infra12:06
openstackgerritCarlos Camacho proposed openstack-infra/infra-ansible: TripleO CI ansible deps  https://review.openstack.org/31172612:08
AJaegeranteaya: get well soon!12:08
anteayaoh I'm not sick, just very very very very very tired12:08
anteayabut thank you :)12:08
anteayaAJaeger: I see your airplane's windshield wiper issue must have gotten addressed?12:09
*** aysyd has joined #openstack-infra12:09
AJaegeranteaya: yeah, took 1.5 hours to do that - and then 30 mins of paperwork plus 15 mins of refuelling the plan ;(12:10
AJaegeranteaya: So, is there a better wish? Take your time and recover fully?12:10
*** baoli has joined #openstack-infra12:11
anteayaAJaeger: glad you finally got on your way12:11
anteayaAJaeger: yeah, I guess that is a good wish12:11
anteayaboth are great, thank you12:12
AJaegeranteaya: yeah, so was I...12:12
anteayajust the get well soon phrase is usually the response after being informed someone is either ill or injured12:12
AJaegerAt least not spending the night at the airport ;)12:12
anteayano kidding12:12
anteayacongratulations for not spending the night at the airport :)12:12
*** baoli_ has joined #openstack-infra12:12
*** heyongli__ has quit IRC12:12
anteayathere are greeting cards with the phrase 'get well soon' printed on them which you usually send to someone when they are in the hospital12:13
anteayaspeaking of, yolanda, I hope your family is well12:14
AJaegeryeah, knew about that - but didn't know what the best way was to wish you all the best ...12:14
yolandaanteaya, thanks12:14
anteayayolanda: I am sorry you couldn't make it to summit but I understand the circumstances12:14
* AJaeger was at least not creative12:14
anteayaAJaeger: ah thank you12:14
anteayaAJaeger: ha ha ha12:14
yolandawe had better moments, but she is recovering12:14
yolandai have to be at the hospital for 7-10 days more at leat12:14
yolandaleast12:14
*** e0ne has quit IRC12:14
anteayayolanda: understood, glad she is doing well12:15
AJaegeryolanda: ;(12:15
anteayayolanda: blessed healing for her and I hope she continues to do well12:15
anteayayolanda: and take your time12:15
*** baoli has quit IRC12:16
yolandaanteaya, many thanks, much appreciated12:16
*** kgiusti has joined #openstack-infra12:16
*** weshay has joined #openstack-infra12:17
*** heyongli_ has joined #openstack-infra12:17
*** tlian has joined #openstack-infra12:17
*** dtardivel has joined #openstack-infra12:19
anteayayolanda: :)12:20
openstackgerritCarlos Camacho proposed openstack-infra/infra-ansible: Adding dependencies for paramiko 2.0 when installing ansible  https://review.openstack.org/31172612:21
*** heyongli_ has quit IRC12:23
*** heyongli_ has joined #openstack-infra12:23
AJaegerany project-config core willing to push https://review.openstack.org/311701 on their review queue, please? This does not run releasenotes jobs on liberty for openstack-manuals, we only enabled this for mitaka.12:24
AJaegerccamacho, Regarding 311726 let's not add more workarounds but work on a good plan first - and wait for fungi to see what he has to say...12:25
*** dprince has joined #openstack-infra12:25
ccamachosure12:25
*** EricGonczer_ has quit IRC12:30
*** edmondsw has joined #openstack-infra12:33
*** heyongli_ has quit IRC12:33
*** heyongli_ has joined #openstack-infra12:33
*** psilvad has joined #openstack-infra12:35
*** psilvad has quit IRC12:36
*** psilvad has joined #openstack-infra12:36
*** Na3iL has quit IRC12:38
*** timl0101 has joined #openstack-infra12:39
*** jheroux has joined #openstack-infra12:40
*** jheroux has quit IRC12:42
*** jheroux has joined #openstack-infra12:43
*** Hal has joined #openstack-infra12:43
*** Hal is now known as Guest3904112:43
*** heyongli_ has quit IRC12:43
*** heyongli_ has joined #openstack-infra12:43
*** pfallenop has quit IRC12:44
*** gordc has joined #openstack-infra12:45
openstackgerritCarlos Camacho proposed openstack-infra/infra-ansible: Adding dependencies for paramiko 2.0 when installing ansible  https://review.openstack.org/31172612:47
openstackgerritPaul Belanger proposed openstack-infra/zuul: Pin paramiko < 2.0.0  https://review.openstack.org/31162612:48
pabelangerAJaeger: ccamacho: fungi: why we need to pin paramiki<2.0.0 ^12:48
pabelangerregardless of installing missing libs, zuul won't function12:49
ccamachopabelanger, paramiko 2.0 uses cryptography12:49
*** rfolco has joined #openstack-infra12:49
ccamachoand need other deps12:49
cbaderanteaya, question on dib-image build do you know if someone is working on fix for ffi.h error on creation of build?12:49
ccamachocheck https://review.openstack.org/#/q/Ia7e0d64214cfd5412ec236ee4bdd6a076821d258,n,z12:49
*** pfallenop has joined #openstack-infra12:50
pabelangerccamacho: we need to audit paramiko 2.0 and see what is all needed.  I'd rather pin ATM until we can see what all needs to be updated12:50
*** krtaylor has joined #openstack-infra12:51
*** jaypipes has joined #openstack-infra12:51
ccamachopabelanger, question, the dependecies hsould be added in openstack-infra/infra-ansible¬†/¬†setup_env.sh  ?12:51
pabelangerccamacho: they should be added into jenkins_slave.pp12:51
pabelangerin openstack-infra/system-config12:52
pabelangerand / or bindep-fallback.txt12:52
*** heyongli_ has quit IRC12:53
ccamachopabelander, like https://review.openstack.org/#/c/311711/3/jenkins/data/bindep-fallback.txt but jenkins_slave.pp does not exist12:54
*** heyongli_ has joined #openstack-infra12:54
pabelangerccamacho: sorry, jenkins_params.pp and jenkins.pp12:55
zxiiroDo you have to have an entry in default_data.json to appear in stackalytics?12:55
pabelangerAJaeger: I think bindep-fallback.txt might be a good place.  Especially since we want to stop using jenkins_params.pp and jenkins.pp12:55
*** nelsnelson has quit IRC12:56
pabelangerzxiiro: I believe so12:56
ccamachopabelanger, mmm i see12:56
ccamachoin bindep-fallback libffi is already included12:57
pabelangerWe need to do some optimizations for DIBs. Since bindep-fallback only runs when jenkins launches.12:58
pabelangerI am adding a package-install to the nodepool-base element13:00
*** sdake has quit IRC13:00
pabelangerthat should help13:00
*** _ari_ has joined #openstack-infra13:00
*** hrybacki has quit IRC13:01
*** hrybacki has joined #openstack-infra13:01
*** sdake has joined #openstack-infra13:01
ccamachopabelanger, thanks13:03
ccamachoI will abandone mi tests then If you are getting into install the packages13:04
*** heyongli_ has quit IRC13:04
*** sdake_ has joined #openstack-infra13:04
*** heyongli_ has joined #openstack-infra13:04
anteayacbader: I have no idea, my suggest is read the channel backscroll for yourself: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/13:05
*** sdake has quit IRC13:06
*** aysyd has quit IRC13:06
*** aysyd has joined #openstack-infra13:07
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add new zuul dependencies  https://review.openstack.org/31173413:07
pabelangerthat should do it, just testing with ubuntu-trusty and centos-7 now13:08
*** _amrith_ is now known as amrith13:09
*** deadnull_ has joined #openstack-infra13:10
*** samuelBartel has joined #openstack-infra13:11
*** rbradfor_home is now known as rbradfor13:11
*** Goneri has joined #openstack-infra13:12
*** kengo_sakai has joined #openstack-infra13:12
AJaegerpabelanger: and added to devstack/files13:12
AJaegerzxiiro: no, you should appear directly there...13:13
*** tpsilva has joined #openstack-infra13:13
*** heyongli_ has quit IRC13:14
*** heyongli_ has joined #openstack-infra13:14
*** rkukura has quit IRC13:15
AJaegerzxiiro: http://stackalytics.com/?user_id=zxiiro13:16
*** kengo_sakai has quit IRC13:17
*** julim has joined #openstack-infra13:19
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: workaround to deploy libffi-devel and  openssl-devel  https://review.openstack.org/31173713:20
zxiiroAJaeger: thanks, I was asking in regards to one of my interns not showing up but I just realized the issue. We had to search Mitaka since his patches were merged a few months ago.13:20
*** julim has quit IRC13:20
*** pilgrimstack has joined #openstack-infra13:21
*** heyongli_ has quit IRC13:24
*** heyongli_ has joined #openstack-infra13:24
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: workaround to deploy libffi-devel and  openssl-devel  https://review.openstack.org/31173713:25
*** pradk has joined #openstack-infra13:29
*** baoli_ has quit IRC13:30
cbaderanteaya, thank you13:30
fungiAJaeger: pabelanger: i approved the paramiko<2.0.0 pin in zuul for now until someone has time to dig into making it compatible. as a side effect that will also avoid us needing libffi-dev and openssl-dev for a little while until we decide if we want to have dib installs for /usr/zuul-env use prebuilt wheels of cryptography from our wheel mirror13:31
*** jaosorior has joined #openstack-infra13:32
*** yolanda has quit IRC13:32
AJaegerfungi, I've seen some changes flying by for the issue like 311737 and 311726 - and do not know what's the best way is and neither have time to dig...13:32
*** doug-fish has joined #openstack-infra13:33
openstackgerritMerged openstack-infra/zuul: Pin paramiko < 2.0.0  https://review.openstack.org/31162613:34
pabelangerfungi: works for me.  Adding wheels to dibs would be great, but likely a longer term goal.  Also testing new pkg-map for nodepool-base element ATM13:34
*** heyongli_ has quit IRC13:34
*** heyongli_ has joined #openstack-infra13:35
*** links has quit IRC13:35
*** yolanda has joined #openstack-infra13:36
fungii need to head to an appointment but should be back in a couple hours13:37
*** otherwiseguy has joined #openstack-infra13:37
pabelangerack13:37
EmilienMpabelanger: when you have time this week for the tarballs, please ping me13:38
pabelangerEmilienM: Ha ha, I was just about to ping you about it13:38
EmilienMok, here is the etherpad: https://etherpad.openstack.org/p/fpHDMxrKuQ13:38
pabelangerEmilienM: great, will start at the top again13:38
EmilienMlet's try one command, yeah13:38
pabelanger++13:38
*** kushal has joined #openstack-infra13:39
*** jaosorior has quit IRC13:41
*** tonytan4ever has joined #openstack-infra13:44
*** heyongli_ has quit IRC13:45
*** heyongli_ has joined #openstack-infra13:45
*** nelsnelson has joined #openstack-infra13:46
*** |-paul-| has quit IRC13:47
EmilienMpabelanger: puppet-aodh tarball name is wrong13:47
EmilienMpuppet-aodh.tar.gz should be with the tag13:47
EmilienMsee http://tarballs.openstack.org/puppet-aodh/13:47
pabelangerEmilienM: http://logs.openstack.org/3f/3f0e8910b2703ed7f23ec22a8ec86e575c5f922c/release/puppet-aodh-tarball/manual_000/console.html13:47
pabelangerthat is the log file13:47
EmilienMok, /me looking13:47
anteayacbader: welcome13:47
pabelangerEmilienM: just testing puppet-ceilometer now, then stopping13:48
EmilienMhttp://logs.openstack.org/3f/3f0e8910b2703ed7f23ec22a8ec86e575c5f922c/release/puppet-aodh-tarball/manual_000/console.html#_2016-05-02_13_45_30_19513:48
anteayacbader: the third party meeting is in the #openstack-meeting channel at 15:00 utc today, you could consider attending13:48
pabelangerEmilienM: also, there is an issue with publisher13:48
cbaderanteaya, I have been attending just not saying anything yet just watching.13:48
EmilienMpabelanger: the "dist/*.whl" thing?13:49
*** ccamacho has quit IRC13:49
anteayacbader: okay, glad you have been attending13:49
*** dmsimard|afk is now known as dmsimard13:49
pabelangerEmilienM: ya13:49
pabelangerEmilienM: which causes jenkins to report a failure13:49
cbaderanteaya, I have this thread up on a monitor all day watching and looking at commits. I has been a great help.13:50
anteayacbader: oh I'm so glad to hear that13:50
anteayawell done13:50
cbaderanteaya, I spent all weekend trying to get my ci back up after dib-images crashed thursday night with no luck. so looking for some clue. will ask today.13:51
anteayacbader: great, I look forward to seeing you in the meeting13:51
EmilienMpabelanger: the thing is {name}-branch-tarball JJB is doing a mv, so the branch tarballs have the right name. But when it's regular tarball job, it does not perform any mv, so I wonder how to put the tag in the name13:52
cbaderanteay, found fix for vms and they are up but dib is still bad13:52
anteayahopefully one of the other operators might have some insight13:52
anteayacbader: understood, well done on finding a fix for your vms13:52
cbaderanteaya, andrew has been a help13:52
EmilienMpabelanger: if you look at jenkins/jobs/python-jobs.yaml13:52
anteayacbader: I'm glad to hear that13:52
*** mriedem has joined #openstack-infra13:53
pabelangerEmilienM: http://logs.openstack.org/71/71f6212daf6224d162fa45d136a2092e79e28437/release/puppet-aodh-tarball/manual_000/console.html#_2016-05-02_13_41_27_69313:54
*** Guest25915 is now known as zeus`13:54
pabelangerEmilienM: looks like puppet module build . is naming the file correctly13:54
*** sigmavirus24_awa is now known as sigmavirus2413:54
pabelangerEmilienM: do you have a recent branch-tarball log?13:55
*** heyongli_ has quit IRC13:55
*** hashar has joined #openstack-infra13:55
*** heyongli_ has joined #openstack-infra13:55
pabelangerEmilienM: Oh, I see the issue13:56
*** e0ne has joined #openstack-infra13:56
pabelangerEmilienM: http://logs.openstack.org/39/39d3950cf9a4b07153667c48cd53c041e8b2d483/release/puppet-ceilometer-tarball/manual_000/console.html#_2016-05-02_13_48_49_903 show why we need to rename13:56
EmilienMright13:56
pabelangerso 2 questions13:56
EmilienMwe need to skip the rename if we run a tag job13:57
pabelanger1): why don't you change the metadata.json name from openstack-ceilometer to puppet-ceilometer?13:57
pabelanger2): we need to add version into for the move command13:57
*** inc0 has joined #openstack-infra13:58
EmilienMfor 1), I think puppetlabs named this module because we have the https://forge.puppetlabs.com/openstack namespace13:58
EmilienMand yes for 2), but how to find if whether or not we are running a tag job13:58
pabelanger1) okay13:59
dimsAnyone have an idea on when we can get our pypi mirroring back on its feet?14:00
EmilienMbecause the mv is useful for the branch-tarball jobs14:00
*** mwagner_lap has quit IRC14:00
pabelangerEmilienM: so, puppet module build . is doing the right thing, we just need to sed openstack with puppet14:00
dimsfungi : clarkb : yolanda : ^^14:00
EmilienMpabelanger: that's what I'm doing, see jenkins/scripts/run-tarball.sh14:00
*** mtanino has joined #openstack-infra14:00
*** oanson has quit IRC14:01
pabelangerdims: we are waiting for the mirror to restore.  300GB @ 12Mbps is like 2.5 days14:01
dimspabelanger : ah i see. do we hold releases until then?14:01
pabelangerEmilienM: right, let me test something locally14:01
pabelangerdims: AFS won't let us release, that is the issue ATM.14:02
dimsack pabelanger14:02
pabelangerSo, if I understand correctly, afs is reverting to the last know good state14:02
mordredwe reverted something in AFS?14:02
dimspabelanger : ack thanks i'll tell ttx and dhellmann that we'll have to wait until things are back to normal to make releases14:02
mordredalso, good morning14:03
*** kzaitsev_mb has quit IRC14:03
pabelangermordred: I believe the vos release was killed due to some confusion between roots14:03
mordredyay!14:03
mordredwell, hopefully jeblair will be up soon14:03
pabelangerI _think_ afs is restoring from last state. But need some help to confirm that14:04
anteayamordred: good morning14:04
*** heyongli_ has quit IRC14:05
*** jamesmcarthur has joined #openstack-infra14:05
*** heyongli_ has joined #openstack-infra14:05
bswartzinfra team: I'm trying to figure out how to create a specs repo...14:06
bswartzI looked at the history of project-config and I see some examples of other projects that have done it14:07
bswartzbut I don't understand where the initial content for the repo comes from14:07
anteayabswartz: have you read this? http://docs.openstack.org/infra/manual/creators.html14:07
*** baoli has joined #openstack-infra14:07
bswartzanteaya: yes that page came up in my googling and I didn't see anything specific about creation of the specs repo14:08
anteayathat is the guide to create any repo14:08
anteayajust call yours specs-foo14:08
bswartzokay I'll peruse that page in more detail14:08
anteayaand you probably don't need a launchpad account for bugs against a specs repo14:08
anteayabut up to you14:08
bswartzit would be openstack/manila-specs14:08
anteayaI figured that was what you were heading towards14:09
mordredmorning anteaya !14:09
anteaya:)14:09
anteayasorry s/specs-foo/foo-specs14:09
pabelangerEmilienM: okay, I need a log for branch-tarball please14:09
pabelangerEmilienM: have a patch, just need to confirm it will work14:09
EmilienMpabelanger: let me find one14:10
*** xarses_ has quit IRC14:10
*** jamesmcarthur has quit IRC14:10
EmilienMpabelanger: http://logs.openstack.org/e3/e3c649f0877e2deb594864b2aacf5fc480cd4ffa/post/puppet-ceilometer-branch-tarball/14:10
*** sdake_ is now known as sdake14:11
*** ddieterly has joined #openstack-infra14:12
*** mtanino has quit IRC14:12
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge  https://review.openstack.org/31175014:12
pabelangerEmilienM: anything newer?14:12
EmilienMpabelanger: not afik14:13
EmilienMpabelanger: let me search14:13
*** e0ne has quit IRC14:13
EmilienMpabelanger: http://logs.openstack.org/a0/a0f957f2e85f8bd533b82675dd15aca0d48ba4f4/post/puppet-neutron-branch-tarball/14:13
EmilienMfrom yesterday!14:13
yolandamordred, so about afs... there is one restore task that seems active, and there is one forwardmulti task that seems inactive since this morning. No idea if that's expected or not14:14
*** e0ne has joined #openstack-infra14:14
EmilienMpabelanger: we still have the pip install ansible thing broking our CI, should we wait for a new DIB build & upload? or find a workaround?14:15
*** heyongli_ has quit IRC14:15
*** nmagnezi has quit IRC14:15
*** heyongli_ has joined #openstack-infra14:16
bswartzanteaya: would it be best to create a skeleton of manila-specs on my personal github and then import that using the "upstream" yaml tag? or could we create a completely empty repo and build the skeleton after the repo is created?14:16
pabelangerEmilienM: we pinned paramiko < 2.0.0 new images builds will pick it up shortly14:18
*** rbrndt has joined #openstack-infra14:18
EmilienMpabelanger: cool14:19
clarkbpabelanger: but not for ansible right?14:19
yolandamordred, can you take a look at https://review.openstack.org/311610 ? magnum+shade jobs again14:19
pabelangerclarkb: EmilienM: oh, wait. ansible is broken too?14:19
openstackgerrityolanda.robla proposed openstack-infra/shade: Add baymodel list and search call for shade+magnum  https://review.openstack.org/30800914:19
EmilienMyeah14:19
clarkbpabelanger: yes, because it depends on paramiko if I understand correctly14:19
*** mwagner_lap has joined #openstack-infra14:19
pabelangerwhere is that broken?14:20
pabelangeronly knew of zuul14:20
EmilienMhttp://logs.openstack.org/50/311750/1/check-tripleo/gate-tripleo-ci-f22-ha/01518a0/console.html14:20
clarkbdevstack-gate is my guess bht I am still half asleep14:20
EmilienMclarkb: good guess ;-) it's in there.14:20
pabelangerI am testing another patch to add depends for nodepool-base element, but still testing14:20
*** xyang1 has joined #openstack-infra14:22
pabelangerguess we need to add ansible to wheel mirror14:23
pabelangeror ask #ansible to upload a wheel14:23
clarkbor install the libffi-dev and other deps14:23
pabelangerI don't have bandwidth atm to check on it let me finish up this14:23
clarkbor pin whatever it is in ansible install14:24
*** esker has joined #openstack-infra14:25
*** heyongli_ has quit IRC14:26
EmilienMclarkb: could we just install the dep in devstack-gate in the meantime?14:26
*** heyongli_ has joined #openstack-infra14:26
AJaegerbswartz: starting from empty repo should just work fine...14:27
clarkbEmilienM: that would be a quick workaround and should work14:27
AJaegerbswartz: run the openstack-dev/specs-cookiecutter to set it up14:27
*** ajmiller has joined #openstack-infra14:27
*** dprince has quit IRC14:29
sdakehey quick question14:31
sdakeis it possible via acls to restrict +2's to limited directories in a repo14:31
clarkbno14:32
sdakerelated to kolla repo and the idea of a kolla-k8s-core review team14:32
sdakeclarkb thanks14:32
openstackgerritPaul Belanger proposed openstack-infra/project-config: Update puppet logic for run-tarball.sh  https://review.openstack.org/31175414:32
*** ajmiller has quit IRC14:32
clarkbwe can restrict by git refs not by file path14:32
pabelangerEmilienM: okay ^ should do what we need. But need to test it14:32
sdakeclarkb ok thanks14:32
*** ajmiller has joined #openstack-infra14:32
jeblairmordred: i'm up14:32
sdakeso the onl yway to restrict is raelly trust14:33
anteayajeblair: good morning14:33
jeblairmordred: i have not caught up on scrollback14:33
anteayabswartz: you can create a completely empty repo and build everything in the repo after it is created, that is the way I would suggest14:34
AJaegersdake: yes, trust. And keep in mind that reverting is always an easy option.14:34
openstackgerritEmilien Macchi proposed openstack-infra/devstack-gate: devstack-vm-gate-wrap: deploy libffi-dev & openssl-dev  https://review.openstack.org/31175514:34
*** salv-orlando has joined #openstack-infra14:34
EmilienMclarkb: ^ it probably needs review & adjustment14:34
*** burgerk has joined #openstack-infra14:35
*** heyongli_ has quit IRC14:36
*** heyongli_ has joined #openstack-infra14:36
EmilienMpabelanger: commented14:38
pabelangerEmilienM: replied14:40
pabelangerI also have to run an errand for a hour, will be back shortly14:40
*** jang has joined #openstack-infra14:40
EmilienMpabelanger: you missed the "fi", line 3314:41
EmilienMpabelanger: no problem, ttyl14:41
jeblairi get the feeling that some folks want me to help with something about afs mirrors, but i'd really appreciate a summary of the issue.14:41
*** deadnull_ is now known as _deadnull14:42
*** flepied has quit IRC14:43
anteayajeblair: my read of backscroll has led me to the conclusion that yolanda is the only one who has a handle on the afs issue, she seems to have seen something but isn't quite certain what action to take to fix it14:44
*** xarses_ has joined #openstack-infra14:44
anteayaand I am unable to summarize the 'something'14:44
jeblairi also saw that someone said something may have gone wrong due to 'confusion between roots'14:44
jeblairwhich makes me especially hesitant to wade in without someone actually explaining the problem14:45
anteayaI saw that as well14:45
anteayaI can understand your perspective14:45
anteayaindeed there is a lack of detail on this issue14:45
*** ramishra has quit IRC14:46
*** heyongli_ has quit IRC14:46
*** heyongli_ has joined #openstack-infra14:46
openstackgerritPaul Belanger proposed openstack-infra/project-config: Update puppet logic for run-tarball.sh  https://review.openstack.org/31175414:47
*** ramishra has joined #openstack-infra14:47
anteayaso, for deprecating the sprints page when we get to it: https://wiki.openstack.org/wiki/Sprints14:47
anteayaa page in a repo should be pretty straightforward14:47
anteayabut what repo?14:47
anteayanew repo? existing repo?14:48
*** asselin has joined #openstack-infra14:48
anteayavirtual sprints could probably go into the same repo, on a separate page: https://wiki.openstack.org/wiki/VirtualSprints14:48
AJaegeranteaya: for scheduling? irc-meeting ;)14:48
anteayagood point14:48
AJaegeranteaya: seriously: I don't think we have something that really fits...14:49
anteayaagreed14:49
anteayawe might have to create something14:49
anteayabut creating a new repo per page or two might be a poor use of new repos14:49
mordredjeblair: I do not have any additional clarity and agree with anteaya's read on the backscroll. I have not begun attempting to investigate the situation for largely the same reason14:49
AJaegerthe nice thing about the wiki is the self-service14:49
anteayaall the things from the wiki repo?14:49
anteayaAJaeger: so is gerrit14:49
AJaegeranteaya: gerrit needs at least one core to review. Which has its advantages but can be a burden as well...14:50
yolandahi jeblair, so afs seems to have a problem since 29 of april, failing with vos release14:50
jeblairi did check the log, but only saw the problem statement: https://wiki.openstack.org/wiki/Infrastructure_Status14:50
anteayabut the patch is in gerrit14:50
jeblairyolanda: yes, that much i know :)14:50
jeblairyolanda: thanks for the log entry :)14:50
*** ayoung has joined #openstack-infra14:50
anteayaAJaeger: folks can read the patch prior to it being merged14:50
yolandawhen i checked, no vos release process was running, but even with that, the volumes were locked14:51
yolandai restarted the fileservers and then release ran again, but it's there forever. vos status shows a restore task for one of the mirrors, that seems to be active, and one forwardmulti task in another, that seems to be inactive14:51
yolandai don't know how to troubleshoot that14:51
AJaegeranteaya: yes, agreed. I don't have a strong opinion right now, to me it looks like we should write this one down and see what other information needs to move from the wiki and then decide what to do with it.14:52
AJaegeranteaya: but if you have a strong opinion, go for it;)14:52
anteayaAJaeger: fair enough14:52
anteayaI am so out of strong opinons today14:53
* AJaeger as well...14:53
*** salv-orlando has quit IRC14:53
anteayamy cat is back in my lap and I'm trying to avoid strong opinions for as long as I can14:53
*** reed has quit IRC14:53
*** reed has joined #openstack-infra14:53
anteayaand just enjoy my cat14:53
jeblairyolanda: somebody in scrollback said something about killing or deleting locks or vos release commands because of confusion between roots.  do you know what that was about?14:54
yolandajeblair, i was working with pabelaner on that yesterday, he may be talking about that14:55
jeblairyolanda: yes;  what was the confusion that happened?14:55
yolandathere was one vos_release running when i restarted the service, then we were talking about that release should have timeout in 30 minutes, and we killed the process and restarted again14:56
yolandaso looks as the process started again yesterday14:56
yolandait doesn't seem to timeout after 30 minutes, but not sure if that's progressing properly now14:56
sdakeAJaeger ia gree revert wins on bad actors :)14:56
*** heyongli_ has quit IRC14:56
sdakeplus removal from the acls if needed14:56
*** heyongli_ has joined #openstack-infra14:57
yolandajeblair, one thing that happened is that afs ran out of quota, that may have caused the initial problem on 29 april14:57
sdakeAJaeger can we have a different acl for the same repo ?14:57
sdakethe reason iask is one of the propoosals is to have kolla-core and kolla-k8s-core with same permissions on the repo but with different rights in the project14:57
jeblairyolanda: the vos release is not in a timeout -- it is permitted to take as long as it takes; however, if it takes too long, its authentication tokens will expire and will not clean up after itself and get stuck in a state like this14:57
anteayasdake: how are you differenciating between permissions and rights?14:58
*** _deadnull is now known as deadnull_14:58
sdakerights = voting rights for policy14:58
jeblairyolanda: running out of quota on one volume (like ceph) should never cause a problem with another (like pypi); and it should also not cause a problem with vos release14:58
sdakepermissions = +2/-2 on commits to the repo14:58
anteayasdake: do you mean approval?14:58
*** deadnull_ has quit IRC14:58
jeblairyolanda: from looking at the logs on the 29th, it seems like there was a network issue between dfw and ord, and the vos release failed.14:58
*** mtanino has joined #openstack-infra14:59
notmorgano/ mornin all14:59
sdakeanteaya apologies lost context there, could you add some :)14:59
AJaegersdake: yes, you could have two groups on the repo; example: both groups can +2 but onl yone can +A.14:59
anteayasdake: right now most acls have +2 and +1 workflow for core reviewers14:59
yolandajeblair, what can we do on that cases? tried vos release several times, and it seems to stuck all the time14:59
anteayasdake: are you wanting a different group to have +1 workflow?14:59
sdakeAJaeger oh winning14:59
sdakeanteaya thats a good solution i didn't know existed15:00
jeblairyolanda: i suspect that because of that, subsequent vos release actions are requiring more time to complete -- either because it has more work to do to clean up or because the underlying volume has changed enough that it is taking longer to complete the transfer than our authentication tokens allow15:00
*** mriedem is now known as mriedem_meeting15:00
AJaegersdake: let me give you a reference...15:00
yolandajeblair, seems it's doing a complete release, not incremental15:00
anteayasdake: I was asking if that was what you wanted, not suggesting you do that15:00
jeblairyolanda: so i think we need to run a manual vos release on the fileserver in screen, so that we can avoid it being interrupted15:00
ansiwenAJaeger: do I have to change the upper-constraints.txt as well?15:00
AJaegersdake: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/trove-dashboard.config has two groups - but both have the same permissions15:00
jeblairyolanda: it always says "complete release" even if it's an incremental one.15:01
sdakeanteaya yes got it but it seems like a sensible workaround15:01
AJaegeransiwen: for BAbel? Yes.15:01
anteayasdake: workaround for what, what are you trying to do?15:01
yolandajeblair, can you help with that? my wifi is so bad to run anything  on servers15:01
AJaegersdake: you can adopt that trove-dashboard setup and do something similar if needed.15:01
sdakebtw, can infra handle the case where kolla has 80 gates which are short running (20 minutes)15:01
AJaegersdake: But my recommendation : KISS.15:01
AJaegerand social contract ;)15:01
jeblairyolanda: well, right now there is one running on mirror-update15:01
sdakeAJaeger right - i like social contract model15:01
AJaegerI doubt you have many problems.15:02
sdakeAJaeger i agree15:02
anteayasdake: do you mean jobs?15:02
AJaegersdake: On the other hand, looking at the k8s thread: You had many many people that want to be cores - that's really a lot.15:02
yolandajeblair, yes, running since this morning, and vos status is showing that two tasks i mentioned15:02
anteayainfra has one and only one gate pipeline15:02
ansiwenAJaeger: yes, for babel15:02
*** EricGonczer_ has joined #openstack-infra15:02
AJaegersdake: Add none, see who reviews, and speed add after one month the first three ;)15:02
sdakeAJaeger yes - but i suspect half of them will dorp out in the first 30-60 days15:02
jeblairyolanda: i only see the forwardmulti transaction15:03
sdakeAJaeger yar - but we want to move super fast15:03
ansiwenAJaeger: i'm confused, because upper-constrains still points to 2.2.0, but 2.3.4 is out15:03
yolandarestore finished?15:03
*** mwagner_lap has quit IRC15:03
jeblairyolanda: oh the restore is on ord15:03
yolandayes15:03
ansiwenAJaeger: probably I don't understand the meaning of that file15:03
jeblairyolanda: i believe those are two sides of the same transaction15:04
AJaegeransiwen: 2.3.3 had problems, so we didn't go up...15:04
yolandaand the forwardmulti says not active since some hours15:04
AJaegeransiwen: we blacklisted 2.3.2, 2.3.3 - and then it was summit and nobody approved the go to 2.3.4 ;)15:04
jeblairyolanda: that is -- dfw is transmitting a copy of a volume (forwardmulti) to ord (restore)15:04
openstackgerritEmilien Macchi proposed openstack-infra/devstack-gate: devstack-vm-gate-wrap: handle ansible dependencies  https://review.openstack.org/31175515:04
*** mriedem_meeting is now known as mriedem15:04
*** samuelBartel has quit IRC15:04
yolandajeblair, so you think everything is fine then?15:04
*** arxcruz has quit IRC15:04
AJaegeransiwen: submit the change and add me as reviewer ;)15:04
ansiwenAJaeger: so do I remove the antry altogether, or set it to the current release (2.3.4)15:04
AJaegeransiwen: in upper-constraints: Set it to 2.3.415:05
ansiwenAJaeger: ok15:05
tonytan4everHi folks,  I encountered a problem when installing oslo-config: AssertionError: oslo-config .dist-info directory not found15:05
tonytan4everIs there anything changed with oslo-config ?15:05
*** EricGonc_ has joined #openstack-infra15:05
AJaegertonytan4ever: I suggest to ask on the oslo channel...15:05
tonytan4everOkay thank.15:06
AJaegertonytan4ever: #openstack-oslo15:06
jeblairyolanda: using iftop, i can see an 11mbit stream from dfw to ord on the afs3-volser port.  so i believe that the current vos release is still running.15:06
*** heyongli_ has quit IRC15:07
yolandaok so just slowness caused by a wrong volume state then?15:07
jeblairyolanda: if it finishes before its tokens expire, everything will be fine.  if it takes longer than that, we will probably need to manually vos release on the fileserver15:07
*** heyongli_ has joined #openstack-infra15:07
yolandaok understood15:07
jeblairyolanda: i suspect we may have a *lot* of changes in pypi because of the pathname change...15:07
jeblairso we may have a couple of problems compounding -- a lot of changes to sync on top of a failed release at an unlucky time.15:08
yolandajeblair, what is length of token expiration?15:08
jeblairyolanda: 10 hours by default, i think?15:08
yolandarelease would take more...15:09
openstackgerritSven Anderson proposed openstack/requirements: Require Babel>=2.3.4  https://review.openstack.org/31176115:09
*** EricGonczer_ has quit IRC15:09
jeblairyolanda: it looks like that is about 36 minutes from now, yeah?15:09
*** hrybacki_ has joined #openstack-infra15:09
jeblairyolanda: so i think we should expect it to fail15:09
*** simonmcc_ has joined #openstack-infra15:09
*** rackertom_ has joined #openstack-infra15:10
*** |GonZo2000| has joined #openstack-infra15:10
jeblairyolanda: i think we should disable puppet on mirror-update, then disable the bandersnatch cron entry, then wait for it to fail, then manually grab the bandersnatch lock, then vos release on the fileserver in screen, then re-enable the cronjob and puppet on mirror-update15:10
jeblairthat way we have control over the vos release (because of our holding the lockfile)15:11
*** burgerk_ has joined #openstack-infra15:11
*** jaranovich_ has joined #openstack-infra15:11
*** petems_ has joined #openstack-infra15:11
*** charz_ has joined #openstack-infra15:11
jeblairand when it's done, we can run bandersnatch and vos release manually until we are happy15:11
*** EricGonc_ has quit IRC15:11
jeblairand, for the future to make this more robust -- i think we should stop doing vos releases on the mirror worker, and instead have a gearman job running on the fileserver to do them, so they don't timeout15:12
jeblairif that sounds good, i can do the steps i outlined above15:12
*** f1ller has joined #openstack-infra15:12
yolandajeblair, a gearman job instructed by zuul?15:13
*** r-mibu_ has joined #openstack-infra15:13
jeblairyolanda: that would be ideal, but doesn't currently work for the mirror-update jobs (though it would for the wheel mirror jobs)15:13
*** links has joined #openstack-infra15:14
*** links has quit IRC15:14
jeblairyolanda: an alternative would actually be to make a job queue in afs (files in a directory)15:14
*** bryan_att_ has joined #openstack-infra15:14
*** spiffxp_ has joined #openstack-infra15:14
* yolanda needs to learn more about afs, just reading about it these days15:14
yolandajeblair, your steps look sane to me15:14
*** cargonza_ has joined #openstack-infra15:15
*** rackertom has quit IRC15:15
jeblairhttps://etherpad.openstack.org/p/fix-afs15:15
*** rackertom_ is now known as rackertom15:15
*** phschwartz_ has joined #openstack-infra15:15
jeblairokay, i will start on those now15:15
*** lindsayk- has joined #openstack-infra15:15
nibalizergod morning15:15
nibalizerer good morning eep15:16
* nibalizer nits jeblair's etherpad15:16
*** mtreinish_ has joined #openstack-infra15:16
* jeblair names nibalizer in etherpad15:16
*** dirk_ has joined #openstack-infra15:17
*** mriedem1 has joined #openstack-infra15:17
nibalizerhah15:17
*** heyongli_ has quit IRC15:17
*** tpsilva_ has joined #openstack-infra15:17
nibalizerin all seriousness i can run this procedure if you want15:17
*** sulo_ has joined #openstack-infra15:17
*** ashtokolov_ has joined #openstack-infra15:17
jeblaircool15:18
*** cfarquhar has joined #openstack-infra15:18
*** cfarquhar has quit IRC15:18
*** cfarquhar has joined #openstack-infra15:18
jeblairi have done the first 2 steps15:18
jeblairnext is wait...15:18
*** fdegir_ has joined #openstack-infra15:18
jeblairso i think it will be good for others to be able to pick up the process15:18
jeblairso yeah, let's coordinate in here and on the etherpad15:18
jeblair#status log steps to fix the pypi mirror problem in progress: https://etherpad.openstack.org/p/fix-afs15:19
openstackstatusjeblair: finished logging15:19
jeblairi'm going to grab breakfast15:19
*** maximov_ has joined #openstack-infra15:19
*** psachin has quit IRC15:20
*** dangers has joined #openstack-infra15:20
*** davidlenwell has quit IRC15:20
yolandathanks for helping with that, using my mobile connection now so i cannot volunteer to run anything essential15:20
*** skolekonov_ has joined #openstack-infra15:20
*** doug-fis_ has joined #openstack-infra15:21
nibalizeryah thanks jeblair15:21
*** mnencia_ has joined #openstack-infra15:21
*** charz has quit IRC15:21
*** dtantsur|afk has quit IRC15:21
*** hrybacki has quit IRC15:21
*** simonmcc has quit IRC15:21
*** GonZo2000 has quit IRC15:21
*** mancdaz has quit IRC15:21
*** jaranovich has quit IRC15:21
*** petems has quit IRC15:21
*** lindsayk has quit IRC15:21
*** mancdaz has joined #openstack-infra15:21
*** filler_ has quit IRC15:21
*** bryan_att has quit IRC15:21
*** briancline has quit IRC15:21
*** dirk has quit IRC15:21
*** briancline has joined #openstack-infra15:21
*** burgerk has quit IRC15:21
*** whoops has quit IRC15:21
*** clarkb has quit IRC15:21
*** r-mibu has quit IRC15:21
*** skolekonov|away has quit IRC15:21
*** tpsilva has quit IRC15:21
*** cargonza has quit IRC15:21
*** mtreinish has quit IRC15:21
*** mnencia has quit IRC15:21
*** rcernin has quit IRC15:21
*** cfarquhar_ has quit IRC15:21
*** msuriar has quit IRC15:21
*** phschwartz has quit IRC15:21
*** spiffxp has quit IRC15:21
*** ashtokolov has quit IRC15:21
*** fdegir has quit IRC15:21
*** maximov has quit IRC15:21
*** sulo has quit IRC15:21
*** briancline has quit IRC15:21
*** weshay has quit IRC15:21
*** dangers_` has quit IRC15:21
*** mancdaz has quit IRC15:21
*** zul has quit IRC15:21
*** lindsayk_ has quit IRC15:21
*** reed has quit IRC15:21
*** timfreund has quit IRC15:21
*** mriedem has quit IRC15:21
*** jlvillal has quit IRC15:21
*** mtreinish_ is now known as mtreinish15:21
*** jaranovich_ is now known as jaranovich15:21
*** dtantsur has joined #openstack-infra15:21
*** jlvillal has joined #openstack-infra15:21
*** reed has joined #openstack-infra15:22
*** doug-fish has quit IRC15:22
*** dtantsur has quit IRC15:22
*** dtantsur has joined #openstack-infra15:22
*** mnencia_ is now known as mnencia15:22
*** rcernin has joined #openstack-infra15:22
*** simonmcc_ is now known as simonmcc15:22
*** briancline has joined #openstack-infra15:22
*** heyongli_ has joined #openstack-infra15:22
*** weshay has joined #openstack-infra15:22
*** tpsilva_ is now known as tpsilva15:22
*** hrybacki_ is now known as hrybacki15:23
*** maximov_ is now known as maximov15:23
*** ashtokolov_ is now known as ashtokolov15:23
*** mancdaz has joined #openstack-infra15:23
*** dirk_ is now known as dirk15:23
*** fdegir_ is now known as fdegir15:23
*** cargonza_ is now known as cargonza15:23
*** zul has joined #openstack-infra15:24
*** petems_ is now known as petems15:24
*** bryan_att_ is now known as bryan_att15:24
*** kmartin has joined #openstack-infra15:25
*** lindsayk_ has joined #openstack-infra15:26
*** heyongli_ has quit IRC15:27
jeblairinfra-root: my availability will be severely limited this week, sorry.15:27
*** amitgandhinz has joined #openstack-infra15:28
*** heyongli_ has joined #openstack-infra15:28
*** davidlenwell has joined #openstack-infra15:28
*** armax has joined #openstack-infra15:28
openstackgerrityolanda.robla proposed openstack-infra/shade: Add baymodel list, get and search call for shade+magnum  https://review.openstack.org/30800915:29
*** nadya has joined #openstack-infra15:29
anteayajeblair: hope it is because you are doing things you enjoy15:29
*** rcernin has quit IRC15:30
jeblairAJaeger, pleia2, StevenK: i got the BEST email on friday:  http://paste.openstack.org/show/495881/   :)15:30
cbadermmedvede, I can't get into the openstack-third-party-ci chat.15:30
StevenKjeblair: \o/15:30
mmedvedecbader: ok, can discuss here15:30
StevenKI guess I can destroy my Transifex account too15:31
cbadermmedvede, ok15:31
*** mriedem1 is now known as mriedem15:31
cbadermmedvede, I was looking at the patch https://review.openstack.org/#/c/311080/15:32
cbadermmedvede, sorry have to step away for second boss here15:33
*** qb_ has joined #openstack-infra15:34
*** yolanda has quit IRC15:35
*** salv-orlando has joined #openstack-infra15:36
*** nbloom has joined #openstack-infra15:36
mmedvedecbader: ok, so I am checking the latest nodepool build logs, and it looks to be fixed. Last ffi.h errors were last seen yesterday15:36
*** heyongli_ has quit IRC15:37
nbloomHi, can someone help me. A custom job I configured seems to fail somewhere on the gate and I can't figure out what is the cause. Log under http://logs.openstack.org/21/295021/4/experimental/gate-vitrage-dsvm-nv/a7fc25c/15:37
*** heyongli_ has joined #openstack-infra15:38
*** chlong has quit IRC15:38
nbloomsorry, on the experimental pipeline, not on the gate **15:38
*** mriedem has quit IRC15:38
*** kzaitsev_mb has joined #openstack-infra15:39
*** whoops has joined #openstack-infra15:39
jlkHi, I need some assistance. I have a project, giftwrap, and currently it is set to run "python-jobs" and "pypi-jobs", and it also uses templates "merge-check, python-jobs, publish-to-pypi, and python3-jobs". HOwever in one of those it's calling a docs jobs, and I'd like it to not call that job. What's the right way to opt out of the docs jobs?15:39
*** mriedem has joined #openstack-infra15:40
pabelangerjeblair: I likely added to the confusion for afs mirrors.  Yolanda asked me to vos release which me not fully understanding what was going on, as such I unlocked the volume when a transaction was going on.  From everything I was seeing / reading up on, it appeared the AFS mirror was restoring from the last know good transaction15:40
anteayanbloom: so if you look here: http://logs.openstack.org/21/295021/4/experimental/gate-vitrage-dsvm-nv/a7fc25c/console.html#_2016-05-02_15_21_51_98415:41
cbadermmedvede, so you have a successful dib-image build15:41
anteayanbloom: you will see that devstack failed15:41
anteayaand logs are in logs/devstacklog.txt.gz15:41
anteayanbloom: when you then look at logs/devstacklog.txt.gz15:41
*** msuriar has joined #openstack-infra15:42
anteayanbloom: you see http://logs.openstack.org/21/295021/4/experimental/gate-vitrage-dsvm-nv/a7fc25c/logs/devstacklog.txt.gz#_2016-05-02_15_21_55_20615:42
pleia2jeblair: yeah :) you should have seen the folks in the i18n work session, they were pretty excited about finally getting to delete it15:42
mmedvedecbader: no, but it works for infra team. Might be due to https://review.openstack.org/#/c/311080/ now installing those missing libraries. My configuration is different, so I might have to install the dependencies somewhere else15:42
anteayanbloom: it also looks like your current build is having trouble cloning: http://logs.openstack.org/21/295021/4/experimental/gate-vitrage-dsvm-nv/a7fc25c/logs/devstacklog.txt.gz#_2016-05-02_15_21_54_20215:42
pleia2(also, my time this week is limited as well)15:43
anteayapleia2: I hope you too are doing things you enjoy15:43
*** fedexo has joined #openstack-infra15:43
*** pilgrimstack has quit IRC15:44
cbadermmedvede, so I will have to fix somewhere else and not use upstream for infra team?15:44
mmedvedecbader: we could still try adding the dependencies in puppet manfiest to make them explicit15:44
cbadermmedvede, sorry which puppet manifest?15:45
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Create stackviz element and script in nodepool"  https://review.openstack.org/31176815:45
cbadermmedvede, the patch https://review.openstack.org/#/c/311080/ is the one I was looking at to see if I could add that somewhere.15:46
pabelangerinfra-root: I propose we revert stackviz element as our DIBs are broken as a result^. Logs can be found on http://nodepool.openstack.org15:46
nbloomanteaya thanks :) I'll check this out though I don't really understand how this can be as I use the same configuration when running devstack ./stack.sh on my VM...15:46
*** sputnik13 has joined #openstack-infra15:46
*** e0ne has quit IRC15:47
pabelangerproject-config-core as well ^15:47
anteayanbloom: well if you have set cloning to pull from the wrong source our system will error our15:47
*** kzaitsev_mb has quit IRC15:47
anteayanbloom: the same won't happen locally for you15:47
*** heyongli_ has quit IRC15:48
*** bpokorny has joined #openstack-infra15:48
*** heyongli_ has joined #openstack-infra15:48
nbloomanteaya15:48
nbloomanteaya15:48
nbloomanteaya then it's a problem with the cloning source?15:48
anteayanbloom: yes15:48
nbloomanteaya thanks15:49
anteayanbloom: https://git.openstack.org/cgit/openstack-dev/devstack/tree/functions-common#n516 the Cloning not allowed in this configuration error comes from two loops15:49
anteayanbloom: welcome15:49
nibalizerpabelanger: i don't see the explossion15:50
nibalizeron ubuntu-trusty15:50
nibalizerjeblair: as for waiting for bandersnatch to fail... it hasn't written anything to its log file in the 12 hours it's been running15:51
nibalizerso if we're just waiting for that process to leave 'ps' then that's cool15:51
*** vincentll has quit IRC15:51
anteayajlk: the python-jobs group has a docs job: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n47915:52
jlkanteaya: yeah, I'd like to opt-out of that particular job in the group15:52
anteayajlk: the best way to not run the docs job is in zuul/layout.yaml specifiy which of the python jobs you wish to run and don't mention the docs job15:52
jlkI'm not sure if I should make a new group, make a new template, or transpose all the jobs into my project job15:52
pabelangernibalizer: can't link but here is the error in the log15:53
pabelangernibalizer: 2016-05-02 15:13:33,861 INFO nodepool.image.build.ubuntu-trusty: /usr/bin/env: node: No such file or directory15:53
*** pilgrimstack has joined #openstack-infra15:53
pabelangereach OS is failing in a different way, but all related to stackviz15:53
jlkanteaya: so I would take out  "python-jobs" from the template: block of my project, and instead write out the check/gate/whatever jobs from that template that I would want?15:53
jeblairnibalizer: yes; also, i believe a log rotation/compression happened underneath the bandersnatch process, so when it does fail, we may not see anything15:53
jlkanteaya: (this is the other confusing part, in my job config it' slisted as a template, but it's also a job group?15:53
pabelanger2016-05-02 14:27:33,193 INFO nodepool.image.build.centos-7: sudo: npm: command not found is another15:54
anteayajlk: yes in the zuul/layout.yaml file15:54
*** apoorvad has joined #openstack-infra15:54
anteayajlk: can you link your job config please?15:54
anteayaif I am to respond to a definition question15:55
cbadermmedvede, so looks like wheel_mirror_slave.pp already has libffi-dev under ensure_packages but missing openssl-devel15:55
jlkanteaya: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n635115:55
fungipabelanger: thanks for spotting. i was trying to see if it was succeeding but it was previously masked by the zuul-env build issues with paramiko 2.0.015:55
anteayathank you15:56
mmedvedecbader: that is not the manifest that is used to configure jenkins slaves15:56
cbadermmedvede, oh15:56
*** yaume_ has joined #openstack-infra15:56
anteayajlk: right now in zuul/layout.yaml you are using templates15:56
cbadermmedvede, which on is15:56
mmedvedecbader: look at single_use_slave.pp15:56
jlkanteaya: correct15:56
*** yaume has quit IRC15:56
nibalizerjeblair: makes sense15:57
*** ddieterly is now known as ddieterly[away]15:57
anteayajlk: in the jenkins/jobs/python-jobs.yaml file: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n47915:57
*** lakshmiS has joined #openstack-infra15:57
mmedvedecbader: it uses slave_common.pp. The ffi.h error did happen when zuul env was being installed15:57
*** abregman has quit IRC15:58
anteayajlk: python-jobs is a job group, consisting of a number of jobs, which have been defined above in that file15:58
*** heyongli_ has quit IRC15:58
AJaegerjeblair: it was a pleasure ;)15:58
cbadermmedvede, I see that15:58
*** dmk0202 has quit IRC15:59
*** yolanda has joined #openstack-infra15:59
pabelangermmedvede: https://review.openstack.org/#/c/311734/ is the path I am testing right now.  that should properly address the build dependencies for zuul, since we use diskimage-builder to install it15:59
*** Guest37590 has quit IRC15:59
jlkanteaya: okay, so my use of "python-jobs" doesn't relate to https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n294 at all?15:59
AJaegernbloom: read http://jaegerandi.blogspot.de/2016/02/templates-in-openstacks-zuul.html15:59
AJaegernbloom: sorry, that was meant for jlk15:59
AJaegerjlk, read http://jaegerandi.blogspot.de/2016/02/templates-in-openstacks-zuul.html15:59
jlkwell now I'm confused16:00
jlkam I using a template like AJaeger is saying, or a job group like anteaya is saying?16:00
fungijlk: you're conflating jenkins configuration with zuul configuration. two pieces of software, two sets of configuration16:01
jlkle sigh16:01
mmedvedecbader: see pabelanger 's comment. That is better way to manage the dependency if you use dib16:01
AJaegerjlk: Jenkins != Zuul. Jenkins has job-groups, Zuul has templates. Two different kind of languages16:01
openstackgerritJames E. Blair proposed openstack/gertty: Add support for auth-type=form  https://review.openstack.org/16946716:02
openstackgerritJames E. Blair proposed openstack/gertty: Add matchers for external references on launchpad  https://review.openstack.org/29559216:02
openstackgerritJames E. Blair proposed openstack/gertty: Use short titles in breadcrumbs  https://review.openstack.org/31003616:02
openstackgerritJames E. Blair proposed openstack/gertty: Make 'title' attribute available on dialog widgets  https://review.openstack.org/28974816:02
*** lakshmiS has quit IRC16:02
openstackgerritJames E. Blair proposed openstack/gertty: Add navigation breadcrumb footer  https://review.openstack.org/28825016:02
openstackgerritJames E. Blair proposed openstack/gertty: Add extra columns to change list if there is room  https://review.openstack.org/31003716:02
openstackgerritJames E. Blair proposed openstack/gertty: Fix rendering error in change screen  https://review.openstack.org/29542316:02
openstackgerritJames E. Blair proposed openstack/gertty: Add configuration information to docs  https://review.openstack.org/28624016:02
openstackgerritJames E. Blair proposed openstack/gertty: Add an option to disable the breadcrumb footer  https://review.openstack.org/31177016:02
openstackgerritJames E. Blair proposed openstack/gertty: Show potential completions  https://review.openstack.org/31177116:02
openstackgerritJames E. Blair proposed openstack/gertty: Make change list searchable  https://review.openstack.org/31177216:02
openstackgerritJames E. Blair proposed openstack/gertty: Make project list searchable  https://review.openstack.org/31177316:02
openstackgerritJames E. Blair proposed openstack/gertty: Add a short title to diff screen  https://review.openstack.org/31177416:02
openstackgerritJames E. Blair proposed openstack/gertty: Add form authentication info to docs  https://review.openstack.org/31177516:02
jlkwell, it's pretty unclear to me what I need to do to "make my project work"16:02
jlkmy project doesn't work. It runs a job that doesn't work. I'd like to not run that job.16:02
AJaegerjeblair: two more and my screen is scrolled off ;)16:02
jeblairAJaeger: i had a good flight16:02
*** kjorgensen has joined #openstack-infra16:02
AJaegerjlk: read my blog post, it explains it bottom up ;)16:02
AJaegerjeblair: great!16:02
fungiAJaeger: more specifically, zuul has "project-templates"16:03
jlkAJaeger: except it doesn't explain at all "how do I use this thing, but opt out of that thing"16:03
jlkor what I should do if one job from a template is not desired16:03
*** heyongli_ has joined #openstack-infra16:03
anteayajlk: I already told you above16:03
anteayajlk: make a cup of tea and read what has been suggested again16:03
fungijlk: in the zuul configuration you need to manually expand the project-template you're using and omit any undesired entries16:04
AJaegerjlk: it does it the other way round: It starts with no jobs and adds them and then finally uses the template...16:04
jlkalright, here's what's unclear though.16:04
*** thorst has quit IRC16:04
jlkanteaya told me to read about the job groups16:04
jlkbut it seems like AJaeger is telling me to expand the template16:04
anteayajlk: I told you to specify the jobs you want in zuul/layout.yaml16:04
anteayaand omit the job you don't want16:05
anteayaas fungi has also said16:05
jlk[10:52:05]  <anteaya>jlk: the python-jobs group has a docs job: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n47916:05
* AJaeger needs to step out a bit...16:05
jlk[10:52:38]  <anteaya>jlk: the best way to not run the docs job is in zuul/layout.yaml specifiy which of the python jobs you wish to run and don't mention the docs job16:05
anteayayeah I'm out too16:05
* anteaya makes a cup of tea16:05
jlkI'm sorry, that read to me that I need to look at the job group and expand that16:05
jlkI'm sorry for not getting it.16:05
* AJaeger makes family dinner...16:05
*** EricGonczer_ has joined #openstack-infra16:05
cbadermmedvede, I am looking at it.16:05
*** sridhar_ram has joined #openstack-infra16:05
*** lakshmiS has joined #openstack-infra16:06
*** kjorgensen has quit IRC16:06
*** kjorgensen has joined #openstack-infra16:06
jeblairjlk: i think you've got it -- this is correct:  15:53 < jlk> anteaya: so I would take out  "python-jobs" from the template: block of my project, and instead write out the check/gate/whatever jobs from that template that I would want?16:06
jlkjeblair: thanks, I think I was thrown off-course by the first mention of the python-jobs file, which is a job group.16:07
jeblairjlk: then i think the conversation spiraled into helpfulness :)16:07
*** salv-orlando has quit IRC16:08
*** EricGonczer_ has quit IRC16:08
*** heyongli_ has quit IRC16:08
jeblairjlk: out of curiosity, why no docs?16:08
jlkI just don't have infra right now to build them out, that'll come soonish16:09
jlkbut having the docs job is blocking any chnage from coming in16:09
*** heyongli_ has joined #openstack-infra16:09
jeblairgotcha16:09
fungijlk: an example would be openstack-infra/lodgeit which runs python27 and pep8 but not docs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n325716:10
*** hughhalf has quit IRC16:10
openstackgerritJesse Keating proposed openstack-infra/project-config: Opt out of docs jobs for giftwrap  https://review.openstack.org/31177816:10
fungiso instead of using the project-template named "python-jobs" in the zuul layout, it just specifies which jobs from that project-template it actually needs/can use16:10
sdakeso, kolla wnats to implemenet ~80 gates for kolla16:10
sdakeshort running jobs - 20 mins16:10
*** EricGonczer_ has joined #openstack-infra16:10
sdakewill that be a problem?16:10
fungisdake: 80 gates? i really hope you mean 80 jobs16:11
jlkfungi: okay, that looks like what I just submitted16:11
sdakedefine job for me fungi :)16:11
* fungi worries "gate" the new "smurf" and we are going to gate the gates with gateity gating16:11
fungisdake: a "job" is a work unit currently defined in our jenkins configuration, which runs a series of commands and reports back a result16:11
sdakeyes 80 jobs16:12
*** jordanP has quit IRC16:12
sdakewe have 1 gat e;)16:12
sdakewell 2 if yo ucount the humans ;)16:12
*** links has joined #openstack-infra16:12
fungisounds exciting. i hope they succeed at least 999 times out of 1000 or you're probably never going to be able to merge changes easily16:12
sdakefungi all our jobs are non-voting16:13
sdakebecuase we lack mirrors in infra repo for our upstream stuff16:13
sdakethe work has started here in infra16:13
sdakebut incomplete16:13
*** kzaitsev_mb has joined #openstack-infra16:13
fungiahh, cool. nondeterministic failures in job results become crippling the more voting jobs you run16:14
sdakefugni right - we want to get to the point wher ewe have 1 voting job for functional teting16:14
sdakethe rest would be sort of advisory ;)16:14
sdakebut that wont implode infra I assume?16:14
sdake(the 80 jobs016:14
*** gyee has joined #openstack-infra16:14
*** gyee has quit IRC16:15
*** tqtran has joined #openstack-infra16:15
cbadermmedvede, the issue pabelanger is for dependencies for zuul will this work for dependencies on dib builder?16:17
*** sulo_ is now known as sulo16:18
*** ddieterly[away] is now known as ddieterly16:18
*** EricGonczer_ has quit IRC16:18
anteayasdake: infra would not implode, your patches will take a very long time to get all the nodes it needs to finish testing16:18
*** fawadkhaliq has joined #openstack-infra16:18
*** heyongli_ has quit IRC16:18
jeblairsdake: that's about 26 machine-hours, or about 2x the amount of time we spend testing a change to nova16:18
sdakeanteaya could you put a WAG on that in terms of time16:18
*** EricGonczer_ has joined #openstack-infra16:19
anteayasdake: well I don't know what a WAG is, I don't acroymn16:19
sdakewild ass guess16:19
anteayabut I'm going to guess jeblair's estimate is useful to your question16:19
anteayawell I don't like that language so wouldn't use it16:19
anteayabut again I think jeblair's statement is helpful here16:20
*** hughhalf has joined #openstack-infra16:20
*** heyongli_ has joined #openstack-infra16:21
mmedvedecbader: your dib build fails because zuul dependencies fail to install, right? If so, that should fix the problem.16:21
*** Swami has joined #openstack-infra16:22
cbadermmedvede, I made the fix on ci and I am trying the build will see if it works thanks for the help.16:22
fungicbader: not sure which "this" you're referring to, but for the moment zuul is declaring itself incompatible with the new version of paramiko which now drags in cryptography (for other reasons)16:22
cbaderfungi, thank you for the clarification, I am still new to this and haven't figured out all the connections.16:22
pabelangerand, if other projects are getting hit with it, updating bindep-fallback.txt is also an option16:23
*** jistr has quit IRC16:23
fungiwhen zuul eventually gets patched to support paramiko 2.0.0 (assuming it does, rather than switching to a different ssh implementation) then we'll likely need to revisit the indirect libffi-dev and openssl-dev dependencies16:23
fungipabelanger: i expect any jobs which are running on our workers won't see this unless they're not using our wheel mirrors16:24
*** asselin_ has joined #openstack-infra16:24
fungithose header packages are needed to build the cryptography package, but using a prebuilt wheel of cryptography instead sidesteps building16:24
*** gyee has joined #openstack-infra16:25
pabelangerfungi: exactly16:25
mmedvedeyes, our third-party CI was hit with it, we are not using pre-built wheels16:25
*** yaume_ has quit IRC16:25
jeblairsdake: i think the answer to your question depends on how often changes are proposed to your project, and how busy the system is with other projects.  an occasional change launching 80 jobs isn't terrible.  if kolla has many changes, each of which consumes 26-machine hours, it will have an impact on other projects and itself.16:26
fungimmedvede: hit with it running jobs, or just hit with it building images?16:26
cbaderfungi, with jobs and building images for me.16:26
fungicbader: which jobs/projects? i'm curious to see how they address it16:27
jeblairsdake: our current budget is 19392 machine-hours-per-day.  (this is not an SI unit, but it should be).  overall, we are at about half-capacity.  some weeks we are at capacity.16:27
mmedvedefungi: both for me too16:27
cbaderfungi, noticed on friday all jobs against cinder patches started failing, looked around and notice my dib builde hadn't work since thursday night. go fix for vms friday night.16:28
mmedvedefungi: I patched jobs right away, now fixing image builds. We are not using dibs, and have our own version of system-config, so it is easy local fix16:28
*** xarses_ is now known as xarses16:28
*** thorst has joined #openstack-infra16:29
*** heyongli_ has quit IRC16:29
*** xnox has quit IRC16:29
*** heyongli_ has joined #openstack-infra16:29
fungimtreinish: timothyb89: (i don't see austin81 in here at the moment) do you have time to dig into the stackviz build issues we're seeing in our image updates? i'm going to approve the temporary revert in https://review.openstack.org/311768 and we can reintroduce it once you iron out those kinks16:29
cbaderfungi, I looked in log for image build and saw libffi error, but didn't know how to integrate vm fix to that process.16:29
*** ifarkas has quit IRC16:29
*** EricGonczer_ has quit IRC16:30
pabelangerfungi: do you mind looking at 311768? We need to revert stackviz to merge our zuul dependency issue16:32
pabelangeranteaya: thanks for the reviewing too16:32
pabelangerwoah16:32
anteayapabelanger: sure16:33
fungipabelanger: see my above pings to mtreinish and timothyb8916:33
*** sputnik13_ has joined #openstack-infra16:33
*** sputnik13 has quit IRC16:33
pabelangerfungi: ack16:33
anteayapabelanger: thanks for figuring out the issue and offering the revert16:33
pabelangeranteaya: fungi: I think we're need some more test coverage for stackviz. Once I have some spare bandwidth, I'll see if we can add something into the gate for all our OSes16:34
*** nbloom has quit IRC16:35
mtreinishfungi: I am around now16:35
*** ccarmack has joined #openstack-infra16:36
openstackgerritMerged openstack-infra/project-config: Revert "Create stackviz element and script in nodepool"  https://review.openstack.org/31176816:38
anteayapabelanger: um okay16:38
*** amrith is now known as _amrith_16:39
dimsanteaya : we are trading some proposals on kolla project. one idea that came up was feature branches. can we have a separate ACL group for a specific feature branch?16:39
dimsAJaeger : ^^16:39
*** heyongli_ has quit IRC16:39
*** heyongli_ has joined #openstack-infra16:39
anteayadims: repos have one acl, for each repo16:39
AJaegerdims, yes. Similar like on stable/16:39
anteayadims: now within one acl file you can specify different things, such as AJaeger's point16:40
dimsack. so it's possible :)16:40
anteayabut it is one acl file16:40
anteayayes it is possible16:40
dimsgot it16:40
*** yamahata has joined #openstack-infra16:40
dimsinc0 : ^^16:40
*** acabot has quit IRC16:40
inc0thanks dims, anteaya16:41
inc0and AJaeger ;)16:41
*** dprince has joined #openstack-infra16:42
dims:)16:42
AJaegerinc0, dims: Just check how it's done for stable/ branches in projects16:42
dimsAJaeger : yep16:42
*** tphummel has joined #openstack-infra16:42
*** tonytan4ever has quit IRC16:44
dmsimardIs there any chance we could pass the --prefix parameter to the meetbot irclogs2html implementation for eavesdrops so that it would include the channel name ? i.e, https://github.com/mgedmin/irclog2html/blob/cfefff0d191a17cbdab4bab4c99a638ce9d15105/src/irclog2html/logs2html.py#L198-L20016:44
dmsimardFor example, in http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ it would say "IRC logs for #openstack-infra"16:45
*** harlowja has joined #openstack-infra16:45
mtreinishfungi: looking at the logs it can't find the npm binary on redhat or node on ubuntu16:45
mtreinishwhich is odd, because I thought they were added to the package lists16:46
fungimtreinish: unless that's running before package installation (or in the wrong context? chroot vs host?)16:46
anteayadmsimard: I don't have any argument for not doing so, so I guess it comes down to whoever offers that patch and how they impliment it16:46
*** pradk has quit IRC16:46
dmsimardanteaya: I tried to look at how that specific script was called but couldn't find it .. all I saw was a cron setup in puppet-meetbot but it doesn't particularly speak to me16:47
fungimtreinish: anyway, i mainly just don't want to iterate on it in production when each new attempt means another day of missing image updates16:47
dmsimardI'm willing to send a patch, just need to know where16:47
*** xnox has joined #openstack-infra16:48
anteayadmsimard: have you looked at all at codesearch.openstack.org16:48
anteayait may help you find the call to that script, if indeed there is a call16:48
dmsimardanteaya: yup: http://codesearch.openstack.org/?q=irclog2html&i=nope&files=&repos=16:49
dmsimardthat's where I got the cron from16:49
pabelangermtreinish: fungi: Maybe a path issue? I also wonder if it would be possible to not use sudo npm install, but rather sandbox our npm installs into a local sandbox.  I admit, I don't have much experience with npm to know if possible16:49
*** oanson has joined #openstack-infra16:49
anteayadmsimard: ah great, then at this point you know more than I16:49
*** heyongli_ has quit IRC16:49
*** heyongli_ has joined #openstack-infra16:49
anteayadmsimard: have you looked at prior patches to the repo?16:49
dmsimardit looks like it could be hidden in supybot or something, there is a mention of ChannelLogger but it doesn't reveal a whole lot either http://codesearch.openstack.org/?q=ChannelLogger&i=nope&files=&repos=16:50
anteayadmsimard: sometimes they can be helpful16:50
EmilienMpabelanger: is this workaround acceptable for ansible? https://review.openstack.org/#/c/311755/16:50
anteayadmsimard: that is a distinct possibility16:50
EmilienMpabelanger: otherwise, can I just submit a patch in OOO JJB? so we can quickly fix it16:51
pabelangerEmilienM: I think so, we can ask fungi what he thinks16:51
mtreinishpabelanger: the other idea I had was to add a post job to stackviz to build the static part and just have nodepool put that on the disk from the log server16:51
*** daemontool has joined #openstack-infra16:52
fungipabelanger: mtreinish: worth looking at the npm installation macro in our jjb config for hints. i know there are some path differences too based on where the initial distro packaged npm installs the subsequent npm when bootstrapping16:52
pabelangermtreinish: Personally, I am happy with that approach. Also makes debugging failures a little easier16:53
*** fedexo has quit IRC16:53
fungipabelanger: EmilienM: i take it ansible also uses paramiko?16:54
pabelangerfungi: Ya16:54
*** sputnik13_ is now known as sputnik1316:54
fungii'm curious how they're handling the sudden header deps in their pip-installed ecosystem16:54
*** acabot has joined #openstack-infra16:54
pabelangerI was going to pop over to #ansible and see if they can also do a wheel16:55
pabelangernot sure that will fix all our issues16:55
fungipabelanger: well, it's cryptography that would need to be wheeled, and the least-common-denominator "anylinux" wheels pypi recently started supporting intentionally punt on problematic libs like openssl16:56
pabelangerfungi: Ah, okay.16:56
pabelangerfungi: at first pass: https://review.openstack.org/#/c/311734/ works on ubuntu-trusty, testing centos-7 now.  If we want to go down that path in the future16:57
AJaegerif any project-config core could please later review 311675 and 311701 for openstack-manuals, it would be great. Those changes are needed for us to move forward with getting rid of DocBook XML building16:58
*** heyongli_ has quit IRC16:59
ccarmackfungi:  Hi, I'm working on nova cells v2, and we would like to add a multi-node job for cells.  Right now there is a single-node job for cells, but multi-node would be more realistic.  Does this seem ok?17:00
ccarmackalaski, bauzas ^17:00
*** hdd has joined #openstack-infra17:00
EmilienMfungi, pabelanger: should I abandon my patch?17:01
EmilienMI'm trying to restablish tripleo CI which is broken for 2 days17:01
*** hashar has quit IRC17:01
EmilienMmaybe I can just send a quick fix in ooo jjb, but I'm not sure if anyone else is affected17:02
openstackgerritSridhar Ramaswamy proposed openstack-infra/irc-meetings: Move Tacker team meeting to compensate DST  https://review.openstack.org/31178917:02
*** heyongli_ has joined #openstack-infra17:02
pabelangerEmilienM: is the devstack-gate failing on other clouds?  I was under the impression everybody is affected17:03
openstackgerritJesse Keating proposed openstack-infra/project-config: Opt out of docs jobs for giftwrap  https://review.openstack.org/31177817:03
EmilienMpabelanger: I have no idea17:03
*** oanson has quit IRC17:04
fungiEmilienM: pabelanger: looks like ansible in at least some of our jobs is using paramiko 1.16.0? http://logs.openstack.org/43/308343/1/gate/gate-tempest-dsvm-trove/e89bc95/console.html#_2016-05-02_16_13_07_64417:05
*** sputnik13 has quit IRC17:05
pabelangerEmilienM: could you link the tripleo failure again?17:06
anteayaAJaeger: I'm already +2 on 311701, going to have to wait for someone else to review17:07
*** clarkb has joined #openstack-infra17:07
openstackgerritMerged openstack-infra/project-config: openstack-manuals: Adjust jobs  https://review.openstack.org/31167517:08
*** pfallenop has quit IRC17:08
EmilienMpabelanger: it's every jobs, let me find a link17:10
*** heyongli_ has quit IRC17:10
EmilienMpabelanger: http://logs.openstack.org/37/311737/2/check-tripleo/gate-tripleo-ci-f22-ha/88f25da/console.html#_2016-05-02_13_27_21_94317:10
*** sputnik13 has joined #openstack-infra17:10
*** heyongli_ has joined #openstack-infra17:10
*** yamahata has quit IRC17:10
*** yamahata has joined #openstack-infra17:11
*** oanson has joined #openstack-infra17:11
pabelangerEmilienM: fungi: The issue is, tripleo is not using our wheel mirror. And since mirror.pypi is behind we are still pinned to paramiko 1.16.0. That explains why all our other jobs are not failing17:12
openstackgerritChristopher Aedo proposed openstack-infra/project-config: Disable App Catalog periodic job  https://review.openstack.org/31179017:12
EmilienMpabelanger: I see17:12
EmilienMpabelanger: so can I just patch JJB in the meantime? so we can fix it very quickly in the meantime DIB is fixed?17:12
*** pradk has joined #openstack-infra17:12
EmilienMor should we go for the devstack gate patch?17:12
EmilienMI prefer option #1 since it only affects OOO CI17:13
pabelangerEmilienM: fungi: We should likely fix devstack-gate for tripleo, once our pypi mirror resync it will expose it.17:13
*** trown is now known as trown|lunch17:13
*** pfallenop has joined #openstack-infra17:14
*** sputnik13 has quit IRC17:14
pabelangerEmilienM: As a long term plan, I want to work towards common CI for tripleo.  See https://review.openstack.org/#/c/311721/ for a first step towards that, replacing the fedora-22 jenkins slave with our centos-7 slave17:15
pabelangerEmilienM: will send out a ML post about that17:15
*** markvan has quit IRC17:16
*** markvan has joined #openstack-infra17:16
fungipabelanger: EmilienM: yeah, so if `pip install ansible` doesn't work in vanilla cases right now i agree that devstack-gate installing ansible that way means we should make sure it also drags in any system package prerequisites17:16
fungipabelanger: EmilienM: https://github.com/ansible/ansible/issues/1566517:17
*** johnny has joined #openstack-infra17:17
*** jerryz has joined #openstack-infra17:18
*** johnny is now known as Guest6636317:18
fungiwhich points out that we could manually install a lower version of paramiko in the virtualenv before installing ansible into it17:18
*** sputnik13 has joined #openstack-infra17:19
pabelangerfungi: right, we should infact see if ansible works with paramiko too17:19
fungior just change the pip command to `/tmp/ansible/bin/pip install paramiko==1.16.0 ansible==$ANSIBLE_VERSION` presumably17:19
pabelangerparamiko 2.0.017:19
mordredfunny thing is - ansible actually uses openssh for things we do17:19
mordredparamiko is the fallback for "is your system so broken that we can't find a usable ssh client"17:20
*** piet has joined #openstack-infra17:20
*** heyongli_ has quit IRC17:20
*** heyongli_ has joined #openstack-infra17:20
fungiright, similarly zuul could call out to /usr/bin/ssh rather than relying on paramiko?17:20
anteayado we have any need to set any kind of limit on grafana dashboards?17:21
*** sputnik13 has quit IRC17:21
mordredit could, but that would be a decent amount of work17:21
mordredI'd suggest pinning paramiko for zuul17:21
anteayaeither as a total quantity or on a per project basis?17:21
anteayamotivation for my question: https://review.openstack.org/#/c/307511/417:21
fungimordred: agreed (and we already did for now as even with paramiko 2.0.0 successfully installed zuul breaks on it)17:21
mordred++17:22
fungitheir api changed in non-backward-compatible ways in addition to adding install-time build deps on c library headers17:22
mordredyay! another success for semver17:22
mordredI mean, at least they _told_ us they were releasing a completely incompatible thing17:23
anteayamordred: mr silver lining17:23
pabelangerclarkb: fungi: I've started a manually building new nodepool dibs17:23
*** markvan has quit IRC17:23
mordredanteaya: that's what people call me17:24
anteayaha ha ha17:24
mordredanteaya: right after they say "that guy who is always upbeat, positive and who gives criticism in a completely constructive manner at all times"17:24
*** daemontool_ has joined #openstack-infra17:24
*** fitoduarte has joined #openstack-infra17:24
anteayamordred: yeah right after that17:25
openstackgerritMerged openstack-infra/project-config: Remove translations from operations-guide  https://review.openstack.org/31166017:25
*** hughhalf has quit IRC17:25
*** tosky has quit IRC17:26
*** daemontool has quit IRC17:27
*** hughhalf has joined #openstack-infra17:27
*** david-lyle has joined #openstack-infra17:28
*** david-lyle has quit IRC17:29
*** baoli has quit IRC17:29
anteayapabelanger: can you think of any reason why would have a limit on how many grafana dashboards we create?17:29
*** sputnik13 has joined #openstack-infra17:30
*** david-lyle has joined #openstack-infra17:30
*** heyongli_ has quit IRC17:30
*** heyongli_ has joined #openstack-infra17:30
mordredanteaya: there is a limit to the number of bytes of information that can be stored across the sum of all of the computers in the world ... that might be a limit ;)17:31
pabelangeranteaya: maybe if the dashboards are impacting graphite.o.o we want to limit grafana dashboards.17:31
AJaegerthanks for reviews, anteaya !17:31
anteayamordred: indeed, how very helpful17:31
fungiEmilienM: pabelanger: so i think a saner temporary workaround for d-g is probably to update the pip install command per above. i've tested the syntax from my comment and it successfully avoids installing cryptography as expected17:31
*** markvan has joined #openstack-infra17:32
anteayapabelanger: so do I have any way of evaluating that? or just allow dashboards until we have a problem and revert some or call a halt?17:32
anteayaAJaeger: welcome :)17:32
*** sputnik13 has quit IRC17:33
*** links has quit IRC17:33
*** eil397 has joined #openstack-infra17:34
*** _amrith_ is now known as amrith17:35
pabelangeranteaya: I think we'd need to monitor catci.o.o to see how much load graphite.o.o is under then decide to limit dashboards or increase resources.  But, at this point in time, I'd personally like to see more dashboards in grafana.o.o17:35
*** acabot has quit IRC17:35
*** zeih_ has quit IRC17:35
openstackgerritPaul Belanger proposed openstack-infra/ansible-puppet: Remove connection: localhost  https://review.openstack.org/31087817:36
anteayapabelanger: more dashboards it is17:36
*** zeih has joined #openstack-infra17:36
anteayapabelanger: lets revisit if/when we need to17:36
anteayaand thank you17:36
pabelangermordred: nibalizer: fungi: ^ here is the patch phschwartz_ and myself came up with in austin. However, we don't have a good way to test it outside of production17:36
pabelangeranteaya: np17:36
pabelangermordred: nibalizer: fungi: that is for our OOM issues with ansible-playbook on puppetmaster.o.o17:37
mtreinishAJaeger, fungi, pabelanger: https://review.openstack.org/#/c/311147/117:38
*** inc0 has quit IRC17:39
mordredpabelanger: looking17:39
mtreinishI'm not sure how many of those jobs actually output subunit, but on fri AJaeger and I were looking at adding it to the translation jobs17:39
*** piet has quit IRC17:39
*** inc0 has joined #openstack-infra17:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Install constrainted version of babel  https://review.openstack.org/30980917:39
mordredpabelanger: we'll need to make sure root can ssh into itself on puppetmaster to do that17:39
*** piet has joined #openstack-infra17:39
fungimtreinish: seems reasonable17:40
AJaegermtreinish: yes, this is needed for https://review.openstack.org/311459 - thanks!17:40
mordredpabelanger: which it cannot17:40
*** ddieterly is now known as ddieterly[away]17:40
*** bardia has joined #openstack-infra17:40
mordredpabelanger: other than that - I think that change is completely safe to just apply17:40
openstackgerritEmilien Macchi proposed openstack-infra/devstack-gate: devstack-vm-gate-wrap: pin paramiko==1.16.0  https://review.openstack.org/31175517:40
pabelangermordred: I thought that was the case, but phschwartz_ mention ansible would simply reuse the current connect17:40
mordredpabelanger: as it should not affect functionality17:40
EmilienMfungi, pabelanger: https://review.openstack.org/31175517:40
*** heyongli_ has quit IRC17:40
mordredpabelanger: there is no current connect17:40
fungimtreinish: clarkb may want to have a look at it as well, but i assume there's nothing "special" about log collection for the post pipeline17:41
*** heyongli_ has joined #openstack-infra17:41
*** zeih has quit IRC17:41
pabelangermordred: ack17:41
mordredpabelanger: we just need to put the authorized_keys file in place17:41
EmilienMadding a comment17:41
mtreinishdhellmann: ^^^ you might want to look at AJaeger's patch too, you were asking me about that for release jobs17:41
pabelangermordred: right17:41
fungiEmilienM: lgtm, and as mordred mentioned, in our case the ansible calls in devstack-gate should never end up using paramiko anyway17:42
openstackgerritEmilien Macchi proposed openstack-infra/devstack-gate: devstack-vm-gate-wrap: pin paramiko==1.16.0  https://review.openstack.org/31175517:42
*** pvaneck has joined #openstack-infra17:42
pabelangerfungi: EmilienM: +A17:43
mordredfungi: I'm running the FIP clean script17:43
openstackgerritMerged openstack-infra/project-config: grafana: added a dashboard with periodic jobs for neutron  https://review.openstack.org/30751117:44
mordredthere were 3 leaked osic fips and bluebox seemed totally leaked17:44
mordredscuse me - 2 leaked fips17:44
mordredCleaned 2 FIPs from osic-cloud117:44
mordredCleaned 44 FIPs from bluebox-sjc117:44
EmilienMfungi, pabelanger: thx guys17:45
fungimordred: so the osic leak is basically solved but bbox is still up in the air?17:45
mordredyah17:46
EmilienMI just hope I did not break anything17:46
mordredfungi: working theory also is that older openstack does this worse than newer17:46
fungiEmilienM: it's about the most innocuous solution we have17:47
openstackgerritBen Kero proposed openstack-infra/system-config: add trusty support for paste.openstack.org  https://review.openstack.org/31123517:47
fungiEmilienM: so should be completely safe (but i think our test jobs will tell us if it's not since d-g is self-testing)17:47
mordredfungi: and our clouds from newest to oldest are osic, bluebox, tripleo - and they exhibit the behavior in "not much, medium, constant"17:47
*** acabot has joined #openstack-infra17:47
EmilienMfungi: could I have your feedback on https://review.openstack.org/#/c/301375/ if you have time this week?17:47
EmilienMthis is not urgent ^17:47
*** oanson has quit IRC17:49
*** rbrndt has quit IRC17:49
*** fawadkhaliq has quit IRC17:50
*** heyongli_ has quit IRC17:51
*** asselin__ has joined #openstack-infra17:51
*** heyongli_ has joined #openstack-infra17:51
*** permalac has quit IRC17:52
*** inc0_ has joined #openstack-infra17:52
*** inc0 has quit IRC17:52
*** akshai has joined #openstack-infra17:52
*** julim has joined #openstack-infra17:52
*** deadnull_ has joined #openstack-infra17:53
*** asselin has quit IRC17:53
*** deadnull_ has quit IRC17:54
*** piet has quit IRC17:55
fungiEmilienM: the example we discussed for considering not moving the script for a proposal job into project-config was in the requirements repo, and on reconsideration it turned out that having one branchless copy of the tool in project-config was probably a saner solution anyway due to modifications having to get backported to each branch of the script otherwise. i see that17:55
fungiopenstack/puppet-openstack-integration also has stable branches... do you need multiple branch support in that job?17:55
EmilienMfungi: for now, no. We aim to change the Puppetfile only on master.17:56
EmilienMfungi: ie: proposal job will only run on master17:56
dprincepabelanger: hi, re. https://review.openstack.org/#/c/311217/ how about slave_common.pp in system-config?17:57
EmilienMfungi: so I move it?17:58
anteayamordred: have you any interest in this patch any longer? https://review.openstack.org/#/c/175229/17:59
*** piet has joined #openstack-infra17:59
mordredanteaya: nope. abandoned17:59
anteayamordred: thank you and the system-config change it depended on?17:59
mordreduhm18:00
openstackgerritSteven Dake proposed openstack-infra/project-config: Make oracle gates for Kolla always runs  https://review.openstack.org/31180318:00
*** sputnik13 has joined #openstack-infra18:00
mordredanteaya: I mean, that change is probably useful to someone, but I'm not really giving it mental energy at this point18:00
pabelangerdprince: we have a few reviews open to address that.  311734 for example fixes in our dib images.  However, until tripleo migrates to that, we'll need either slave.pp or slave_common.pp.  Would need to look deep into those.18:00
dprincepabelanger: yep, understood this is a "quick fix" sort of thing. And totally agree with moving towards the clean DIB build too18:01
*** heyongli_ has quit IRC18:01
dprincepabelanger: I will look and post it now18:01
anteayamordred: fair enough, this is the system-config patch that was consided a depends on for the patch you abandoned: https://review.openstack.org/#/c/175227/18:01
*** heyongli_ has joined #openstack-infra18:01
pabelangerdprince: EmilienM did just make a fix to devstack-gate, so what other issues are you seeing?18:01
mordredanteaya: yah.18:01
EmilienMdprince: yeah, my patch should fix tripleo CI18:02
EmilienMdprince: https://review.openstack.org/31175518:02
dprincepabelanger: we are blocked because our TripleO images haven't had libffi-devel and openssl-devel since Friday18:02
dprinceEmilienM: okay, I will reference that here I suppose18:03
dprincehttps://bugs.launchpad.net/tripleo/+bug/157680718:03
openstackLaunchpad bug 1576807 in tripleo "ffi.h: No such file or directory" [Critical,Triaged]18:03
*** sputnik13 has quit IRC18:03
dprinceEmilienM: sorry I missed your fix. Thanks for posting it18:03
EmilienMdprince: no worries, I should have updated the LP18:03
pabelangerdprince: EmilienM: Ya, that should fix it for now. Until we see what ansible wants to do18:03
EmilienMpabelanger: do we need to wait for a new image to be built?18:04
dprincepabelanger: is there a way to force an image rebuild for TripleO now?18:04
*** jed56 has quit IRC18:04
fungidprince: yeah, i'm unconvinced that adding more system packages is necessarily the solution until the ansible community confirms paramiko 2.0.0 support is really intentional18:04
EmilienMmaybe we clone devstack-gate, let me check18:04
pabelangerI believe once the commit merges, it should unblock.18:05
pabelangerbut easy to test18:05
EmilienMdprince: no need, scripts/bighammer.sh clones devstack-gate18:05
fungiEmilienM: dprince: normal devstack-gate jobs see changes take effect as soon as they merge, since d-g is cloned and updated at job start time18:05
EmilienMpabelanger: you're right, TripleO CI git clones devstack-gate18:05
EmilienMfungi: yeah18:05
*** trown|lunch is now known as trown18:05
pabelangerI am doing new image builds too, so I have no problems kicking off tripleo if needed18:05
pabelangerEmilienM: ack18:05
dprincefungi: cool, sounds like we are close then18:05
dprincethanks guys18:05
jeblair(also, d-g uses zuul-cloner so depends-on should work)18:07
*** ccamacho has joined #openstack-infra18:07
AJaegerproject-config cores, https://review.openstack.org/310929 contains a first version of review guidelines - comments and reviews are welcome.18:08
JayFIs there a place I can see exactly what size VMs you all use for check/gate jobs? I want to do some benchmarking around Ironic jobs and want to be sure I'm using the right instnace size18:09
*** sputnik13 has joined #openstack-infra18:10
nibalizerJayF: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml is the nodepool config file18:10
fungiJayF: the providers section of our nodepool configuration contains hints for flavor selection across different providers we use http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n12118:10
JayFexactly what Iw as looking for, thanks to both of you18:11
nibalizerit doesn't say exactly what we use but it gives you enough information18:11
*** heyongli_ has quit IRC18:11
nibalizerthe tl;dr is every test vm gets 8G of ram18:11
JayFI mean, I'm a racker, so that tells me 8gb instance :)18:11
*** heyongli_ has joined #openstack-infra18:11
JayFwhich is what I'll use18:11
fungiright, it in fact doesn't know exactly what to use (because that can change across providers and over time) so instead it uses some heuristics to pick something suitable18:12
*** sputnik13 has quit IRC18:12
fungiJayF: specifically 8gb "performance" (and our images are pvhvm)18:12
fungiin rax18:12
JayFawesome; now time to see if I can get pxe booting to be faster :)18:13
*** akshai has quit IRC18:13
mordredJayF: it's also worth noting that we upload externally built images, and it's entirely possible that we are not adding a magic "run faster" flag on them18:14
*** akshai has joined #openstack-infra18:14
mordredsince apparently image flags have an effect on how hypervisors execute them18:14
*** nadya has quit IRC18:14
JayFmordred: My benchmarking is going to be around which qemu pxe provider pxe boots things faster18:14
mordredkk18:14
JayFmordred: so I think my results will be valid, and I have enough anecdotal evidence about how long it takes that if I see anything too offbase, I'll raise a red flag18:14
mordredsweet!18:15
*** nadya has joined #openstack-infra18:15
bswartzAJaeger: I ran the cookiecutter thing and it crashed with an error18:20
* bswartz sighs18:20
*** heyongli_ has quit IRC18:22
*** xarses has quit IRC18:22
* bswartz roflcopters18:22
bswartzthere is a bug in the cookie cutter repo18:22
*** xarses has joined #openstack-infra18:22
*** xarses has quit IRC18:23
*** xarses has joined #openstack-infra18:23
*** Sukhdev has joined #openstack-infra18:24
*** heyongli_ has joined #openstack-infra18:24
openstackgerritBen Swartzlander proposed openstack-dev/specs-cookiecutter: Fix spelling error in template.rst  https://review.openstack.org/31180618:25
bswartz^ hope this is an easy change to merge18:26
*** Sukhdev has quit IRC18:26
notmorganmordred: DUDE I want a magic "run faster" flag on all my cloud resources. (sigh... image flags)18:27
*** Sukhdev has joined #openstack-infra18:27
*** notmorgan is now known as morgan18:28
*** sputnik13 has joined #openstack-infra18:28
*** kzaitsev_mb has quit IRC18:28
*** fawadkhaliq has joined #openstack-infra18:28
* morgan remembers to put -infra meeting on schedule again.18:28
*** ddieterly[away] is now known as ddieterly18:29
*** inc0_ has quit IRC18:29
dimsbswartz : +218:31
*** heyongli_ has quit IRC18:32
*** mkoderer__ has quit IRC18:32
*** heyongli_ has joined #openstack-infra18:32
dimsbswartz : thanks!18:33
*** tphummel has quit IRC18:36
*** madhu_ak has joined #openstack-infra18:37
*** tphummel has joined #openstack-infra18:38
timothyb89mtreinish: fungi: pabelanger: apologies for stackviz issues, a quick look through the logs suggests nodejs-legacy was missing on the ubuntu images18:39
timothyb89mtreinish: fungi: pabelanger: if there's desire to rethink the build process, I'd like to throw in this as an alternative: https://etherpad.openstack.org/p/stackviz-deployment18:41
*** dprince has quit IRC18:42
*** heyongli_ has quit IRC18:42
*** heyongli_ has joined #openstack-infra18:42
*** nmagnezi has joined #openstack-infra18:45
*** baoli has joined #openstack-infra18:45
openstackgerritMerged openstack/gertty: Add configuration information to docs  https://review.openstack.org/28624018:46
openstackgerritMerged openstack/gertty: Add navigation breadcrumb footer  https://review.openstack.org/28825018:46
openstackgerritMerged openstack/gertty: Make 'title' attribute available on dialog widgets  https://review.openstack.org/28974818:46
openstackgerritMerged openstack/gertty: Fix rendering error in change screen  https://review.openstack.org/29542318:47
openstackgerritMerged openstack/gertty: Add matchers for external references on launchpad  https://review.openstack.org/29559218:47
openstackgerritMerged openstack/gertty: Use short titles in breadcrumbs  https://review.openstack.org/31003618:47
openstackgerritMerged openstack/gertty: Add extra columns to change list if there is room  https://review.openstack.org/31003718:47
openstackgerritMerged openstack/gertty: Add an option to disable the breadcrumb footer  https://review.openstack.org/31177018:47
fungimorgan: turbo button as a service18:48
morganfungi: hehe18:48
openstackgerritMerged openstack-dev/specs-cookiecutter: Fix spelling error in template.rst  https://review.openstack.org/31180618:49
*** nadya has quit IRC18:50
*** banix has joined #openstack-infra18:50
*** roxanagh_ has joined #openstack-infra18:50
*** banix has quit IRC18:51
*** banix has joined #openstack-infra18:51
pabelangertimothyb89: I'll look at the etherpad and give comments18:52
fungitimothyb89: on the centos images i gather the failure was different enough to possibly be a second problem18:52
*** heyongli_ has quit IRC18:52
*** inc0 has joined #openstack-infra18:52
*** heyongli_ has joined #openstack-infra18:52
timothyb89fungi: right, that was a race condition in the build script, I just pushed a patch on our end to fix:  https://review.openstack.org/31181018:53
*** rbrndt has joined #openstack-infra18:54
*** hashar has joined #openstack-infra18:54
fungitimothyb89: excellent. can you also propose a revert of the revert in project-config which adds the nodejs-legacy package when needed? (or have you already and i missed it?)18:55
*** roxanagh_ has quit IRC18:55
timothyb89fungi: I'm a bit confused about that actually ... in this it seems to be installed already: https://review.openstack.org/#/c/279317/12/nodepool/elements/stackviz/pkg-map.json18:56
*** esker has quit IRC18:56
timothyb89fungi: but in the logs we got: /usr/bin/env: node: No such file or directory18:56
fungitimothyb89: did you see it getting installed earlier in the log?18:56
timothyb89fungi: it used the name configured, so it said it was installing 'nodejs'18:57
timothyb89fungi: 2016-05-02 15:39:08,230 INFO nodepool.image.build.debian-jessie: installing nodejs from stackviz18:58
fungiwas the apt-get stdout included after that? shoul dmention the actual package name(s) installed18:59
timothyb89fungi: ah, no, I'm not seeing it there18:59
nibalizerthe bandersnatch process is still .... there19:00
openstackgerritJoshua Harlow proposed openstack/requirements: UC bumps for pslo releases for week of may 2nd  https://review.openstack.org/31181319:00
nibalizerbut it hasn't put anything new in its log file and stracing it, it is just waiting19:00
openstackgerritJoshua Harlow proposed openstack/requirements: UC bumps for oslo releases for week of may 2nd  https://review.openstack.org/31181319:01
*** esker has joined #openstack-infra19:01
*** heyongli_ has quit IRC19:02
thingeepleia2: ping19:03
*** heyongli_ has joined #openstack-infra19:03
*** piet has quit IRC19:03
*** xarses_ has joined #openstack-infra19:04
timothyb89fungi: hmm, I take it that file should be just `pkg-map` and not `pkg-map.json`? :)19:04
*** xarses has quit IRC19:05
thingeefungi: ping19:05
*** ccamacho has quit IRC19:05
fungitimothyb89: good question. git ls-files says we have two other elements which just call the file "pkg-map" with no extension19:05
fungithingee: clothe thy ping, heathen19:05
fungitimothyb89: so anyway, that would likely explain it19:05
timothyb89fungi: sounds good to me :) any objections if I put the patch up again with corrections?19:06
*** esker has quit IRC19:06
*** jbryce has quit IRC19:06
fungitimothyb89: no objections--quite the opposite, that would be most welcome19:07
thingeefungi, pleia2 not sure who all was part of this summit discussion https://etherpad.openstack.org/p/newton-infra-landing-page-for-contributors ... my apologies for not attending as I had a conflict. But I wanted to say the part that says the work I was doing was "geared towards slide deck + room full" ...19:07
timothyb89fungi: great! thanks for the help!19:07
thingeefungi, pleia2 my work is actually the stuff under how to prepare - http://docs.openstack.org/upstream-training/#how-to-prepare where IRC, account setup and git setup are the essential things19:07
fungithingee: i think the mention of your guide work was mostly in regards to theming and related resources we might want to link19:08
fungithingee: but also, thanks for the clarification19:08
*** jbryce has joined #openstack-infra19:08
thingeefungi: ok, just quoting the etherpad here since there seems to be confusion. But the value of this documentation is to give people something they glance at and feel like they can complete things in five minutes. As oppose to existing documentation that is pretty lengthy, can be intimidating, and is attempting to cover every edge case you'd hit along the way.19:09
fungiyikes, opening our image build logs in firefox is a terrible idea :/19:09
fungithingee: sounds good to me. the discussion was two-pronged. we want to move the how to contribute page off the wiki. it also covers lots of non-code-contributor getting started situations that don't need gerrit/git19:11
*** hdd has quit IRC19:11
thingeefungi: I'm trying to solve for those people that might not be a developer, but want to contribute to OpenStack in other ways. Or even the person that just has a two line fix and doesn't want to spend half a day getting started with our process.19:11
thingeefungi: +1 for moving off the wiki19:11
fungilong live "project.openstack.org"19:12
*** nmagnezi has quit IRC19:12
jeblairthingee: oh, i did not realize you were doing that19:12
thingeefungi: what's the reasoning behind "project" being used here?19:12
*** oanson has joined #openstack-infra19:13
jeblairthingee: we put a lot of effort into making http://docs.openstack.org/infra/manual/developers.html exactly that -- the shortest and simplest steps for a developer to get started19:13
*** heyongli_ has quit IRC19:13
*** heyongli_ has joined #openstack-infra19:13
dirkdims: could you merge https://review.openstack.org/#/c/291769/ ?19:14
*** yolanda has quit IRC19:14
openstackgerritTim Buckley proposed openstack-infra/project-config: Create stackviz element and script in nodepool  https://review.openstack.org/31181519:14
thingeejeblair: yeah, I'd love to collaborate on how we can make things more friendly for people. I've tried these docs with a variety of people in the mentor programs and upstream and felt it was well received.19:14
jeblairthingee: should we just delete the infra manual?19:14
dimsdirk : need a second core19:14
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints  https://review.openstack.org/30137519:14
EmilienMfungi: I moved the script ^19:14
timothyb89fungi: pabelanger: mtreinish: revised patch is here: https://review.openstack.org/31181519:15
dirkdims: apevec is core and he +1ed it19:15
jeblairthe main reason i wrote the infra manual was that we had duplicate documentation, much of which was outdated, wrong, or needlessly complex, and i thought it was important to have just one place with the steps for setting things up as a new developer19:16
dimsdirk : apologies i meant requirements-core folks https://review.openstack.org/#/admin/groups/131,members19:16
thingeejeblair: not at all. I think we can collaborate together on keeping what we have, but make it so things *feel* more digestable. I think it's just maybe separating things from one big page as it is right now.19:16
jeblairthingee: ok cool, that sounds good19:17
jeblairthingee: maybe each of the developer's guide sections should be different sphinx pages?19:18
thingeejeblair: yes19:18
fungii can certainly see where adding more of a "choose your own adventure" feel and hiding less commonly needed details could help19:18
dirkdims: ok, I see. I was apparently looking at the wrong core list.19:18
*** dtardivel has quit IRC19:19
jeblairfungi, thingee: yeah, as thingee says that, i can see how one might mistake the entire developers guide (most of which is a reference) with the minimal 'getting started' section, which is a small (but very important) fraction of the whole19:19
jeblairaccording to my scrollbar, it's about 25%19:20
jeblairand the transition from getting started to development workflow would make a really good page split19:20
thingeejeblair: From the contributor on-board page, I was picturing something static that would allow you to pick what area you want to contribute to, as fungi mentioned "choose your own adventure" kind of feel. No matter what you choose there would be essential stuff like communication (explaining ML and IRC http://docs.openstack.org/upstream-training/irc.html) and then account setup19:21
thingee(http://docs.openstack.org/upstream-training/accounts.html) (http://docs.openstack.org/upstream-training/git.html) ... of course we don't have to use what I worked on here, but that's eventually the bigger picture of things19:21
*** jaosorior has joined #openstack-infra19:22
thingeejeblair: if you pick translator for example, it would give the essential links and then content for the specific stuff the i18n team specifically wants.19:22
*** heyongli_ has quit IRC19:23
*** baoli has quit IRC19:23
*** rockyg has joined #openstack-infra19:23
*** heyongli_ has joined #openstack-infra19:23
fungii'm on board as long as we can keep _one_ place where those steps are documented/encoded so that we don't wind up back in the bad old days of multiple walkthroughs which drifted/skewed/diverged19:23
thingeefungi: +119:24
thingeeThis on-boarding page, and good communication (especially in the docs of starting a new project and mentioning it there) ... would hopefully encourage that idea of one place.19:24
fungiwe've spent lots of time attempting to squash all alternative walkthroughs so that people got a consistent set of steps (particularly around the terrible nuances of gerrit and foundation accounts, cla, et cetera)19:25
jeblairyeah, which is why i'm (seriously) suggesting that maybe we should delete the infra manual :)19:25
AJaegerfungi, I agree - and try constantly to keep things like http://docs.openstack.org/contributor-guide/quickstart/first-timers.html short but it's longer than I loe ;(19:25
*** kzaitsev_mb has joined #openstack-infra19:25
*** rockyg has quit IRC19:26
thingeeAJaeger: I love these docs. I spoke to anne before I started these upstream ones and unfortunately just discovered them on the last day of the summit.19:26
AJaegerthingee: I'm trying hard that we don't duplicate everything in the contributor guide...19:27
thingeeAJaeger: I might've just didn't explain things well for her to point out to me that something similar already exists.19:27
*** rockyg has joined #openstack-infra19:27
*** mylu has joined #openstack-infra19:28
*** yamahata has quit IRC19:28
*** ddieterly is now known as ddieterly[away]19:28
AJaegerjeblair: will you sent a patch to split up the developer's guide page?19:29
thingeeAJaeger: so if the on-board docs had the "essential stuff" like communication, getting account setup, git, workflow, would the docs team be able to use that and in addition to the on-board page having other references to docs team specific tools documentation, etc?19:29
*** fawadkhaliq has quit IRC19:29
*** mylu has quit IRC19:30
AJaegerthingee: I hope so;) Better discuss with annegentle and loquacities.19:30
harlowjaqq, for folks, likely something simple i'm missing here19:31
harlowjai'm trying to create a feature branch for oslo.messaging19:31
harlowjarunning the following19:31
harlowjassh -p 29418 harlowja@review.openstack.org 'gerrit create-branch'  openstack/oslo.messaging feature/dispatch-router 51f3b7b65fff6194bf1c50043bddd649ae6d19ee19:31
harlowjabut ending up getting 'fatal: Cannot create "refs/heads/feature/dispatch-router"'19:31
thingeeAJaeger: not sure if you saw earlier conversation, but the bigger picture here is it's an on-boarding contributor launch page ... choose your own adventure style. Based on what you want to contribute to it would give the essential resources always, and then in that same docs repo we can have page(s) dedicated to very specific contributor resource stuff for different teams.19:31
*** fawadkhaliq has joined #openstack-infra19:31
fungiAJaeger: thingee: jeblair: i suspect it would be less of a deletion of the infra-manual, and more of a (partial?) move of the information, or possibly even we build the project.openstack.org site out of files in infra-manual. we didn't really bikeshed on where the content is maintained, so much as where and how we would publish it19:32
bswartzharlowja: I've gotten that before -- it was acls related19:32
AJaegerthingee: I scanned the earlier discussion19:33
bswartzharlowja: it's possible to create branches inside gerrit IIRC19:33
harlowjahmmmm19:33
*** heyongli_ has quit IRC19:33
*** jaosorior has quit IRC19:33
bswartzI mean thru the UI19:33
AJaegerfungi, thingee : Thanks for explanation. I'm happy to help with this ;)19:33
*** heyongli_ has joined #openstack-infra19:33
*** jaosorior has joined #openstack-infra19:33
jeblairAJaeger: i won't have time this week19:33
thingeeAJaeger: any thoughts? hate it? I think it would help consolidate things, but also finally give us a place that's organized to reference new potential contributors to.19:34
fungiharlowja: i don't see a "create" privilege in openstack-infra/project-config:gerrit/acls/openstack/oslo.messaging.config19:34
AJaegerthingee: I'm all for consolidation ;) So, like the idea.19:34
harlowjahmmmm,  i wonder how the other feature branches for oslo.messaging were made19:34
harlowjahttps://review.openstack.org/#/admin/projects/openstack/oslo.messaging,branches?filter=feature%252F :-/19:34
rcarrillocruzharlowja: unless you belong to 'project bootstrappers' group or you have explicit acl perms on your group, you won't be able to create the branch, no...19:34
rcarrillocruzprobably someone from infra did for oslo team?19:34
clarkbI want to see that the releases team has blankey create perms19:34
thingeeAJaeger: excellent. Just waiting on jeblair and fungi's blessing and I guess I'll start an email to explain things.19:34
clarkbs/see/say/19:34
thingeeAJaeger, fungi, jeblair before going to far with things like splitting, etc19:35
fungiharlowja: gerrit admins and the release management team can create branches by default, others can also create branches on a project-by-project basis with the right acls19:35
harlowjak, so i guess it must of been gerrit admins19:35
fungithose pesky gerrit admins19:35
harlowja:)19:35
*** nadya has joined #openstack-infra19:35
harlowjasoooo, can i request a feature branch for oslo.messaging, or should i just get the acl?19:35
AJaegerthingee: make a cross-project spec ;)19:36
thingeeAJaeger: that's a good idea. I can do that19:37
bswartz"Give me ACLs or give me death!" --Patrick Henry19:37
dirkdims: could you help me understand http://logs.openstack.org/68/311468/1/check/gate-requirements-integration-dsvm/2075ca2/console.html ?19:37
dirkdims: this package definitely exists in that version on pypi.. is there some mirroring cronjob somewhere that needs to be triggered?19:37
harlowjaif i can just request the feature branch, that'd be fine, if someone can do it, `'gerrit create-branch'  openstack/oslo.messaging feature/dispatch-router 51f3b7b65fff6194bf1c50043bddd649ae6d19ee ` can be the commmand someone runs :-P19:38
dimsdirk : the pypi mirroring is broken, infra folks here are working on resurrecting the mirrors19:38
clarkbharlowja: please ask the release team for release managed repos19:38
harlowjak19:39
dimsdirk : the pypi mirrors are listed here - http://docs.openstack.org/infra/system-config/bandersnatch.html19:40
openstackgerritDirk Mueller proposed openstack/requirements: Add spec_cleaner for rpm-packaging style checks  https://review.openstack.org/31146819:40
thingeeAJaeger: maybe here, if the docs team was fine with it becoming a shared thing since they already own it for how to contribute to docs. http://git.openstack.org/cgit/openstack/openstack-manuals/tree/doc/contributor-guide19:40
*** mylu has joined #openstack-infra19:42
thingeeAJaeger, fungi: and this would build to project.openstack.org ... which I'm still not sure why we decided on project here?19:42
AJaegerthingee: best reach out to loquacities on that one. As I said: I'm for consolidation but haven't had enough convincing arguments yet. But if we move forward with project.o.o, I'll happily help to push that.19:42
dirkdims: ouch, ok19:42
*** xarses_ is now known as xarses19:42
AJaegerthingee: I think the idea was to put it on a separate website with an easy to remember name.19:43
AJaegerthingee: and project was the best idea from that session - which I couldn't attend either19:43
*** ccamacho has joined #openstack-infra19:43
*** heyongli_ has quit IRC19:43
fungithingee: there was far too much bike shedding over coming up with a name that indicated this was a landing portal for the upstream project-teams, without ending up with a name that creates a battlefield over atcs vs. non-atcs drama19:43
thingeeI'm trying to make the connection of telling a new contributor go to project.o.o - seems like contributor.o.o would make sense?19:44
*** heyongli_ has joined #openstack-infra19:44
thingeeah19:44
*** mylu has quit IRC19:44
fungithingee: "contributor" was contentious19:44
thingeefungi: I think whatever the non-atc wg comes up with, we can add that in?19:44
thingeewould it still be a problem then? meet up organizing, etc19:45
fungithingee: concern was that documentation for the non-atcs should be under the control of a neutral body and not under the thumb of the tc19:45
*** baoli has joined #openstack-infra19:45
thingeewelcome.o.o ? :P19:46
* thingee takes out some paint buckets19:46
fungiheh19:46
* dims shows up with some brushes19:46
*** baoli_ has joined #openstack-infra19:47
fungianyway, the conflation is that www.openstack.org is the site for the openstack foundation, so project.openstack.org seemed to be the least overloaded and most descriptive name for a landing site for the openstack project19:47
thingeeok well I'll leave that open in the spec and walk away for a week or so19:47
* AJaeger grabs his camera to record the painting session19:47
thingeefungi: kind of seems project.o.o should be navigator realted19:47
thingeethat's my initial confusion with it19:48
AJaegerthingee: or just say, a common page like project.o.o19:48
thingeeproject navigator*19:48
fungithingee: ttx and jeblair had a lot to say on the matter of the site name, so will probably need to be brought in on further naming conversations when they're not vacationing19:48
thingeeAJaeger: I think it's a valid concern that project.o.o can be confused for http://www.openstack.org/software/project-navigator/19:49
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin  https://review.openstack.org/26158719:49
thingeefungi: sure. so I do have some blessing to start cross-project spec on this?19:49
thingeethe idea of moving infra-manual dev doc content over to whatever location?19:50
EmilienMpabelanger: why do we workflow https://review.openstack.org/#/c/311754/ ? are you still testing something?19:50
thingeejeblair: ^19:50
fungithingee: what's cross-project about it?19:50
fungidoesn't seem like it needs changes in lots of projects19:50
*** regXboi has joined #openstack-infra19:50
pabelangerEmilienM: Ya, I want to test it tonight. Make sure we don't break the world tomorrow19:51
thingeefungi: well if it's a one place for learning how to contribute to different efforts, it involves projects, i18n, some working groups and how you contribute to them... etc19:51
*** baoli has quit IRC19:51
AJaegerdocs will come into it. Have some specification writtenup would be nice but it doesn't need a full blown cross-spec.19:51
fungigot it. i have no opposition to a cross-project openstack/openstack-specs spec describing a possible implementation we can pick apart19:52
jeblairi do not think i will have time to contribute19:52
EmilienMpabelanger: ok19:52
*** ddieterly[away] is now known as ddieterly19:52
thingeejeblair: no problem. just wanted to make sure with the little details and direction we have right now that it sounded ok to you.19:53
fungithough it also still seems like it could wait a week or so for the key people from that session to be back at computers19:53
*** _ari_ has quit IRC19:53
thingeefungi: sure no problem. I can delay putting it out. I'll just privately start the spec so things are ready.19:53
*** matrohon has joined #openstack-infra19:53
*** lucasagomes has quit IRC19:53
*** heyongli_ has quit IRC19:54
AJaegerput it out - let's just give some time for others to come back before you promote it...19:54
*** heyongli_ has joined #openstack-infra19:54
*** yamahata has joined #openstack-infra19:58
*** lucasagomes has joined #openstack-infra20:00
*** chem has quit IRC20:02
*** apoorvad has quit IRC20:02
*** matrohon has quit IRC20:03
*** heyongli_ has quit IRC20:04
*** jamesmcarthur has joined #openstack-infra20:05
*** matrohon has joined #openstack-infra20:05
openstackgerritSarafraj Singh proposed openstack/requirements: Bumping version of websockify>=0.8.0  https://review.openstack.org/31182920:06
*** heyongli_ has joined #openstack-infra20:06
jlkCan I get a workflow +2 on https://review.openstack.org/#/c/311778/ ?20:07
*** baoli has joined #openstack-infra20:08
*** baoli_ has quit IRC20:08
*** sdake_ has joined #openstack-infra20:08
*** sdake has quit IRC20:08
*** baoli has quit IRC20:08
*** baoli has joined #openstack-infra20:09
*** jamesmcarthur has quit IRC20:10
pabelangerEmilienM: with puppet tarballs, we need puppet-foo-master.tar.gz, puppet-foo-stable-mitaka.tar.gz and puppet-foo-stable-liberty.tar.gz for each branch right?20:10
*** rbrndt_ has joined #openstack-infra20:10
EmilienMpabelanger: yes20:11
EmilienMpabelanger: it used to work until now20:11
EmilienMonly tags are broken20:11
*** rbrndt has quit IRC20:11
pabelangerEmilienM: right, I am just looking for how puppet-foo.tar.gz is copied into master, stable-mitaka and stable-liberty. Since run-tarballs doesn't do that now20:12
EmilienMpabelanger: it's in the branch-tarball publisher IIRC20:12
EmilienMpabelanger: see in python-jobs.yaml20:12
pabelangerEmilienM: perfect20:13
*** heyongli_ has quit IRC20:14
pabelangerEmilienM: I just need to confirm what a ZUUL_REFNAME looks like for a tag20:14
pabelangerthen happy to remove -1 workflow20:14
*** heyongli_ has joined #openstack-infra20:14
*** kzaitsev_mb has quit IRC20:15
pabelangerZUUL_REFNAME=refs/tags/1.19.22 IIRC is the format20:15
pabelangerEmilienM: okay, looks good20:17
*** nadya has quit IRC20:17
pabelangerEmilienM: the only other issue, is we may need to update the '{name}-tarball' job for puppet, since we don't have any wheels to upload.  That is causing the failure on the job returning a success20:18
EmilienMright20:18
*** mylu has joined #openstack-infra20:19
*** jamesmcarthur has joined #openstack-infra20:21
openstackgerritPradeep Kilambi proposed openstack-infra/tripleo-ci: Add aodh endpoints to enable-tls  https://review.openstack.org/31183320:21
*** mylu has quit IRC20:21
*** marcusvrn__ has joined #openstack-infra20:22
*** markmcd has joined #openstack-infra20:22
*** serverascode_ has joined #openstack-infra20:23
*** vdrok_ has joined #openstack-infra20:23
*** notmorgan has joined #openstack-infra20:24
*** notmorgan has quit IRC20:24
*** notmorgan has joined #openstack-infra20:24
*** pradk has quit IRC20:24
*** Ng_ has joined #openstack-infra20:24
*** morgan has quit IRC20:24
*** notmorgan is now known as morgan20:24
*** xgerman_ has joined #openstack-infra20:24
*** heyongli_ has quit IRC20:24
*** heyongli_ has joined #openstack-infra20:25
*** sridhar_ram has quit IRC20:25
*** mgkwill_ has joined #openstack-infra20:25
*** sheeprine_ has joined #openstack-infra20:25
*** igorbelikov_ has joined #openstack-infra20:26
*** dougwig_ has joined #openstack-infra20:26
*** Sukhdev has quit IRC20:27
*** andreykurilin___ has joined #openstack-infra20:27
*** warpc_ has joined #openstack-infra20:28
*** sylwesterB_ has joined #openstack-infra20:28
*** AJaeger has quit IRC20:28
*** GheRivero_ has joined #openstack-infra20:28
clarkbmtreinish: re https://review.openstack.org/#/c/311256/ you should be able to add a [ -d .tox/$venvname ] to devstack and switch on which of the paths is in use20:28
*** afazekas_ has joined #openstack-infra20:28
*** bapalm_ has joined #openstack-infra20:28
*** LinuxJedi_ has joined #openstack-infra20:28
clarkbmtreinish: then after you switch (remember you have to do it in each branch too), you can remove that and go back to the common venv20:28
*** cbader has quit IRC20:29
*** niska` has quit IRC20:29
*** GheRivero has quit IRC20:29
*** xgerman has quit IRC20:29
*** sylwesterB has quit IRC20:29
*** cgross has quit IRC20:29
*** bapalm has quit IRC20:29
*** Ng has quit IRC20:29
*** peterlisak has quit IRC20:29
*** teran has quit IRC20:29
*** sheeprine has quit IRC20:29
*** flip214 has quit IRC20:29
*** odyssey4me has quit IRC20:29
*** kaisers has quit IRC20:29
*** andreykurilin has quit IRC20:29
*** mgkwill has quit IRC20:29
*** marcusvrn_ has quit IRC20:29
*** markmcd1 has quit IRC20:29
*** afazekas has quit IRC20:29
*** dstanek has quit IRC20:29
*** warpc has quit IRC20:29
*** morgabra has quit IRC20:29
*** mmedvede has quit IRC20:29
*** vdrok has quit IRC20:29
*** bookwar has quit IRC20:29
*** serverascode has quit IRC20:29
*** fungi has quit IRC20:29
*** dougwig has quit IRC20:29
*** dtroyer has quit IRC20:29
*** igorbelikov has quit IRC20:29
*** andreykurilin__ has quit IRC20:29
*** LinuxJedi has quit IRC20:29
*** LinuxJedi_ is now known as LinuxJedi20:29
*** bookwar_ has joined #openstack-infra20:29
clarkbmtreinish: I really do like the common venv idea though, makes a lot of sense for what tempest is trying to do I think and will make jobs a lot quicker20:29
*** cbader has joined #openstack-infra20:29
*** andreykurilin has joined #openstack-infra20:29
*** sylwesterB_ is now known as sylwesterB20:29
*** niska has joined #openstack-infra20:29
*** vdrok_ is now known as vdrok20:29
*** xgerman_ is now known as xgerman20:29
*** warpc_ is now known as warpc20:30
mtreinishclarkb: yeah, that should work. I'll push up a depends on patch(es) for that20:30
*** kaisers has joined #openstack-infra20:30
*** cgross has joined #openstack-infra20:30
*** mgkwill_ is now known as mgkwill20:30
*** marcusvrn__ is now known as marcusvrn_20:30
*** peterlisak has joined #openstack-infra20:30
*** dtroyer has joined #openstack-infra20:30
*** GheRivero_ is now known as GheRivero20:30
*** andreykurilin___ is now known as andreykurilin__20:30
*** dstanek has joined #openstack-infra20:31
clarkbfungi: pabelanger: I am done running errands for the day what is paramiko/ansible situation and are there changes I should be reviewing to help fix that?20:31
*** igorbelikov_ is now known as igorbelikov20:31
*** Ng_ is now known as Ng20:31
*** flip214 has joined #openstack-infra20:31
pabelangerclarkb: fungi: we landed a fix for devstack-gate. Actually EmilienM did20:31
*** baoli has quit IRC20:31
*** morgabra has joined #openstack-infra20:32
pabelangerclarkb: but it appears only tripleo is affected at the moment. They are not using our pypi mirrors, which don't have the new paramiko release yet20:32
clarkbha, the unhappy mirrors are good for something20:32
nibalizeroh did i miss a painting session20:33
*** banix has quit IRC20:33
nibalizerthingee: call it portal.openstack.org, everyone needs a portal20:33
*** fungi has joined #openstack-infra20:33
*** AJaeger has joined #openstack-infra20:34
*** dougwig_ is now known as dougwig20:34
thingeenibalizer: only if it's running off of php nuke.20:34
*** serverascode_ is now known as serverascode20:34
pabelangerclarkb: ++20:34
*** odyssey4me has joined #openstack-infra20:35
*** heyongli_ has quit IRC20:35
*** heyongli_ has joined #openstack-infra20:35
pabelangerclarkb: I have finished centos-7 and ubuntu-trusty dibs pinning paramiko in zuul.  ubuntu-xenial building now, with fedora-23 and debian-jessie to follow20:35
*** hdd has joined #openstack-infra20:35
*** jamesmcarthur has quit IRC20:36
pabelangerI'm also testing ubuntu-precise locally to see what happens20:36
clarkbmtreinish: can we WIP https://review.openstack.org/#/c/297446/ based on what happens iwth common tempest env and plugin support in devstack?20:36
clarkbpabelanger: are you uploading those images too or just building them for the moment?20:36
pabelangerclarkb: just building now, but looks like nodepool is uploading them as I go. Since the original uploads failed20:37
*** teran has joined #openstack-infra20:37
mtreinishclarkb: done20:38
*** sileht has quit IRC20:39
*** jamesmcarthur has joined #openstack-infra20:39
*** jamesmcarthur has quit IRC20:41
*** sileht has joined #openstack-infra20:41
*** pradk has joined #openstack-infra20:41
*** jamesmcarthur has joined #openstack-infra20:41
*** rockyg has quit IRC20:41
*** denisra has quit IRC20:43
*** matrohon has quit IRC20:43
*** jamesmcarthur has quit IRC20:43
*** heyongli_ has quit IRC20:45
*** jaosorior has quit IRC20:45
*** heyongli_ has joined #openstack-infra20:45
clarkbianw: comment on https://review.openstack.org/#/c/296171/2 want to turn that around quickly? I can approve if you are around today20:47
*** mylu has joined #openstack-infra20:48
*** kzaitsev_mb has joined #openstack-infra20:48
*** mylu has quit IRC20:49
*** apoorvad has joined #openstack-infra20:51
*** mmedvede has joined #openstack-infra20:52
*** pradk has quit IRC20:53
*** piet has joined #openstack-infra20:53
openstackgerritMerged openstack-infra/devstack-gate: devstack-vm-gate-wrap: pin paramiko==1.16.0  https://review.openstack.org/31175520:54
*** kzaitsev_mb has quit IRC20:54
*** yamamoto has joined #openstack-infra20:55
*** heyongli_ has quit IRC20:55
*** heyongli_ has joined #openstack-infra20:55
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Add manila-specs repo  https://review.openstack.org/31185320:56
clarkbI am flushing out as many of the d-g changes as I think I can with single reivew based on comments from other cores and things that were approved but needed small corrections20:56
clarkbhopeflly this will let the localrc -> local.conf work happen more smoothly over the next week or two20:57
*** Goneri has quit IRC20:57
*** rfolco has quit IRC20:57
*** inc0 has quit IRC20:58
*** dmk0202 has joined #openstack-infra21:00
openstackgerritJeff Grafton proposed openstack-infra/jenkins-job-builder: Update Google Cloud Storage publisher  https://review.openstack.org/31008021:00
*** sdake_ has quit IRC21:03
*** sdake has joined #openstack-infra21:03
*** kgiusti has left #openstack-infra21:04
openstackgerritJeff Grafton proposed openstack-infra/jenkins-job-builder: Update workspace-cleanup to support newer options  https://review.openstack.org/31008421:05
*** heyongli_ has quit IRC21:05
*** esker has joined #openstack-infra21:05
*** skolekonov__ has joined #openstack-infra21:06
*** heyongli_ has joined #openstack-infra21:06
*** degorenko has quit IRC21:07
*** burgerk__ has joined #openstack-infra21:08
clarkbjlvillal: re https://review.openstack.org/#/c/290171/1, how hard would it be to have ironic just log in the logs dir then delete that code entirely?21:08
*** tojuvone_ has joined #openstack-infra21:08
clarkbjlvillal: if that is an easy change it might be preferable but if it isn't easy I can push 290171 through21:08
*** ccarmack1 has joined #openstack-infra21:08
*** netsin has joined #openstack-infra21:09
*** apoorvad has quit IRC21:09
*** Sukhdev has joined #openstack-infra21:09
*** netsin has quit IRC21:09
*** amrith has quit IRC21:09
openstackgerritMerged openstack-infra/project-config: Opt out of docs jobs for giftwrap  https://review.openstack.org/31177821:09
*** apoorvad has joined #openstack-infra21:09
*** eliqiao_ has quit IRC21:09
*** netsin has joined #openstack-infra21:09
*** rodrigods has quit IRC21:10
*** Guest39041 has quit IRC21:10
*** rodrigods has joined #openstack-infra21:10
*** amrith has joined #openstack-infra21:10
*** Sukhdev has quit IRC21:10
*** ccarmack has quit IRC21:11
*** burgerk_ has quit IRC21:11
*** trown is now known as trown|outtypewww21:11
*** tojuvone has quit IRC21:12
*** dmk0202 has quit IRC21:12
*** eliqiao has joined #openstack-infra21:12
*** jlvillal has quit IRC21:12
*** jlvillal has joined #openstack-infra21:12
*** skolekonov__ has quit IRC21:13
jlkclarkb: hi, are project-config things continuously deployed? Like should I expect that ^^ will be reality in the very near future?21:14
*** amrith is now known as _amrith_21:14
*** xyang1 has quit IRC21:14
clarkbyes21:14
clarkbjlk: but also you could just add a simple set of docs to your project if it is urgent21:15
jlkand to get reviews to retest, its it just comment with "recheck"21:15
clarkbyes21:15
jlkperfect. Not super urgent, just wanted to understand the process21:15
openstackgerritVipin Balachandran proposed openstack/requirements: Add defusedxml to requirements  https://review.openstack.org/31185721:15
clarkbmordred: looks like the bluebox fips that leaked last week are gone, did you clean those out?21:15
mordredclarkb: I did21:15
*** |-paul-| has joined #openstack-infra21:15
mordredclarkb: there are some new ones21:15
mordredbluebox-sjc1 [u'dda930bf-3fc6-4223-8532-adeb48a3d76d', u'f0353ce5-6ce4-4f26-a73c-73addee86ddf']21:16
*** heyongli_ has quit IRC21:16
*** heyongli_ has joined #openstack-infra21:16
*** ldnunes has quit IRC21:17
*** BobBall has quit IRC21:17
clarkbmordred: I want a project called bilge-pump now21:19
mordredclarkb: I can totally rename my clean-fips.py script to bilge-pump :)21:19
clarkbwe can have it restart jenkinses for leaked thread cleanup, and delete leaked fips in clouds21:19
mordredclarkb: oooh. yeah. your thing is better21:19
*** pradk has joined #openstack-infra21:19
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge  https://review.openstack.org/31185921:19
jeblairmordred, pabelanger, nibalizer: i'm thinking that what will happen with the vos release is that it will run to completion before it fails to update the vldb record21:20
jeblairmordred, pabelanger, nibalizer: so i'm starting to wonder if we should kill it, then manually unlock the volume, then vos release again21:21
jeblairsince it's clear at this point, it's doing a complete resync21:21
jeblairand it's already passed the 10 hour credential timeout21:21
clarkbmanually unlock means run ro off of whatever is currently there?21:22
clarkbzaro: looking at review.o.o memory use things are still looking good after the upgrade21:22
clarkbzaro: 21GB mean memory use. Max is 30GB (30GB is the current limit)21:22
clarkbzaro GC time is almost non existant21:23
*** kzaitsev_mb has joined #openstack-infra21:23
pabelangerjeblair: I'll defer to you, but sounds reasonable21:23
*** aysyd has quit IRC21:23
nibalizerjeblair: like pabelanger im hardly an expert21:24
nibalizerbut sure21:24
nibalizeri support killing it21:24
pabelangerclarkb: only fedora-23 and debian-jessie left for dib-image-build21:24
* nibalizer has no faith21:24
pabelangerclarkb: err image-build21:24
nibalizerno faith in things still working normally21:24
*** BobBall has joined #openstack-infra21:25
*** asettle has joined #openstack-infra21:25
*** BobBall is now known as Guest7871721:25
clarkbjeblair: did you see that the zuul 2.5 stack failed in the gate on its first attempt in?21:26
clarkbjeblair: I continued to review the changes but didn't approve any more of them.21:26
*** heyongli_ has quit IRC21:26
openstackgerritMerged openstack-infra/devstack-gate: Update multinode setup doc to VXLAN  https://review.openstack.org/28688021:26
*** julim has quit IRC21:26
*** heyongli_ has joined #openstack-infra21:26
pabelangerianw: if you are around tonight, would use some help debugging: http://paste.openstack.org/show/495912/21:26
pabelangerianw: trying precise ubuntu-minimal image21:27
mordredjeblair: I agree with nibalizer and pabelanger21:27
mordredjeblair: I am unsure of the full consequences of killing and restarting the release after manually removing the lock21:27
mordredjeblair: but as the current operation cannot succeed21:27
mordredjeblair: I would be hard pressed to describe how it could be a WORSE outcome21:27
nibalizeryah21:28
nibalizerthere is also some afs stuff in the mirror.1.gz log file21:28
*** lakshmiS has quit IRC21:29
nibalizer/var/log/bandersnatch/mirror.log.1.gz, even21:29
nibalizerStarting ForwardMulti from 536870932 to 536870932 on afs01.ord.openstack.org (full release).21:29
nibalizerwhatever that means21:29
nibalizerme thinks we should put more 'date' in our scripts21:29
jeblairclarkb: no, the unlock doesn't affect which volumes are used -- currently the vldb says "dfw has new, ord has old" so clients will use dfw and ignore ord21:30
jeblairclarkb: when we unlock the volume, those flags will persist21:30
jeblairclarkb: and the next vos release will copy the (existing) clone volume to ord (again)21:31
jeblairclarkb: and when that completes, assuming it still has credentials (which it should if we run it on the fileserver) it will update the vldb and clear the old/new flags21:31
jeblairso both copies will be used21:31
clarkbgotcha so manual run is mostly to run it on the fileserver with local creds21:32
*** weshay has quit IRC21:33
jeblairclarkb: i did not see the failure, but the zuulv2.5 code is not touched at all by tests, so i suspect either nondeterminism or external forces at play.21:33
jeblairif anyone wants to look into that or recheck this week, that's fine.  otherwise, i will get back to zuulv2.5 next week21:34
jeblairnibalizer: the last dated entry is when the vos release started; all of the undated stuff after that is the output of the vos release command21:34
*** jheroux has quit IRC21:35
clarkbI will likely continue to try working through the stack doing reviews.21:35
jeblairnibalizer: "Starting ForwardMulti" means "copy this volume from here to there"21:35
*** heyongli_ has quit IRC21:36
*** heyongli_ has joined #openstack-infra21:36
*** yamamoto has quit IRC21:37
*** Hal has joined #openstack-infra21:37
nibalizerjeblair: what do the numbers mean? are they versions?21:37
*** yamamoto has joined #openstack-infra21:38
nibalizeri guess it is the same number21:38
jeblairnibalizer: there are volume ids21:38
*** Hal is now known as Guest3185921:38
jeblairnibalizer: so yeah, it's the volume id of the ro volume, which is the same on both servers21:38
*** oanson has quit IRC21:38
jeblair(a vos copy would start the same kind of transaction, but with different volume ids)21:38
nibalizerso i sitll need more dns entries...21:39
nibalizercan we jetison all of infracloud? that stuff is all getting renamed ya?21:39
*** markvan has quit IRC21:39
nibalizercrinkle: yo<tab> rcarrillocruz ^?21:40
clarkbnibalizer: it is all getting reIP'd so even if it doesn't get renamed we are still needing new records for everything21:40
*** yamamoto_ has joined #openstack-infra21:40
funginibalizer: clarkb: crinkle: rcarrillocruz: my recollection was we already agreed it was fine to delete all the current *.ic.openstack.org a and aaaa resource records since they'll have to be updated with new addresses later anyway21:41
pabelangerclarkb: jeblair: I only had an issue with 307515 in the stack while testing. The rest seemed fine. Even added a few patches a top it for ansible21:42
fungiand creating vs modifying them is an equivalent amount of work21:42
fungiand recreating them means we can do it in a separate zone and not hit the maximum record limit for zones in rackspace's dns manglement implementation21:42
*** yamamoto has quit IRC21:43
nibalizeryah21:43
rcarrillocruzYup, fine by me21:43
clarkbnibalizer: can you point me at the traceback/story for the launch node fail you ran into?21:43
clarkbnibalizer: I want ot make sure that gets fixed asap21:44
*** thorst has quit IRC21:44
* clarkb checks email21:44
funginibalizer: as to your earlier complaints about being unable to figure out how to get the rackdns client to list more than the first 100 records, i'm pretty sure i came to the conclusion some time ago that it's a bug (doesn't understand the pagination in their api maybe?)21:45
clarkbhttps://storyboard.openstack.org/#!/story/2000569 email so good21:45
nibalizerfungi: yah21:45
nibalizerhttps://github.com/kwminnick/rackspace-dns-cli/issues/321:45
nibalizerreported 2ish years ago21:45
nibalizerso i tried to patch it, found that setup.py install fails on that project these days..21:46
*** thorst has joined #openstack-infra21:46
*** heyongli_ has quit IRC21:46
*** heyongli_ has joined #openstack-infra21:47
rcarrillocruzsigh, I wish we had some cloud that used Designate v221:47
jeblairnibalizer, clarkb, mordred, pabelanger: i killed the vos release; killed davolser on each of the fileservers (to abort the transfer); unlocked the volume (vos unlock mirror.pypi); and ran vos release on the fileserver in screen21:48
mordredjeblair: woot21:48
nibalizerjeblair: wooo21:48
nibalizerafs wizardry: incanted21:48
*** Sukhdev has joined #openstack-infra21:48
jeblair(bos restarte davolserv automatically, so manually restarting it is not necessary)21:48
nibalizerthanks for hopping in on that21:48
jeblairi also have the bandersnatch lock in screen on mirror-update21:49
jeblairso i will re-enable puppet now21:49
*** jang has quit IRC21:49
*** ccamacho has quit IRC21:49
*** Sukhdev has quit IRC21:49
jeblairnibalizer: np -- i'm trying to help as much as i can today, because i'm only less available as the week goes on21:49
nibalizersure21:50
*** thorst has quit IRC21:50
nibalizerim sure you'll come back on monday to paul and i knowning more about afs haha21:50
openstackgerritMerged openstack-infra/shade: Add Designate recordsets support  https://review.openstack.org/30650221:50
*** amitgandhinz has quit IRC21:51
jeblairi hope so, but only because it was the most fun thing you could think to do :)21:51
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: tripleo.sh: deploy missing deps to run rdopkg  https://review.openstack.org/31186821:51
openstackgerritMerged openstack-infra/devstack-gate: support py3  https://review.openstack.org/30457221:51
jeblair#status log manual vos release of pypi mirror started in screen on fileserver; see https://etherpad.openstack.org/p/fix-afs21:51
openstackstatusjeblair: finished logging21:51
*** yamamoto_ has quit IRC21:53
*** yamahata has quit IRC21:53
*** hashar has quit IRC21:54
*** yamahata has joined #openstack-infra21:54
*** nelsnelson has quit IRC21:54
*** yamamoto has joined #openstack-infra21:55
openstackgerritMerged openstack-infra/devstack-gate: Fix '-f: command not found' error  https://review.openstack.org/27146321:56
openstackgerritMerged openstack-infra/devstack-gate: Consider to log a package list for pip3  https://review.openstack.org/24601721:56
openstackgerritMerged openstack-infra/devstack-gate: Remove doubled configuration in local.conf  https://review.openstack.org/29747521:56
*** banix has joined #openstack-infra21:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: tripleo.sh: deploy missing deps to run rdopkg  https://review.openstack.org/31186821:57
*** heyongli_ has quit IRC21:57
*** earlephilhwoer has quit IRC21:57
*** heyongli_ has joined #openstack-infra21:57
*** tpsilva has quit IRC21:57
*** earlephilhower has joined #openstack-infra21:58
*** Sukhdev has joined #openstack-infra21:58
openstackgerritClark Boylan proposed openstack-infra/system-config: Properly handle volumes in launch node  https://review.openstack.org/31187221:58
clarkbnibalizer: ^ I *think* that will fix it but it is untested21:58
clarkbnibalizer: not sure if you are booting new instances right now, but if so maybe give that a go21:59
*** regXboi has quit IRC21:59
*** tonytan4ever has joined #openstack-infra22:00
*** pilgrimstack has quit IRC22:01
*** ddieterly is now known as ddieterly[away]22:01
nibalizerwell we are still out of dns22:02
clarkbnibalizer: ya but you can delete records easy enough22:02
clarkbwant me to help?22:02
nibalizerso kinda need to https://xkcd.com/1205/ the fixing dns22:02
nibalizeri think morgan is on the rackdns thing22:03
nibalizerwe do need to spin up a new puppetdb server once i work the kinks out22:03
nibalizerand a new storyboard-dev server22:03
*** ddieterly[away] is now known as ddieterly22:03
*** thorst has joined #openstack-infra22:03
clarkbnibalizer: its about 2 minutes of button clicking if you just want it done22:03
clarkbderp I forgot to put the story in my change /me updates things22:04
nibalizerclarkb: oh? i deleted one record and it was anoying22:04
pabelangerjeblair: TIL: kill davolser22:04
nibalizerhundreds sounds :( :(22:04
*** earlephilhower has quit IRC22:04
clarkbnibalizer: I think you just select them then do a bulk op22:04
*** earlephilhower has joined #openstack-infra22:05
*** ddieterly has quit IRC22:05
openstackgerritClark Boylan proposed openstack-infra/system-config: Properly handle volumes in launch node  https://review.openstack.org/31187222:05
nibalizerclarkb: ahh22:06
clarkbok story and commit updated22:06
clarkbif you can use the above change to see if that makes stuff happy that would be excellent22:07
*** heyongli_ has quit IRC22:07
*** mriedem has quit IRC22:07
*** heyongli_ has joined #openstack-infra22:07
jeblairpabelanger: ya, 'vos release' basically just updates the vldb, then asks the sending volume server to copy a volume to the receiving volume server, then they handle it on their own while the vos release command sits there and waits for completion; so killing 'vos release' doesn't actually cause the volume servers to stop their work22:07
*** thorst has quit IRC22:08
*** pvaneck has quit IRC22:08
jeblairpabelanger: (and when vos release receives completion, it updates the vldb again -- but if it has been too long, it tries to do that with expired tokens -- which is the situation we are in that's causing us to have to do this)22:08
pabelangerjeblair: okay, good to know. not that I'll be killing processes between ord and dfw often22:08
jeblairyeah, it's not a good habit to get into22:09
pabelangerjeblair: makes sense22:09
clarkband all this started due to a network issue between ord and dfw?22:09
clarkbwe have good evidence that this is a chronic problem at least outside of afs :/22:09
jeblairclarkb: i think so -- at least, according to the log, i believe a final vldb update failed22:10
pabelangerI'm curious what the interlink between ord and dfw looks like22:10
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Stop execution if ansible fails to install  https://review.openstack.org/31187622:11
pabelangeror is it a cross-connect22:11
clarkbjlvillal: you around now? did you see my question about the ironic old logs change?22:11
jeblairi've been mulling over the best way to rejigger the vos releases so we don't have to do this manually; i think i have some good ideas, and can work on that when i'm back22:11
mordred++22:11
*** fawadkhaliq has quit IRC22:11
* mordred has heard some of the ideas and supports them22:11
jlvillalclarkb: Oh hey. I did not see your question.22:11
* jlvillal looks at scrollback...22:11
*** fawadkhaliq has joined #openstack-infra22:11
openstackgerritMerged openstack-infra/devstack-gate: Fixed link to l3 agent in tempest-logs  https://review.openstack.org/31090222:12
clarkbjlvillal: and for this new change, maybe we can try making d-g use errexit with an exit handler for log cleanup22:12
clarkbjlvillal: though that is bound to make things unhappy until we clean up all the things that have been slipping through22:12
jlvillalclarkb: Okay. I'll have to learn about that.22:13
*** yamamoto has quit IRC22:13
clarkbpretty sure non zero exit codes are expected from the workspace setup function for example :/22:13
jlvillalclarkb: I don't see your question on the ironic old log changes in my scrollback.22:14
clarkbjlvillal: basically can we make ironic log to the log dir and avoid moving anything around at all?22:14
jlvillalWell at least not addressed to me, that I can see22:14
clarkbjlvillal: it is a bit odd that ironic needs a special log dir which leads to these issues22:14
jlvillalclarkb: It has been that way for as long as I can remember. They put the baremetal VM logs into a separate directory.22:15
clarkbjlvillal: ya just asking if we can stop22:15
*** markvoelker has quit IRC22:15
clarkbthen we can delete code instead of adding it22:15
jlvillalHmmm. Maybe, would require a few changes in different places I think.22:16
jlvillalSince there is a hook in place that renames the log file when the VM is shut-down.22:16
jlvillalclarkb: https://github.com/openstack/ironic/blob/master/devstack/files/hooks/qemu22:17
*** heyongli__ has joined #openstack-infra22:17
clarkbjlvillal: looks like if we set LOG_DIR properly it would just work?22:17
jlvillalclarkb: Possibly/probably. Not sure.22:18
jlkargh22:18
jlkis there a way to keep nose unittests from testing a particular python import?22:19
jlkor should I disable the py34 job somehow22:19
clarkbjlvillal: https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L686-L688 appears to be where that ends up getting actually written22:19
mordredjlk: well... we don't usually use nose anywhere anymore - but it is usually possible to control what testr does if you're using that ...22:19
*** yamamoto has joined #openstack-infra22:19
mordredjlk: repo in question and I can take a look?22:19
jlkhttps://review.openstack.org/#/c/31186222:20
clarkbjlvillal: and there is an rm -rf associated with that path :(22:20
clarkbjlvillal: it is just annoying that we solve this problem for devstack then other services decide they are more specialer and we need to resolve it again22:20
jlkit's giftwrap, and nose is chewing through imports and finding bad syntax22:20
openstackgerritDirk Mueller proposed openstack/requirements: Add spec_cleaner for rpm-packaging style checks  https://review.openstack.org/31146822:21
jlvillalclarkb: Understandable. I'm not sure if I have time right now to work on it. But I can put it on my todo list22:21
clarkbjlvillal: ok, I will go ahead and +2 the d-g change22:21
mordredjlk: yah - I'm guessing it doesn't have .tox in its excludes list22:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move ubuntu-xenial voting for infra jobs  https://review.openstack.org/31187822:21
mordredjlk: I'm looking now22:21
jlvillalclarkb: Thanks22:21
jrollclarkb: keep in mind we can generate a whole bunch of logs there22:21
jrollclarkb: typical gate job should only be... maybe 10?22:21
*** heyongli_ has quit IRC22:21
jrollbut from a day of gate debugging...22:22
jroll$ ls ../ironic-bm-logs/ | wc -l22:22
devanandajroll: won't that change when we are able to do tempest full though?22:22
jroll16722:22
jrollyep22:22
clarkbjroll: ya but you only ever want those logs if you are ironicing anyways and they are only generated in that case22:22
clarkbjroll: basically its fine ebcause if ironic then ironic logs22:22
clarkbif not ironic then no logs22:22
clarkbbut this copy pasta of code that is only different because ironic has a special logs dir is silly22:22
jrollclarkb: yeah, I like spammy things, I don't mind. but some people can't handle extra log messages so how will they deal with a million files? :)22:22
devanandaclarkb: having them not pollute the main log dir is appealing to me22:23
jrollsee :)22:23
clarkbdevananda: you can still do that22:23
devanandaoh?22:23
clarkbdevananda: and put them in logs/ironic/22:23
jrollanyway, got a link to this patch?22:23
* JayF calls it, 2y from now someone is going to say it's silly for us to have dozens of bm console logs in the parent dir22:23
clarkbhttps://review.openstack.org/#/c/290171/1/functions.sh is the patch22:23
jrollthanks22:23
jrollOH22:23
clarkbits literally mv path/ironic-logs-are-specialer path/logs/ironic-logs22:23
jrolllogs/ironic-bm-logs/ is appealing to me22:23
jrollyeah22:24
clarkbinstead of just using logs/ironic-logs from the get go22:24
clarkblike a sane person22:24
jroll+2 do it, but let's still call it ironic-bm-logs22:24
jrollfor muscle memory sake if nothing else22:24
jroll(those are the qemu console logs for the VMs we use)22:24
devanandaalso lets mv, not cp22:25
mordredjlk: https://review.openstack.org/31188022:25
jrollclarkb: I don't think that patch is doing the right thing, though22:25
*** burgerk__ has quit IRC22:25
mordredjlk: feel free to squash that into your change if it's helpful22:25
mordredjlk: I do not need the stackalytics stats :)22:25
jlklol22:25
*** jamielennox|away is now known as jamielennox22:25
clarkbjroll: what is the issue?22:26
jrollclarkb: that's still a special case?22:26
mordredjlk: oh! wait. it's still not excluding22:26
devanandaah, nvm. everything else here is copying22:26
mordredone sec22:26
jlkyeah, this.. yeah.22:26
mordredjlk: there are at least 2 issues here which are legit22:26
clarkbjroll: I think it is doing what the new side does and this just adds in support for grenade things22:26
mordredjlk: in that the underlying library is actually not python3 compatible22:26
clarkbjroll: functionally it should be fine22:27
clarkbjroll: if you agree I will go ahead and approve so that logs function then we can worry about cleaning stuff up and rearranging22:27
jlkthat's what I'm seeing now22:27
jlkI think I need to disable the 34 job22:27
jlkor non-vote it22:27
mordredjlk: yah. or update to a newer gitpython22:27
mordredyou're pinning to a very old one22:27
*** heyongli__ has quit IRC22:27
jrollclarkb: I mean, that patch is fine, but why not make that path under logs/ to begin with22:27
mordredbut you could do that as a follow on when you're not trying to get something else done22:27
jlkwas just noticing that22:28
jlknot sure why22:28
clarkbjroll: ya I think it may be partially a devstack thing22:28
*** heyongli__ has joined #openstack-infra22:28
jroll-echo "IRONIC_VM_LOG_DIR=$BASE/$localrc_oldnew/ironic-bm-logs" >>"$localrc_file"22:28
jroll+echo "IRONIC_VM_LOG_DIR=$BASE/$localrc_oldnew/logs/ironic-bm-logs" >>"$localrc_file"22:28
jroll(no clue why that's in d-s-g)22:28
clarkbdevstack is splatting into a gross place and we are trying to rectify?22:28
*** david-lyle has quit IRC22:28
jrollnope, it's all dsg22:28
pabelangerclarkb: welp, time to pin debian-jessie mirrors to something other then http://httpredir.debian.org/debian/22:28
mordredjlk: I get two fixable errors with gitpython 2.0.122:28
pabelangerclarkb: 2 failure today, with more yesterday22:28
jrollclarkb: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L34222:29
jlknod22:29
clarkbjroll: ya because of how devstack works22:29
jlkI'm finding other things to fix up too22:29
clarkbjroll: see line 818 of the patch in question22:29
jlkI think this is going to be a chain of commits22:29
clarkbjroll: basically d-g is doing the ugly thing because everything else does the ugly thing22:29
jrollclarkb: I guess, idk why it can't be a devstack config though22:29
jrollclarkb: I thought the original intent was "put these in the right place to begin with so we don't need to do this extra cp"22:30
jrollbut I see it now22:30
mordredjlk: I've got a clean 34 run now - checking I didn't break 2722:30
jlkI'm getting close, we're racing :)22:30
* clarkb is sad that we have to do this much path munging to copy logs properly22:30
jroll++22:31
mordredjlk: https://review.openstack.org/311882 BOOM!22:31
mordredjlk: :)22:31
clarkbjroll: one time I wanted /etc/openstack/nova etc so that we could just copy /etc/openstack, that didn't get very far22:31
jrollclarkb: that's why I thought you were trying to *not* do this extra munging, I figured everything else was happy :/22:31
jrollheh22:31
mordredjlk: now, I have NO IDEA if giftwrap actually works with latest GitPython :)22:31
jlkmordred: I had those, just not in a commit yet22:31
jlkyeah I'm going to test that too22:31
jrollclarkb: anyway, I gotta run but that seems fine22:32
craige o/22:32
clarkbjroll: we are looking at some fairly largish changes to devstack and d-g this cycile so trying to get on top of current changes and see if we can make things bettererer22:32
mordredjlk: I have excitement for our new giftwrap overlords22:32
mordredlook it's a craige22:32
clarkbbefore we make everything merge conflict22:32
jrollclarkb: awesome22:32
mordredclarkb: speaking of those22:32
mordredclarkb: I've been noodling on the mirror information devstack-gate/devstack thing22:32
mordredclarkb: hoping to have a proposal for specifics by tomorrow (I've hated all of my ideas today)22:32
openstackgerritPaul Belanger proposed openstack-infra/project-config: Pin debian-jessie deb mirror to ftp.us.debian.org  https://review.openstack.org/31188322:33
pabelangerclarkb: I believe that is the correct syntax ^22:33
*** lucasagomes has quit IRC22:36
clarkbpabelanger: there is something funny about http://ftp.fqdn22:36
pabelangerclarkb: don't think so, I was reading from https://www.debian.org/mirror/list22:37
*** heyongli__ has quit IRC22:38
clarkbya it is correct, I just get a kick out of url and hostname not matching22:38
clarkbwe are going to talk http to the ftp server22:38
*** heyongli__ has joined #openstack-infra22:38
* fungi enjoys the history there22:38
*** fawadkhaliq has quit IRC22:38
mordredSTAB STAB STAB STAB STAB22:38
fungiit's "we stuck an http service on our ftp server, serving the same file tree so you could get it via this new-fangled protocol"22:39
mordredI thought we raised the openid auth timeout on gerrit?22:39
pabelangerwe could do ftp vs http22:39
pabelangerdealers choice22:39
clarkbmordred: requires a gerrit restart22:39
clarkbmordred: not sure if anyone has done that second step since change went in22:39
mordredI just lost a non-trivial comment I tried to to write on a review because gerrit has this obsession with logging me out and making me click 3 times to log in again22:40
*** fawadkhaliq has joined #openstack-infra22:40
mordredI would like to set it to never expire22:40
clarkbmordred: there is a "fix for that"22:40
mordredbecause the expiration is completely useless22:40
clarkbhit go anonymous, copy comments, then log in22:40
mordredit solves absolutely nothing22:40
clarkband supposedly newer gerrit will save the stuff to a cookie and restore it post log in22:40
fungiwe've surprisingly _not_ had to restart gerrit recently, thanks to bigger server22:40
pabelangernice22:40
mordredclarkb: awesome. it's great that it doesn't do that across the login boundary22:41
mordredsince "go anonymous" would not imply to me that I wouldn't lose my comment22:41
mordredRAGE STAB STAB22:41
* mordred goes to delete review.openstack.org22:41
*** xarses has quit IRC22:41
*** pradk has quit IRC22:41
*** lucasagomes has joined #openstack-infra22:41
*** banix has quit IRC22:42
*** Guest31859 has quit IRC22:42
StevenKmordred: SURE, that will fix EVERYTHING22:42
dstufftmordred: hilariously, Logged In -> Logged Out is arguably the time when you _should_ be clearing session data.22:43
anteayayay for not having to restart gerrit recently?22:43
dstufft(Assuming "Go Anonymous" is some logged out thing)22:43
*** pradk has joined #openstack-infra22:44
clarkbdstufft: your session expires after 24 hours currently and when you try to do a thing that requires a new login eg leaving a comment it asks you to login (and implicitly lose your data) or go anonymous (and implicitly keep the data)22:44
dstufftclarkb: oh, so this isn't a log out button?22:44
clarkbnope22:45
dstufft(can you tell I totally use gerrit a lot)22:45
dstufftok ignore me then ;P22:45
fungijust google developers thinking it's insane anyone would stay logged into a web service for 24 hours22:45
mordreddstufft: this is the "for some reason we think making you click a button once a day is more better"22:45
clarkbthis was one of the things I pointed out when upstream finally went "well what is wrong with the new interface bceause we think it is amazing"22:46
StevenKBut it turns into a button, a redirect another button and another redirect22:46
clarkband they responded with "well we did fix that one issue very recently, please wait for next gerritrelease"22:46
jlkmordred: new fixes, I had to change some code to make this actually work: https://review.openstack.org/#/c/31186222:46
clarkbsupposedly the mobile diff view will stop sucking in a newer release too22:46
anteayacan you gertty on your phone?22:47
clarkbanteaya: you can ssh to a gertty running elsewhere22:47
dstufftmordred: forcing a new login periodically is not entirely wrong, though if you're going to do that you should take care to not lose state when people submit on old pages22:47
clarkbwhich I have considered doing, but have never really seriously tried22:47
*** heyongli__ has quit IRC22:48
nibalizerok morgan and i are throwing in the towel22:48
* nibalizer presses buttons22:48
anteayaclarkb: ah yes22:49
*** piet has quit IRC22:49
nibalizerhaha now the web ui is erroring22:49
nibalizeryayyyy22:49
*** |-paul-| has quit IRC22:49
anteayaclarkb: I also have never tried22:49
anteayanibalizer: because that is what web ui's do22:49
*** akshai has quit IRC22:50
nibalizer16 out of 18 requests succeeded!22:51
*** heyongli_ has joined #openstack-infra22:53
jeblairmordred: you could change the session cache duration, but i guess it's easier to just be mad at everyone22:54
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Add manila-specs repo  https://review.openstack.org/31185322:54
nibalizeri'm wondering if we need ppbelectsionsregistration.openstack.org or ppbelectionsregistration.openstack.org22:55
morganfixed it!22:55
nibalizertestme.openstack.org also seems like maybe we don't need it22:55
morganpython-novaclient<=2.15.0 (cc nibalizer )22:55
nibalizerwoo22:55
*** davidlenwell has quit IRC22:55
anteayajeblair: he has a reputation to maintain22:56
*** rodrigods has quit IRC22:56
*** rodrigods has joined #openstack-infra22:56
morganjeblair: i vote mad at everyone22:56
morganjeblair: :P22:56
jeblairmordred: http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/templates/gerrit.config.erb22:57
*** rbrndt_ has quit IRC22:57
jeblairmordred: it *used* to be 1 day, but then lifeless set it to 2 minutes.22:57
jlkso, if I'm the PTL, and one of hte few engineers working on a project, is it wrong to +2 or +workflow my own changes?22:57
*** rbrndt_ has joined #openstack-infra22:58
jeblairmordred: you approved that change: https://review.openstack.org/29553022:58
*** rbrndt_ has quit IRC22:58
*** heyongli_ has quit IRC22:58
*** heyongli_ has joined #openstack-infra22:58
jeblairmordred: considering all of that, i think that, in retrospect, that rage was not warranted and would appreciate less rageing and stabbing.23:00
morganjlk: really depends23:00
morganjlk: if you have critical mass of cores from disparate companies, we tend to have a soft rule about +2/+A23:01
morganjlk: but to be perfectly honest, I am of the opinion we trust people in the community until they give us a reason not to23:01
mordredjeblair: fair enough23:01
morganjlk: sometimes you need to +2/+A your own things at times23:01
morganjlk esp. with a very small core team23:02
jlkalright, I'll use it sparingly23:02
morganjlk: ++23:02
nibalizeroh sweet, blessed pagination23:03
jeblairmordred: maybe the timeout should be increased to something like 2-3, or even 7 days?23:03
*** davidlenwell has joined #openstack-infra23:03
openstackgerritMerged openstack-infra/irc-meetings: Add Moshe Levi as sriov meeting chair  https://review.openstack.org/31147223:04
jeblairi'm not sure i dig '2 months'23:04
*** xarses has joined #openstack-infra23:04
nibalizerfungi: i'm thinking im just gonna post our entire openstack.org dns to a paste for everyone to look at23:04
jeblair(which is likely what lifeless was going for)23:04
nibalizerthere are definitely some records we can lose23:04
nibalizerunless you can think of an issue with doing that23:04
*** doug-fis_ has quit IRC23:04
*** doug-fish has joined #openstack-infra23:05
mordredjeblair: yah - I believe he was going for 2 months, which seems fine to me - but my opinion on appropriate length should likely be discounted at the moment23:06
mordredjeblair: I'm happy to follow that change with one that sets to something more appropriate23:06
*** thorst has joined #openstack-infra23:07
*** heyongli_ has quit IRC23:08
nibalizerERROR: OverLimit Retry... (HTTP 413)23:10
nibalizerhow silly of me, thinking i could delete 1 record every 3 seconds23:11
*** heyongli_ has joined #openstack-infra23:11
funginibalizer: yeah, i sent 'round a list of those idk subdomains a while back (maybe a year ago?) and removed the ones people got back to me saying were unneeded. unfortunately a bunch of those are possibly created by the foundation staff and i never heard any confirmation i could delete them so left them there23:11
funginibalizer: and making a paste is fine--there's nothing "secret" in our domains23:12
mordredfungi: what about the secret domains though?23:12
fungimordred: since i don't know about them, i'm guessing they're extremely secret23:12
bswartzopenstack.xxx?23:13
funginibalizer: i'm cool with cleanup, just cautious since the infra team aren't the only people creating records in that domain23:14
mordredbswartz: sadly, someone already registered openstack.ninja23:15
*** markvoelker has joined #openstack-infra23:16
openstackgerritMonty Taylor proposed openstack-infra/puppet-gerrit: Change default login cache length to 7 days  https://review.openstack.org/31188823:16
fungimordred: your repeated mention of "the gathering" makes me think of something other than openstack project teams ;)23:16
* mordred taps three forests and summons a wall23:17
nibalizerhttp://dpaste.com/2W5F0HC23:17
*** ccarmack1 has quit IRC23:17
*** larainema has quit IRC23:17
nibalizer(our paste server doesn't allow them to be as long as the file is)23:17
bkeroThe troll has flying. It attacks.23:17
nibalizeri'm getting rid of the infacloud ones (currently suffering from the api rate limiting blues)23:18
bswartzomg ninjas23:18
*** larainema has joined #openstack-infra23:18
*** heyongli_ has quit IRC23:19
*** heyongli_ has joined #openstack-infra23:19
*** fawadkhaliq has quit IRC23:20
*** markvoelker has quit IRC23:21
pabelangerclarkb: all dibs rebuilt for today. However, I have not uploaded any images to clouds but nodepool image-list appears happy (0 building)23:21
pabelangerclarkb: I wonder if that is not correct23:21
*** dingyichen has joined #openstack-infra23:21
*** yamamoto has quit IRC23:21
clarkbpabelanger: you can check the builder log to see if uploads all failed or whatever23:21
clarkbaround now is when I expect them to be done though23:22
*** Sukhdev has quit IRC23:22
clarkbI would like to move our cron back a few hours too, /me will propose that change now23:22
funginibalizer: yes, that particular dns service regularly makes me want to just set up some bind 10 servers. and then i remember that i'd need to provide some mechanism for random other non-infra people at the foundation to be able to make near-instantaneous changes to the domain23:22
*** yamamoto has joined #openstack-infra23:23
*** tonytan4ever has quit IRC23:23
jlkquestion on work flow23:23
*** mylu has joined #openstack-infra23:23
jlksomebody sends up a review.  It doesn't pass CI due to other issues. I fix those other issues (which moves master along). Does the original review have to be rebased, or will a recheck work?23:23
fungijlk: recheck23:24
nibalizerfungi: so vexxhost23:24
nibalizeralso lol @ 10 requests/second rate limiting23:24
pabelangerclarkb: okay, I see the uploads. neat23:24
fungijlk: zuul attempts to merge the commit to the tip of the target branch, and tests it that way23:24
jlkperfect23:25
*** thorst has quit IRC23:25
*** fawadkhaliq has joined #openstack-infra23:25
pabelangerclarkb: also, https://review.openstack.org/#/c/311721/ when you have spare cycles.  Starting to change out fedora-22 on tripleo CI23:25
*** Sukhdev has joined #openstack-infra23:25
openstackgerritClark Boylan proposed openstack-infra/project-config: Move image update cron back  https://review.openstack.org/31189023:26
pabelangerhttps://review.openstack.org/#/c/294339/ could also use a review, that is our nodepool-builder worker settings23:26
clarkbpabelanger: ^ something like that to shift everything back23:26
pabelangerclarkb: checking23:26
*** pahuang has joined #openstack-infra23:27
pabelangerclarkb: WFM23:27
*** thorst has joined #openstack-infra23:28
*** heyongli_ has quit IRC23:29
*** heyongli_ has joined #openstack-infra23:29
asselin_fungi, jlk really? I though that was only for dependent pipeline, gate checks. I though check was exactly what the patch is based off of...similar to git review -d23:30
*** mylu has quit IRC23:30
*** mylu has joined #openstack-infra23:30
jlkcrap, i need a gpg key...23:30
clarkbasselin_: yes, it tests forward state even in check23:31
*** heyongli_ has quit IRC23:31
fungiasselin_: really. there's little point in it testing your change with something other than the branch tip23:31
*** sputnik13 has quit IRC23:31
clarkbthat forward state is based off of the target branch though, not off of a proposed future23:31
*** gyee has quit IRC23:32
*** keedya has joined #openstack-infra23:33
jeblairclarkb, asselin_, jlk: what clarkb says is true, except you can also use depends-on to pull another patch into the test scenario in check23:33
*** dangers is now known as dangers_away23:34
fungiand also it stacks your parent changes if there are any in between the change and the branch tip23:35
*** Guest66363 has quit IRC23:35
*** mylu has quit IRC23:36
*** keedya has quit IRC23:37
*** keedya has joined #openstack-infra23:38
asselin_ok, good to know23:38
openstackgerritMerged openstack-infra/irc-meetings: Remove the ambassadors meeting  https://review.openstack.org/31092823:39
*** sdake_ has joined #openstack-infra23:41
*** mriedem has joined #openstack-infra23:41
openstackgerritMerged openstack-infra/irc-meetings: Move Tacker team meeting to compensate DST  https://review.openstack.org/31178923:41
*** yamamoto has quit IRC23:42
*** sdake has quit IRC23:43
*** gordc has quit IRC23:43
lifelessjeblair: argh. 2minutes? EIFAIL23:44
lifelessjeblair: yes, I was going for 2 months23:44
lifelessjeblair: is there a fix-up patch already up, or should I submit one?23:45
jeblairlifeless: mordred has https://review.openstack.org/31188823:45
lifelessjeblair: ack, thanks23:45
*** flepied has joined #openstack-infra23:45
fungioh, yikes23:45
jeblairso, erm, nobody restart gerrit :)23:46
*** doug-fish has quit IRC23:46
*** banix has joined #openstack-infra23:46
fungiroger that23:47
jeblairin fact...23:47
fungiapproving as soon as i see check results pass for this23:47
*** doug-fish has joined #openstack-infra23:47
clarkbhttps://gerrit-review.googlesource.com/Documentation/config-gerrit.html#cache.name.maxAge says 'days' is good23:47
jeblair#status log due to an error in https://review.openstack.org/295530 which will be corrected in https://review.openstack.org/311888 gerrit should not be restarted until the second change lands23:47
openstackstatusjeblair: finished logging23:47
jeblairinfra-root: ^23:47
* jeblair uses tools23:47
fungitools!23:48
clarkbhttp://logs.openstack.org/88/311888/1/check/gate-puppet-gerrit-puppet-beaker-rspec-ubuntu-trusty/c684427/console.html#_2016-05-02_23_45_51_91023:48
clarkbI am thinking that test fail is due to broken tests/manifests23:48
jeblairclarkb: i was afraid of that, but haven't quite gotten into the habit of finding where the errors actually start in the beaker tests23:49
*** heyongli_ has joined #openstack-infra23:49
clarkbjeblair: it is right where I linked, looks like the reindex step timed out?23:49
jeblair(search for "Failures:" seems to approximate that)23:49
jeblairclarkb: yeah, i'm just saying i scanned a few hundred lines and gave up :)23:49
clarkbwhich is odd considering there should be nothing to index23:49
jeblairi also saw some bouncycastle errors and thought "internet?")23:50
*** doug-fis_ has joined #openstack-infra23:51
*** mylu has joined #openstack-infra23:51
*** chlong has joined #openstack-infra23:52
*** doug-fish has quit IRC23:52
clarkbthe reindex fail would fail to create the init script symlink if there is a subscribe and refreshonly23:52
clarkbactually if the init fails then there may not be anything to symlink to, reading the manifest I Think that is actually how it is failing. The init (index step specifically) fails leaving the symlink dangling23:54
*** yamamoto has joined #openstack-infra23:54
jeblairclarkb: yeah, i think it may actually be the bouncycastle thing23:55
*** Swami has quit IRC23:55
clarkbhttp://logs.openstack.org/88/311888/1/check/gate-puppet-gerrit-puppet-beaker-rspec-ubuntu-trusty/c684427/console.html#_2016-05-02_23_40_42_50823:55
clarkbjeblair: ^ and bouncy castle causes that I think?23:55
jeblairclarkb: bouncecastle causes initial-init to fail, which causes reindex to fail (because it is not initted)23:55
*** doug-fis_ has quit IRC23:55
*** sdake_ has quit IRC23:55
jeblairya23:55
clarkbhttps://downloads.bouncycastle.org/java/bcprov-jdk16-144.jar is a 40423:55
*** yamamoto has quit IRC23:55
jeblairthatll do it23:56
*** mtanino has quit IRC23:56
fungiwhoops23:56
*** sdake has joined #openstack-infra23:56
* fungi starts to rant about people deleting releases of things, and then realizes it won't help23:56
*** sdake has quit IRC23:56
*** sdake has joined #openstack-infra23:57
jeblairthere's bouncycastle stuff in the puppet module -- does our gerrit think it needs something newer and so downloads it instead?23:58
*** doug-fish has joined #openstack-infra23:58
*** piet has joined #openstack-infra23:58
jeblairalso, different jdk23:58
*** sdake has quit IRC23:58
fungithe release of gerrit we're running wants a newer bc than the debs on trusty, if memory serves23:59
clarkbjeblair: yes iirc precise/trusty have not been able to keep up with a new enough bouncy castle to do gerrit things23:59
*** heyongli_ has quit IRC23:59
clarkbhttps://www.bouncycastle.org/releasenotes.html if your release number and version number don't match...23:59
jeblairclarkb: right, but the thing we have in puppet is 1.51 on jdk1.5; gerrit is downloading 1.44 on jdk 1.623:59
*** heyongli_ has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!