Monday, 2014-02-24

*** openstack has joined #openstack-infra00:03
*** yassine has quit IRC00:04
*** oubiwan__ has quit IRC00:05
*** matsuhashi has joined #openstack-infra00:09
*** Hunner has quit IRC00:12
*** Hunner has joined #openstack-infra00:20
*** milki_ is now known as milki00:23
*** jcooley_ has quit IRC00:31
*** fifieldt has joined #openstack-infra00:32
*** gokrokve has joined #openstack-infra00:34
*** gokrokve has quit IRC00:39
morganfainberglifeless, fungi, mordred, clarkb, this review seems wedged somehow.  it's not hitting zuul, do i need to un-approve it then recheck?00:45
morganfainbergit looks like the verify job was completely lost at one point...and never requeued00:45
*** mriedem has joined #openstack-infra00:46
*** davidhadas_ has joined #openstack-infra00:46
morganfainbergi can do a new patchset that changes something to force it back through check if needed.00:46
morganfainbergbut not sure if you guys want to look at it first00:47
lifelessI would recheck no bug; not that that might be case sensitive :)00:47
morganfainberglifeless, lol let me try it w/ case corrected00:47
morganfainberglifeless, but if thats not working i'll goahead and do a trivial change.00:48
*** davidlenwell_ has joined #openstack-infra00:48
morganfainberglifeless *grumble* typo in casesensitivity :P00:48
*** juice- has joined #openstack-infra00:48
morganfainberglifeless, thanks00:48
morganfainbergjenkins seems to be chewing on it again00:49
*** fifieldt has quit IRC00:49
*** flaper87 has quit IRC00:49
*** bodepd_ has quit IRC00:49
*** plomakin_ has quit IRC00:49
*** andre__ has quit IRC00:49
*** dosaboy has quit IRC00:49
*** davidhadas has quit IRC00:49
*** dachary has quit IRC00:49
*** ogelbukh has quit IRC00:49
*** bookwar has quit IRC00:49
*** lyxus has quit IRC00:49
*** vipuls has quit IRC00:49
*** juice has quit IRC00:49
*** jang1 has quit IRC00:49
*** davidlenwell has quit IRC00:49
*** juice- is now known as juice00:51
*** fifieldt has joined #openstack-infra00:52
*** flaper87 has joined #openstack-infra00:52
*** bodepd_ has joined #openstack-infra00:52
*** plomakin_ has joined #openstack-infra00:52
*** andre__ has joined #openstack-infra00:52
*** dosaboy has joined #openstack-infra00:52
*** dachary has joined #openstack-infra00:52
*** ogelbukh has joined #openstack-infra00:52
*** bookwar has joined #openstack-infra00:52
*** lyxus has joined #openstack-infra00:52
*** vipuls has joined #openstack-infra00:52
*** jang1 has joined #openstack-infra00:52
*** dolphm_503 is now known as dolphm00:52
*** alexpilotti has quit IRC00:54
*** oubiwan__ has joined #openstack-infra01:01
*** oubiwan__ has quit IRC01:06
*** nosnos has joined #openstack-infra01:07
*** sdake_ has joined #openstack-infra01:15
*** sdake_ has quit IRC01:23
*** zhiwei has joined #openstack-infra01:25
*** Guest633 has quit IRC01:30
zhiweiclarkb: hi01:33
*** gokrokve has joined #openstack-infra01:34
openstackgerritA change was merged to openstack-infra/config: Rename Openstack to OpenStack
*** gokrokve has quit IRC01:38
*** nati_ueno has joined #openstack-infra01:42
*** thomasbiege has joined #openstack-infra01:45
*** nati_ueno has quit IRC01:47
*** unicell has joined #openstack-infra01:48
*** morganfainberg is now known as morganfainberg_Z01:56
*** morganfainberg_Z is now known as morganfainberg01:57
*** oubiwan__ has joined #openstack-infra02:02
*** oubiwan__ has quit IRC02:07
*** masayukig has joined #openstack-infra02:08
openstackgerritlifeless proposed a change to openstack-infra/config: Install libffi-dev needed for python-glanceclient
*** yamahata has joined #openstack-infra02:10
lifelessclarkb: around ? ^ is that sensible? Or does jenkins have sudo access?02:11
Alex_Gaynorlifeless: did that come up via pyOpenSSL?02:11
lifelessAlex_Gaynor: probably -
lifeless2014-02-24 01:17:25.540 | Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->python-glanceclient)02:11
Alex_GaynorLooks like yeah.02:11
lifelessStevenK: ah, we do have sudo02:12
StevenKAh, nice.02:13
lifelesslocal tripleo-ci change proposed02:16
lifelessthat should fail in the image02:16
lifelessif it does then we can land it and go to the image one and recheck02:16
*** yaguang has joined #openstack-infra02:20
*** CaptTofu has quit IRC02:26
openstackgerritZang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP
*** Ryan_Lane1 has joined #openstack-infra02:31
*** gokrokve has joined #openstack-infra02:34
*** thomasbiege has quit IRC02:34
*** vkozhukalov has joined #openstack-infra02:35
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters.
*** gokrokve has quit IRC02:39
*** sarob has quit IRC02:49
*** sarob has joined #openstack-infra02:50
*** CaptTofu has joined #openstack-infra02:53
*** sarob has quit IRC02:54
*** dolphm is now known as dolphm_50302:57
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Add oslo.vmware
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Rename slave_template to single_use_slave
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Support filtering by review id(s)
*** dolphm_503 is now known as dolphm02:59
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
*** pcrews has joined #openstack-infra03:01
*** oubiwan__ has joined #openstack-infra03:03
*** davidhadas has joined #openstack-infra03:05
*** davidhadas_ has quit IRC03:08
*** oubiwan__ has quit IRC03:08
openstackgerritA change was merged to openstack-infra/devstack-gate: Start compressing config files too
openstackgerritA change was merged to openstack-infra/devstack-gate: Add support for running tempest serially without tenant isolation
*** dolphm is now known as dolphm_50303:14
*** pcrews has quit IRC03:15
openstackgerritA change was merged to openstack-infra/devstack-gate: Support grenade-svm-rolling rename to *-partial-ncpu
*** sarob has joined #openstack-infra03:17
*** coolsvap has joined #openstack-infra03:20
*** CaptTofu has quit IRC03:22
*** dolphm_503 is now known as dolphm03:23
*** mwagner_lap has joined #openstack-infra03:26
*** SpamapS_ is now known as SpamapS03:30
*** SpamapS has quit IRC03:30
*** SpamapS has joined #openstack-infra03:30
*** pmathews has joined #openstack-infra03:33
*** talluri has joined #openstack-infra03:33
*** gokrokve has joined #openstack-infra03:34
*** pballand has joined #openstack-infra03:35
*** thomasbiege has joined #openstack-infra03:37
*** matsuhashi has quit IRC03:38
*** gokrokve has quit IRC03:39
*** dstanek has joined #openstack-infra03:40
*** david_lyle_ has joined #openstack-infra03:40
*** david_lyle_ is now known as david_lyle03:43
*** pballand has quit IRC03:49
*** pballand has joined #openstack-infra03:50
*** pballand has quit IRC03:54
*** miqui_ has quit IRC04:01
*** mriedem has quit IRC04:02
*** talluri has quit IRC04:02
*** lnxnut has joined #openstack-infra04:05
*** Hunner has quit IRC04:06
*** Hunner has joined #openstack-infra04:12
*** nati_ueno has joined #openstack-infra04:14
*** matsuhashi has joined #openstack-infra04:22
*** nati_ueno has quit IRC04:23
*** thomasbiege has quit IRC04:29
*** sirushti is now known as shortstop04:45
*** sarob has quit IRC04:49
*** zhiyan_ is now known as zhiyan04:49
*** sarob has joined #openstack-infra04:50
*** afazekas has joined #openstack-infra04:50
*** sarob_ has joined #openstack-infra04:51
*** sarob has quit IRC04:54
*** sarob_ has quit IRC05:00
*** amotoki has joined #openstack-infra05:01
*** chandan_kumar has joined #openstack-infra05:08
*** sarob has joined #openstack-infra05:09
*** chandan_kumar has quit IRC05:12
*** matsuhas_ has joined #openstack-infra05:14
*** lnxnut has quit IRC05:14
*** matsuhashi has quit IRC05:16
*** lnxnut has joined #openstack-infra05:22
*** CaptTofu has joined #openstack-infra05:23
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client
*** lnxnut has quit IRC05:26
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client
*** matsuhashi has joined #openstack-infra05:27
*** CaptTofu has quit IRC05:27
*** matsuhas_ has quit IRC05:30
*** nicedice_ has quit IRC05:30
openstackgerritmark mcclain proposed a change to openstack-infra/reviewstats: Add Oleg Bondarev to Neutron core
*** matsuhas_ has joined #openstack-infra05:33
*** matsuhashi has quit IRC05:36
*** dolphm is now known as dolphm_50305:43
*** nosnos_ has joined #openstack-infra05:43
*** nosnos has quit IRC05:43
*** jcooley_ has joined #openstack-infra05:44
*** jcooley_ has quit IRC05:47
*** jcooley_ has joined #openstack-infra05:49
*** jcooley_ has quit IRC05:49
openstackgerritKhai Do proposed a change to openstack-infra/config: remove flaky gerrit tests
*** kashyap has joined #openstack-infra05:55
*** chandan_kumar has joined #openstack-infra05:55
*** matsuhas_ has quit IRC06:00
*** matsuhashi has joined #openstack-infra06:01
*** mrda is now known as mrda_away06:03
*** lnxnut has joined #openstack-infra06:05
*** matsuhas_ has joined #openstack-infra06:05
*** matsuhashi has quit IRC06:05
*** lnxnut has quit IRC06:06
*** lnxnut has joined #openstack-infra06:07
*** lnxnut has quit IRC06:11
*** sarob has quit IRC06:17
*** pmathews has quit IRC06:17
*** sarob has joined #openstack-infra06:18
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers
*** matsuhas_ has quit IRC06:21
*** matsuhashi has joined #openstack-infra06:22
*** matsuhashi has quit IRC06:22
*** sarob has quit IRC06:22
*** matsuhashi has joined #openstack-infra06:23
*** jcooley_ has joined #openstack-infra06:25
*** jishaom has joined #openstack-infra06:28
*** jcoufal has joined #openstack-infra06:30
*** nosnos_ has quit IRC06:32
*** nosnos has joined #openstack-infra06:32
*** matsuhashi has quit IRC06:33
*** ArxCruz has quit IRC06:36
*** matsuhashi has joined #openstack-infra06:38
*** jcooley_ has quit IRC06:42
*** dolphm_503 is now known as dolphm06:44
*** pblaho has joined #openstack-infra06:49
*** vkozhukalov has quit IRC06:51
*** gokrokve has joined #openstack-infra06:51
*** gokrokve_ has joined #openstack-infra06:52
*** dolphm is now known as dolphm_50306:53
*** gokrokve has quit IRC06:55
*** saju_m has joined #openstack-infra06:58
*** Hunner has quit IRC07:01
*** Hunner has joined #openstack-infra07:02
*** nati_ueno has joined #openstack-infra07:03
tchaypoI'm making a change to openstack_project::slave and want to test it on both precise and fedora19 to make sure it works on both supported platforms, so I find myself writing a multi-box vagrant file. It's not a lot of work, but I'm wondering if someone has done something like this for testing changes already..07:04
*** lnxnut has joined #openstack-infra07:05
*** lnxnut has quit IRC07:06
*** lnxnut has joined #openstack-infra07:06
*** nati_ueno has quit IRC07:10
*** lnxnut has quit IRC07:11
*** gokrokve_ has quit IRC07:12
*** CaptTofu has joined #openstack-infra07:24
*** CaptTofu has quit IRC07:28
*** e0ne has joined #openstack-infra07:31
*** DinaBelova has joined #openstack-infra07:34
*** nati_ueno has joined #openstack-infra07:36
*** davidhadas has quit IRC07:38
*** e0ne_ has joined #openstack-infra07:41
jishaomI'm testing my jobs using zuul+gearman, but I meet an error, error info "zuul.Scheduler: Run handler sleeping ,DEBUG zuul.Gearman: Looking for lost builds". It always shows 'Looking for lost builds' and zuul can't work. Did anyone meet this error and how to fix it?07:42
*** flaper87 has quit IRC07:42
*** flaper87 has joined #openstack-infra07:42
*** e0ne has quit IRC07:43
*** luqas has joined #openstack-infra07:46
*** luqas has quit IRC07:49
*** matsuhashi has quit IRC07:52
*** matsuhashi has joined #openstack-infra07:52
*** gokrokve has joined #openstack-infra07:53
*** jamielennox is now known as jamielennox|away07:54
*** dstanek_afk has joined #openstack-infra07:54
*** plomakin has joined #openstack-infra07:58
*** bodepd has joined #openstack-infra07:59
openstackgerritjames Polley proposed a change to openstack-infra/config: Make sure gawk is installed
*** jang has joined #openstack-infra07:59
*** fesp has joined #openstack-infra08:00
*** dizquierdo has joined #openstack-infra08:00
*** dachary_ has joined #openstack-infra08:02
*** vipul has joined #openstack-infra08:02
*** dstanek has quit IRC08:03
*** fifieldt has quit IRC08:03
*** flaper87 has quit IRC08:03
*** bodepd_ has quit IRC08:03
*** plomakin_ has quit IRC08:03
*** andre__ has quit IRC08:03
*** dosaboy has quit IRC08:03
*** dachary has quit IRC08:03
*** ogelbukh has quit IRC08:03
*** bookwar has quit IRC08:03
*** lyxus has quit IRC08:03
*** vipuls has quit IRC08:03
*** jang1 has quit IRC08:03
*** dachary_ is now known as dachary08:04
*** fesp is now known as flaper8708:04
*** dosaboy has joined #openstack-infra08:04
*** morganfainberg is now known as morganfainberg_Z08:06
*** rlandy has joined #openstack-infra08:06
*** lnxnut has joined #openstack-infra08:06
*** oubiwan__ has joined #openstack-infra08:08
*** gcb has joined #openstack-infra08:08
*** lyxus has joined #openstack-infra08:08
*** andre__ has joined #openstack-infra08:09
*** andre__ has quit IRC08:09
*** oubiwan__ has quit IRC08:09
*** ogelbukh has joined #openstack-infra08:10
*** lnxnut has quit IRC08:11
*** matrohon has joined #openstack-infra08:12
*** bookwar has joined #openstack-infra08:15
*** fifieldt has joined #openstack-infra08:15
*** andre__ has joined #openstack-infra08:15
*** oubiwan__ has joined #openstack-infra08:15
*** oubiwan__ has quit IRC08:16
*** luqas has joined #openstack-infra08:17
*** sarob has joined #openstack-infra08:19
*** jcooley_ has joined #openstack-infra08:21
*** jgallard has joined #openstack-infra08:23
*** sarob has quit IRC08:23
*** jcooley_ has quit IRC08:28
*** ociuhandu has joined #openstack-infra08:37
*** vkozhukalov has joined #openstack-infra08:37
*** garyk has joined #openstack-infra08:37
*** boris-42_ has quit IRC08:37
*** hashar has joined #openstack-infra08:40
*** mattymo has joined #openstack-infra08:42
*** rossella-s has joined #openstack-infra08:42
*** DinaBelova is now known as DinaBelova_08:43
*** davidhadas has joined #openstack-infra08:45
*** davidhadas_ has joined #openstack-infra08:46
*** davidhadas__ has joined #openstack-infra08:46
*** persia has joined #openstack-infra08:50
*** persia is now known as Guest8827108:51
*** sahid has joined #openstack-infra08:51
*** Guest88271 has quit IRC08:51
*** Guest88271 has joined #openstack-infra08:51
*** Guest88271 is now known as persia08:51
*** tteggel_ is now known as tteggel08:53
*** dizquierdo has quit IRC08:54
*** nati_ueno has quit IRC08:57
*** yassine has joined #openstack-infra09:01
*** yassine_ has joined #openstack-infra09:01
*** nati_ueno has joined #openstack-infra09:04
*** fbo_away is now known as fbo09:04
*** lnxnut has joined #openstack-infra09:05
*** zhiwei has quit IRC09:06
*** Guest86727 is now known as bobba09:09
*** bobba is now known as BobBall09:10
*** lnxnut has quit IRC09:10
*** derekh has joined #openstack-infra09:17
*** sarob has joined #openstack-infra09:20
*** CaptTofu has joined #openstack-infra09:25
*** sarob has quit IRC09:26
*** CaptTofu has quit IRC09:29
*** gcb has quit IRC09:31
*** zhiwei has joined #openstack-infra09:34
*** matel has quit IRC09:35
*** luqas has quit IRC09:35
*** nati_ueno has quit IRC09:36
*** afrittoli has joined #openstack-infra09:37
*** afrittoli has quit IRC09:38
*** andreaf has joined #openstack-infra09:38
*** johnthetubaguy has joined #openstack-infra09:39
*** johnthetubaguy1 has joined #openstack-infra09:41
openstackgerritTimur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-devstack job
*** johnthetubaguy has quit IRC09:42
ihrachyshub_cap: ping09:43
*** nati_ueno has joined #openstack-infra09:43
*** saju_m has quit IRC09:46
*** e0ne_ has quit IRC09:50
*** saju_m has joined #openstack-infra09:50
*** e0ne has joined #openstack-infra09:51
*** gcb has joined #openstack-infra09:54
openstackgerritChangBo Guo(gcb) proposed a change to openstack/requirements: Remove vim header
*** jp_at_hp has joined #openstack-infra09:58
*** mkerrin has joined #openstack-infra10:02
*** lnxnut has joined #openstack-infra10:05
*** luqas has joined #openstack-infra10:05
*** rlandy has quit IRC10:08
*** lnxnut has quit IRC10:10
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field
*** protux has joined #openstack-infra10:12
*** rlandy has joined #openstack-infra10:19
*** jhesketh has quit IRC10:20
*** ildikov_ has joined #openstack-infra10:21
*** sarob has joined #openstack-infra10:23
*** jhesketh_ has quit IRC10:24
*** e0ne_ has joined #openstack-infra10:26
*** masayukig has quit IRC10:27
*** jishaom has quit IRC10:27
*** matsuhashi has quit IRC10:27
*** sarob has quit IRC10:27
*** CaptTofu has joined #openstack-infra10:28
*** e0ne has quit IRC10:29
*** markmc has joined #openstack-infra10:29
*** dizquierdo has joined #openstack-infra10:35
*** talluri has joined #openstack-infra10:43
openstackgerritIvan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves
*** nati_ueno has quit IRC10:49
*** lcostantino has joined #openstack-infra10:50
*** alexpilotti has joined #openstack-infra10:51
*** jlibosva has joined #openstack-infra10:52
*** zenoway has joined #openstack-infra10:53
*** alexpilotti has quit IRC10:57
*** matsuhashi has joined #openstack-infra10:59
*** homeless_ has joined #openstack-infra11:00
*** cyeoh has quit IRC11:01
*** markmc has quit IRC11:01
*** homeless has quit IRC11:02
*** markmc has joined #openstack-infra11:02
*** vipul has quit IRC11:03
*** vipul has joined #openstack-infra11:04
*** cyeoh has joined #openstack-infra11:05
*** yamahata has quit IRC11:07
*** vkozhukalov has quit IRC11:12
*** saju_m has quit IRC11:12
*** bodepd_ has joined #openstack-infra11:12
*** vkozhukalov_ has joined #openstack-infra11:12
*** jcooley_ has joined #openstack-infra11:12
*** matsuhashi has quit IRC11:13
*** SpamapS_ has joined #openstack-infra11:14
*** zigo_ is now known as zigo11:14
*** plomakin_ has joined #openstack-infra11:14
*** matsuhashi has joined #openstack-infra11:14
*** jcooley_ has quit IRC11:14
*** ociuhandu has quit IRC11:14
*** nibalizer has joined #openstack-infra11:14
*** jaypipes has quit IRC11:14
*** jp_at_hp has quit IRC11:14
*** andreaf has quit IRC11:15
*** coolsvap has quit IRC11:15
*** matrohon has quit IRC11:16
*** SpamapS has quit IRC11:17
*** plomakin has quit IRC11:17
*** bodepd has quit IRC11:17
*** julienvey has quit IRC11:17
*** dhellmann has quit IRC11:17
*** jp_at_hp1 has joined #openstack-infra11:17
*** andreaf has joined #openstack-infra11:17
*** samalba has quit IRC11:19
*** max_lobur has quit IRC11:19
*** jaypipes has joined #openstack-infra11:20
*** nibalize1 has quit IRC11:20
*** coolsvap has joined #openstack-infra11:20
*** jp_at_hp1 has quit IRC11:20
*** jp_at_hp1 has joined #openstack-infra11:20
*** andreaf has quit IRC11:20
*** andreaf has joined #openstack-infra11:20
*** dhellmann has joined #openstack-infra11:20
*** samalba has joined #openstack-infra11:20
*** sarob has joined #openstack-infra11:23
*** fifieldt has quit IRC11:25
*** matrohon has joined #openstack-infra11:26
*** saju_m has joined #openstack-infra11:27
*** sarob has quit IRC11:31
*** dizquierdo has quit IRC11:35
*** nosnos has quit IRC11:41
*** coolsvap has quit IRC11:45
*** dizquierdo has joined #openstack-infra11:48
*** alexpilotti has joined #openstack-infra11:48
*** ociuhandu has joined #openstack-infra11:51
*** yaguang has quit IRC11:54
*** zenoway has quit IRC11:55
*** rfolco has joined #openstack-infra11:56
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field
*** jcooley_ has joined #openstack-infra11:59
*** lcostantino has quit IRC12:01
*** lcostantino has joined #openstack-infra12:02
*** saju_m has quit IRC12:02
*** jcooley_ has quit IRC12:04
*** boris-42_ has joined #openstack-infra12:05
*** mattymo has quit IRC12:06
*** mattymo has joined #openstack-infra12:06
*** lcostantino has quit IRC12:06
*** lcostantino has joined #openstack-infra12:07
*** CaptTofu has quit IRC12:07
*** lcostantino has quit IRC12:08
*** DinaBelova_ is now known as DinaBelova12:10
*** matsuhashi has quit IRC12:14
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field
*** ArxCruz has joined #openstack-infra12:17
*** zhiwei has quit IRC12:18
*** dolphm_503 is now known as dolphm12:20
*** lcostantino has joined #openstack-infra12:24
*** zhiwei has joined #openstack-infra12:27
*** eharney has joined #openstack-infra12:30
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller
openstackgerritAlexander Jones proposed a change to openstack-infra/git-review: Fix parsing of SCP-style URLs, as these are valid in Git itself
openstackgerritA change was merged to openstack-infra/elastic-recheck: Improved timestamp parsing
*** CaptTofu has joined #openstack-infra12:35
*** luqas has quit IRC12:38
*** zhiwei has quit IRC12:38
*** talluri has quit IRC12:39
*** weshay has joined #openstack-infra12:39
*** talluri has joined #openstack-infra12:40
*** hashar has quit IRC12:40
*** heyongli has joined #openstack-infra12:41
*** talluri has quit IRC12:44
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field
*** dolphm is now known as dolphm_50312:45
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Add cryptography needed by python-openstackclient
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
*** Adri2000_ is now known as Adri200012:49
ihrachysfungi: ping12:52
ihrachysfungi: regarding unmaintained stable/havana trove branch and its removal12:53
*** che-arne has joined #openstack-infra12:54
*** smarcet has joined #openstack-infra12:54
sdaguehmmmm -
*** saju_m has joined #openstack-infra12:55
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field
sdague500k ERROR log lines on success builds in last 24 hrs12:55
*** jcooley_ has joined #openstack-infra12:57
*** lcostantino has quit IRC12:57
*** dims has joined #openstack-infra12:58
*** ArxCruz has quit IRC12:59
*** dkranz has quit IRC13:01
*** sandywalsh has quit IRC13:02
*** ArxCruz has joined #openstack-infra13:04
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller
*** jcooley_ has quit IRC13:05
*** CaptTofu has quit IRC13:10
*** heyongli has quit IRC13:11
*** gcb has quit IRC13:12
*** lcostantino has joined #openstack-infra13:14
*** dolphm_503 is now known as dolphm13:14
*** david_lyle has quit IRC13:14
*** hashar has joined #openstack-infra13:14
*** sandywalsh has joined #openstack-infra13:14
*** pdmars has joined #openstack-infra13:14
*** zul has quit IRC13:26
*** zul has joined #openstack-infra13:27
*** dims has quit IRC13:28
*** thomasbiege has joined #openstack-infra13:31
*** lcostantino has quit IRC13:31
*** NikitaKonovalov is now known as NikitaKonovalov_13:31
*** enikanorov_ has joined #openstack-infra13:31
*** locke105 has joined #openstack-infra13:32
*** bnemec_ has joined #openstack-infra13:33
*** NikitaKonovalov_ is now known as NikitaKonovalov13:33
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
*** jpeeler has joined #openstack-infra13:35
*** jpeeler has quit IRC13:35
*** jpeeler has joined #openstack-infra13:35
*** fbo is now known as fbo_away13:36
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
*** zhiyan- has joined #openstack-infra13:38
*** wayneseguin has joined #openstack-infra13:38
*** jd__` has joined #openstack-infra13:38
openstackgerritAlex Gaynor proposed a change to openstack-dev/pbr: Declare support for Python versions in setup.cfg
*** rahmu_ has joined #openstack-infra13:39
*** StevenK_ has joined #openstack-infra13:39
*** sbadia_ has joined #openstack-infra13:39
*** enikanorov has quit IRC13:40
*** EntropyWorks has quit IRC13:40
*** wayneeseguin has quit IRC13:40
*** jd__ has quit IRC13:40
*** changbl has quit IRC13:40
*** jasond`` has quit IRC13:40
*** bnemec has quit IRC13:40
*** sbadia has quit IRC13:40
*** locke1051 has quit IRC13:40
*** dansmith has quit IRC13:40
*** zhiyan has quit IRC13:40
*** pabelanger has quit IRC13:40
*** stevebaker has quit IRC13:40
*** greghaynes has quit IRC13:40
*** mancdaz has quit IRC13:40
*** rahmu has quit IRC13:40
*** mrda_away has quit IRC13:40
*** wayneseguin is now known as wayneeseguin13:40
*** jd__` is now known as jd__13:42
*** rahmu_ is now known as rahmu13:42
*** Mithrand1r has joined #openstack-infra13:42
*** ilyashakhat has joined #openstack-infra13:42
*** dims has joined #openstack-infra13:42
*** EntropyWorks has joined #openstack-infra13:42
*** sbadia_ is now known as sbadia13:42
*** dansmith has joined #openstack-infra13:43
*** sbadia has quit IRC13:43
*** Guest50062 has joined #openstack-infra13:43
*** saju_m has quit IRC13:43
*** loquacities has quit IRC13:43
*** shortstop has quit IRC13:43
*** skraynev has quit IRC13:43
*** sirushti has joined #openstack-infra13:43
*** skraynev has joined #openstack-infra13:44
*** saju_m has joined #openstack-infra13:44
*** che-arne has quit IRC13:44
*** DinaBelova is now known as DinaBelova_13:44
*** lcostantino has joined #openstack-infra13:44
*** psedlak has quit IRC13:44
*** psedlak has joined #openstack-infra13:45
*** loquacities has joined #openstack-infra13:45
*** Vivek_ has joined #openstack-infra13:45
*** jasond`` has joined #openstack-infra13:45
*** pdmars has quit IRC13:45
*** mbacchi has joined #openstack-infra13:46
*** sileht_ has joined #openstack-infra13:48
*** pabelanger has joined #openstack-infra13:50
*** stevebaker has joined #openstack-infra13:50
*** greghaynes has joined #openstack-infra13:50
*** mancdaz has joined #openstack-infra13:50
*** mrda_away has joined #openstack-infra13:51
*** dhellmann_ has joined #openstack-infra13:51
*** sirushti is now known as shortstop13:51
*** mbacchi has quit IRC13:51
*** mbacchi has joined #openstack-infra13:51
*** changbl has joined #openstack-infra13:51
*** Mithrandir has quit IRC13:51
*** ilyashakhat_ has quit IRC13:51
*** Vivek has quit IRC13:51
*** StevenK has quit IRC13:51
*** dhellmann has quit IRC13:51
*** dhellmann_ is now known as dhellmann13:51
*** sileht has quit IRC13:51
*** exz has quit IRC13:51
*** julienvey has joined #openstack-infra13:51
*** che-arne has joined #openstack-infra13:52
*** exz has joined #openstack-infra13:52
*** DinaBelova_ is now known as DinaBelova13:52
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
*** jcoufal has quit IRC13:53
*** matrohon has quit IRC13:53
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Test for email-ext publisher
*** mrodden has quit IRC13:53
*** jcoufal has joined #openstack-infra13:54
*** dprince has joined #openstack-infra13:55
*** pdmars has joined #openstack-infra13:56
*** jcooley_ has joined #openstack-infra13:56
*** matrohon has joined #openstack-infra13:57
*** CaptTofu has joined #openstack-infra13:58
*** yamahata has joined #openstack-infra13:58
*** mestery has quit IRC13:58
*** boris-42_ has quit IRC14:00
*** mkoderer has joined #openstack-infra14:00
*** mestery has joined #openstack-infra14:02
*** jcooley_ has quit IRC14:02
*** luqas has joined #openstack-infra14:03
*** saju_m has quit IRC14:05
*** kiall has quit IRC14:06
*** thomasem has joined #openstack-infra14:06
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Auth Token Middleware
*** kiall has joined #openstack-infra14:07
*** mrodden has joined #openstack-infra14:08
*** dkranz has joined #openstack-infra14:08
fungiihrachys: did you contact hub_cap about that?14:09
fungiihrachys: i'm all for deleting branches from trove if their ptl requests it14:09
*** jeckersb_gone is now known as jeckersb14:10
openstackgerritChangBo Guo(gcb) proposed a change to openstack-dev/pbr: Remove copyright from empty files
fungisdague: that sounds to me like we should just rebuild openstack from scratch. the current incarnation is nothing but error messages14:12
sdaguewell, volume is against us :) I don't know the 24hr message catalog is probably hundreds of millions14:13
*** lnxnut has joined #openstack-infra14:13
openstackgerritJoão Vale proposed a change to openstack-infra/jenkins-job-builder: Add support for TestNG publisher.
*** mayu has joined #openstack-infra14:14
mayujaypipes: hi, there ?14:15
*** amotoki has quit IRC14:15
mayujaypipes: hi, I sent a email to you for help, there are ssh key file error on  jenkins master node.14:17
openstackgerritJoão Vale proposed a change to openstack-infra/jenkins-job-builder: Fix unused variable in ci-skip wrapper.
ihrachysfungi: he doesn't respond, but let's wait another day or two14:19
*** mfer has joined #openstack-infra14:21
mayufungi: hi, there ?14:21
fungimayu: hi14:22
mayufungi: I apply a service account last time14:22
openstackgerritJoão Vale proposed a change to openstack-infra/jenkins-job-builder: Add attachment pattern expression to email-ext.
*** mriedem has joined #openstack-infra14:22
*** luqas has quit IRC14:23
mayufungi: there are problem on my ssh key files, I need to generate a new one14:23
mayufungi: I need your help to refresh my service account public key info14:24
fungimayu: are you trying to use /home/jenkins/.ssh/ from your jenkins master?14:25
*** homeless_ has quit IRC14:25
mayufungi: right14:25
*** miqui has joined #openstack-infra14:26
mayufungi: the key file is generated from another vm, I copy it to the jenkins master14:26
fungimayu: from a shell in that account, did you try to 'ssh -p 29418 gerrit ls-files' and received an authentication error?14:27
mayufungi: no14:27
*** homeless has joined #openstack-infra14:28
fungiif the /home/jenkins/.ssh/id_rsa and /home/jenkins/.ssh/ files were generated together and the /home/jenkins/.ssh/ is the one i entered into gerrit got you, it should work14:28
*** DinaBelova has quit IRC14:28
*** SergeyLukjanov has quit IRC14:28
*** ruhe has quit IRC14:28
*** NikitaKonovalov has quit IRC14:28
*** julim has joined #openstack-infra14:28
fungier, the one i entered into gerrit for you14:28
*** DinaBelova_ has joined #openstack-infra14:29
*** DinaBelova_ is now known as DinaBelova14:29
*** NikitaKonovalov has joined #openstack-infra14:29
*** ruhe has joined #openstack-infra14:29
mayufungi: this is the whole steps.14:29
*** SergeyLukjanov has joined #openstack-infra14:30
*** roey_ has joined #openstack-infra14:30
*** zehicle_at_dell has quit IRC14:30
*** roey_ has left #openstack-infra14:31
mayufungi: ssh -p 29418 gerrit stream-events, exec that command, I got the gerrit event14:31
*** lcostantino has quit IRC14:31
*** roeyc has joined #openstack-infra14:31
fungimayu: that sounds like the keyfiles are correct then. this is with the decrypted private key (after you ran ssh-keygen -f id_rsa -p on it)?14:32
*** lcostantino has joined #openstack-infra14:32
*** Mithrand1r is now known as Mithrandir14:33
fungilooks like step 7 in your paste says that was the case, okay14:33
*** ociuhandu has quit IRC14:33
mayufungi: you mean, the decrypted process cause the problem14:34
fungii don't think you need new keys... have you tried restarting the jenkins service on your jenkins masters? they may cache copies of the original keyfiles in memory14:34
*** lcostantino has quit IRC14:35
openstackgerritA change was merged to openstack-dev/cookiecutter: Adjust _TRUE_VALUES to be consistent with OpenStack projects.
mayuthere are still errors in /var/log/zuul/debug.log14:35
fungioh, i see, at the end of your paste you're using zuul anyway, so it's zuul which is connecting to gerrit, not the jenkins masters14:35
mayufungi: as you see on the paste14:35
fungithe paste doesn't seem to include the errors you're getting from zuul, however14:36
*** prad_ has joined #openstack-infra14:36
mayufungi: the error14:37
*** blamar has joined #openstack-infra14:38
fungimayu: did you possibly save your rsa keys onto the zuul server as id_dsa and instead if id_rsa and
*** markmcclain has joined #openstack-infra14:41
openstackgerritA change was merged to openstack-dev/hacking: Updated from global requirements
mayufungi: you mean, change the file name ?14:41
*** andreaf has quit IRC14:41
*** sandywalsh has quit IRC14:41
fungimayu: i'm just trying to figure out what could cause that error. you're trying to use an rsa keypair but paramiko is raising an exception complaining about an invalid dsa key14:43
mayufungi: or, regenerate the key pairs?14:43
mayufungi: right14:43
*** dstanek_afk is now known as dstanek14:44
fungizuul should be able to use keyfiles in the same format as jenkins, and you've confirmed you're able to authenticate using it so i don't think you need to regenerate them14:44
mayufungi: jaypipes use rsa keypair, I follow his blog to construct.14:45
fungihow did you copy the files? if you did it by cut-and-paste in your terminal, maybe the file is misformatted (stray newlines?)14:45
mayuscp the id_rsa and files14:45
fungiokay, that should have been fine14:46
fungilooking around on the internet, it seems like "not a valid dsa private key file" is a misleading error paramiko produces whenever it can't understand your keyfiles (regardless of whether they're rsa, dsa, ecdsa, whatever)14:47
mayuright, paramiko raise the exception14:48
*** dkliban has joined #openstack-infra14:49
fungii'm still hunting around for any likely causes of that14:49
sdaguefungi: can I get your eyes on - ?14:49
mayuIt is keyfile problem14:49
fungimayu: do the md5sums of your id_rsa and on your zuul server match the files on your jenkins servers where you're able to connect to gerrit?14:50
sdaguemayu: does your keyfile have a password on it?14:50
fungimayu did an 'ssh -f id_rsa -p' to strip the passphrase. just wondering whether the copy on the zuul server is from before or after doing that14:51
mayuReseting passphase as empty, id_rsa must be modified.14:52
mayufungi: there is no password now.14:52
fungisdague: is 75683 replacing 75534?14:52
*** gokrokve has quit IRC14:53
*** gokrokve has joined #openstack-infra14:53
*** jcooley_ has joined #openstack-infra14:53
fungimayu: in a shell on your zuul server, as the same user under which your zuul process runs, try running ssh -p 29418 gerrit stream-events14:53
sdaguefungi: yeh, from what I can tell 75534 doesn't actually work14:54
mayufungi: after14:54
fungisdague: right. i would have expected a takeover of the old change, but this is fine14:54
sdagueyeh, I guess I didn't quite realize what jeblair was doing with that change, and I just went roaring through a bunch of fix ups this weekend14:55
mayufungi: it works, the gerrit event output to the screen14:55
fungisdague: and we get the doubled timestamps in the console log like with 75534, so that we'll still have them when we're no longer relying on jenkins. lgtm14:55
sdaguefungi: yeh, if you go up the patch series, I actually drop outputing the devstack logs to console eventually as well14:56
*** talluri has joined #openstack-infra14:56
mayufungi: sorry, I got a miss14:56
*** luqas has joined #openstack-infra14:56
*** sileht_ is now known as sileht14:57
fungisdague: grenade-specific setup logs look to be timestamped too. excellent14:57
mayufungi: I did not run that as zuul user14:57
*** sandywalsh has joined #openstack-infra14:57
*** sileht has joined #openstack-infra14:57
sdagueI think I got all the logs14:57
sdagueonce that is in, we can get this sorted -
fungimayu: if your zuul service runs as the user "zuul" then you need to make sure the decrypted keypair gets copied into the ~zuul/.ssh/ directory14:58
mayufungi: I will try. if it works, how do i do, if it doesn't work, how do i do14:58
*** boris-42_ has joined #openstack-infra14:58
*** jcooley_ has quit IRC14:58
mayufungi: I think you get the point14:59
*** mwagner_lap is now known as mwagner_bbl14:59
SergeyLukjanovmorning folks15:00
openstackgerritSalvatore Orlando proposed a change to openstack-infra/elastic-recheck: Remove neutron-heat-slow job from query for bug 1253896
*** mrodden has quit IRC15:00
*** simonmcc_ is now known as simonmcc15:00
sdaguemorning SergeyLukjanov15:01
SergeyLukjanovfungi, sdague, could you please take a look at ? it's about avoiding pbr's override while proposing requirements updates15:01
fungimayu: it looks like we actually configure our zuul process to use a key at /var/lib/zuul/ssh/id_rsa15:01
mayufungi: 26th,Feb is my deadline for contruction of third-party ci. Thanks for your help. I will go to company lab to finish the work. keep in touch15:01
*** jnoller has joined #openstack-infra15:02
sdagueSergeyLukjanov: +A15:03
sdagueSergeyLukjanov: then I'll hit you up for this -  :)15:03
mayufungi: you mean, the file in /var/lib/zuul/ssh/id_rsa, I had replaced this file with the passwordless one.15:03
sdagueas I'd really like to get those timestamps in so we can adjust the indexer15:03
fungimayu: correct15:03
*** sahid has quit IRC15:04
mayuI had exec command find / -name id_rsa, and replace the proper ones15:05
SergeyLukjanovsdague, thanks15:06
SergeyLukjanovsdague, looking at your timestamps patch15:06
SergeyLukjanovsdague, my knowledge of awk == google15:07
sdagueyeh, it's lifted largely from stack.sh15:07
*** markmcclain has quit IRC15:07
sdaguewith one minor format change15:07
*** oubiwan__ has joined #openstack-infra15:08
SergeyLukjanovsdague, yup, find it15:08
SergeyLukjanovsdague, so, it'll just add timestamp prefix for each line15:08
*** oubiwan__ has quit IRC15:09
sdagueso the sublogs will have timestamps15:10
sdaguebecause they don't get them from jenkins15:10
SergeyLukjanovsdague, ok, +A15:10
*** Steap has joined #openstack-infra15:11
mayufungi: hi, I will go to company and try again. If it doesn't work, please help to refresh my service account public key.15:11
*** mayu has quit IRC15:13
*** eharney has quit IRC15:13
*** thomasbiege has quit IRC15:14
*** oubiwan__ has joined #openstack-infra15:14
*** rwsu has joined #openstack-infra15:14
*** vishy has quit IRC15:15
*** dkliban is now known as dkliban_br15:20
*** dkliban_br is now known as dkliban_brb15:20
*** DuncanT- has joined #openstack-infra15:22
*** avishay has joined #openstack-infra15:22
*** DuncanT- has left #openstack-infra15:22
avishayhi all, we have a bunch of patches failing, it looks like this is the culprit:
avishaythere's a patch with one +2 already, would appreciate it if someone could take a look15:23
*** amir2 is now known as asadoughi15:23
*** jgrimm has joined #openstack-infra15:25
fungiavishay: can you point me to a recent failure? i just want to make sure we're on the right track with that (though i approved the --no-use-wheels patch to pypi-mirror just now anyway because it's sane for the moment)15:25
*** DuncanT- has joined #openstack-infra15:26
*** david_lyle_ has joined #openstack-infra15:27
*** rcleere has joined #openstack-infra15:28
mordredfungi: ++ to --no-use-wheels15:28
ildikov_fungi: here is one console output for this failure:
*** andre__ has quit IRC15:30
*** dkliban_brb is now known as dkliban15:31
fungiildikov_: intersting. i wonder why that patch isn't hitting the issue consistently15:31
*** johnthetubaguy1 is now known as johnthetubaguy15:31
fungioh, you're adding a requirement for suds, so maybe that's changing the import order15:32
*** markmcclain has joined #openstack-infra15:32
sdagueso the wheels mirror issue fixes that?15:32
ildikov_fungi: we tried to figure out with my colleague what could go wrong, but we did not succeed15:33
sdaguefungi: so I don't entirely understand why this isn't failing every run15:37
sdaguebut very clearly it's new15:37
ildikov_fungi: I've checked some requirements.txt files, but I saw 1.4.1 in every project I checked, so I do not know where the 1.5.2 requirement comes from exactly15:37
fungiildikov_: most likely suds or something suds requires is pulling in a newer version of it15:37
sdaguefwiw it's all rax iad nodes15:38
*** andreaf has joined #openstack-infra15:39
*** sc68cal has quit IRC15:39
avishayi have no understanding of the issue, just know that our stuff is failing on it :)15:39
*** fbo_away is now known as fbo15:40
*** wenlock has joined #openstack-infra15:40
ildikov_fungi: a-ha, I see, so it's more complicated, than I thought15:40
*** tomhe has joined #openstack-infra15:40
mordredfungi, sdague: the solum guys have a feature in the bot that's in their channel which notices urls and pastes shortened versions back into channel in response15:41
mordredwe should get us some of that15:41
mordredI mean, next time any of us have a pile of free tie15:41
sdaguebut the issue at hand15:41
*** sc68cal has joined #openstack-infra15:41
fungimordred: i have a pile of something. pretty sure it's not free time15:42
mordredfungi: :)15:42
fungisdague: that is a very strange correlation. we build images the same in each region, so not sure why only iad would exhibit it15:42
*** jcooley_ has joined #openstack-infra15:42
sdaguefungi: no idea15:42
fungioh! unless maybe there's something different about the ubuntu base image in iad!15:42
mordredfungi: maybe there was a temporary glich in the matrix at image create tie?15:43
fungii wonder if there's a system package of python-six installed on the nodes in that region15:43
*** zhiyan- is now known as zhiyan_15:45
*** beagles is now known as beagles_brb15:45
jeblairfungi, mordred, sdague: good morning.  anything urgent i should look into while you're checking out the six thing?15:46
*** oubiwan__ has quit IRC15:47
sdaguethat's sort of the urgent thing I think15:48
tomheRegarding recent improvements on event processing in Zuul: We're having performance problems on event processing on our Zuul setup. Right now we're running on 951d8f3 from Mon Feb 10 20:40:46. This weekend I tried 264b06 from Fri Feb 21 08:28:58 to test the new event processing, but I couldn't get Zuul to launch any jobs on Jenkins. They showed up on the status page as queued, but nothing more happened. Any hints on why Zuul15:48
mordredjeblair: my coffee isn't very good - can you fix that?15:48
sdagueinteresting our pip freeze says: suds==0.4.115:49
sdaguebut that's not an installable version for me locally15:49
*** thomasbiege has joined #openstack-infra15:49
*** mdenny has joined #openstack-infra15:49
jeblairsdague: i'm reading your reply to hashar and not understanding why his answer isn't the answer to your question...15:49
hasharhello :-]15:49
sdagueok, well I wasn't parsing it this morning. :)15:50
jeblairsdague: it looks like he suggests a single timeout value in jjb that gets interpolated in 2 places (plugin and d-g parameter), and then in d-g you can subtract the appropriate time15:50
fungitomhe: are you running the new zuul-merger daemon?15:50
*** dripton has joined #openstack-infra15:51
fungitomhe: see the NEWS file and updated documentation in the openstack-infra/zuul repo15:51
fungijeblair: nothing urgent i think. just housekeeping15:51
jeblairsdague: oh, he actually doesn't mention the part about it being a parameter to d-g; i think i must have filled that in.  but i think it can be done.15:51
tomhefungi: No, I hoped that it was started the same way as the Gearman server. Separate tool?15:51
sdagueyeh, that's why I was confused. because basically that's what I was suggesting, and it seemed kind of circular from there.15:51
fungitomhe: since it's expected to possibly run on a separate server, it has a separate initscript15:52
jeblairthough the trick is that the timeout should be based on the job, not the project15:52
sdagueI know what I want, which is that variable to get to the environment15:52
fungitomhe: and pay attention to the minor configuration changes you'll need to make for the new [merger] section15:52
sdagueI just don't know how to make that happen15:52
tomhefungi: Thanks, I'll go back to the docs.15:52
jeblairand i don't think we can set up a variable interpolation from within the job-template15:52
jeblairso maybe it isn't the answer :(15:52
tomhefungi: I have the configuration in place, just missed that the merger was started separately15:53
fungijeblair: sdague: also, i pondered that more. moving options for how the job script is launched into hidden (from a cut-and-paste standpoint) variables makes it harder for people to recreate these outside our infrastructure15:54
fungid-g currently falls back to a 60-minute timeout if none is provided, so if we stop adding that in the shell snippet for the job then you have to remember to look up what it is and insert it separately15:55
fungiif you're running something which will take more than 60 minutes anyway15:55
*** andre__ has joined #openstack-infra15:55
jeblairfungi: yeah.  i think ideally the solution keeps the value within the job-template definition15:56
sdaguefungi: honestly, I think removing copy paste is kind of important here, because we keep having errors in this template when people miss updating one of the variables15:56
sdaguethe current double entry is just error prone15:56
*** pcrews has joined #openstack-infra15:57
*** NikitaKonovalov is now known as NikitaKonovalov_15:57
sdaguejeblair: so do we actually need variable interpolation in the template? is there not a way to create jenkins config that just injects an environment variable directly?15:58
jeblairsdague: i think that's what hashar was saying isn't possible -- because the timeout plugin only accepts a direct value15:58
hasharunfrotunately :(-15:58
jeblairsdague: (so you can get an env variable to the builder, but not the jenkins timeout plugin)15:59
hasharbut maybe you could have the JJB {timeout} passed down to the script15:59
hasharas a parameter/env variable15:59
sdaguejeblair: sure, but can't we have JJB take the timeout variable and as it to the env15:59
*** sdake_ has joined #openstack-infra15:59
sdagueand just keep doing what it's doing with it to the timeout plugin15:59
fungido we need to use the evinject plugin for that?16:00
sdague2014-02-24 14:00:57.451 | [EnvInject] - Loading node environment variables.16:00
sdagueI mean, that's in every run16:00
sdaguethis six thing is so confusing16:01
sdaguebecause I don't see anywhere, anyone specifies six>=1.5.2 in this run -
jeblairsdague: i completely agree that this is a good idea -- i would have liked to implement it that way originally if i could have.  but you said "we keep having errors in this template when people miss updating one of the variables".  when has that happened?16:01
jeblairsdague: all of the jobs in devstack-gate.yaml are correct16:01
openstackgerritsahid proposed a change to openstack/requirements: Updates six to the last version
sdaguejeblair: I remember dealing with 2 of these issues in jan16:02
fungisdague: i'm going to hold a node from iad and ord and compare them16:03
sdaguebasically every time we bulk change one of the job classes, we seem to miss one of the inner variables16:03
sdaguealso, from other investigations, the inner variables are all actually wrong16:03
sdaguebecause of where they are applied16:03
*** jeblair has quit IRC16:04
*** jeblair has joined #openstack-infra16:04
sdaguewhich was actually the point of -
*** dolphm is now known as dolphm_50316:05
*** oubiwan__ has joined #openstack-infra16:05
*** testing has joined #openstack-infra16:06
sdaguebasically there is an assumption that 5 minutes is enough time to handle the outer setup, but it actually turns out that it's not. The workspace setup is very variable, and the cleanup is almost 4 minutes on it's own.16:07
jeblairsdague: i don't remember that happening.  perhaps it happened while i was away.16:07
sdaguejeblair: I think it was16:07
*** testing has quit IRC16:07
*** rlandy_ has joined #openstack-infra16:08
jeblairsdague: of course -- the 5 mins was just an estimate, naturally it would need to change and i'm very happy to update this to the approach in your proposed change16:08
*** eharney has joined #openstack-infra16:08
sdaguehonestly, the envinject thing is mostly an error proofing idea16:08
sdagueit's the least of my concerns, it just feels dirty to have to duplicate it16:09
sdaguebut if we are going to dup it, we should at least actually match up the variables to the same values manually and do the offset subtraction for cleanup in d-g, because when we realize cleanup takes 6 minutes, we need to not have to update every job definition.16:10
*** jraim_ is now known as jraim16:10
*** mgagne1 has joined #openstack-infra16:12
*** rlandy has quit IRC16:14
sdagueso anyway -,n,z a bunch of changes I did yesterday which hopefully help clean up a lot of things, and address issues like this. Comments welcomed. I'm also happy to reorder the series if there are pieces you think are slam dunks, and others that need to be discussed16:14
*** malini1 has joined #openstack-infra16:14
sdaguebut all of this is sort of trivial right now to the fact that nothing seems to pass on rax iad nodes, and I have no idea why :)16:16
mattoliverauMorning everyone16:16
anteayamattoliverau: morning16:17
anteayayou are up early/late16:17
davidlenwell_Morning gentlemen, Do I need to do anything with this review other than be patient?
*** davidlenwell_ is now known as davidlenwell16:18
openstackgerritA change was merged to openstack-infra/devstack-gate: timestamp sublogs
anteayadavidlenwell: if you don't mind accepting an answer from a none gentleman, no, just be patient16:19
*** KurtMartin has quit IRC16:19
mattoliverauanteaya: I'm in the US this week and next, so in your time zone.. but very jetlagged :(16:19
anteayamattoliverau: ah yes that is right, texas16:19
anteayahappy jetlag to you16:19
*** lcostantino has joined #openstack-infra16:19
anteayahow was your trip?16:20
davidlenwellanteaya: I'm happy to accept answers from gentlemen and ladies or some other class of human I am unaware of16:20
*** beagles_brb is now known as beagles16:21
openstackgerritJulien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency
mattoliverauYup texas :) The trip was good, just long. Today is my second day, so I think I'll crash in a few hours.16:22
anteayaso if I am reading backscroll correctly is the bug dejour and we still don't know why it is still cropping up in rax-iad16:22
anteayaand the bug bot seems to be awol16:22
*** dolphm_503 is now known as dolphm16:22
*** julienvey has left #openstack-infra16:22
jeblairi believe the freenode ddos's are continuing16:23
anteayamattoliverau: good plan, hard to think when your brain is still in a different timezone16:23
anteayajeblair: oh great16:23
*** kmartin has joined #openstack-infra16:23
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add query for six issue
*** mayu has joined #openstack-infra16:25
*** eharney_ has joined #openstack-infra16:26
mattoliveraubrb, going to find some more coffee!16:26
*** eharney has quit IRC16:27
*** chuck__ has joined #openstack-infra16:28
openstackgerritDoug Hellmann proposed a change to openstack-dev/cookiecutter: Add src and bug links to README
*** yassine has quit IRC16:29
*** talluri has quit IRC16:30
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove neutron-heat-slow job from query for bug 1253896
mayufungi: hi there16:31
fungidhellmann: any clue what's triggering (a second-order dependency being pulled in by openstackclient?)16:32
fungimayu: hi, did you make any progress?16:32
*** eharney_ is now known as eharney16:33
*** thomasbiege has quit IRC16:33
mayufungi: I copy ssh keypairs to ~zuul/.ssh directory. zuul debug.log error
mayufungi: It is AuthenticationException: Authentication failed. not Dsa key file problem16:34
fungimayu: well, that's an improvemement i think. i'll check the gerrit logs to see if i can tell why it failed to authenticate16:34
mayufungi: greate16:35
mayufungi: great16:35
bnemec_So there's no sense rechecking patches that failed due to the six bug until merges, right?16:36
*** bnemec_ is now known as bnemec16:37
fungimayu: your logs are clearly not in utc. what is your utc offset?16:37
fungimayu: are you utc +8?16:37
mayufungi: switch to user zuul, ssh -p 29418 "huawei-ci" gerrit stream-events, I can get gerrit event16:37
*** lcheng has joined #openstack-infra16:37
mayufungi: yes, utc+816:37
*** jcooley_ has quit IRC16:38
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Content-Type can now be set for email-ext publisher
*** chuck__ has quit IRC16:38
anteayabnemec: that would be a safe way to go16:40
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1268274
*** luqas has quit IRC16:40
anteayait is in check and almost done there16:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add query for six issue
bnemecanteaya: Okay, thanks.16:41
*** markmcclain has quit IRC16:41
*** pmathews has joined #openstack-infra16:41
fungii couldn't find any obvious system installs of six when comparing an iad and a dfw devstack-precise slave16:41
fungiboth seemed to have six==1.4.1 installed according to pip freeze16:41
*** e0ne_ has quit IRC16:41
*** dkliban is now known as dkliban_afk16:42
sdaguefungi: yep, I don't think that's the issue. I think something thinks it needs six >= 1.5.216:42
*** e0ne has joined #openstack-infra16:42
sdaguethat is being pulled in16:42
anteayafungi: how long after 74579 merges before folks can recheck, is there a dissemination phase for this one?16:42
*** Ryan_Lane1 has quit IRC16:42
anteayawe would have to wait for mirrors to rebuild, correct?16:42
mayufungi: I am waiting for you. Is there some clue be found in gerrit log ?16:43
fungimayu: i am still looking. i haven't found any obvious indication that gerrit saw an ssh connection from your zuul server at the time in your logs... what is the ip address of your server?16:43
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1282876
*** bearhand1 is now known as comstud16:45
*** garyk has quit IRC16:45
mayuinternal external 202.96.134.*16:45
*** pmathews has quit IRC16:45
mayufungi: internal external 202.96.134.*16:45
*** relaxdiego has joined #openstack-infra16:46
*** luqas has joined #openstack-infra16:46
*** e0ne has quit IRC16:46
fungimayu: found you... [2014-02-24 16:25:36,439 +0000] a377d880 'Huawei openstack ci' - AUTH FAILURE FROM user-not-found16:48
*** esker has joined #openstack-infra16:48
*** pmathews has joined #openstack-infra16:49
fungimayu: you need to change your zuul configuration to use "huawei-ci" instead of "Huawei openstack ci" as your ssh username16:55
mayufungi: ok16:55
mayufungi: I will try16:55
*** pblaho has quit IRC16:55
*** malini1 has quit IRC16:55
*** markmc has quit IRC16:56
*** nati_ueno has joined #openstack-infra16:57
*** malini has joined #openstack-infra16:57
*** dolphm is now known as dolphm_50316:57
*** nati_ueno has quit IRC16:57
*** tjones has joined #openstack-infra16:58
*** dolphm_503 is now known as dolphm16:58
*** yassine_ has quit IRC16:58
*** jlibosva has quit IRC16:59
mayufungi: good news, there are not error in zuul debug.log16:59
*** homeless has quit IRC17:00
*** matrohon has quit IRC17:00
*** zul has quit IRC17:00
*** miqui has quit IRC17:00
*** dkranz has quit IRC17:00
*** dkranz has joined #openstack-infra17:01
*** dangers has joined #openstack-infra17:01
*** gyee has joined #openstack-infra17:01
*** relaxdiego has quit IRC17:04
*** kmartin has quit IRC17:04
*** lcostantino has quit IRC17:04
*** rcleere has quit IRC17:04
*** mestery has quit IRC17:04
*** jcoufal has quit IRC17:04
*** locke105 has quit IRC17:04
*** rfolco has quit IRC17:04
*** ogelbukh has quit IRC17:04
*** lyxus has quit IRC17:04
*** jang has quit IRC17:04
*** davidlenwell has quit IRC17:04
*** dangers_away has quit IRC17:04
*** asadoughi has quit IRC17:04
*** mattoliverau has quit IRC17:04
*** wenlock has quit IRC17:04
*** EntropyWorks has quit IRC17:04
*** pabelanger has quit IRC17:04
*** stevebaker has quit IRC17:04
*** greghaynes has quit IRC17:04
*** mancdaz has quit IRC17:04
*** eharney has quit IRC17:04
*** rlandy_ has quit IRC17:04
*** oubiwan__ has quit IRC17:04
*** sdake_ has quit IRC17:04
*** andre__ has quit IRC17:04
*** jgrimm has quit IRC17:04
*** gokrokve has quit IRC17:04
*** dkliban_afk has quit IRC17:04
*** blamar has quit IRC17:04
*** SergeyLukjanov has quit IRC17:04
*** ruhe has quit IRC17:04
*** NikitaKonovalov_ has quit IRC17:04
*** DinaBelova has quit IRC17:04
*** mfer has quit IRC17:04
*** lnxnut has quit IRC17:04
*** changbl has quit IRC17:04
*** dhellmann has quit IRC17:04
*** Vivek_ has quit IRC17:04
*** shortstop has quit IRC17:04
*** Guest50062 has quit IRC17:04
*** dims has quit IRC17:04
*** Mithrandir has quit IRC17:04
*** jd__ has quit IRC17:04
*** zhiyan_ has quit IRC17:04
*** ArxCruz has quit IRC17:04
*** mattymo has quit IRC17:04
*** samalba has quit IRC17:04
*** SpamapS_ has quit IRC17:04
*** vipul has quit IRC17:04
*** protux has quit IRC17:04
*** davidhadas__ has quit IRC17:04
*** davidhadas_ has quit IRC17:04
*** davidhadas has quit IRC17:04
*** bookwar has quit IRC17:04
*** luqas has quit IRC17:04
*** mdenny has quit IRC17:04
*** david_lyle_ has quit IRC17:04
*** prad_ has quit IRC17:04
*** julim has quit IRC17:04
*** thomasem has quit IRC17:04
*** mkoderer has quit IRC17:04
*** CaptTofu has quit IRC17:04
*** pdmars has quit IRC17:04
*** dprince has quit IRC17:04
*** exz has quit IRC17:04
*** che-arne has quit IRC17:04
*** enikanorov_ has quit IRC17:04
*** weshay has quit IRC17:04
*** dstanek has quit IRC17:04
*** Hunner has quit IRC17:04
*** hemnafk has quit IRC17:04
*** bradm has quit IRC17:04
*** akscram has quit IRC17:04
*** rahmu has quit IRC17:04
*** dizquierdo has quit IRC17:04
*** jaypipes has quit IRC17:04
*** nibalizer has quit IRC17:04
*** plomakin_ has quit IRC17:04
*** rossella-s has quit IRC17:04
*** dosaboy has quit IRC17:04
*** flaper87 has quit IRC17:04
*** mwagner_bbl has quit IRC17:04
*** vladan has quit IRC17:04
*** antonym has quit IRC17:04
*** spiffxp_ has quit IRC17:04
*** kashyap has quit IRC17:04
*** jcooley has quit IRC17:04
*** esmute has quit IRC17:04
*** belliott has quit IRC17:04
*** adarazs_ has quit IRC17:04
*** jog0_ has quit IRC17:04
*** jooools1 has quit IRC17:04
*** therve has quit IRC17:04
*** luisg has quit IRC17:04
*** guitarzan has quit IRC17:04
*** mburned_` has quit IRC17:04
*** michchap has quit IRC17:04
*** aude has quit IRC17:04
*** Steap has quit IRC17:04
*** jeremyb has quit IRC17:04
*** mugsie has quit IRC17:04
*** medieval1 has quit IRC17:04
*** yamahata_ has quit IRC17:04
*** HenryG has quit IRC17:04
*** katyafervent_awa has quit IRC17:04
*** dstufft has quit IRC17:04
*** Xurong has quit IRC17:04
*** krtaylor has quit IRC17:04
*** gema has quit IRC17:04
*** lifeless has quit IRC17:04
*** geekinutah1 has quit IRC17:04
*** mika has quit IRC17:04
*** dkehn_ has quit IRC17:04
*** jesusaurus has quit IRC17:04
*** primeministerp has quit IRC17:04
*** gpocentek has quit IRC17:04
*** GheRivero has quit IRC17:04
*** BobBall has quit IRC17:04
*** mordred has quit IRC17:04
*** ihrachys has quit IRC17:04
*** dtroyer has quit IRC17:04
*** Daviey has quit IRC17:04
*** sc68cal has quit IRC17:04
*** boris-42_ has quit IRC17:04
*** sileht has quit IRC17:04
*** mrda_away has quit IRC17:04
*** StevenK_ has quit IRC17:04
*** derekh has quit IRC17:04
*** EmilienM has quit IRC17:04
*** tteggel has quit IRC17:04
*** dolphm has quit IRC17:04
*** Adri2000 has quit IRC17:04
*** jraim has quit IRC17:04
*** westmaas has quit IRC17:04
*** maurosr has quit IRC17:04
*** Shrews has quit IRC17:04
*** clarkb has quit IRC17:04
*** koolhead17 has quit IRC17:04
*** chmouel has quit IRC17:04
*** alaski has quit IRC17:04
*** soren has quit IRC17:04
*** dragondm has quit IRC17:04
*** fungi has quit IRC17:04
*** jdurgin has quit IRC17:04
*** zigo has quit IRC17:04
*** pvo has quit IRC17:04
*** Guest34377 has quit IRC17:04
*** wendar has quit IRC17:04
*** jbryce has quit IRC17:04
*** russellb has quit IRC17:04
*** jlk has quit IRC17:04
*** adam_g has quit IRC17:04
*** ttx has quit IRC17:04
*** leifmadsen has quit IRC17:04
*** saschpe has quit IRC17:04
*** tristanC has quit IRC17:04
*** msuriar_ has quit IRC17:04
*** mikal has quit IRC17:04
*** beagles has quit IRC17:04
*** rockstar has quit IRC17:04
*** jeckersb has quit IRC17:04
*** hughsaunders has quit IRC17:04
*** cp16net has quit IRC17:04
*** kevinbenton has quit IRC17:04
*** saper has quit IRC17:04
*** isviridov has quit IRC17:04
*** fbo has quit IRC17:04
*** ChanServ has quit IRC17:04
*** morganfainberg_Z has quit IRC17:04
*** mayu has quit IRC17:04
*** tomhe has quit IRC17:04
*** roeyc has quit IRC17:04
*** ildikov_ has quit IRC17:04
*** jgallard has quit IRC17:04
*** jerryz has quit IRC17:04
*** malini has quit IRC17:04
*** psedlak has quit IRC17:04
*** anteaya has quit IRC17:04
*** ekarlso has quit IRC17:04
*** tru_tru has quit IRC17:04
*** tchaypo has quit IRC17:04
*** gothicmindfood has quit IRC17:04
*** portante has quit IRC17:04
*** hdd has quit IRC17:04
*** jamielennox|away has quit IRC17:04
*** pleia2 has quit IRC17:04
*** hub_cap has quit IRC17:04
*** gmoro has quit IRC17:04
*** niska has quit IRC17:04
*** comstud has quit IRC17:04
*** med_ has quit IRC17:04
*** jamespage has quit IRC17:04
*** notmyname has quit IRC17:04
*** dkehn has quit IRC17:04
*** bogdando has quit IRC17:04
*** sdake has quit IRC17:04
*** shardy has quit IRC17:04
*** NobodyCam has quit IRC17:04
*** devananda has quit IRC17:04
*** zaro has quit IRC17:04
*** Ng has quit IRC17:04
*** SlickNik has quit IRC17:04
*** mriedem has quit IRC17:04
*** jpeeler has quit IRC17:04
*** smarcet has quit IRC17:04
*** alexpilotti has quit IRC17:04
*** jp_at_hp1 has quit IRC17:04
*** vkozhukalov_ has quit IRC17:04
*** bodepd_ has quit IRC17:04
*** cyeoh has quit IRC17:04
*** mkerrin has quit IRC17:04
*** johnthetubaguy has quit IRC17:04
*** persia has quit IRC17:04
*** dachary has quit IRC17:04
*** juice has quit IRC17:04
*** openstackgerrit has quit IRC17:04
*** Hefeweizen has quit IRC17:04
*** marun has quit IRC17:04
*** simonmcc has quit IRC17:04
*** Alex_Gaynor has quit IRC17:04
*** AaronGr has quit IRC17:04
*** milki has quit IRC17:04
*** DuncanT- has quit IRC17:04
*** david-lyle has quit IRC17:04
*** oubiwann has quit IRC17:04
*** spredzy_ has quit IRC17:04
*** mtreinish has quit IRC17:04
*** gaelL has quit IRC17:04
*** bknudson has quit IRC17:04
*** sdague has quit IRC17:04
*** yolanda has joined #openstack-infra17:04
*** dprince has joined #openstack-infra17:06
*** yolanda is now known as 23LAAA2TD17:06
*** EntropyWorks has joined #openstack-infra17:06
*** wenlock has joined #openstack-infra17:06
*** sarob has joined #openstack-infra17:06
*** rfolco has joined #openstack-infra17:06
*** miqui has joined #openstack-infra17:06
*** homeless has joined #openstack-infra17:06
*** malini has joined #openstack-infra17:06
*** luqas has joined #openstack-infra17:06
*** eharney has joined #openstack-infra17:06
*** mayu has joined #openstack-infra17:06
*** rlandy_ has joined #openstack-infra17:06
*** oubiwan__ has joined #openstack-infra17:06
*** sdake_ has joined #openstack-infra17:06
*** andre__ has joined #openstack-infra17:06
*** mdenny has joined #openstack-infra17:06
*** sc68cal has joined #openstack-infra17:06
*** tomhe has joined #openstack-infra17:06
*** david_lyle_ has joined #openstack-infra17:06
*** jgrimm has joined #openstack-infra17:06
*** Steap has joined #openstack-infra17:06
*** boris-42_ has joined #openstack-infra17:06
*** sileht has joined #openstack-infra17:06
*** gokrokve has joined #openstack-infra17:06
*** dkliban_afk has joined #openstack-infra17:06
*** blamar has joined #openstack-infra17:06
*** prad_ has joined #openstack-infra17:06
*** roeyc has joined #openstack-infra17:06
*** SergeyLukjanov has joined #openstack-infra17:06
*** ruhe has joined #openstack-infra17:06
*** NikitaKonovalov_ has joined #openstack-infra17:06
*** DinaBelova has joined #openstack-infra17:06
*** julim has joined #openstack-infra17:06
*** mfer has joined #openstack-infra17:06
*** lnxnut has joined #openstack-infra17:06
*** thomasem has joined #openstack-infra17:06
*** CaptTofu has joined #openstack-infra17:06
*** pdmars has joined #openstack-infra17:06
*** 18WAFV1VI has joined #openstack-infra17:06
*** exz has joined #openstack-infra17:06
*** che-arne has joined #openstack-infra17:06
*** changbl has joined #openstack-infra17:06
*** dhellmann has joined #openstack-infra17:06
*** mrda_away has joined #openstack-infra17:06
*** mancdaz has joined #openstack-infra17:06
*** greghaynes has joined #openstack-infra17:06
*** stevebaker has joined #openstack-infra17:06
*** pabelanger has joined #openstack-infra17:06
*** Vivek_ has joined #openstack-infra17:06
*** psedlak has joined #openstack-infra17:06
*** shortstop has joined #openstack-infra17:06
*** Guest50062 has joined #openstack-infra17:06
*** dims has joined #openstack-infra17:06
*** Mithrandir has joined #openstack-infra17:06
*** StevenK_ has joined #openstack-infra17:06
*** rahmu has joined #openstack-infra17:06
*** jd__ has joined #openstack-infra17:06
*** zhiyan_ has joined #openstack-infra17:06
*** enikanorov_ has joined #openstack-infra17:06
*** ArxCruz has joined #openstack-infra17:06
*** weshay has joined #openstack-infra17:06
*** mattymo has joined #openstack-infra17:06
*** dizquierdo has joined #openstack-infra17:06
*** samalba has joined #openstack-infra17:06
*** jaypipes has joined #openstack-infra17:06
*** nibalizer has joined #openstack-infra17:06
*** plomakin_ has joined #openstack-infra17:06
*** SpamapS_ has joined #openstack-infra17:06
*** vipul has joined #openstack-infra17:06
*** ildikov_ has joined #openstack-infra17:06
*** protux has joined #openstack-infra17:06
*** derekh has joined #openstack-infra17:06
*** davidhadas__ has joined #openstack-infra17:06
*** davidhadas_ has joined #openstack-infra17:06
*** davidhadas has joined #openstack-infra17:06
*** rossella-s has joined #openstack-infra17:06
*** jgallard has joined #openstack-infra17:06
*** bookwar has joined #openstack-infra17:06
*** dosaboy has joined #openstack-infra17:06
*** flaper87 has joined #openstack-infra17:06
*** dstanek has joined #openstack-infra17:06
*** Hunner has joined #openstack-infra17:06
*** kashyap has joined #openstack-infra17:06
*** mwagner_bbl has joined #openstack-infra17:06
*** tchaypo has joined #openstack-infra17:06
*** EmilienM has joined #openstack-infra17:06
*** tteggel has joined #openstack-infra17:06
*** vladan has joined #openstack-infra17:06
*** dolphm has joined #openstack-infra17:06
*** Adri2000 has joined #openstack-infra17:06
*** hemnafk has joined #openstack-infra17:06
*** antonym has joined #openstack-infra17:06
*** spiffxp_ has joined #openstack-infra17:06
*** belliott has joined #openstack-infra17:06
*** jcooley has joined #openstack-infra17:06
*** esmute has joined #openstack-infra17:06
*** adarazs_ has joined #openstack-infra17:06
*** jog0_ has joined #openstack-infra17:06
*** jooools1 has joined #openstack-infra17:06
*** therve has joined #openstack-infra17:06
*** luisg has joined #openstack-infra17:06
*** jraim has joined #openstack-infra17:06
*** guitarzan has joined #openstack-infra17:06
*** akscram has joined #openstack-infra17:06
*** bradm has joined #openstack-infra17:06
*** mburned_` has joined #openstack-infra17:06
*** michchap has joined #openstack-infra17:06
*** anteaya has joined #openstack-infra17:06
*** ekarlso has joined #openstack-infra17:06
*** tru_tru has joined #openstack-infra17:06
*** aude has joined #openstack-infra17:06
*** gothicmindfood has joined #openstack-infra17:06
*** portante has joined #openstack-infra17:06
*** jeremyb has joined #openstack-infra17:06
*** mugsie has joined #openstack-infra17:06
*** medieval1 has joined #openstack-infra17:06
*** yamahata_ has joined #openstack-infra17:06
*** HenryG has joined #openstack-infra17:06
*** katyafervent_awa has joined #openstack-infra17:06
*** dstufft has joined #openstack-infra17:06
*** krtaylor has joined #openstack-infra17:06
*** Xurong has joined #openstack-infra17:06
*** gema has joined #openstack-infra17:06
*** lifeless has joined #openstack-infra17:06
*** geekinutah1 has joined #openstack-infra17:06
*** mika has joined #openstack-infra17:06
*** dkehn_ has joined #openstack-infra17:06
*** jesusaurus has joined #openstack-infra17:06
*** primeministerp has joined #openstack-infra17:06
*** gpocentek has joined #openstack-infra17:06
*** GheRivero has joined #openstack-infra17:06
*** BobBall has joined #openstack-infra17:06
*** mordred has joined #openstack-infra17:06
*** ihrachys has joined #openstack-infra17:06
*** Daviey has joined #openstack-infra17:06
*** dtroyer has joined #openstack-infra17:06
*** hdd has joined #openstack-infra17:06
*** jamielennox|away has joined #openstack-infra17:06
*** pleia2 has joined #openstack-infra17:06
*** hub_cap has joined #openstack-infra17:06
*** westmaas has joined #openstack-infra17:06
*** maurosr has joined #openstack-infra17:06
*** Shrews has joined #openstack-infra17:06
*** clarkb has joined #openstack-infra17:06
*** koolhead17 has joined #openstack-infra17:06
*** chmouel has joined #openstack-infra17:06
*** alaski has joined #openstack-infra17:06
*** dragondm has joined #openstack-infra17:06
*** soren has joined #openstack-infra17:06
*** fungi has joined #openstack-infra17:06
*** jdurgin has joined #openstack-infra17:06
*** zigo has joined #openstack-infra17:06
*** pvo has joined #openstack-infra17:06
*** Guest34377 has joined #openstack-infra17:06
*** wendar has joined #openstack-infra17:06
*** jbryce has joined #openstack-infra17:06
*** russellb has joined #openstack-infra17:06
*** jlk has joined #openstack-infra17:06
*** adam_g has joined #openstack-infra17:06
*** ttx has joined #openstack-infra17:06
*** gmoro has joined #openstack-infra17:06
*** niska has joined #openstack-infra17:06
*** comstud has joined #openstack-infra17:06
*** med_ has joined #openstack-infra17:06
*** leifmadsen has joined #openstack-infra17:06
*** jerryz has joined #openstack-infra17:06
*** jamespage has joined #openstack-infra17:06
*** notmyname has joined #openstack-infra17:06
*** dkehn has joined #openstack-infra17:06
*** saschpe has joined #openstack-infra17:06
*** tristanC has joined #openstack-infra17:06
*** msuriar_ has joined #openstack-infra17:06
*** mikal has joined #openstack-infra17:06
*** beagles has joined #openstack-infra17:06
*** rockstar has joined #openstack-infra17:06
*** jeckersb has joined #openstack-infra17:06
*** hughsaunders has joined #openstack-infra17:06
*** bogdando has joined #openstack-infra17:06
*** sdake has joined #openstack-infra17:06
*** morganfainberg_Z has joined #openstack-infra17:06
*** cp16net has joined #openstack-infra17:06
*** kevinbenton has joined #openstack-infra17:06
*** saper has joined #openstack-infra17:06
*** isviridov has joined #openstack-infra17:06
*** fbo has joined #openstack-infra17:06
*** ChanServ has joined #openstack-infra17:06
*** shardy has joined #openstack-infra17:06
*** NobodyCam has joined #openstack-infra17:06
*** devananda has joined #openstack-infra17:06
*** zaro has joined #openstack-infra17:06
*** Ng has joined #openstack-infra17:06
*** SlickNik has joined #openstack-infra17:06
*** sets mode: +o ChanServ17:06
*** kiall has quit IRC17:06
*** DuncanT- has joined #openstack-infra17:06
*** mriedem has joined #openstack-infra17:06
*** jpeeler has joined #openstack-infra17:06
*** smarcet has joined #openstack-infra17:06
*** alexpilotti has joined #openstack-infra17:06
*** jp_at_hp1 has joined #openstack-infra17:06
*** vkozhukalov_ has joined #openstack-infra17:06
*** bodepd_ has joined #openstack-infra17:06
*** cyeoh has joined #openstack-infra17:06
*** mkerrin has joined #openstack-infra17:06
*** johnthetubaguy has joined #openstack-infra17:06
*** persia has joined #openstack-infra17:06
*** dachary has joined #openstack-infra17:06
*** juice has joined #openstack-infra17:06
*** openstackgerrit has joined #openstack-infra17:06
*** Hefeweizen has joined #openstack-infra17:06
*** marun has joined #openstack-infra17:06
*** simonmcc has joined #openstack-infra17:06
*** Alex_Gaynor has joined #openstack-infra17:06
*** AaronGr has joined #openstack-infra17:06
*** milki has joined #openstack-infra17:06
*** david-lyle has joined #openstack-infra17:06
*** oubiwann has joined #openstack-infra17:06
*** spredzy_ has joined #openstack-infra17:06
*** gaelL has joined #openstack-infra17:06
*** mtreinish has joined #openstack-infra17:06
*** bknudson has joined #openstack-infra17:06
*** sdague has joined #openstack-infra17:06
*** dkranz has quit IRC17:06
*** dkranz has joined #openstack-infra17:06
*** dangers has quit IRC17:06
*** dangers has joined #openstack-infra17:06
*** oubiwan__ has quit IRC17:06
fungimayu: glad it's working!17:07
*** 23LAAA2TD has quit IRC17:07
*** yolanda_ has joined #openstack-infra17:07
mayufungi: I will test a job on sandbox17:08
*** zul has joined #openstack-infra17:09
*** hemnafk is now known as hemna17:09
*** enikanorov_ has quit IRC17:10
*** che-arne has quit IRC17:10
*** roeyc has quit IRC17:10
*** pmathews has quit IRC17:10
pleia2good morning17:10
*** pmathews has joined #openstack-infra17:10
jeblairpleia2: hi there17:10
*** thomasem has quit IRC17:11
*** jraim has quit IRC17:11
*** 18WAFV1VI has quit IRC17:11
*** thomasem has joined #openstack-infra17:11
openstackgerritMonty Taylor proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror
openstackgerritMonty Taylor proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv management into a class
*** davidlenwell has joined #openstack-infra17:11
mordredI started trying to track down why something in pypi mirror wasn't building correctly and ran straight into a terrible pile of code I wrote17:12
mordredthere is an attempt to try to make it suck less - no functional difference intended17:12
*** gokrokve has quit IRC17:13
*** wchrisj has joined #openstack-infra17:13
pleia2returned from scale, we can apply to current version of sysadmin-codereview: git tag -s -m "SCaLE12x, 2014" 2014-scale12x-sysadmin-codereview17:13
pleia2or Southern California Linux Expo 12x if we prefer17:13
openstackgerritIlya Sviridov proposed a change to openstack-infra/config: Added new MagnetoDB project to Stackforge
*** jraim_ has joined #openstack-infra17:14
*** kmartin has joined #openstack-infra17:14
jeblairpleia2: i will do that17:14
pleia2jeblair: thank you17:14
fungijeblair: beat me to it17:14
fungipleia2: so is what you presented there?17:15
pleia2fungi: yep17:15
*** jraim_ is now known as jraim17:15
fungii still love that photo of our notepads. at this point we'd need at least a third sheet for that17:15
*** cadenzajon has joined #openstack-infra17:16
jeblairpleia2: tag pushed17:16
pleia2I think elasticsearch and logging could use their own pad at this point17:16
*** lcostantino has joined #openstack-infra17:17
anteayamordred: rebase needed for both patches17:17
*** rfolco has quit IRC17:18
*** rfolco has joined #openstack-infra17:18
fungipleia2: jobs worked...
*** krotscheck has joined #openstack-infra17:18
fungiand it appears on the index17:18
*** dizquierdo has quit IRC17:18
*** andre__ has quit IRC17:20
*** openstackgerrit has quit IRC17:20
*** Ryan_Lane1 has joined #openstack-infra17:20
*** openstackgerrit has joined #openstack-infra17:20
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git
*** lcostantino has quit IRC17:21
*** andre__ has joined #openstack-infra17:21
anteayapleia2: very nice17:21
pleia2clarkb: had a nice chat with the community manager for elasticsearch, she's hoping we could write an article for their blog about how we use it - if you don't mind, I will take a stab at writing, you can do technical editing and we can co-publish17:22
*** lcostantino has joined #openstack-infra17:22
pleia2anteaya: thanks17:22
*** rfolco_ has joined #openstack-infra17:23
dhellmannfungi: no idea, the tests are running for me here :-/17:23
mayuclarkb: hi17:24
fungidhellmann: right, mostly just wondering what second-or-greater-order dependency for openstackclient is demanding six>=1.5.2 (pretty sure it's nothing we release)17:24
*** rfolco has quit IRC17:24
mayuclarkb: following jay's blog, there is a problem for sandbox test.17:25
*** davidhadas__ has quit IRC17:25
*** davidhadas_ has quit IRC17:25
*** davidhadas has quit IRC17:25
dhellmannfungi: I don't know the full origin of that bug, but when I install openstackclient I get six 1.5.2 -- do you know where it was running with an older version?17:26
mayuclarkb: can you help to analysis what's wrong with it.17:26
* dhellmann clicks links17:26
*** rfolco_ has quit IRC17:26
anteayamayu jaypipes has been active in #openstack-neutron today, he might still be there17:26
fungidhellmann: we're testing with 1.4.1 because of a bug in pypi-mirror (which a fix for is in the process of landing)17:26
anteayaclarkb hasn't been around yet, mayu17:27
jeblairmordred: when and how would you like to deal with this change?
mayuanteaya: you are so kind17:27
*** lcostantino has quit IRC17:27
fungidhellmann: but clearly something we're pulling in is sometimes insisting on newer versions of six17:27
*** relaxdiego has joined #openstack-infra17:27
dhellmannfungi: ok, I'll have to look into it further17:28
fungidhellmann: and more disturbingly from my perspective, only seems to happen when we run jobs in rax-iad (which suggests there's something different about the base ubuntu precise image in that region)17:28
dhellmannfungi: do you have any idea if our six lower bound is accurate, aside from this?17:28
jaypipesmayu: I am here.17:28
*** rfolco has joined #openstack-infra17:29
fungidhellmann: no clue, other than six==1.4.1 has been working for tests for a while since the openstackclient exceptions only just started cropping up17:29
mayujaypipes: Following your blog. sandbox test doesn't work17:29
jaypipesmayu: in #openstack-neutron...17:29
mordredjeblair: we're running that change at HP - so I'm comfortable with it - but perhaps we want to wait until puppetboard since it's a pretty pervasive change?17:30
mayujaypipes: ok17:30
*** mestery has joined #openstack-infra17:30
*** terrylhowe has joined #openstack-infra17:30
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror
fungimordred: jeblair: in which case, we have up and running, but ci-puppetmaster still has an outstanding change in review to configure it to start reporting there17:31
* fungi finds link17:31
jeblairmordred: well, you have the access to monitor it yourself, and it fixes a problem we're having in infra, so i don't think it needs to wait for puppetboard.17:31
*** nicedice has joined #openstack-infra17:31
jeblairmordred: but perhaps it should wait until after i3?17:31
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror
mordredjeblair: kk17:31
*** eharney has quit IRC17:32
*** hashar has quit IRC17:32
jeblairfungi: that change is +2 from me; i want to keep the pupetboard project moving17:32
fungijeblair: agreed. i just realized i hadn't tried it with --noop on ci-puppetmaster yet to confirm it won't have trouble applying, so doing that now17:33
jeblairmfer: you said "a codebase exists for these" in  would you like to populate the repos in gerrit from an existing git repo somewhere, or do you want them to start empty?17:33
*** cadenzajon has quit IRC17:34
*** cadenzajon has joined #openstack-infra17:34
mferjeblair we can populate from empty. for a clean log we'll be destroying the history17:34
mferthere were some slip ups in commits of things that's should be in the repo history17:35
jeblairmfer: ok17:35
jeblairfungi, mfer, anteaya: what do i need to know about manage-projects?17:36
*** rfolco has quit IRC17:36
*** eharney has joined #openstack-infra17:36
jeblairsorry, mordred, not mfer ^17:36
*** eharney is now known as Guest689617:36
anteayajeblair: mordred offered some patches yesterday17:36
fungijeblair: mordred had some changes proposed. i can't remember whether i've finished reviewing them yet17:36
mordredjeblair: they do not fix any bugs we've seen currently in infra - those atches address an issue I saw running internally - the gitlab patch can be ignored17:37
mordredjeblair: as of now, I believe we're still at running it and trying to find what's going wrong17:37
anteayajeblair: this one looks the most relevant:
jeblairthen i'll leave them at the bottom of the queue17:37
*** vkozhukalov_ has quit IRC17:37
anteayamordred: but you have 75691 dependant on the gitlab change17:37
mordredanteaya: I can rebase that17:37
anteayaI think hitting github less would be worth merging17:37
*** oubiwann_ has joined #openstack-infra17:38
*** rfolco has joined #openstack-infra17:38
jeblairmordred, fungi: i'm concerned about that change because people do update descriptions, etc, and expect them to change17:38
*** oubiwann_ has quit IRC17:39
fungiin which case hitting github less means finding a way for manage-projects to know what's changed from one run to the next, and retry things which failed previously17:39
mordredjeblair: I could be convinced otherwise ... we're just hitting github a lot for a reasonably infrequent occurance. I don't feel strongly about this patch17:39
jeblairmordred: it just seems to substitute one problem for another17:40
*** jcoufal has joined #openstack-infra17:40
*** oubiwann_ has joined #openstack-infra17:40
*** Guest6896 is now known as eharney17:41
*** lcostantino has joined #openstack-infra17:41
*** yolanda_ has quit IRC17:41
*** oubiwann_ has quit IRC17:42
*** relaxdiego has quit IRC17:43
*** enikanorov has joined #openstack-infra17:44
*** talluri has joined #openstack-infra17:44
*** reed has joined #openstack-infra17:44
*** jgriffith has joined #openstack-infra17:44
*** jang has joined #openstack-infra17:44
*** atiwari has joined #openstack-infra17:44
*** yassine has joined #openstack-infra17:44
*** rcleere has joined #openstack-infra17:44
*** locke105 has joined #openstack-infra17:44
*** ogelbukh has joined #openstack-infra17:44
*** lyxus has joined #openstack-infra17:44
*** asadoughi has joined #openstack-infra17:44
*** mattoliverau has joined #openstack-infra17:44
funginibalizer: a couple of oddities with the puppetdb server... which i've enumerated in a comment on
openstackgerritA change was merged to openstack-infra/config: Modernize ATC list format
funginibalizer: if you have ideas/suggestions for how we should be tackling those, it would be most appreciated17:46
*** oubiwann_ has joined #openstack-infra17:46
jeblairhas anyone notified tripleo that it looks like we're not getting nodes from their cloud?17:47
jeblairfungi: ?17:47
jeblair(i just joined the tripleo channel so have no backlog)17:47
fungijeblair: i hadn't noticed. i'll give them a heads up17:47
jeblairfungi: i can do it17:47
*** garyk has joined #openstack-infra17:48
fungiyeah, their /topic usually mentions any problems they're aware of, but in this case it's not talking about anything which i expect should affect the tripleo-ci cloud17:48
*** lcostantino has quit IRC17:50
*** lcostantino has joined #openstack-infra17:50
*** lcostantino has quit IRC17:50
*** lcostantino has joined #openstack-infra17:50
*** jgriffit2 has joined #openstack-infra17:51
*** chandan_kumar has quit IRC17:51
fungithe last time this came up, lifeless suggested raising the alarm for everyone listed in
*** sarob_ has joined #openstack-infra17:51
jeblairi recall, so i did that, approximately at the same time you did i think.  :)17:52
*** jgriffith has quit IRC17:52
jeblairit would be cool if they also had highlighting set for "tripleo-admin" or something.  :)17:52
*** chandan_kumar has joined #openstack-infra17:53
*** reed has quit IRC17:53
*** gokrokve has joined #openstack-infra17:53
*** sarob has quit IRC17:53
*** reed has joined #openstack-infra17:54
*** jgriffit2 is now known as jgriffith17:54
nibalizerfungi: sure give me a sec17:54
sdagueso is there any further progress on the six issue?17:55
clarkbpleia2: nice sounds good17:55
*** jgallard has quit IRC17:55
*** terrylhowe has left #openstack-infra17:55
*** sarob_ has quit IRC17:56
sdagueclarkb: - I have timestamps on the sublogs now, so I think this is right to get them into ES17:56
*** relaxdiego has joined #openstack-infra17:56
mordredhave I mentioned I hate setuptools/easy_install?17:57
*** dizquierdo has joined #openstack-infra17:57
fungisdague: i'm still unclear on what suddenly started demanding six>=1.5.2 within python-openstackclient and why it's only cropping up in rax-iad, but the fix to pypi-mirror which should get it into our mirror is about 15 minutes out from merging now17:57
clarkbsdague: we need to write logstash rules to parse them too17:58
clarkbsdague: also we should stop logging to console log if we want to consume them that way17:58
sdagueclarkb: they should be the same as console17:58
fungisdague: then the mirror runs need to fire, and i'll keep tabs on them to make sure this solved the failure we were hitting for it17:58
jeblairfungi: can you add to your list?  it's not urgent but is ossg related so i want your eyes on it.17:58
*** oubiwann_ has quit IRC17:59
sdagueclarkb: so once you get to here in my d-g changes - we stop having duplicate content in the index17:59
clarkbsdague: cool17:59
sdagueI matched their date format exactly to console to hopefully make it easy17:59
openstackgerritA change was merged to openstack-infra/config: Make gantt unit tests voting
fungijeblair: sure. to answer your question, i think the reason it ended up targeting the openstack/ repo hierarchy is because the contents may be published to the docs site (i need to review that recent ml thread)18:01
*** luqas has quit IRC18:01
*** lcostantino has quit IRC18:01
jeblairfungi: it was in stackforge/18:01
fungier, ahh18:01
jeblairfungi: the ossg is like an official thing, yeah?18:02
jeblairactually, are they a program?18:02
fungiyes, i agree with you now that i've reread your comment and refrained from inserting words which you did not use ;)18:02
jeblairfungi: it's okay to insert funny words18:02
jeblairfungi: actually, they aren't a program according to openstack/governance18:03
fungijeblair: they are not an official program. i think the upshot of the ml thread was that this was a documentation-oriented repo and would be under the docs program umbrella18:03
nibalizerfungi: can you give me 'cat /etc/puppetdb/conf.d/jetty.ini | grep ssl-host' and 'facter -p fqdn' on puppetdb?18:03
fungiwhich would make anyone publishing security notes a docs contributor18:03
*** derekh is now known as derekh_afk18:03
*** jgriffith has quit IRC18:03
funginibalizer: you bet! on it18:03
jeblairfungi: cool, we should probably start keeping an eye out to make sure that openstack/governance gets updated in these cases18:04
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip
fungijeblair: i'm in full agreement18:04
*** sabari_ has joined #openstack-infra18:04
*** julim_ has joined #openstack-infra18:04
*** david_lyle_ is now known as david_lyle18:04
funginibalizer: (i included /etc/hosts too because you almost certainly were going to ask for that next)18:05
*** lcostantino has joined #openstack-infra18:06
*** julim has quit IRC18:07
*** Ryan_Lane1 has quit IRC18:07
openstackgerritA change was merged to openstack-infra/pypi-mirror: Do not download wheels when running "pip install"
fungisdague: ^ that was the probable pypi-mirror fix18:07
*** dprince has quit IRC18:08
nibalizerfungi: wow, so i guess when jetty does a listen on a name, it just finds the first entry in /etc/hosts that matches (which is that ten net addr) and binds to that18:08
sdaguefungi: cool, will that trigger now?18:08
nibalizeris that how you read it?18:08
mordredjeblair, fungi ^^ YorikSar's patches there are much nicer looking than mine18:08
funginibalizer: that is precisely my suspicion based on the evidence18:09
sdaguebecause of clean check enforcement that issue basically has kept the gate super small all day - but it's our top issue -
funginibalizer: perhaps there's a way to tell jetty to bind to all interfaces?18:09
openstackgerritA change was merged to openstack-infra/config: Allow for etherpad title to be parameterized
*** harlowja has joined #openstack-infra18:10
nibalizerfungi: would you rather I talk you through a one line change, or go through gerrit for it?18:10
nibalizeri think if we put ssl-host = it'll listen on all interfaces, and if it doesn't listen on ipv6 well ... grump18:11
*** dripton has quit IRC18:11
clarkbare we not proxying with something like apache?18:11
*** kmartin has quit IRC18:11
funginibalizer: i can test that, but we'll still need to punch it through review once we're sure of the solution18:11
openstackgerritA change was merged to openstack-infra/config: Incompatible chrome extension has been fixed
*** nati_ueno has joined #openstack-infra18:12
*** jgriffith has joined #openstack-infra18:12
*** ryanpetrello has joined #openstack-infra18:13
*** e0ne has joined #openstack-infra18:13
nibalizerclarkb: if you're talking to me, puppetdb just runs its own server using jetty, and we aren't proxying it18:14
*** dripton has joined #openstack-infra18:14
clarkbnibalizer: fungi: maybe we should proxy it?18:14
fungiclarkb: nibalizer: i think i agree. apache will listen on ipv4 and ipv6 addresses. telling jetty to bind to will probably only bind to all ipv4 addresses18:15
nibalizerthat wouldn't be trivial, since puppetdb uses client cert authentication, we'd have to terminate the ssl and verify the cert at the apache vhost, then proxy the traffic with another cert to puppetdb18:15
fungiahh, nope, actually listens to all...18:16
fungitcp6       0      0 :::8081                 :::*                    LISTEN18:16
fungiso this is viable18:16
nibalizerokay, hoorah for specail ip addrs18:16
*** lcostantino has quit IRC18:17
nibalizerokay i'll fire up a gerrit review to make that change (by the way, what do we call the gerrit pull request things?)18:17
nibalizerclarkb: fungi unless you want to further explore proxying18:17
jeblairnibalizer: 'propose a change' ?18:17
clarkbnibalizer: 'changes'18:17
nibalizerokay, thanks18:18
openstackgerritA change was merged to openstack-infra/config: Allow devstack-precise-check for non-voting jobs
*** sabari_ is now known as sabari18:19
*** morganfainberg_Z is now known as morganfainberg18:20
funginibalizer: so that's one of two problems i noticed18:20
openstackgerritA change was merged to openstack-infra/config: python-ceilometerclient: make the py33 gate voting
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting
funginibalizer: i can reach 8081/tcp on puppetdb's ipv4 address now, but ci-puppetmaster actually seems to want to reach puppetdb's ipv6 address instead, even though ci-puppetmaster has no global ipv6 addresses or routes itself18:21
openstackgerritA change was merged to openstack-infra/config: Remove depreciated htmlify-screen-log
openstackgerritA change was merged to openstack-infra/config: oslo-incubator: gate on py33
funginibalizer: i can work around that by deleting the aaaa rr for but it probably implies a bug in the module18:22
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Set puppetdb server to listen on all interfaces
funginibalizer: unless that's just an (unexpected) artifact of testing with --noop, which i would also be willing to believe18:22
nibalizerso, you're testing my change to add puppetdb to the master with noop? and in the diff you're seing an ipv6 address put in puppetdb.yaml or some other file?18:24
funginibalizer: i'm seeing a "no route to host" for the connection attempt18:25
*** ildikov_ has quit IRC18:25
jeblairpleia2: what do you think of the comment i left on ?18:25
funginibalizer: which strongly suggests that it's trying to reach the ipv6 address (the log only seems to mention the destination by dns name, not ip address)18:26
funginibalizer: though it could just be a fluke of testing via --noop18:26
openstackgerritA change was merged to openstack-infra/config: Add documentation jobs for taskflow
*** kmartin has joined #openstack-infra18:27
pleia2jeblair: you're probably right :)18:27
pleia2I have a fedora server sitting here, so I can actually check18:27
*** cadenzajon has quit IRC18:28
*** salv-orlando has joined #openstack-infra18:33
*** johnthetubaguy has quit IRC18:34
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting
*** dripton has quit IRC18:35
*** esker has quit IRC18:35
*** rossella-s has quit IRC18:35
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Fix pip on py3k/pypy nodes
*** lcostantino has joined #openstack-infra18:37
jeblairmordred: updated your change there ^18:38
pleia2jeblair: re: should we just register it for them with the openstackinfra account? if they can't even figure out registration I'm not sure we want them with +F anyway :)18:38
jeblairpleia2: oh, no they understand -- the problem is that they _can't_ register it because they don't have ops and there are too many people in channel18:39
nibalizerfungi: are you getting 'Failed to connect to puppetdb; sleeping 2 seconds before retry'18:39
*** dkliban_afk is now known as dkliban18:39
pleia2jeblair: oh, should I follow up with freenode staff, or is someone handling it?18:39
*** yamahata has quit IRC18:40
*** yolanda_ has joined #openstack-infra18:40
fungijeblair: i wonder if that channel is available on oftc ;)18:41
*** hemna has quit IRC18:41
nibalizerfungi: so the puppetdb module implements a type/provider to provide a pre-flight check against the puppetdb server inside the puppet code. I've never seen this before. I would say there is a good chance this is noop being useless18:41
jeblairfungi: i already registered it there.  :)18:41
funginibalizer: makes sense. we can just give it a shot once we merge the jetty config fix18:41
jeblairpleia2: 1 sec18:42
*** jgriffit2 has joined #openstack-infra18:42
*** hemna has joined #openstack-infra18:42
*** SpamapS_ is now known as SpamapS18:43
funginibalizer: mainly it was the "no route to host" which worried me, because ci-puppetmaster *does* have a route to the ipv4 address, but not the ipv6 address, and it really looked from the logs like it was actually trying to connect, thus my suspicion...18:43
*** sabari has quit IRC18:43
*** mdenny has quit IRC18:43
*** rfolco has quit IRC18:43
*** sabari has joined #openstack-infra18:43
*** SpamapS has quit IRC18:43
*** SpamapS has joined #openstack-infra18:43
*** mdenny has joined #openstack-infra18:43
funginibalizer: though in retrospect, it could also have been trying to connect to the rfc-1918 eth1 interface of the puppetdb server18:43
fungii didn't fire up tcpdump. but i will if the issue persists18:43
openstackgerritA change was merged to openstack-infra/config: Add tempest coverage job to post
*** rfolco has joined #openstack-infra18:44
*** dripton_ has joined #openstack-infra18:44
*** jgriffith has quit IRC18:44
jeblairpleia2: so rather than just getting ops on that one channel, bswartz has graciously attempted to fix the global problem by attempting to help us fix the group registration situation for openstack18:45
pleia2jeblair: awesome \o/18:45
*** mrmartin has joined #openstack-infra18:45
jeblairpleia2: sadly, we've been trying and failing to do that for years18:45
*** cadenzajon has joined #openstack-infra18:45
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting
nibalizerthat would be weird, im used to linux not using v6 if it doesn't have v618:45
pleia2jeblair: yeah, I spoke with another freenode admin about it last week when he was in town, he gave me the epic rundown of their rewrite of the GC system18:45
jeblairpleia2: so the current state is that the staffer that he was working with believes that i am who i say i am, and says that the groups folks are ready to handle it18:46
pleia2jeblair: I told him to use a spreadsheet :)18:46
jeblairpleia2: but that was last week.18:46
* pleia2 nods18:46
openstackgerritA change was merged to openstack-infra/config: fix indentation in install-buck macro
jeblairpleia2: no kidding (spreadsheet)18:46
jeblairpleia2: i would bug them today, except ddos18:46
jeblairpleia2: so i'm thinking let it subside a bit and then ping mquin again (the staffer that mswartz found who has confirmed my identity)18:47
pleia2jeblair: sounds good, in the meantime do we want to let these patches go through so folks can get bots?18:47
jeblairpleia2: or anyone else who you might know who would actually respond to you :)18:47
funginibalizer: i concur, but we have seen that some applications try to open direct sockets anyway, rather than leaving it up to the system libs to decide how to get somewhere, so not entirely outside the realm of possibilities18:48
*** dprince has joined #openstack-infra18:48
pleia2Corey is responsive, and he works from SF often so we go out for whiskey :)18:48
jeblairpleia2: no; i don't want to have anything to do with any channels we don't have access to18:48
pleia2jeblair: ok, good18:48
jeblairpleia2: people don't seem to be following the directions to talk to us before creating a channel.  maybe we need to rework that.18:49
fungipleia2: perhaps expense several whiskies for him and then turn him loose with his netops hat on18:49
pleia2fungi: haha18:49
*** esker has joined #openstack-infra18:49
*** leifmadsen has quit IRC18:50
*** dizquierdo has quit IRC18:51
*** rlandy_ has quit IRC18:51
clarkbfungi: I like this idea :P18:54
*** chenxu_ has joined #openstack-infra18:56
chenxu_folks, my patch is failing jenkins test on totally unrelated errors:
chenxu_I only touched vif related code, but the check-grenade-dsvm is failing on volume related stuff18:57
chenxu_any ideas?18:57
*** yolanda_ has quit IRC18:59
*** coolsvap has joined #openstack-infra18:59
anteayaERROR: Exception raised: (six 1.4.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('six>=1.5.2'))19:03
fungianteaya: are you sure it's not that the fix for bug 1223469 needs to be backported to stable/havana?19:03
*** jlibosva has joined #openstack-infra19:03
fungianteaya: that's non-voting19:03
anteayathen I am wrong19:03
anteayaand no, I am not sure19:03
fungianteaya: actually, you did have the right bug, it's just cropping up in a more obscure place than we usually see it!
fungiStderr: 'Traceback (most recent call last):\n  File "/usr/local/bin/cinder-rootwrap", line 5, in <module>\n    from pkg_resources import require\n  File "build/bdist.linux-x86_64/egg/", line 2720, in <module>\n    iter_entry_points = working_set.iter_entry_points\n  File "build/bdist.linux-x86_64/egg/", line 592, in resolve\n    return to_activate    # return list of19:05
fungidistros to activate\npkg_resources.VersionConflict: (six 1.4.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse(\'six>=1.5.2\'))\n'19:05
*** amrith has joined #openstack-infra19:06
anteayacorrect by fluke19:06
*** rlandy_ has joined #openstack-infra19:07
fungianteaya: however it gives us a new clue as to where the six>=1.5.2 is coming from, i think19:07
anteayamore data19:07
sdagueoh... ffs19:08
*** NikitaKonovalov_ is now known as NikitaKonovalov19:08
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx
sdagueis this because oslo.rootwrap isn't in the gr sync?19:08
fungithat's what i was checking next19:08
sdagueyeh, it's not in projects.txt19:08
*** rcarrillocruz has joined #openstack-infra19:09
*** esker has quit IRC19:09
fungihowever it's got >=1.4.119:09
rcarrillocruzguys, it seems there's no way to change the gerrit username, the one that is linked in my launchpad profile . Who should I contact, do you know?19:09
fungircarrillocruz: why do you need to change it?19:09
fungircarrillocruz: you can tell git what ssh username to use for a particular remote host19:10
rcarrillocruzfor the sake of consistency, i changed my launchpad id, but that didn't change the gerrit username, it still refers to the old username19:10
jeblairclarkb: do you want to look at
fungircarrillocruz: gerrit removed the ability for users to change their ssh username once it's set. that was way back in gerrit 2.1 i think19:11
openstackgerritA change was merged to openstack-infra/config: Added requirements gates for Murano repositories
*** ok_delta has joined #openstack-infra19:11
anteayamorning zaro19:11
anteayahow was montreal?19:11
rcarrillocruzk, nm then, thanks fungi19:12
zaroanteaya: not there19:12
openstackgerritA change was merged to openstack-infra/config: fix a typo
anteayazaro: weren't you at confoo?19:12
reednagging for review :
* anteaya is confused19:12
reedfungi, jeblair: pretty please :)19:13
rcarrillocruzhave you seen codysomerville online today?19:13
zaroanteaya: it's this week.19:13
*** sarob has joined #openstack-infra19:13
chenxu_this seems more directly related19:13
clarkbjeblair: sure, I am almost done writing a simple logstash watchdog, will review once I have pushed this change19:13
*** chandan_kumar has quit IRC19:14
chenxu_fungi: ^^19:14
*** NikitaKonovalov is now known as NikitaKonovalov_19:14
SpamapSjeblair: not sure why I'm attached to
anteayazaro: ah, safe travels to you then19:14
jeblairreed: i'm reviewing it now, actually19:15
*** NikitaKonovalov_ is now known as NikitaKonovalov19:15
reedjeblair, wonderful19:15
* reed keeps fingers crossed19:15
jeblairreed: sadly, i'm having trouble asking gerrit to diff patchset 7-1319:15
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest
*** MarkAtwood has joined #openstack-infra19:15
fungichenxu_: looks like what we're trying to fix in
jeblairso it's going a bit slower than i would like19:16
harlowjahas anyone seen the following, maybe just something simple wrong,
chenxu_fungi: I was referring to this one
harlowjaerror: 'source_dir' must be a directory name (got `/home/jenkins/workspace/gate-taskflow-docs/doc/source`)19:17
harlowjathis source_dir is actually a directory19:17
chenxu_fungi: which has this line 2014-02-24 17:07:07.204 | FAIL: setUpClass (tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON)19:17
chenxu_can I do two rechecks on two different bugs?19:17
fungichenxu_: you can, though you'd do it in separate comments (can't put them both in the same comment)19:18
harlowjadhellmann do u have any idea about the above, if u are free19:18
chenxu_fungi: cool, thx19:18
dhellmannharlowja: there's a known distutils bug related to that19:18
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Add token header to requests
* dhellmann looks for link19:18
harlowjadhellmann ya, odd that it works locally fine19:18
dhellmannharlowja: it has to do with filesystem encoding or something, I think19:19
dhellmannsometimes the filename is a unicode string instead of a bytestring19:19
zaroclarkb: now i know why upstream gerrit doesn't run tests in their CI..
dhellmannharlowja: the fix is to ensure that the right version of sphinx is used, and not allow the beta in19:20
harlowjadhellmann, k, let me repoke this, perhaps the requirements adjustment that went in fixed it19:20
*** rfolco has quit IRC19:20
openstackgerritZiad Sawalha proposed a change to openstack-infra/config: Add satori as a new project
*** resker has joined #openstack-infra19:22
clarkbzaro: nice19:23
* anteaya goes for a walk19:24
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add a simple watchdog for logstash-indexer service
clarkbjeblair: ^19:25
clarkbnow I will review horizon tx change19:25
fungisdague: anyway, we've got a separate bug (probably a bug in pycparser's packaging/setup) preventing pypi-mirror from making it through the grizzly branch run and moving on to later branches. though i'm unclear on why it's finding cliff in the requirements list on the grizzly branch...
*** markmcclain has joined #openstack-infra19:25
*** vkozhukalov_ has joined #openstack-infra19:25
jeblairsdague: in i don't understand why you matched the console log format instead of the devstack log format?19:26
fungiunless it's not actually checking out stable/grizzly and just running on master after all19:27
openstackgerritA change was merged to openstack-infra/config: Set a ratelimit for tripleo-test-cloud.
jeblairsdague: surely since you also want to index the devstack log, we shoud have all the sublogs use that format instead?19:27
clarkbwait we want to index the devstack log?19:27
jeblairclarkb: why not?  we do now by virtue of having it in the console log19:27
clarkbdid that end up on the console log before?19:27
clarkboh right its d-g logs that don't end up in console log19:27
jeblairclarkb: so if we remove it from the console log, we should index it19:28
fungioic, xattr requires cffi and not the other way around19:29
sdaguejeblair: mostly readability and the fact that I knew the format was already parsed19:29
jeblairsdague: what's the plan for devstack logs then?19:30
sdagueI stared at the logs this weekend, and felt the | helped19:30
jeblairsdague: do you want to change the format in devstack to match this then?19:30
sdaguewas going to see what dtroyer thought of it19:30
sdagueI did match the time resolution19:31
jeblairsdague: that's fair.  ultimately, i'd like d-g and devstack to have the same format.19:31
openstackgerritA change was merged to openstack-infra/config: remove flaky gerrit tests
*** hashar has joined #openstack-infra19:31
dtroyerI think you guys are the ones that changing the devstack log format affects the most...19:32
jeblairdtroyer: fortunately, i think we can change the timestamp format right now without affecting us, since i believe we're actually getting the timestamps from the console log19:32
dtroyerI hadn't looked at that change yet…will after this meeting19:33
*** rfolco has joined #openstack-infra19:33
jeblairso a switch in devstack to match that format, then to have logstash start parsing/indexing the separate logfile instead of console should be smooth19:33
sdaguedtroyer: so the remaining question is just about adding the | , though realistically I'd rather not change that format until we actually get close to that patch landing, just so we don't double pipe people for a long time19:34
sdagueso I guess that brings up the question of the rest of the series19:34
harlowjaclarkb was there a bug created for that mysql permission issue that happened friday (if u remember)19:34
clarkbharlowja: I didn't create one19:35
harlowjak, np19:35
harlowjacreating one19:35
clarkbharlowja: though its a simple change if we just want to give the openstack_citest user ALL privs19:35
clarkbmordred: ^19:35
*** davidhadas has joined #openstack-infra19:35
harlowjabrb, one sec, fogot a meeting19:35
jeblairsdague: looks like it currently assumes the devstack log is the same format19:35
*** davidhadas__ has joined #openstack-infra19:36
*** davidhadas_ has joined #openstack-infra19:36
*** dolphm is now known as dolphm_50319:37
sdaguehow is console.html getting tagged?19:38
*** yolanda_ has joined #openstack-infra19:39
clarkbjeblair: reviewed horizon change, let me know if you think my comment doesn't need to be addressed19:39
SteapHow often is the OpenStack PyPI mirror udpated ?19:40
jeblairclarkb: seems reasonable19:40
jeblairSteap: at least daily or as needed by requirements changes or openstack-related releases19:41
*** dstanek_afk has joined #openstack-infra19:41
jeblairsdague: what do you mean?19:41
*** NikitaKonovalov is now known as NikitaKonovalov_19:42
sdagueso it looks like on the logstash side we need to tag things as console.html to get the log parsing to work anyway19:42
Steapjeblair: ok, thanks :)19:42
sdaguenow I'm curious where that will happen19:42
*** jcooley_ has joined #openstack-infra19:42
*** davidhadas has quit IRC19:42
*** jlibosva has quit IRC19:42
sdagueI also updated this with the additional format -
fungisdague: eureka! i overlooked that the failing changes are all on devstack-precise-check-rax-iad nodes (not devstack-precise-rax-iad) which we know have somewhat different and more stripped-down images. so chances are the difference triggering this is that something which isn't installed system-wide is getting updated by devstack instead19:43
*** dstanek has quit IRC19:43
sdaguefungi: oh, interesting19:43
sdagueany idea what that might be?19:44
*** oubiwann_ has joined #openstack-infra19:44
lifelessjeblair: controller node just threw an NMI19:44
fungisdague: it's a fairly long list of debs on normal images which aren't on the pvhvm images. a couple dozen are python-.* debs too. hard to say what the trigger is19:44
lifelessjeblair: rebooting and we'll see if its a kernel / driver issue (and thus cleared) or deeper hardware fault19:44
zarosdague: so your looking to query for the build timeout?19:45
sdaguezaro: I'd love to :)19:45
sdagueinstead of us having to add it explicitly to every job19:45
*** jgrimm has quit IRC19:46
zarosdague: isn't already added to every job? how do you have a timeout without adding to a job?19:47
zarosdague: ohh you mean using the envinject it into every job?19:48
sdaguewe effectively duplicate the same static piece of info for every job19:49
sdagueso that we can do the inner timeout19:49
*** hashar has quit IRC19:49
fungiwhat if jjb had a parameter which turned on a feature to duplicate the entire job configuration in an envvar passed to the worker via envinject?19:51
clarkbfungi: re those different images. If we can make the pvhvm image happy we should kill the -check images and use pvhvm everywhere19:51
sdaguethat work work19:51
sdaguewould work19:51
fungiclarkb: yeah, but now we have evidence that it's weird in some ways19:51
sdaguehonestly, I don't care how it gets there, or even what it's called once it's there. I just find it silly that we are duping it :)19:51
fungiclarkb: would be nicer if rackspace would make them more consistent19:52
jeblairjnoller: ^ pvhvm images != normal images == not entirely expected behavior19:53
clarkbsdague: to be fair we are duping timeouts19:53
clarkbsdague: not just the values but the implementations19:53
*** garyk has quit IRC19:53
fungiso that we don't get strange differences in userspace like we ran into here19:53
*** dolphm_503 is now known as dolphm19:53
sdagueclarkb: right, but if I included some python code that had 2 constants defined for the same purpose, you'd -1 that19:53
sdagueanyway, this isn't a high priority thing19:53
fungijnoller: right... the ubuntu precise pvhvm and non-pvhvm images... do a dpkg -l in both and see all teh extra stuff which is installed by default on the latter19:53
sdagueit was just an annoyance once I started staring at this yesterday19:54
sdagueso I figured I'd ask if there was a way to clean it up19:54
clarkbsdague: I'd -1 because there were two separate implementation of the same thing19:54
clarkbbut we can't get away from that because EJENKINS19:54
sdagueI guess19:54
sdagueduping constants still seems silly19:54
sdaguebut in reality - getting this in will help much more than fixing the duplication issue -
sdaguebecause, it actually turns out our inner time outs are all wrong anyway19:56
*** thomasem has quit IRC19:56
zarothat was the work around i was thinking of but i agree it's not a good thing.19:56
sdagueand they aren't actually working like anyone thinks they are19:56
clarkb:/ the delta between them is too small?19:56
sdaguebut more importantly, the timeout is just wrapped around the gate_hook19:57
sdagueand there is 3-8minutes setup time before that, and 3-4 minutes clean up19:58
zarosdague: i don't know a good solution off the top of my head.  but if you think it's something to work on.  enter a bug and i'll look into it.19:58
sdagueso we need to stop doing the math in the job template, just pass the same value down to the jobs, calculate how much time is left before we start the timer, and cut the 5 minutes off for cleanup in d-g19:58
sdaguewhich is basically 7572619:59
*** thomasem has joined #openstack-infra19:59
*** dolphm is now known as dolphm_50319:59
*** rossella-s has joined #openstack-infra19:59
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Don't blow up on a MemoryError
*** hashar has joined #openstack-infra20:00
*** nati_uen_ has joined #openstack-infra20:00
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add a simple watchdog for logstash-indexer service
clarkbthat should pass check tests20:01
SergeyLukjanovfungi, mordred, clarkb, jeblair, folks, what do you think about adding more templates in layout.yaml? like check-requrements, tarballs, pypi-release20:01
SergeyLukjanovwe already have docs template with two jobs, and IMO it could make layout.yaml much simpler20:02
openstackgerritA change was merged to openstack-infra/config: Uses python-jobs template in zuul for python-tuskarclient.
fungiSergeyLukjanov: i think that was intended, but nobody's had time to create them yet20:02
*** tjones has quit IRC20:02
*** dstanek_afk is now known as dstanek20:02
SergeyLukjanovfungi, ok, if mordred, clarkb, jeblair will not have any objection, I'll start doing it20:02
*** tjones has joined #openstack-infra20:03
*** ArxCruz has quit IRC20:03
*** pdmars_ has joined #openstack-infra20:03
*** nati_ueno has quit IRC20:03
*** cody-somerville has joined #openstack-infra20:03
*** cody-somerville has quit IRC20:03
*** cody-somerville has joined #openstack-infra20:03
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting
*** pdmars has quit IRC20:06
sdaguefungi: can you kick openstackrecheck bot20:06
sdagueI expect in all the netsplits it didn't recover well20:07
sdaguealso, clarkb we're still about 2 hrs behind on indexing? -
*** SumitNaiksatam has joined #openstack-infra20:07
*** tjones has quit IRC20:07
*** chenxu_ has quit IRC20:08
openstackgerritA change was merged to openstack-infra/reviewstats: Don't blow up on a MemoryError
*** dolphm_503 is now known as dolphm20:09
*** pdmars has joined #openstack-infra20:09
*** pdmars_ has quit IRC20:09
jeblairSergeyLukjanov: ++20:10
* jeblair -> lunch20:10
*** dangers is now known as dangers_away20:10
sdagueSergeyLukjanov: yeh, I was thinking about those layout changes when I was trying to drop the unit tests. Because it would be good to make an integrated gate template that includes the basic devstack and grenade jobs we expect everyone to run20:12
SergeyLukjanovsdague, added to my backlog20:12
SergeyLukjanovsdague, good idea20:12
mordredSergeyLukjanov: sounds awesome20:12
sdagueSergeyLukjanov: awesome20:13
*** tomhe has quit IRC20:13
*** dstanek_afk has joined #openstack-infra20:15
*** dstanek has quit IRC20:15
*** dstanek_afk is now known as dstanek20:16
jog0_sdague: ping20:18
jog0_sdague: so the grenade changes20:19
sdagueit seems super weird that the service lists are hard coded to me20:19
jog0_sdague: agreed20:20
sdagueis there a way not to do that that isn't crazy hard?20:20
jog0_I am going to fix that one20:20
sdagueok, cool20:20
*** jog0_ is now known as jog020:20
jog0sdague: I think there is20:20
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware
sdagueok, so lets do that. I realistically don't want to hold this one up too much, because it would be useful to nova to gate on it20:21
jog0so  Ithink i did it that way because of
jog0so you had some further concerns about
sdagueso what's the job definition for what's going to run this?20:23
jog0so funny you should mention that... that code landed already20:23
*** mrda_away is now known as mrda20:24
sdagueno tempest?20:24
*** Sukhdev has joined #openstack-infra20:25
jog0as no no tempest patch needed for this20:25
jog0as in*20:25
*** lcostantino has quit IRC20:26
Sukhdevfolks - I am seeing the following error when i clone nova - any idea why? ---- 2014-02-24 12:01:03 + git clone git:// /opt/stack/nova20:26
Sukhdev2014-02-24 12:01:16 Cloning into '/opt/stack/nova'...20:26
Sukhdev2014-02-24 12:02:16 fatal: The remote end hung up unexpectedly20:26
Sukhdev2014-02-24 12:02:16 fatal: early EOF20:26
Sukhdev2014-02-24 12:02:16 fatal: recursion detected in die handler20:26
SpamapSsdague: Installed /opt/stack/python-glanceclient20:26
SpamapSProcessing dependencies for python-glanceclient==
SpamapSerror: Installed distribution six 1.4.1 conflicts with requirement six>=1.5.220:26
*** lcostantino has joined #openstack-infra20:26
openstackgerritA change was merged to openstack-infra/config: Fix log Footer README's for 'check-tempest-dsvm'
sdagueSpamapS: yeh, so we are still trying to figure out what is calling for that version20:26
sdagueand failing20:26
SpamapSoh so this isn't actually that the mirror has wheel issues?20:27
sdagueso the issue might have moved after that20:27
fungiSpamapS: yeah, it's a twofold problem20:27
sdagueyou guys have a super minimal imagE?20:27
*** gokrokve has quit IRC20:27
fungiSpamapS: well, threefold now because we can't build an updated mirror for other reasons, so i've gone off in a corner to reproduce it and see if i can fix it now20:27
*** gokrokve has joined #openstack-infra20:28
*** cadenzajon has quit IRC20:28
*** mwagner_bbl is now known as mwagner_wfh20:28
SpamapSsdague: if you checkout diskimage-builder and tri[Cpleo-image-elements you can reproduce with just 'disk-image-create -a i386 -u ubuntu openstack-clients' .. I think.. testing that theory now.20:28
sdagueso one of our dependencies, which is already installed by a basic precise server install, isn't in the phvm images on rax20:29
fungiSpamapS: if we knew what was demanding six>=1.5.2, it would help ease some related concerns at least20:29
sdagueso when pip installs the latest and greatest version of it, it requires new six20:29
*** resker has quit IRC20:29
sdagueor at least that's my guess20:30
sdaguehowever, the logs have not been very helpful in figuring that out20:30
sdagueif yuo have a broken system can you grep the python tree to find what's requiring new six?20:30
SpamapSsdague: Yeah one second20:30
fungii can at least confirm that pypi-mirror itself is caching six-1.5.2.tar.gz20:32
fungiwhen i run it manually20:32
*** gokrokve has quit IRC20:32
*** ildikov_ has joined #openstack-infra20:33
fungistill waiting to see if it tickles the same error we're hitting when it's run within the jenkins job20:33
anteayaSukhdev: I just checked my nova repo from git.o.o and was able to update correctly20:34
anteayaSukhdev: are you using a proxy?20:34
SpamapSsdague: Ok I have a broken chroot20:35
SpamapSsdague: with a broken virtualenv ...20:35
*** harlowja is now known as harlowja_away20:35
sdagueSpamapS: well, you guys have a broken that looks like our broken20:35
jog0sdague: looping back to  I am going to make the ENABLED_SERVICES change , but you had some other concerns on that patch20:35
SpamapSsdague: ^^20:36
SpamapSI said the same thing out loud20:36
fungiSpamapS: sdague: yep, that's one of the system-installed debs on the non-pvhvm precise images20:36
*** DinaBelova is now known as DinaBelova_20:37
*** ok___delta has joined #openstack-infra20:37
*** enikanorov_ has joined #openstack-infra20:37
sdaguehmmm... no that doesn't help20:37
SpamapS0.14 uploaded yesterday20:37
*** malini is now known as malini_afk20:38
*** enikanorov has quit IRC20:38
*** kynes has joined #openstack-infra20:39
sdaguehmmmm.... but why only the rax nodes hitting this?20:39
sdaguejog0: so, actually, I think ENABLED_SERVICES is all I need on it20:40
sdagueI was thinking about patch 2 incorrectly20:40
sdagueI +Aed patch #220:40
fungigrr... when i run it by hand i get pypimirror/mirror/openstack/six/six-1.5.2.tar.gz20:41
fungii'm tempted to move aside the local caches on the mirror27 slave and try retriggering that update20:43
*** coolsvap has quit IRC20:43
fungii was completely unable to reproduce the traceback in
SpamapSso.. is this going to be another one where we pin everybody to pyOpenSSL <=0.13 ?20:44
fungier, in whatever20:44
*** ok_delta has quit IRC20:44
fungiSpamapS: no, i think once pypi-mirror is getting six 1.5.1 properly installed into our mirror, it should sort itself out20:44
anteayafungi is Client disconnected before sending all data to backend a manifestation of the mirror error?20:44
fungier, 1.5.220:44
fungianteaya: i have no idea what you said there ;)20:45
anteayaI am seeing an error in
*** jgriffit2 is now known as jgriffith20:45
anteayapart of the output is Client disconnected before sending all data to backend20:45
*** lcheng has quit IRC20:45
anteayaI am wondering if this is a manifestation of the mirror error20:45
anteayait is affecting a nova patch in the gate20:46
anteayasorry was trying to be consise and I missed20:46
fungianteaya: looks like glance --os-auth-token is failing. no idea why. would need to dig into the glance logs for that job next, probably20:47
openstackgerritSahdev Zala proposed a change to openstack-infra/config: New StackForge project heat-translator
anteayaokay I will look in glance bugs to see if it is known20:47
*** relaxdiego has quit IRC20:48
*** dprince has quit IRC20:49
clarkbback from lunch20:49
anteayaso far no known glance --os-auth-token bugs filed that I can find20:51
fungianteaya: you probably need to look into the glance screen logs to see what error they're actually hitting, and then possibly recurse into screen logs for other services as indicated20:52
*** nati_uen_ has quit IRC20:52
*** pdmars_ has joined #openstack-infra20:52
*** tjones has joined #openstack-infra20:52
*** ok___delta has quit IRC20:53
clarkbsdague: yes we are way behind on indexing, I guess you missed the comments late last week20:53
sdaguefungi: I guess given that neither image has python-openssl, I'm still confused why we hit it on only the phvm images20:53
anteayaI'm not sure what changes I need to do to the url to get those logs20:53
clarkbsdague: tl;dr is we index too much stuff and our cluster is too small20:53
sdagueclarkb: I guess I did20:53
clarkbsdague: are are now indexing about as much data as we were before we stopped indexing DEBUG level logs20:53
clarkbsdague: which is unhappy times20:53
sdagueany idea what's filling that up?20:53
*** emagana_ has joined #openstack-infra20:54
*** tjones has quit IRC20:54
clarkbsdague: I think console logs mostly20:54
*** tjones has joined #openstack-infra20:54
clarkbbut also possibly the new libvirt logs, though they seemed relatively small20:54
clarkbsdague: I think its more volume20:54
clarkbwe are running a lot of tests especially with the new check rules20:54
anteayafungi: how do I get to teh screen logs for a job that is still in the gate?20:54
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting
*** pdmars has quit IRC20:55
clarkbsdague: curent vague plan is to work on using rax performance nodes that are bigger than our current nodes to incrase the size of the cluster20:55
*** nati_ueno has joined #openstack-infra20:55
*** pdmars_ has quit IRC20:56
fungianteaya: look at the end where it says that it's copying logs to the logserver, but change everything up through /srv/static/logs/ to
mattoliverauclarkb and/or fungi would one of you take a look at it's for a guy here at work :)20:56
*** pdmars has joined #openstack-infra20:56
clarkbsdague: anywas we are seeing a lot of memory pressure in the cluster and lots of cache evictions and so on, this appears to be affecting the rate at which we can index20:56
anteayafungi: thanks, will try20:57
clarkbsdague: so nodes with more memory should relieve that and quicker nodes (yay pvhvm) will be an overall improvement20:57
sdagueclarkb: so there might be other culprits too20:57
sdaguelike the ceilometer logs are enormous, and garbage20:57
clarkbwe don't index those20:57
clarkband if we do that is a bug /me double checks20:58
*** harlowja_away is now known as harlowja20:58
clarkbya we don't20:58
anteayaI've always wondered how to do that20:58
*** ociuhandu has joined #openstack-infra20:59
*** chenxu_ has joined #openstack-infra20:59
*** banix has joined #openstack-infra20:59
*** banix has quit IRC21:00
*** denis_makogon_ has joined #openstack-infra21:00
*** oubiwann_ has quit IRC21:00
mattoliverauthanks clarkb21:01
*** denis_makogon_ is now known as denis_makogon21:01
clarkbmattoliverau: I havne't looked yet :P21:01
chenxu_Just had another issue with turbo-hipster21:01
clarkbnow I feel compelled, I see how this works :)21:01
chenxu_db migration taking too long21:01
*** esker has joined #openstack-infra21:01
sdagueclarkb: interesting21:01
chenxu_doesn't look like an existing bug21:01
fungimattoliverau: ^21:02
*** lcheng has joined #openstack-infra21:02
fungimikal: ^21:02
fungii don't see jhesketh around at the moment21:02
*** relaxdiego has joined #openstack-infra21:03
anteayafungi: check me here
anteayaI'm seeing glance is failing due to a swiftclient socket error21:03
fungianteaya: at which point you want to look at the swift screen logs, probably21:04
anteayathe only swift screen with stacktrace is
anteayapkg_resources.DistributionNotFound: ceilometer21:04
anteayaam I on the right track?21:04
Sukhdevanteaya: I am using devstack ( and as a part of it, it does git clone of nova. Most of the times it works OK - once in a while it fails with this error21:04
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client
anteayaSukhdev: are you behind a proxy or firewall when you run
Sukhdevanteaya: I believe so21:05
*** jgrimm has joined #openstack-infra21:06
anteayayou are going to either have to use a vm that is not behind a proxy or firewall21:06
*** vkozhukalov_ has quit IRC21:06
Sukhdevanteaya: However, it has been running just fine - no change in settings. I have started to see this issue as of yesterday21:06
anteayaor address the proxy/firewall that is blocking git.o.o21:06
clarkbsdague: also, I think the fails we had before where we lost logs were masking that we had a lot more logs21:06
anteayanotmyname: ping21:06
clarkbsdague: so we will address this by using more HP21:06
anteayaSukhdev: has your proxy/firewall changed since yesterday?21:07
clarkbfungi: speaking of, are you going to have time to talk about spinning up performance nodes for that today?21:07
Sukhdevanteaya:good point  - do not think so21:07
*** dolphm is now known as dolphm_50321:07
notmynameanteaya: pong21:07
clarkbI am about to switch to reviewing as many changes as possible so should be able to context switch ES stuff relatively easily21:07
anteayacan you try on a node that is not behind a proxy or firewall21:07
anteayanotmyname: hi we have a nova patch failing in the gate21:07
anteayathe breadcrumb trail leads me to swift21:08
Sukhdevanteaya: will give it a try21:08
fungiclarkb: perhaps. let's see how far i get with pypi-mirror21:08
anteayaSukhdev: if you still have problems when you are free of the firewall then let me know21:08
mordredfungi: whatcha doing with pypi-mirror?21:08
anteayaif no, then you need to investigate your settings21:09
notmynameanteaya: last line on that looks like ceilometer isn't installed21:09
Sukhdevanteaya: will do - thanks21:09
fungimordred: continuing to try to reproduce and debug why it's not updating the mirror21:09
anteayanotmyname: I concur21:09
anteayaSukhdev: thank you21:09
*** andreaf has quit IRC21:09
fungimordred: the mirror job logs with tracebacks i linked you to earlier21:09
anteayanotmyname: what happens now?21:09
Sukhdevanteaya: do we not have neutron meeting today? no body is on that channel21:09
anteayaSukhdev: I see that they are21:09
anteaya #openstack-meeting21:10
anteayatry connecting again21:10
anteayayou might have been caught in the net split21:10
notmynameanteaya: install ceilometer, I'd guess. but I'm also guessing you've tried that?21:10
sdagueclarkb: so, maybe we can start tuning down the xtracing in devstack21:10
Sukhdevanteaya: oops my bad I was looking in the openstack-neutron channel - thanks for pointing it out21:10
anteayaSukhdev: np21:10
sdagueit's probably worth seeing what pruning changes we can do there21:11
anteayanotmyname: I had not no, this is on a nova patch failing in the gate21:11
*** cadenzajon has joined #openstack-infra21:11
clarkbsdague: ++ also removing the error'd logs from the console log21:11
*** hogepodge has joined #openstack-infra21:11
openstackgerritgordon chung proposed a change to openstack-infra/config: enable doc generation for pycadf
sdaguewell, I kind of want to keep the error logs in the console log21:11
clarkbsdague: its redundant21:11
sdaguebecause people actually seemed to be paying attention to it21:11
clarkbput them in a file is fine21:11
clarkbI really dislike them in the console log21:11
clarkbbecause most of them are noise and it makes it really hard to see where in the console something broke21:12
notmynameanteaya: I'd guess that at some point you have some devstack setup logs for that instance. and I'd also guess that there may have been a problem during the setup where ceilometer wasn't installed for some reason21:12
sdagueclarkb: well, actually once we started dumping them there I noticed core developers working on them to get rid of  them21:12
*** chadlung has joined #openstack-infra21:12
anteayanotmyname: devstack setup logs:
*** mrmartin has quit IRC21:12
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Update README
anteayanotmyname: correct 2014-02-24 20:38:13.60921:13
*** lcheng has quit IRC21:13
clarkbsdague: hrm, that is valuable21:14
*** melwitt has joined #openstack-infra21:14
*** lcheng has joined #openstack-infra21:14
sdagueyeh, I think the ugliness is a motivator, so I'm hesitant to lose that21:14
sdagueI do think we could probably trim a lot of redundant out of there by tuning down xtrace on well known functions21:15
*** gokrokve has joined #openstack-infra21:17
clarkbsdague: yeah that devstack change lgtm, I will give it a propre review and +121:17
*** dolphm_503 is now known as dolphm21:18
sdagueI'm now actually curious how small we could get console with a few iterations of this21:18
openstackgerritA change was merged to openstack-infra/config: Publish link to docs-drafts for successful builds
mikalSo, turbo hipster...21:20
mikalFailures like those are rare, a recheck should resolve them if its a slow test node21:20
openstackgerritA change was merged to openstack-infra/config: Make sure gawk is installed
openstackgerritA change was merged to openstack-infra/config: Uses python-jobs template in zuul for tuskar.
notmynameanteaya: I see a _lot_ of different errors in those logs. also it's very difficult to find in there any logs about what was installed or not. I've got something else going on, so I'm not sure I have any answers for you. assuming swift not starting is the cause of the other errors, then looking for why ceilometer isn't installed would be the next step21:21
anteayanotmyname: /node21:21
anteayanod even21:21
anteayaokay thanks21:21
anteayawill continue21:21
mferjeblair did you ever get a chance to look at
anteayawould be a manifestation of the mirror issues?21:22
jeblairmfer: thanks, i forgot to update after you answered my q.21:23
fungianteaya: unlikely. also the errors we've been noticing from that problem so far have been entirely on rackspace pvhvm check nodes, and that job ran in hpcloud21:23
anteayahow is ceilometer usually downloaded onto a node?21:24
anteayait comes in the git cache with the image correct/21:24
clarkbanteaya: right21:24
fungianteaya: correct, and it's showing as being a working clone reset to master in the setup workspace log for that job21:24
tchaypomy patch only just got merged, but is already looking stylish again. that was faster than I expected.21:24
anteayaso if pkg_resourses.DistributionNotFound can't find ceilometer?21:25
*** dolphm is now known as dolphm_50321:25
* anteaya checks out the setup workspace log21:25
fungianteaya: then devstack or tempest probably had some trouble making sure it was installed21:25
clarkbanteaya: do you know what the status of new project creations is? eg if we approve changes that create projects does that require a manual manage-projects trigger?21:25
chenxu_mikal: thx.. I did submit a bug report, want me to cancel that, or leave it there?21:26
anteayait requires monitoring and a manual push if need be21:26
*** dolphm_503 is now known as dolphm21:26
anteayait will trigger, it might just not complete correctly, clarkb21:26
fungiclarkb: it will run automagically, and if there's an upstream to import then it seems to work (modulo needing to re-trigger replication got the project in gerrit after giving create-cgitrepos enough time to see the projects.yaml update and rerun)21:26
fungis/replication got the project/replication for the project/21:26
clarkbfungi: anteaya thanks21:27
fungiclarkb: if there is no upstream to import, it currently seems to fail spectacularly21:27
fungiclarkb: but pounding on it with several reruns and removing the git repos it creates in between on the gerrit server seems to eventually make it happen, at least in the one case i ended up cleaning up behind21:28
fungiclarkb: and we do get tracebacks/errors from manage-projects in the syslog now too, courtesy of puppet21:29
anteayathis is interesting: HEAD is now at 2f9300f Merge "Fixed spelling error in Ceilometer"21:29
anteayaI wonder if that plays a role21:29
jeblairYAY! I'm caught up on infra/config reviews to _today_!21:29
*** ociuhandu has quit IRC21:29
clarkbjeblair: wow21:29
clarkbthat is impressive21:29
* clarkb goes back to trying to catch up21:29
fungijeblair: you are waaaay ahead of me on reviews in that case21:30
jeblairclarkb: you can do it!21:30
HenryGHi, sorry, is there a bug for the requires six 1.5.2 error?21:30
jeblairI only did it thanks to SergeyLukjanov's help.  :)21:30
fungioh, wait *just* infra/config, not all the other infra projects21:30
*** juice- has joined #openstack-infra21:30
jeblairfungi: yes.  *just* that.  :)21:30
*** dripton_ is now known as dripton21:30
fungijeblair: heh, still no small task21:30
HenryGanteaya: Thanks!21:30
anteayathanks for asking21:31
HenryGanteaya: can I recheck on that bug, or is there no point?21:31
anteayayou can recheck on that bug21:31
HenryGanteaya: thanks again21:32
fungiHenryG: in another few minutes, the mirror will have six 1.5.221:32
jeblairfungi: what broke?21:32
clarkb++ /me wants to know too and has picked up only snippets on irc21:33
fungibecause apparently the traceback i as looking at was a red herring, and the real reason it didn't update is that on merging a change to pypi-mirror we then proceed to run the mirror job using the copy of pypi-mirror previously installed on the slave by puppet :/21:33
*** lcheng has quit IRC21:33
fungijust retriggering the job seems to have added it21:33
*** mbacchi has quit IRC21:34
fungiit's not on the mirror yet, but it's now appearing in the local tree on teh slave which is about to get rsync'd out to it21:34
*** jhesketh has joined #openstack-infra21:34
openstackgerritClark Boylan proposed a change to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves
fungiand rsync says...21:34
fungi2014-02-24 21:34:17.424 | <f+++++++++ openstack/six/six-1.5.2.tar.gz21:34
clarkbjeblair: ^ fixes a merge conflict in a change you approved21:34
fungiyep, has it now21:35
*** dkliban has quit IRC21:35
fungisdague: ^21:35
*** lcheng has joined #openstack-infra21:35
jeblairjhesketh: hi there21:35
clarkbfungi: oh fun, we can fix that with a virtualenv install for the non periodic triggered jobs maybe?21:35
anteayathis looks important fatal: Couldn't find remote ref refs/zuul/master/Z4f0542f087064faf8b46ce3b08c936f721:35
openstackgerritA change was merged to openstack-infra/config: python-tuskarclient: enable the py33 gate
jeblairanteaya: it's not, that's why we hide it outside the main log.21:36
anteayajeblair: okay then, i will keep looking21:36
fungiclarkb: i expect so21:36
jeblairanteaya: what job?21:36
clarkbjeblair: though now that I think about it maybe that sed line should've been 's/ext3/ext4/g'21:37
clarkbits funny how the brain goes "NO WAIT" when you approve a thing21:37
jeblairanteaya: yeah21:37
anteayajeblair: a nova patch was failing in the gate21:37
clarkbjeblair: fungi: any opinions on needing the g flag?21:37
anteayajeblair: I thought I would dig in in case it was larger than 1 job:
jeblairanteaya: that just means zuul did not prepare a ref for that project.  the message comes from git, and there's not much we can do about it (other than hide it, but also possibly any real errors, completely)21:37
*** oubiwann_ has joined #openstack-infra21:38
jeblairclarkb: probably should21:38
fungiclarkb: can't hurt... let me check a system real quick and see if it has any lines with more than one match21:38
anteayajeblair: okay, well at least I have context now21:38
*** juice- has quit IRC21:38
*** juice- has joined #openstack-infra21:39
openstackgerritClark Boylan proposed a change to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves
clarkbI went ahead and short circuited the approval with a new patchset now with g flag21:39
anteayaokay so in the gate-setup-workspace-new log for that job it had a ceilometer with HEAD is now at 2f9300f21:40
*** jnoller has quit IRC21:40
anteayaso I am not understanding how it couldn't be found by swift21:40
SergeyLukjanovheh, looks like I have a very bad karma -
fungiclarkb: i checked current systems in rax and hpcloud and it looks like it wouldn't make any difference, but it's still good for future-proofing agreed21:41
*** amcrn has joined #openstack-infra21:41
fungianteaya: swift is looking for an installed ceilometer. what you see there is a git repository for ceilometer cached locally om the server. so the missing link is what does the installation of the service (devstack, i'd assume)21:43
anteayaah okay21:43
*** jergerber has joined #openstack-infra21:43
*** jcoufal has quit IRC21:44
*** tjones has quit IRC21:44
fungiclarkb: so performance+pvhvm?21:47
*** thomasem has quit IRC21:47
*** cody-somerville has quit IRC21:47
fungiclarkb: was it the elasticsearch or logstash worker nodes which needed replacing?21:48
clarkbfungi: elasticsearch21:48
clarkbfungi: the logstash worker nodes seem to be mostly ok and trouble lies further down the pipeline21:48
openstackgerritA change was merged to openstack-infra/nodepool: Retry ssh connections on auth failure.
fungiand did we want to scale the cluster out onto new elasticsearch workers, then scale off the old ones, or replace a few at a time (basically scale up then down, reusing old node names)?21:48
fungier, scale up then down, OR reusing old node names?21:49
clarkbfungi: I think we want to add a node, remove old node, and iterate21:49
clarkbfungi: and we can use names like for the new nodes21:49
openstackgerritA change was merged to openstack-infra/nodepool: Add fedora support
*** juice- has quit IRC21:49
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest
openstackgerritA change was merged to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves
fungiclarkb: okay, so the 6 which we currently have?21:50
clarkbfungi: but before we spin up new nodes we probably want to do a couple things. Edit the elasticsearch node list(s) in site.pp so that firewalls are updated appropriately. And we need to make the ES heap size allocation more dynamic. Right now it is hard coded to 16GB iirc21:50
clarkbfungi: the six we currently have a elasticsearch.o.o and elasticsearch2-6.o.o21:51
clarkbzaro: can you address before you travel?21:51
NobodyCamok I was just asked a question that I did not know the answer to. So I am asking here hopping to learn somehting new today. the question is: are there any issues tagging a review with more then one BluePrint?21:51
fungiclarkb: right the same ones i was looking at in the nova list then. perfect21:51
*** juice- has joined #openstack-infra21:52
fungiNobodyCam: ooh, absolutely no idea, but i expect the hook which updates the topic and adds notes to the bp on lp might get confused by that21:52
fungii think it assumes no more than one bp associated with a change21:52
clarkbdo bp links do funcy stuff like that?21:52
clarkbit may just be bugs that do21:52
NobodyCamI didn't know21:52
* anteaya wonders if this is important: Could not find any downloads that satisfy the requirement alembic>=0.4.1 (from ceilometer==2014.1.dev189.g2f9300f)21:52
fungiclarkb: the hook changes the review topic at least21:53
fungii'd have to read back through the script in jeepyb to find out what else21:53
fungisince i don't normally use blueprints, i haven't paid close attention21:53
clarkbsame here21:54
NobodyCamok so Safe answer is don't do it21:54
fungianteaya: perhaps. is that in the devstack log?21:54
clarkbfungi: so ya for ES nodes I think we want to address those two puppet things first, then spin up a machine that joins the cluster21:54
fungiNobodyCam: or try it out on a change to openstack-dev/sandbox with a throwaway blueprint and see what happens21:54
NobodyCamanteaya: sounds like a out of date repo21:54
anteayaNobodyCam: it should be our pypi mirror I think21:54
clarkbfungi: and is first item that needs addressing (just add elasticsearch01.o.o to the lists), is the second thing. We want 16g heap on the21:56
clarkb30G nodes, but probably want 30G heap on 60G nodes and so on21:56
*** protux has quit IRC21:57
*** juice has quit IRC21:57
*** juice- is now known as juice21:57
clarkbthat needs to be made slightly more flexible in puppet, maybe a fact? or just a second elasticsearch_node manifest that sets a different value for the new nodes21:57
fungianteaya: "Cannot fetch index base URL" i think is where the fun is21:57
fungianteaya: possibly that slave ran into network trouble reaching the pypi mirror21:57
fungiclarkb: working on those now21:58
clarkbfungi: rule of thumb from upstream ES is have heap size set to about 50% of total available RAM21:58
clarkbthat gives the OS plenty of room for caches21:59
anteayafungi: so do we have a bug open for that?21:59
fungiyeah, taking a look at the options in flavor-list now21:59
fungianteaya: maybe? i don't recall. it does happen from time to time... internets and all that21:59
clarkbfungi: as far as choices for nodes go I have been leaning towards a 60G performance node with pvhvm image and elasticsearch data on cinder volume(s)21:59
*** e0ne has quit IRC22:00
fungianteaya: putting pypi mirrors in each provider (or maybe in each region/availability zone) could help mitigate it some22:00
clarkbI think doubling the node size from 30G should be a big improvement and if we outgrow the 60G nodes we can move the cinder block mounts22:00
fungiclarkb: oh, fun, so we're adding cinder volumes on these too. excellent22:01
anteayafungi: not seeing an open bug for it, open one or no?22:01
clarkbfungi: well we don't absolutely need to if we go with 120G nodes :)22:01
fungianteaya: i guess open one and if it turns out to be a dupe later when we triage, we can just mark it as such at that point22:01
clarkbthe 120G nodes have enough ephemeral disk, but I think 120G nodes are overkill and going with cinder blocks is more appropriate22:02
clarkbjeblair: ^ thoughts?22:02
fungiclarkb: right. 60g w/ cinder sounds fine22:02
*** smarcet has quit IRC22:02
anteayafungi: here is one that looks close if not exact:
anteayaor are they different?22:03
fungianteaya: nope. unrelated. that's a problem with hpcloud nodes deciding they have ipv6 routes when they don't22:03
anteayanew bug it is22:03
*** rlandy_ has quit IRC22:04
jeblairclarkb: yeah, let's not overkill on ram.22:04
*** mgagne1 is now known as mgagne22:05
anteayafungi: looks like this might be the same bug:
fungianteaya: yep22:05
anteayaI will add a comment to the report22:06
*** dkliban has joined #openstack-infra22:07
anteayanotmyname: thanks for your help, it was a pypi issue22:07
*** emagana_ has quit IRC22:08
notmynameanteaya: ah, good to know. I'm glad you found it :-)22:08
anteayathanks for the chat22:09
*** yassine has quit IRC22:09
fungiclarkb: any reason not to just pass heap_size in from manifests/site.pp temporarily?22:10
clarkbfungi: and have two different node matches? I think that would work well22:10
fungiand then move it to modules/openstack_project/ later once they're consistent22:10
fungii'll do that. no need to overengineer the module for the sake of temporary disparity in heap size between old and new servers22:11
sdagueso my xtrace change only trimmed the console.html by 15-20%22:11
fungisdague: that's 15-20% of a pretty big number though22:12
sdagueit got them down to 1.9MB22:12
sdagueinstead of 2.3MB22:12
*** thomasem has joined #openstack-infra22:12
*** SumitNaiksatam has quit IRC22:12
sdaguehonestly my biggest gripe is the fact that now there are a lot of xtrace lines because I keep turning it off22:13
lifelessjeblair: fungi: tripleo cloud should be functional again now, some cleanup remaining22:14
lifelessjeblair: fungi:
lifelessis our biggest current clue22:14
fungilifeless: don't rule out some n00b in the data center wondering what the nmi button does22:15
lifelessfungi: already checked that :)22:16
lifelessinfra folk had not touched the rack22:16
lifelessor the switches22:16
*** mfer has quit IRC22:16
* fungi has DEFINITELY seen that happen before22:16
funginmi button exposed on the server's bezel is such a bad, bad idea22:16
*** moted has joined #openstack-infra22:17
jeblairfungi, sdague: there are some changes to use six in zuul; should we add six as a direct dependency?  it seems to be getting it from somewhere implecit because it ends up in the venv...22:18
jeblairfungi, sdague: (i ask because you know everything there is to know about six right now)22:18
fungijeblair: i think if zuul directly imports things from six, it should declare that in its reqs22:19
sdagueyes, agreed22:19
fungijust like with any python package22:19
jeblairthat was my thinking as well.  these changes got a lot of +'s though so i thought i'd double check.  :)22:19
fungipeople will + anything (i'm living proof!)22:19
*** miqui has quit IRC22:20
*** nati_ueno has quit IRC22:20
*** matrohon has joined #openstack-infra22:21
lifelessfungi: can you check nodepool thinks the cloud is back ?22:21
*** nati_ueno has joined #openstack-infra22:21
*** sandywalsh has quit IRC22:22
openstackgerritA change was merged to openstack/requirements: Don't update pbr's
openstackgerritA change was merged to openstack-infra/storyboard: Migration to add the openid field
openstackgerritJenkins proposed a change to openstack-dev/pbr: Updated from global requirements
openstackgerritA change was merged to openstack/requirements: Standardize read ops to 'with open' construction
sdaguejeblair: with your devstack +A powers you could +A the log format change -
jeblairsdague: that seems in-scope for me, sure.22:25
*** Sukhdev has quit IRC22:25
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra
*** lcostantino has quit IRC22:26
*** esker has quit IRC22:27
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Create new 30g heap elasticsearch workers
fungiclarkb: that ^ i think. gonna grab dinner and then work on spinning up new virtual machines if the change looks okay22:29
*** lnxnut has quit IRC22:29
* clarkb looks22:30
jeblairsdague: aprvd22:30
fungiclarkb: the elasticsearch module seems to already be configurable but defaults to 16g, so not changing it there in case anyone else is also using it22:31
*** chenxu_ has quit IRC22:31
*** julim_ has quit IRC22:32
openstackgerritA change was merged to openstack-infra/storyboard: Load projects from yaml file
*** jgrimm has quit IRC22:32
clarkbfungi: reviewed22:34
fungilifeless: i see some building for a little while and some very recently ready nodepool nodes in tripleo-ci22:34
*** sandywalsh has joined #openstack-infra22:35
fungiclarkb: any reason not to make the elasticsearch_nodes and discover_nodes lists a second change then, and spin up all 6 new workers at once?22:35
*** lcheng has quit IRC22:36
fungii'm happy to split it up22:36
*** lcheng has joined #openstack-infra22:36
*** ociuhandu has joined #openstack-infra22:37
clarkbfungi: my only concern is that growing the cluster by 2x all at once will cause it to freak out and rebalance all the things then as nodes are removed it will rebalance all over again22:37
clarkbwhereas doing one node at a time should keep the rebalance thrash to a minimum22:37
fungiclarkb: so 6 changes, one for each addition to elasticsearch_nodes and discover_nodes, with corresponding removal changes for the old ones after each?22:38
clarkbfungi: we should be able to do a single removal change to remove nodes (we can manually 'remove' them by turning them off)22:39
clarkbfungi: or, we put up bogus A records for now22:39
clarkbthen as we get IPs kick iptables-peristent22:39
clarkbas I think about that the idea isn't too terrible22:39
*** lcostantino has joined #openstack-infra22:41
*** ianw has joined #openstack-infra22:42
*** yamahata has joined #openstack-infra22:43
fungiwhich, the temporary dummy rrs?22:44
*** denis_makogon has quit IRC22:45
*** yolanda_ has quit IRC22:45
*** dkranz has quit IRC22:45
fungii'll just add those in that case, and remove them before updating dns for each new launch22:45
clarkbfungi: for elasticsearch01 02 03 and so on22:47
*** StevenK_ is now known as StevenK22:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add Zuul Job Queue graph
*** jamielennox|away is now known as jamielennox22:53
*** pdmars has quit IRC22:54
fungiclarkb: right22:55
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add Zuul Job Queue graph
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: MVP Storyboard Client
*** CaptTofu has quit IRC22:57
jeblairsorry about the churn on that; i think i'm happy with ps3 now. ^22:58
*** e0ne has joined #openstack-infra22:58
jeblairand, btw, you can see the tripleo outage on it.22:59
jeblair(waiting climbs while workers falls == provider outage)22:59
*** jergerber has quit IRC23:00
*** e0ne_ has joined #openstack-infra23:00
*** bhuvan has joined #openstack-infra23:02
fungiclarkb: added23:02
fungiclarkb: i set them all as for now. nothing should be listening on lo for that. hopefully making them all the same address won't confuse anything?23:03
bhuvanjeblair: . one quick question ...23:03
clarkbfungi: stuff will listen on that because linux23:03
*** e0ne has quit IRC23:04
clarkbfungi: :) might be better to use bogus 172 or 169.whatever addresses23:04
fungiclarkb: oh, right. bsd brain. i'll make up something else23:04
bhuvanthe current approach is to download the package from sss website directly and install. if there are any security fixes, we might want to change to download new version23:04
bhuvani presume the security fixes will be shipped as part of new package automatically ...23:05
jeblairbhuvan: right, but that requires us to monitor that project so that we notice those security fixes and then update that version23:05
*** e0ne_ has quit IRC23:05
jeblairbhuvan: i'm pretty sure no one's going to do that for an irc log analyzer23:05
bhuvando you propose to bundle that software within our code?23:06
jeblairbhuvan: i would never propose that!23:06
*** oubiwann_ has quit IRC23:06
jeblairbhuvan: so we really need something that's packaged by ubuntu, or else, a continuous deployment mechanism that installs it from their git repo, or something similar23:06
bhuvanhmm, how do we programatically detect for new security updates?23:07
jeblairbhuvan: we install a number of things that aren't distro-packaged in that manner (from git)23:07
*** oubiwann_ has joined #openstack-infra23:07
jeblairbhuvan: it means they are more likely to break, but less likely to sit around on old versions and miss security updates23:07
jeblairbhuvan: but honestly, something packaged by a distro would be ideal for this.23:08
*** weshay has quit IRC23:08
bhuvanunfortunately, sss isn't part of any distro yet23:09
jog0I take it is a known issue23:09
bhuvani'm fix the patch to deploy from git23:09
jeblairjog0: not by me23:09
bhuvanjeblair: i think it's better than downloading package directly. what you think?23:10
jog0jeblair: uh oh23:10
jeblairbhuvan: i think so.  i'd ask some of the other infra core folks for their opinion first.  clarkb and fungi in particular.23:11
fungiclarkb: okay, i've switched them to
*** CaptTofu has joined #openstack-infra23:11
*** rossella-s has quit IRC23:12
clarkbfungi: wfm23:12
clarkbwill remove my -123:12
fungijeblair: bhuvan: yes, i'd rather see the irc stats generation fail than the server get compromised. also deploying from git should probably allow us to do things not as root (whereas installing packages means running maintscripts with root privs)23:13
jog0fungi clarkb: ^23:13
*** rossella-s has joined #openstack-infra23:13
jeblairjog0: looking at nodepool logs, i can tell you that nodepool did not remove that node before the job aborted.23:14
clarkbfungi: I +2'd but did not approve. Mostly because I would like to see the first node get spun up using that in a different puppet env23:15
fungijog0: that's a new one on me23:15
jeblairWARNING: Making bare-precise-hpcloud-az2-1572924 offline temporarily due to the lack of disk space23:15
*** oubiwann_ has quit IRC23:15
jeblairjog0: ^23:15
clarkbfungi: the other potential issue is that elasticsearch will happily get installed and join the cluster before we can attach the cinder volume I think23:15
fungiclarkb: sure, can do23:15
clarkbfungi: Thoughts on how to solve that problem?23:15
fungihmmm... let me think it over while i finish eating23:16
clarkbthis is an issue beacuse it will start indexing and stuff then a new mount will shadow that data23:16
fungiclarkb: can we set the service not to start at boot?23:16
fungiin puppet manifests23:17
*** eharney has quit IRC23:17
jog0jeblair: message:" Looks like the node went offline during the build" AND message:"slave.log \(No such file or directory\)" AND  filename:"console.html"23:17
clarkbfungi: probably? I think the upstream pacakges start the service23:17
jog0if that looks like a good query to you, I will go ahead and file a e-s patch for it23:17
clarkbfungi: you know more about packaging than I, if we can make that work it sounds good to me23:17
fungioh, hrm, right23:17
openstackgerritgordon chung proposed a change to openstack-infra/config: enable doc generation for pycadf
jeblairjog0: yeah.  it might cover slightly more nuanced problems than this, but it's always some kind of jenkins problem so is a good one.23:18
jog0jeblair: kk23:18
*** lcostantino has quit IRC23:18
jog0we don't use that much data by the end of the run23:18
fungiclarkb: what port/protocol do they connect over? can we lock that down more tightly in iptables?23:19
jeblairjog0: yeah, it could be a false error from jenkins (like perhaps its attempt to perform that check encountered a network error)23:19
* jog0 files a bug23:20
jeblairzaro, jog0: i think this might be adverse interaction with gearman-plugin23:20
clarkbfungi: its the es protocol ports 9200-939923:21
clarkbfungi: one half uses http REST the other half a home grown thing23:21
clarkbfungi: what if we do the intial install without your puppet change so that the default manifest is used23:22
*** dims has quit IRC23:22
clarkbfungi: configure cinder, then add the nodes using your manifest23:22
clarkbfungi: that should allow us to prime all of the nodes before we add them to the cluster23:22
fungiclarkb: great idea23:22
fungiclarkb: actually, no23:22
fungithe current pattern is \d*23:23
fungiso it will match the new names too23:23
bhuvanjeblair: fungi: thank you. i'll fix the patch to install latest sss changes using puppet/vcsrepo. the script will be run from cron to generate the stats ...23:23
*** derekh_afk is now known as derekh23:23
jeblairbhuvan: cool, thanks23:23
bhuvanthe script might fail if sss is not already installed23:23
fungiclarkb: so a quick tweak to make the current pattern \d? should make that possible23:23
clarkbfungi: that wfm, though doesn't handle the next time we need to do this23:24
jeblairbhuvan: you can have the cron require the vcsrepo in puppet, that should be good enough23:24
bhuvani presume puppet agent is run as daemon and it'll take care of pulling latest changes from git automatically23:24
jeblairbhuvan: yes23:24
bhuvanjeblair: yep23:24
jeblairjog0, zaro: the incidences of this in the jenkins log strongly correlate with the gearman plugin de-registering jobs23:25
jeblairjog0, zaro: it's possible it's more complicated than that, but it's worth a look.23:26
jeblairjog0, zaro:
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Auth support
bhuvanfungi: as discussed couple of weeks back, review/merge when you find time ...23:26
jeblairjog0: can you throw that in the bug report if you still have it up?23:26
*** tjones has joined #openstack-infra23:27
*** lcheng has quit IRC23:27
*** branen has joined #openstack-infra23:28
jeblairjog0: thanxs23:28
*** hashar has quit IRC23:29
openstackgerritSean Dague proposed a change to openstack-infra/config: index sublog files
sdagueclarkb: that should be the tag fix23:29
*** rossella-s has quit IRC23:32
*** alexpilotti has quit IRC23:32
fungiclarkb: i think i'll see if i can work out a way to make service enabling an option to the module, so we can throw the switch via puppet23:33
clarkbfungi: sounds good23:33
fungibhuvan: thanks and apologies for the delay. i'm trying to get the pull request closer running again first, but trying to figure out why the orgs list api call is coming back with an empty set for our account (i did log into github with the account and it sees it's a member of a group on each org)23:34
clarkbjeblair: for gearman merge workers. if they should be talking to a gerrit replica instead of gerrit itself (assume replication is atomic and instantaneous), zuul can't do that today because the url info comes from gerrit itself irght?23:34
*** zehicle_at_dell has joined #openstack-infra23:34
*** hemna is now known as hemnafk23:34
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Use statsd in logstash client
*** amcrn has quit IRC23:35
jeblairclarkb: ^ what we were missing this wknd23:35
sdagueclarkb: ok, so actually 3.4 M -> 2.7 M with the xtrace changes23:36
sdague2.6 M23:36
bhuvanfungi: thanks23:36
jeblairclarkb: i don't understand the question23:36
sdagueclarkb: if you see other good things to slice out of the logs, now is a good time to speak up23:37
clarkbjeblair: gerrit emits a patchset created event to zuul, zuul kicks off the merge worker to merge that. We don't want the merge worker to do its initial clone or fetches from gerrit directly but a gerrit replica instead23:37
clarkbjeblair: that doesn't seem supported in zuul today23:37
clarkbsdague: I trust your judgement :)23:38
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1284371
jog0jeblair: ^23:38
jeblairclarkb: why don't we want it to fetch from gerrit directly?23:38
clarkbjeblair: because lol firewalls23:38
*** dims has joined #openstack-infra23:39
jeblairclarkb: what firewalls?  our gerrit .....23:39
clarkbjeblair: I am mostly trying to frame the issue and understand where things might break spectacularly23:39
clarkbjeblair: no not our gerrit23:39
clarkbhypothetical hp gerrit23:39
jeblairclarkb: you are using "we" in a way you have not previously used "we" :(23:39
clarkboh sorry23:39
jeblairme too23:39
jeblairclarkb: to answer your question, there may or may not be a race condition there, i would recommend finding a definitive answer to that first.23:41
clarkbjeblair: there definitely is a race condition between gerrit emitting the ssh event and the replica having the data related to that event23:41
lifelessfungi: still seeing queued everywhere :(23:41
fungilifeless: i'll have another look at nodepool23:42
*** dolphm is now known as dolphm_50323:43
*** thomasem has quit IRC23:43
jeblairclarkb: how is it possible for you to connect to gerrit over ssh and not fetch refs over that same ssh connection?23:44
*** sarob_ has joined #openstack-infra23:44
clarkbjeblair: zuul is on one side of the firewall and the zuul merger is on the other23:44
clarkbso zuul gets ssh events fine, but the merger doesn't have ssh access23:45
clarkb* ssh access in23:45
SpamapSFYI tripleo images that build with the mirror are working again.23:45
lifelessfungi: ah - ERROR: Quota exceeded for instances: Requested 1, but already used 100 of 100 instances (HTTP 413) (Request-ID: req-3ad1e797-d33d-47be-87d1-2c63dc24f542)23:46
lifelessfungi: I think I see the problem23:46
fungilifeless: i'm clearing all old nodes23:46
lifelessfungi: I'm going to do a rolling restart of nova-compute to clear out the stuck deleting instances23:46
fungiall tripleo nodes in any state for more than an hour23:46
lifelessSpamapS: can you help me? I'll take novacompute0-4, you do 5-9 ?23:46
*** sarob has quit IRC23:47
jeblairclarkb: zuul does not support this, but you could patch it to have the merger busy-wait for the ref until a timeout.23:47
fungilifeless: we had ~90 nodes which were more than an hour in any given state in your cloud23:47
*** oubiwann_ has joined #openstack-infra23:47
clarkbjeblair: thanks23:48
SpamapSlifeless: sure23:49
*** sarob_ has quit IRC23:49
*** prad_ has quit IRC23:49
*** chadlung has quit IRC23:49
jeblair7 of 9 changes in the gate are devstack or devstack-gate23:50
jeblairsorry 6 of 823:50
lifelessfungi: I issued a bulk (API) delete of ERROR state instances23:50
lifelessSpamapS: shout when thats done please23:51
lifelessSpamapS: I filed about this23:51
lifelessSpamapS: not sure its amenable to 'just go fix' though23:51
SpamapS5,6 done23:51
lifelessok, I'll grab 8, 923:51
SpamapS7 done23:52
jeblairpushed 0.5.3 tag to gear23:52
SpamapShm 7 showing down still23:53
jog0so oslo is wedged!
*** alexpilotti has joined #openstack-infra23:53
SpamapSn/m it came up23:53
jog0jd__: ping23:53
lifelessfungi: there are some unallocated floating-ip's23:54
lifelessfungi: (of nodepools)23:54
lifelessfungi: but all ERROR instances are now cleared23:54
dhellmannjog0: looking23:54
lifelessfungi: and I see a bunch of active slaves23:54
jog02014-02-24 22:06:00.332 | Downloading/unpacking eventlet>=0.13.0 (from oslo.messaging>=1.3.0a4->pycadf>=0.1.9->-r /home/jenkins/workspace/gate-oslo-incubator-python33/requirements-py3.txt (line 14))23:54
*** salv-orlando has quit IRC23:54
lifelessand jobs have attached. Yay. thanks23:54
jog0eventlet!=python 3 friendly23:55
*** salv-orlando has joined #openstack-infra23:55
* jog0 files a bug23:55
jeblairjog0: i think we just merged the change that enabled that job today23:55
fungilifeless: the delete-state nodecount is dropping too23:55
jog0jeblair: yup23:55
jog0this morning23:55
dhellmannyeah, we can fix it in the incubator repo23:56
dhellmannfor some reason the incubator is installing oslo.messaging, which brings in eventlet23:56
jog0dhellmann:  ohh pycadf23:56
dhellmannwtf, why does pycadf require oslo.messaging?23:56
jog0dhellmann: to keep you awake at night23:57
lifelessjeblair: fungi: were you impacted by this outage in any non-trivial way ?23:57
jeblairlifeless: no23:57
* lifeless runs arund like kermit23:57
fungilifeless: flail those kermit arms wildly23:58
jog0dhellmann: so no py33 gate on pyacdf23:59
jeblairclarkb: can you review ?23:59
dhellmannjog0: yeah, I'm testing whether that's enough23:59
jog0ergo it shouldn't be in py33 set in oslo-incubator23:59
*** fbo is now known as fbo_away23:59
*** e0ne has joined #openstack-infra23:59
fungiclarkb: so, there are no trivial mechanisms i can find to preconfigure elasticsearch not to automatically start when the package gets installed. instead i think i'll just move the class instantiation for that pattern to a separate change23:59
clarkbjog0: sure23:59
jeblairclarkb: also, i'd like to get that into prod asap.  do you think there's a chance logstash might catch up tonight?  if not, i'm thinking about declaring queue bankruptcy to get that in23:59
jog0dhellmann: ahh

Generated by 2.14.0 by Marius Gedminas - find it at!