Friday, 2014-09-12

*** david-lyle has quit IRC00:00
wenlockanyone able to review some reviews that i think are close to +3, i'd like to try to wrap those up before weekend:,,
sandywalshright, gotcha00:00
anteayasandywalsh: you like the odd couple too?00:00
*** derekh has joined #openstack-infra00:00
anteayaklugman, quincy?00:00
anteayaodd couple00:00
jeblairfungi, clarkb, mordred: i'm out for the evening00:00
fungiokay, so jenkins-dev has moved cleanly. we're deleting them from the etherpad or leaving them there?00:01
jeblairfungi: del from ep00:01
*** ChuckC_ has quit IRC00:01
*** nelsnelson has joined #openstack-infra00:01
fungithanks jeblair!00:01
jeblairfungi, thank you!00:01
fungii'm mostly out for the evening now as well. cookies, beer and star trek. three great tastes that go great together00:01
anteayafungi: enjoy00:02
jeblairnibalizer: puppet2 hostfile has 35 lines, puppet3 hostfile has 4500:02
fungii'll try to hit some of the low-hanging fruit servers in the morning if there are any left when i get online00:02
jeblairso we're >.5 done00:02
jeblairfungi: the xserves? ;)00:02
sandywalshanteaya: ah, heh, Quincy ... forensic pathologic ... looking for the cause of death ... just like StackTach.00:02
anteayasandywalsh: ah00:02
anteayasee it was klugman that made me wonder if you were branching out a bit00:02
fungijeblair: those are indeed very low-hanging00:03
anteayaI remember that show00:03
sandywalshquincy, quince and klugman00:03
fungijeblair: but sort of an apples to oranges comparison00:03
sandywalshapi, api-implementation and cmdline tools :000:03
anteayasandywalsh: yeah, I am wondering about the quincy, quince naming00:03
anteayasandywalsh: feels too close to each other to me, but meh00:03
*** emagana has quit IRC00:04
sandywalshcache invalidation and naming things ... two hardest problems00:04
*** praneshp has quit IRC00:05
*** oomichi_ has joined #openstack-infra00:05
*** praneshp has joined #openstack-infra00:05
*** oomichi has quit IRC00:08
*** _buzz_ has quit IRC00:08
*** dannywilson has quit IRC00:08
*** rfolco has joined #openstack-infra00:09
*** derekh has quit IRC00:09
*** dannywilson has joined #openstack-infra00:09
*** _nadya_ has joined #openstack-infra00:09
*** nelsnelson has quit IRC00:10
*** ZZelle_ has quit IRC00:10
jrollsdague: if you have a quick sec, super easy devstack review: 12093900:10
openstackgerritSpencer Krum proposed a change to openstack-infra/puppet-storyboard: Template was comparing to :undefined
nibalizerokay i think this ^^ fixes the problem morderd had00:10
*** chuckC has quit IRC00:11
*** signed8bit is now known as signed8bit_ZZZzz00:11
nibalizerwhich is less of a fix and more of making the behaviour be the same on puppet 2 and puppet 300:11
*** signed8bit_ZZZzz has quit IRC00:12
*** dannywilson has quit IRC00:13
nibalizerjeblair: sweetness, i'd say we met our goals for this week :P00:14
*** tomoe has joined #openstack-infra00:14
*** _nadya_ has quit IRC00:14
*** hdd has quit IRC00:15
*** dannywilson has joined #openstack-infra00:16
*** yamamoto has joined #openstack-infra00:16
*** baoli has joined #openstack-infra00:16
*** wenlock has quit IRC00:17
*** dims__ has quit IRC00:17
*** xchu has joined #openstack-infra00:19
*** xchu has left #openstack-infra00:19
*** yamamoto has quit IRC00:20
clarkbok I am back. looks like we are winding down for the day. I probably shouldn't continue to move nodes just in case I trip over myself00:22
mordredclarkb: listen. here's the deal ...00:22
mordreddrink a ton of beer00:22
mordredclarkb: THEN work on more nodes00:22
clarkbmordred: are you sure that is safe? I suppose I won't care at athat point :)00:23
anteayadon't drink and run commands00:23
*** melwitt has quit IRC00:25
*** Vega has joined #openstack-infra00:25
*** Vega is now known as Guest5146300:25
*** MaxV has joined #openstack-infra00:26
*** dannywilson has quit IRC00:26
*** baoli has quit IRC00:29
*** MaxV has quit IRC00:31
*** rhsu has quit IRC00:35
*** weshay has joined #openstack-infra00:35
*** jab has joined #openstack-infra00:38
*** jab has joined #openstack-infra00:38
*** hdd has joined #openstack-infra00:38
*** jab has quit IRC00:40
krotscheckmordred: Where’s that migration script of yours? I want to do a test run00:40
*** mriedem has joined #openstack-infra00:44
*** unicell has joined #openstack-infra00:44
*** MarkAtwood has quit IRC00:44
krotscheckmordred: Durrr… did that merge already?00:45
krotscheckOh it did.00:48
*** david-lyle has joined #openstack-infra00:48
krotscheckAlrightey, let;s see if this is a thing.00:48
*** amcrn has quit IRC00:48
*** bhuvan_ has quit IRC00:50
mordredkrotscheck: yay!00:54
krotscheckmordred: I’m going to wrap a nice UX around this.00:54
krotscheckLike, you know, commandline options.00:55
krotscheckDo we have a convention out there somewhere? My gut is saying storyboard-import <source> <url>00:55
krotscheckwhere source is <launchpad|github|jira| etc>00:55
*** mwagner_lap has quit IRC00:55
*** chuckC has joined #openstack-infra00:56
*** arosen has left #openstack-infra00:56
mordredwell... first stab, I'd say "storyboard-import project" (because I'm a simpleton)00:58
mordredand then --source= which defaults to launchpad00:58
anteayakrotscheck: I just -1'd not for anything you did in your patch, but we have an oppornity to make the file even better00:58
*** hdd has quit IRC00:58
mordredkrotscheck: and, --url which is optional, but which can provide a url in case an import source is something like github or jira which might have a different url00:59
mordredkrotscheck: so that the very simple invocation does the default thing that makes us happy - and that there is an increasing number of ways to say other things00:59
krotscheckOK: Do one thing, do it well.00:59
krotscheckanteaya: kk01:00
mordredkrotscheck: there shoudl also be an option, --unicorn, which if given, wll cause a unicorn to appear01:01
krotscheckHey now, let me learn argparse first.01:01
*** baoli has joined #openstack-infra01:02
mordredis argparse the one we want?01:03
mordredor is it the other one?01:03
mordredkrotscheck: ^^ clarkb ^^ ?01:03
krotscheckoptparse is the old one01:03
mordredok. cool01:04
mordredI can never remember01:04
krotscheckThere’s a 3.7 version called unicornparse that’ll replace it all at PID101:04
mordredkrotscheck: SWEET!01:05
clarkbwe want argparse01:05
*** yamahata has joined #openstack-infra01:05
*** praneshp has quit IRC01:07
* krotscheck needs a non-reserved synonym for import01:12
*** shashankhegde has quit IRC01:13
*** mwagner_lap has joined #openstack-infra01:16
*** yamamoto has joined #openstack-infra01:16
anteayamy thesaurus gives me nothing better than bam!01:16
anteayaI'm so disappointed01:17
clarkbI was channeling whats his name the chef01:17
clarkbwhen you throw spices into a dish you go "bam!"01:17
anteayayeah, I know01:17
anteayaI can't allow my mental energy to follow the thought to think of him01:18
anteayayou can channel him01:18
*** signed8bit has joined #openstack-infra01:21
*** yamamoto has quit IRC01:21
*** signed8bit is now known as signed8bit_ZZZzz01:21
*** signed8bit_ZZZzz is now known as signed8bit01:23
*** hdd has joined #openstack-infra01:23
*** signed8bit has quit IRC01:23
*** Daisy has joined #openstack-infra01:26
*** MaxV has joined #openstack-infra01:26
*** dims_ has joined #openstack-infra01:28
jeblairkrotscheck: for fun, it should also be in  maybe "introduce".  "move_into".01:29
*** MaxV has quit IRC01:31
krotscheckjeblair: “Pink” works for me.01:31
*** dims_ has quit IRC01:31
*** dims_ has joined #openstack-infra01:32
*** homeless has quit IRC01:33
anteayaha ha ha01:34
*** garyh has quit IRC01:34
*** alaski has quit IRC01:36
*** sbalukoff has quit IRC01:38
*** pcrews has quit IRC01:39
*** dims_ has quit IRC01:39
*** marcoemorais has quit IRC01:40
*** jogo has quit IRC01:40
*** dims_ has joined #openstack-infra01:40
*** MaxV has joined #openstack-infra01:42
*** dims__ has joined #openstack-infra01:44
*** dims_ has quit IRC01:44
*** nosnos has joined #openstack-infra01:46
*** MaxV has quit IRC01:47
*** annegentle has joined #openstack-infra01:50
*** dims__ has quit IRC01:51
*** dims_ has joined #openstack-infra01:52
*** hdd has quit IRC01:54
*** baoli has quit IRC01:54
*** baoli has joined #openstack-infra01:55
*** dims_ has quit IRC01:57
*** annegentle has quit IRC01:57
*** patrickeast has quit IRC02:03
*** pcrews has joined #openstack-infra02:05
*** yaguang has joined #openstack-infra02:06
*** ramashri has quit IRC02:10
*** alexpilotti has quit IRC02:11
*** wenlock has joined #openstack-infra02:11
*** baoli has quit IRC02:13
*** baoli has joined #openstack-infra02:14
*** mriedem has quit IRC02:15
*** yamamoto has joined #openstack-infra02:16
*** pcrews has quit IRC02:16
*** annegentle has joined #openstack-infra02:18
*** praneshp has joined #openstack-infra02:21
*** yamamoto has quit IRC02:21
*** Ryan_Lane has quit IRC02:22
*** yamamoto has joined #openstack-infra02:23
*** sunrenjie6 has joined #openstack-infra02:26
*** ianw has quit IRC02:27
*** jyuso has joined #openstack-infra02:33
*** gokrokve has joined #openstack-infra02:36
*** weshay has quit IRC02:36
*** weshay has joined #openstack-infra02:37
*** stevemar has joined #openstack-infra02:41
*** yamamoto has quit IRC02:42
*** MaxV has joined #openstack-infra02:43
*** praneshp has quit IRC02:44
*** baoli has quit IRC02:45
*** baoli has joined #openstack-infra02:45
*** MaxV has quit IRC02:48
*** daya_k has joined #openstack-infra02:51
*** nosnos has quit IRC02:53
*** nosnos has joined #openstack-infra02:53
*** KanagarajM has joined #openstack-infra02:56
*** harlowja is now known as harlowja_away02:56
*** hdd has joined #openstack-infra02:56
*** nosnos has quit IRC02:58
*** rushiagr_away is now known as rushiagr03:07
*** wenlock has quit IRC03:09
*** praneshp has joined #openstack-infra03:12
*** adalbas has quit IRC03:14
*** stevemar has quit IRC03:16
*** stevemar2 has joined #openstack-infra03:16
*** gokrokve_ has joined #openstack-infra03:17
*** yamamoto has joined #openstack-infra03:18
*** gokrokve has quit IRC03:20
*** gokrokve_ has quit IRC03:21
*** praneshp_ has joined #openstack-infra03:22
*** armax has quit IRC03:25
*** armax has joined #openstack-infra03:26
*** armax has quit IRC03:26
*** praneshp has quit IRC03:26
*** praneshp_ is now known as praneshp03:26
*** rushiagr is now known as rushiagr_away03:28
*** rmcall has joined #openstack-infra03:29
*** praneshp has quit IRC03:36
*** praneshp has joined #openstack-infra03:38
*** _nadya_ has joined #openstack-infra03:39
*** praneshp_ has joined #openstack-infra03:41
*** praneshp has quit IRC03:42
*** praneshp_ is now known as praneshp03:42
*** MaxV has joined #openstack-infra03:44
*** MaxV has quit IRC03:48
*** baoli has quit IRC03:51
*** zz_jgrimm is now known as jgrimm03:51
*** amotoki has joined #openstack-infra03:55
*** ramashri has joined #openstack-infra03:57
*** koolhead17 has joined #openstack-infra03:59
*** ramashri has quit IRC04:02
*** rmcall has quit IRC04:03
*** rmcall has joined #openstack-infra04:04
*** AaronGreen has joined #openstack-infra04:07
*** gokrokve has joined #openstack-infra04:07
*** AaronGreen is now known as AaronGr04:07
*** rushiagr_away is now known as rushiagr04:12
*** ramashri has joined #openstack-infra04:12
*** jog0 has joined #openstack-infra04:13
*** nosnos has joined #openstack-infra04:13
*** jog0 is now known as jogo04:13
*** todd_dsm has quit IRC04:14
*** Daisy has quit IRC04:14
*** annegentle has quit IRC04:21
*** annegentle has joined #openstack-infra04:22
*** jogo has quit IRC04:23
*** jog0 has joined #openstack-infra04:23
*** ianw has joined #openstack-infra04:23
*** weshay has quit IRC04:24
*** jog0 has quit IRC04:25
*** jogo has joined #openstack-infra04:25
*** jogo is now known as Guest8613904:25
*** annegentle has quit IRC04:29
*** Guest86139 has quit IRC04:30
*** jogo_ has joined #openstack-infra04:30
*** rfolco has quit IRC04:31
*** bhuvan has joined #openstack-infra04:31
*** wenlock has joined #openstack-infra04:42
*** Daisy has joined #openstack-infra04:44
*** MaxV has joined #openstack-infra04:45
*** jogo_ has quit IRC04:45
*** jogo_ has joined #openstack-infra04:45
*** jogo_ has quit IRC04:46
*** jog0 has joined #openstack-infra04:46
*** jog0 has quit IRC04:46
*** jogo_ has joined #openstack-infra04:47
*** MaxV has quit IRC04:49
*** rhsu has joined #openstack-infra04:49
*** gokrokve has quit IRC04:50
*** mestery has quit IRC04:52
*** jogo_ has quit IRC04:54
*** jogo_ has joined #openstack-infra04:54
*** HenryG is now known as HenryG_afk04:55
*** harlowja_at_home has joined #openstack-infra04:57
jogo_clarkb: I am playing with weechat and have a few questions if your around04:57
jogo_clarkb: for some reason "/script" isn't working04:57
*** tsufiev has quit IRC04:58
*** tsufiev_ has joined #openstack-infra04:58
jogo_Error: unknown command "script" (type /help for help)04:58
jogo_clarkb: any ideas?04:59
clarkbtry using specific language load commands like /python?04:59
clarkbbut going to guess weechat was compiled without script support. I dont actually use any scripts05:00
clarkbmy weechat doesnt have /script but has /python05:00
clarkband ruby and perl05:01
*** _nadya_ has quit IRC05:01
*** asettle has quit IRC05:04
jogo_clarkb: cool it worked05:04
jogo_I wanted the script and its working05:04
*** HenryG_afk has quit IRC05:04
*** alexandra has joined #openstack-infra05:05
*** alexandra is now known as asettle05:06
jogo_clarkb: any scripts you recommend?05:06
clarkbI dont use any. I do recommend autofilters though05:06
clarkbya basically I only see joins parts and quits if I toggle a setting. they are all in my buffers but usually filtered out05:08
clarkbunlike an ignore they are still there if I need to see them05:09
clarkbgah smartfilters05:09
clarkbthats what they are really called05:09
*** jogo_ is now known as jog005:10
*** jog0 is now known as Guest1387505:10
clarkbthen use meta + = to toggle it on off05:11
*** Ryan_Lane has joined #openstack-infra05:11
Guest13875oh neat I have it set that way too05:12
Guest13875gah configuration05:12
*** Guest13875 has quit IRC05:12
*** flashgordon has joined #openstack-infra05:14
*** Ryan_Lane has quit IRC05:14
*** flashgordon has quit IRC05:15
*** HeOS has quit IRC05:16
*** flashgordon has joined #openstack-infra05:18
*** reed has quit IRC05:18
*** flashgordon is now known as jogo05:19
*** jogo has quit IRC05:20
*** jogo has joined #openstack-infra05:22
jogoclarkb: thanks, weechat is pretty nice05:22
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add bare-f20 nodes
*** HenryG has joined #openstack-infra05:23
*** HenryG is now known as HenryG_afk05:24
*** harlowja_at_home has quit IRC05:25
ianwclarkb: ^ puppet-postgresql released so hopefully we can just merge the f20-bare node bring-up with no further changes05:25
*** annegentle has joined #openstack-infra05:28
*** annegentle has quit IRC05:33
*** jogo has quit IRC05:33
*** julim has quit IRC05:33
*** jogo has joined #openstack-infra05:33
*** hdd has quit IRC05:35
*** loki184 has joined #openstack-infra05:35
*** ramashri has quit IRC05:36
*** jogo has quit IRC05:38
*** jogo has joined #openstack-infra05:38
*** Daisy has quit IRC05:39
*** HenryG has joined #openstack-infra05:42
*** vigneshvar has joined #openstack-infra05:42
*** HenryG has quit IRC05:43
*** MaxV has joined #openstack-infra05:45
*** rfolco has joined #openstack-infra05:50
*** MaxV has quit IRC05:50
*** Daisy has joined #openstack-infra05:50
*** morganfainberg is now known as stevemar305:50
*** stevemar3 is now known as morganfainberg05:51
*** emagana has joined #openstack-infra05:55
*** stevemar2 is now known as stevemar05:56
*** wenlock has quit IRC05:57
*** yfried has joined #openstack-infra05:58
*** emagana has quit IRC06:00
*** emagana has joined #openstack-infra06:01
*** emagana_ has joined #openstack-infra06:02
*** jgrimm is now known as zz_jgrimm06:03
*** sunrenjie6 has quit IRC06:04
*** shashankhegde has joined #openstack-infra06:05
*** emagana has quit IRC06:06
*** stevemar has quit IRC06:10
*** ildikov has quit IRC06:17
*** pkoniszewski has joined #openstack-infra06:17
*** flaper87|afk is now known as flaper8706:18
*** shashankhegde has quit IRC06:18
*** emagana has joined #openstack-infra06:24
*** emagana_ has quit IRC06:27
*** annegentle has joined #openstack-infra06:28
*** jamespage_ has joined #openstack-infra06:29
*** emagana has quit IRC06:29
*** andreaf has joined #openstack-infra06:30
*** annegentle has quit IRC06:33
*** rmcall has quit IRC06:35
*** Guest51463 is now known as LuckyVega06:37
*** jcoufal has joined #openstack-infra06:39
*** HeOS has joined #openstack-infra06:42
*** k4n0 has joined #openstack-infra06:45
*** MaxV has joined #openstack-infra06:46
*** Longgeek has joined #openstack-infra06:47
*** sdake has joined #openstack-infra06:50
*** MaxV has quit IRC06:50
*** sdake has quit IRC06:52
openstackgerritA change was merged to openstack-dev/hacking: Updated from global requirements
*** ildikov has joined #openstack-infra07:03
*** mrmartin has joined #openstack-infra07:03
*** mrmartin has quit IRC07:04
*** mrmartin has joined #openstack-infra07:05
*** mrmartin has quit IRC07:05
*** e0ne has joined #openstack-infra07:09
*** penguinRaider has joined #openstack-infra07:16
*** daya_k has quit IRC07:17
*** rhsu has quit IRC07:19
*** dtantsur|afk is now known as dtantsur07:20
*** e0ne has quit IRC07:20
*** arxcruz has joined #openstack-infra07:22
*** berendt has joined #openstack-infra07:23
*** yamamoto has quit IRC07:23
*** praneshp has quit IRC07:24
*** e0ne has joined #openstack-infra07:24
*** koolhead17 has quit IRC07:24
*** e0ne has quit IRC07:26
*** annegentle has joined #openstack-infra07:28
*** niska has quit IRC07:29
*** jp_at_hp has joined #openstack-infra07:29
*** Guest1245 has joined #openstack-infra07:30
*** annegentle has quit IRC07:33
*** niska has joined #openstack-infra07:34
*** ashaeron has joined #openstack-infra07:43
*** jistr has joined #openstack-infra07:47
*** che-arne has quit IRC07:49
*** amotoki has quit IRC07:51
*** amuller has joined #openstack-infra07:51
*** derekh has joined #openstack-infra07:57
*** ndipanov_gone is now known as ndipanoff07:58
*** mmaglana has quit IRC07:58
*** penguinRaider has quit IRC08:00
*** _nadya_ has joined #openstack-infra08:01
*** MaxV has joined #openstack-infra08:01
*** che-arne has joined #openstack-infra08:02
*** ihrachyshka has joined #openstack-infra08:03
*** jlibosva has joined #openstack-infra08:04
*** baoli has joined #openstack-infra08:05
*** _nadya_ has quit IRC08:06
*** lttrl has joined #openstack-infra08:11
*** baoli has quit IRC08:12
*** oomichi has joined #openstack-infra08:12
*** oomichi has quit IRC08:12
*** YorikSar_ has joined #openstack-infra08:13
*** jamespage_ has quit IRC08:13
*** dtantsur is now known as dtantsur|bbl08:15
*** oomichi_ has quit IRC08:16
*** YorikSar has quit IRC08:17
HeOSfungi, when I need to rebase this change ? Once the root admins merge this change (
*** andreykurilin has joined #openstack-infra08:19
*** e0ne has joined #openstack-infra08:20
*** yamahata__ has quit IRC08:22
*** yamahata__ has joined #openstack-infra08:23
*** yamahata has quit IRC08:26
*** annegentle has joined #openstack-infra08:28
*** e0ne has quit IRC08:30
*** annegentle has quit IRC08:33
*** jpich has quit IRC08:34
*** amuller has quit IRC08:35
*** zz_johnthetubagu is now known as johnthetubaguy08:52
*** luqas has quit IRC08:52
*** e0ne has joined #openstack-infra08:52
*** Dafna has joined #openstack-infra08:54
*** jab has joined #openstack-infra08:56
*** jab has joined #openstack-infra08:56
*** camunoz has quit IRC08:57
*** jab has quit IRC08:59
*** eglynn-officeafk is now known as eglynn-office09:00
*** rushiagr is now known as rushiagr_away09:01
*** dizquierdo has joined #openstack-infra09:12
*** rushiagr_away is now known as rushiagr09:13
*** Daisy has quit IRC09:14
*** daya_k has joined #openstack-infra09:16
*** marun has joined #openstack-infra09:16
*** jpeeler has quit IRC09:20
openstackgerritA change was merged to stackforge/dox: Add tests for tox_ini
*** markmc has joined #openstack-infra09:26
*** pelix has joined #openstack-infra09:28
*** annegentle has joined #openstack-infra09:30
*** Azher has joined #openstack-infra09:32
*** annegentle has quit IRC09:35
*** luqas has joined #openstack-infra09:36
*** marun has quit IRC09:39
*** dizquierdo has quit IRC09:39
*** ociuhandu has quit IRC09:41
*** YorikSar_ has quit IRC09:45
*** YorikSar has joined #openstack-infra09:45
*** amotoki has joined #openstack-infra09:49
*** yamahata has joined #openstack-infra09:50
*** dizquierdo has joined #openstack-infra09:51
*** rlandy has joined #openstack-infra09:56
*** _nadya_ has joined #openstack-infra09:56
*** yaguang has quit IRC09:59
*** unicell has quit IRC10:01
*** cnesa9 has joined #openstack-infra10:01
*** dizquierdo has quit IRC10:02
*** jlibosva has quit IRC10:02
*** jlibosva has joined #openstack-infra10:03
*** yamamoto has joined #openstack-infra10:07
*** dims_ has joined #openstack-infra10:11
*** pbelanyi has quit IRC10:12
*** pcm_ has joined #openstack-infra10:13
*** Guest1245 has quit IRC10:13
*** dizquierdo has joined #openstack-infra10:14
*** yamamoto has quit IRC10:14
*** dims_ has quit IRC10:16
*** dims_ has joined #openstack-infra10:16
*** alexpilotti has joined #openstack-infra10:18
*** dims_ has quit IRC10:20
*** yamamoto has joined #openstack-infra10:21
*** andreykurilin has quit IRC10:21
*** jlibosva has quit IRC10:25
*** jlibosva has joined #openstack-infra10:26
*** jlibosva has quit IRC10:27
*** jlibosva has joined #openstack-infra10:27
*** annegentle has joined #openstack-infra10:28
*** YorikSar has quit IRC10:28
*** KanagarajM2 has joined #openstack-infra10:30
*** YorikSar has joined #openstack-infra10:30
*** KanagarajM has quit IRC10:31
*** annegentle has quit IRC10:33
*** yamamoto has quit IRC10:34
*** KanagarajM2 has quit IRC10:34
*** bhuvan has quit IRC10:44
*** kmartin has quit IRC10:44
*** nosnos has quit IRC10:44
*** nosnos has joined #openstack-infra10:45
*** yfried has quit IRC10:47
*** e0ne has quit IRC10:49
*** nosnos has quit IRC10:49
*** [HeOS] has joined #openstack-infra10:52
*** [HeOS] has quit IRC10:52
*** HeOS_ has joined #openstack-infra10:54
*** yfried has joined #openstack-infra10:54
*** rushiagr is now known as rushiagr_away10:56
*** HeOS_ has quit IRC10:58
*** _nadya_ has quit IRC10:59
*** HeOS_ has joined #openstack-infra10:59
*** jcoufal has quit IRC11:00
*** dims_ has joined #openstack-infra11:00
*** Azher has quit IRC11:00
*** yfried is now known as yfried__11:04
*** yfried__ is now known as yfried11:04
*** andreaf has quit IRC11:05
*** andreaf has joined #openstack-infra11:05
*** yamamoto has joined #openstack-infra11:05
*** yamamoto_ has joined #openstack-infra11:08
*** yamamoto has quit IRC11:08
*** Clabbe has quit IRC11:10
*** Clabbe has joined #openstack-infra11:14
*** Dafna is now known as Dafna_lunch11:14
*** jyuso has quit IRC11:15
*** jyuso has joined #openstack-infra11:15
*** adalbas has joined #openstack-infra11:17
*** yfried has quit IRC11:17
*** yfried has joined #openstack-infra11:19
*** yfried has quit IRC11:22
*** adalbas has quit IRC11:22
*** ihrachyshka has quit IRC11:22
*** ihrachyshka has joined #openstack-infra11:22
*** ihrachyshka has quit IRC11:24
*** annegentle has joined #openstack-infra11:28
*** luqas has quit IRC11:32
*** annegentle has quit IRC11:34
*** yfried_ has joined #openstack-infra11:34
openstackgerritzimbatm proposed a change to stackforge/dox: Replace invalid project characters with '-'
openstackgerritzimbatm proposed a change to stackforge/dox: Replace os.getlogin() by our own helper
openstackgerritzimbatm proposed a change to stackforge/dox: Makes the project name configurable
*** dtantsur|bbl is now known as dtantsur11:35
*** ociuhandu has joined #openstack-infra11:36
*** dalgaaf_ has joined #openstack-infra11:39
*** loki184 has quit IRC11:39
*** _nadya_ has joined #openstack-infra11:40
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Switch py33 jobs for oslo.log and oslo.utils to vote
*** lyxus has quit IRC11:44
*** marun has joined #openstack-infra11:44
*** Longgeek has quit IRC11:45
*** Longgeek has joined #openstack-infra11:46
*** Longgeek has quit IRC11:48
*** tty57 has joined #openstack-infra11:48
zaroclarkb: i have someone next to me that cannot sign our ICLA agreement.  error message is "cannot store contact info". do you know anything about that?11:50
*** lyxus has joined #openstack-infra11:50
*** Longgeek has joined #openstack-infra11:51
*** Longgeek_ has joined #openstack-infra11:53
ZZellezaro, hi11:55
ZZellezaro, did he register to
*** Longgeek has quit IRC11:56
*** david-lyle has quit IRC11:57
*** mwagner_lap has quit IRC11:58
*** e0ne has joined #openstack-infra11:58
*** rushiagr_away is now known as rushiagr11:58
*** yfried_ is now known as yfried|afk11:58
ZZellezaro, it's required11:59
*** kgiusti has joined #openstack-infra11:59
zarook.  that was confusing.  he actually started from storyboard and no where along the way did it provide info about this requirement.12:00
*** rushiagr is now known as rushiagr_away12:05
*** aysyd has joined #openstack-infra12:05
*** Clabbe has quit IRC12:05
*** Clabbe has joined #openstack-infra12:08
*** marun has quit IRC12:08
*** russellb has joined #openstack-infra12:11
*** rushiagr_away is now known as rushiagr12:12
ZZellezaro, finally he must update its contact information12:12
*** kgiusti has quit IRC12:14
*** kgiusti has joined #openstack-infra12:16
*** unicell has joined #openstack-infra12:18
*** _nadya_ has quit IRC12:18
*** baoli has joined #openstack-infra12:19
*** mbacchi has joined #openstack-infra12:19
*** vigneshvar has quit IRC12:20
*** dims_ has quit IRC12:21
*** dims_ has joined #openstack-infra12:22
*** dkranz has joined #openstack-infra12:24
*** e0ne has quit IRC12:25
*** kgiusti has quit IRC12:27
*** kgiusti has joined #openstack-infra12:28
*** annegentle has joined #openstack-infra12:28
*** kgiusti has left #openstack-infra12:28
*** hdd has joined #openstack-infra12:30
*** dizquierdo has quit IRC12:31
*** kgiusti has joined #openstack-infra12:31
*** dprince has joined #openstack-infra12:32
*** annegentle has quit IRC12:33
*** _nadya_ has joined #openstack-infra12:36
*** ihrachyshka has joined #openstack-infra12:37
*** bswartz has quit IRC12:38
*** Dafna_lunch is now known as Dafna12:39
*** tty57 has quit IRC12:39
*** _nadya_ has quit IRC12:39
*** Guest1245 has joined #openstack-infra12:43
*** tty57 has joined #openstack-infra12:44
*** julim has joined #openstack-infra12:44
*** luqas has joined #openstack-infra12:46
*** ling-yun has joined #openstack-infra12:46
ling-yunhi stackers. Cinder project has a patch of to fix Jenkins failed. How could we do to expedite the patch in gate routine.12:48
*** HeOS has quit IRC12:48
ling-yunIf could not  expedite the patch to be run in gate, we may need wait more than 10 hours.12:49
*** dmsimard_away is now known as dmsimard12:49
*** HenryG_afk is now known as HenryG12:51
openstackgerritMike Heald proposed a change to openstack-dev/pbr: Retry the integration setup on connection error
*** weshay has joined #openstack-infra12:53
*** yfried|afk is now known as yfried_12:54
*** cdent has joined #openstack-infra12:55
flaper87fungi: I'm able to reproduce that pypy trace. However, the first time I executed it, I reproduced what I've seen in the pypy gate lately
flaper87i believe they're related, though.13:02
flaper87Alex_Gaynor: have you seen this? ^13:02
fungiflaper87: do you have a link to one which was hit in the gate?13:03
flaper87fungi: lemme try to find it13:03
fungiflaper87: because the supposition is that it was fixed after pypy 2.2.113:04
*** tomoe has quit IRC13:04
fungiflaper87: by
*** tomoe has joined #openstack-infra13:05
*** tomoe has joined #openstack-infra13:05
*** ling-yun-1 has joined #openstack-infra13:05
fungiflaper87: relevant details are in bug 1368418 and in the pypy bitbucket issues linked from it13:07
uvirtbotLaunchpad bug 1368418 in pypy "Fatal RPython error: KeyError/Crash in JIT!" [Undecided,New]
*** pbelanyi has joined #openstack-infra13:08
*** ling-yun has quit IRC13:08
*** ramashri has joined #openstack-infra13:08
*** tomoe has quit IRC13:09
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Timex - A time expressions library
*** kgiusti has quit IRC13:11
*** mwagner_lap has joined #openstack-infra13:12
openstackgerritSandy Walsh proposed a change to openstack-infra/config: StackDistiller - A notification extraction and transformation library.
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Sandbox - DevStack-like dev environment for StackTach.v3
*** kgiusti has joined #openstack-infra13:13
flaper87fungi: mmh, can't find it. I got that traceback in the test I just ran13:14
flaper87after that sqlite error, I got the same core dump you got13:14
flaper87and the third time I ran the test it just failed with KeyError but without showing any output13:14
flaper87I'm using the same ubuntu, tox and pypy as mentioned in the bug13:15
fungiflaper87: yeah it seems to be nondeterministic in behavior, possibly varying with different hash seed values13:15
flaper87fungi: oh mmh, good point13:15
flaper87I'll try re-using the same hash13:15
fungiif it's a new enough tox to randomize the hash seed, it will mention what hash seed it used13:15
*** bswartz has joined #openstack-infra13:16
flaper87yeah, it did13:16
flaper87I'll try to reproduce it with the same hash13:16
fungithough i have no idea whether pypy randomizes its seed by default like py3k or holds it at 0 like 2.x13:16
fungiAlex_Gaynor: ^ ?13:16
fungibut yeah, if tox is explicitly passing in a PYTHONHASHSEED envvar then that's a likely cause of variations in behavior13:17
fungiregardless of pypy's default behavior for hash seeds13:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Simport - Simple python import library (not entry-point based)
*** ramashri has quit IRC13:17
*** amuller has joined #openstack-infra13:19
*** rmcall has joined #openstack-infra13:19
*** stevemar has joined #openstack-infra13:20
*** ramashri has joined #openstack-infra13:21
*** mestery has joined #openstack-infra13:22
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for working with OpenStack notifications.
*** rmcall has quit IRC13:23
*** dims_ has quit IRC13:23
*** mfer has joined #openstack-infra13:23
*** dims_ has joined #openstack-infra13:23
*** garyh has joined #openstack-infra13:24
*** mriedem has joined #openstack-infra13:24
*** alaski has joined #openstack-infra13:25
*** dangers_away is now known as dangers13:27
*** amuller has quit IRC13:27
*** esker has joined #openstack-infra13:27
*** dims__ has joined #openstack-infra13:28
*** annegentle has joined #openstack-infra13:28
ZZellefungi, hi13:29
*** esker has quit IRC13:29
*** dims_ has quit IRC13:29
*** adalbas has joined #openstack-infra13:32
*** annegentle has quit IRC13:33
*** dansmith is now known as superdan13:36
*** koolhead17 has joined #openstack-infra13:37
*** bknudson has joined #openstack-infra13:39
*** signed8bit has joined #openstack-infra13:41
*** nelsnelson has joined #openstack-infra13:42
*** annegentle has joined #openstack-infra13:44
fungiwhat's up ZZelle?13:44
ZZellefungi, if you have some time, could you have a look to git-review changes in gerrit?13:45
fungiZZelle: it may end up waiting for the weekend or next week, but i will try to make it a priority13:47
ZZellefungi, thnaks13:47
ZZellefungi, thanks13:47
*** wenlock has joined #openstack-infra13:51
*** e0ne has joined #openstack-infra13:53
*** annegentle has quit IRC13:54
*** doug-fish has left #openstack-infra13:54
*** eharney has joined #openstack-infra13:55
*** ihrachyshka has quit IRC13:56
*** rmcall has joined #openstack-infra13:59
*** hdd has quit IRC14:00
*** ihrachyshka has joined #openstack-infra14:02
*** yfried_ is now known as yfried|afk14:02
*** doug-fish has joined #openstack-infra14:03
*** jcoufal-mobile has joined #openstack-infra14:04
*** dims has joined #openstack-infra14:05
*** r-daneel has joined #openstack-infra14:05
sandywalshhas anyone successfully integrated hosted travis with a stackforge project?14:06
*** DuncanT is now known as DuncanT-away14:07
fungisandywalsh: no idea. we (openstack project infrastructure) generally haven't offered assistance for such endeavors since we already maintain and offer a massive continuous testing platform14:07
openstackgerritIan Cordasco proposed a change to openstack-infra/elastic-recheck: Add elastic-recheck query for bug 1368782
uvirtbotLaunchpad bug 1368782 in openstack-ci "Fixed IPs Test Build Errors Out" [Undecided,New]
sandywalshfungi: gotcha, just thinking about something targeted (lean and mean)14:08
fungii believe as of a couple months ago mordred asserted that we have more than twice the test capacity of travis-ci (just for a comparative data point on scale)14:08
sandywalshgood to know14:09
*** jpeeler has joined #openstack-infra14:10
fungiand also travis-ci doesn't use openstack-based services afaik, so we provide the benefit of dogfooding and generating demonstrative load on openstack service provider member companies too14:11
anteayazaro: we don't expect storyboard as an entrance point:
fungii'm curious to know whether travis-ci is free software, or merely a hosted proprietary service like github14:11
fungioh, cool, so you can actually download and run your own travis instance?14:12
anteayazaro: can you find out a bit more about how he decided storyboard was the entry point for openstack contribution? we need to add a patch to our docs then, both the manual and storyboard14:12
sandywalshthe "runs on openstack" thing is nice, but not a selection criteria. My concern is getting patches blocked due to higher priority projects.14:12
*** eglynn-office is now known as eglynn-openstack14:13
sandywalsh(when something breaks, what gets fixed first)14:13
*** eglynn-openstack is now known as eglynn-officeafk14:13
fungisandywalsh: yep, totally understand. this time in the cycle we tend to be spread a lot more thin than usual14:13
sandywalshfungi: yeah, you guys have a lot on your shoulders ... hate to pile more on there.14:13
*** bnemec is now known as beekneemech14:13
fungiplaying in the same sandbox as integrated openstack projects is not without some drawbacks14:13
sandywalshare there documented hooks we can tap into?14:14
fungisandywalsh: at the moment, the only real event stream is the one gerrit provides, but it is well-documented yes14:14
sandywalshfungi: thanks ... I'll start there. Most likely we'll just stick with the norm. Just want to evaluate the options.14:15
*** rmcall has quit IRC14:16
fungisandywalsh: yeah, what anteaya just linked. you could probably build some sort of bridge which could trigger travis and then report in on your changes (but the infra team would of course appreciate your development time a lot more if it were focused on easing our burden rather than just sidestepping it)14:16
sandywalshfungi: ah, cool ... now I'm thinking about a gerrit -> stacktach event bridge. We could do some cool stuff with that feed.14:16
sandywalshanteaya: thanks14:16
anteayasandywalsh: yes we could14:16
sandywalshfungi: fair point14:17
anteayasandywalsh: if there is any support I can provide do ask14:17
sandywalshwill do14:17
*** dims has quit IRC14:18
*** dims has joined #openstack-infra14:18
*** flaviof is now known as flaviof_zzz14:19
*** pcrews has joined #openstack-infra14:19
*** rmcall has joined #openstack-infra14:20
*** vigneshvar has joined #openstack-infra14:21
*** dims has quit IRC14:22
*** dims__ is now known as dimsum14:22
*** dimsum is now known as dimsum_14:22
anteayasandywalsh: oops:
*** jcoufal-mobile has quit IRC14:23
*** dtroyer has joined #openstack-infra14:25
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for working with OpenStack notifications.
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for OpenStack notifications.
*** radez_g0n3 is now known as radez14:25
*** rlandy has quit IRC14:26
EmilienMsandywalsh: fungi: afik, travis is not open-source: (the Open-Source plan, is the SaaS platform,
*** homeless has joined #openstack-infra14:28
*** rwsu has quit IRC14:28
anteayasandywalsh: you have python-jobs listed but you don't use them:
sandywalshanteaya: is that not needed for the merge check?14:29
anteayasandywalsh: no14:29
anteayamerge check is defined in zuul/layout.yaml14:29
anteayayou don't need python-jobs invoked in projects.yaml for merge check14:29
sandywalshanteaya: does it hurt to have it? We will likely be turning on gate tests in the coming week(s)14:29
fungisandywalsh: EmilienM: and in the readme "Distributed under the MIT license, the same as other Travis CI projects." so sounds like it's free software14:29
anteayanow setting up your node is great for later14:29
*** mestery has quit IRC14:30
EmilienMsandywalsh: fungi indeed, my bad14:30
*** wenlock has quit IRC14:30
*** mestery has joined #openstack-infra14:30
anteayasandywalsh: while it doesn't really hurt there is a wave amounst reviewers to only have the tests in projects.yaml that you use in layout.yaml14:30
*** adalbas has quit IRC14:30
anteayasandywalsh: so you can make a change to both files when you actually call the jobs in layout.yaml14:31
anteayasandywalsh: I'm noticing your other repos don't have python-jobs in projects.yaml14:31
*** zz_jgrimm is now known as jgrimm14:31
sandywalshanteaya: I guess my real question is "am I going to be -1'ed for that" :)14:31
anteayasandywalsh: well I have -1'd two other patches for it14:31
anteayasince it was brought to my attention by someone I am mentoring in their reviews14:32
anteayaand he is correct14:32
*** jungleboyj has joined #openstack-infra14:32
anteayaso I would be inconsistent if I didn't14:32
anteayaand you might get dinged by someone else14:32
sandywalshok, stand by for a flurry14:32
*** dtroyer has quit IRC14:32
*** dtroyer has joined #openstack-infra14:33
* anteaya stands by for a flurry of improved patches14:33
jungleboyjA question for you infra experts ... Is there anyway we can move up the queue?  It is making a mess of Cinder gate jobs right now.14:33
*** shashankhegde has joined #openstack-infra14:33
*** koolhead17 has quit IRC14:33
*** david-lyle has joined #openstack-infra14:33
anteayajungleboyj: this patch is causing the problem or fixing the problem?14:33
jungleboyjanteaya: Fixing the problem.14:33
anteayafungi: ^14:34
*** Sincler has joined #openstack-infra14:34
jungleboyjanteaya: Good morning by the way.  :-)  It is the change to block sqlalchemy-migrate 0.9.214:34
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for OpenStack notifications.
anteayajungleboyj: good morning14:35
jungleboyjanteaya: Been queued for 14 hours.14:35
anteayajungleboyj: :(14:35
anteayasandywalsh: so far we have a flurry of one14:36
fungijungleboyj: i'll move it to the front of the gate shortly, on the grounds that there do seem to be multiple other cinder changes queued ahead of it which are going to cause gate resets and waste additional resources needlessly if i don't14:37
sandywalshanteaya: seems so ... thought I had more. Not so bad. I'm getting good at flying through the repos now.14:37
jungleboyjfungi: You the man!  Thank you!14:38
anteayasandywalsh: yay14:38
anteayahe is the man14:38
*** mestery has quit IRC14:39
*** Longgeek_ has quit IRC14:39
*** mestery has joined #openstack-infra14:39
*** lttrl has quit IRC14:40
anteayasandywalsh: I believe you have one source repo with two branches and one source repo with one branch and one tag14:40
*** datsun180b has joined #openstack-infra14:40
sandywalshanteaya: thought they were all just 'master' ... lemme recheck14:41
anteayasandywalsh: on those repos, and if there are any others with more than just one branch on the source repo, you can comment on those patches that this source repo status is intended14:41
anteayasandywalsh: one has master and swift branches14:41
sandywalshanteaya: k, sec14:41
anteayasandywalsh: and this one has a tag:
*** dizquierdo has joined #openstack-infra14:42
sandywalshanteaya: that's fine14:42
sandywalshanteaya: I check the others too14:42
anteayasandywalsh: can you comment on the patch that that is how you want to consume the source repo14:42
anteayaso reviewers know it isn't a mistake you will ask infra to fix after the repo is created?14:43
*** emagana has joined #openstack-infra14:43
*** SumitNaiksatam has quit IRC14:43
anteayaif memory serves, there were just two patches that had anything other than one branch14:43
openstackgerritMike Heald proposed a change to openstack-dev/pbr: Retry the integration setup on connection error
sandywalshanteaya: nuked the sandbox swift branch14:44
*** SumitNaiksatam has joined #openstack-infra14:44
*** hdd has joined #openstack-infra14:44
anteayasandywalsh: good call14:45
*** adalbas has joined #openstack-infra14:45
anteayasandywalsh: add a comment to that patch to inform subsequent reviewers please14:47
sandywalshanteaya: about the tag?14:47
*** tonytan4ever has joined #openstack-infra14:47
anteayasandywalsh: yes and that you nuked teh swift branch14:47
fungiokay, all the changes at the front of the gate which were going to merge have, and the ceilo change is on its way to failing on gate-tempest-dsvm-neutron-full so i'm promoting the cinder change now14:48
fungijungleboyj: 120907,1 is now next in line for the gate14:49
coolsvaphello all, i had a query, do we enable multiple cinder backends in the default devstack gate? or any specific gate?14:49
jungleboyjfungi: Thank you.14:50
fungicoolsvap: not sure. might be a more appropriate question for #openstack-qa14:50
*** jaypipes is now known as leakypipes\14:50
*** leakypipes\ is now known as leakypipes14:50
*** jp_at_hp has quit IRC14:50
jungleboyjfungi: Just for my edification.  We would have had cinder gate jobs failing without that change.  Right?14:50
fungijungleboyj: sounds like yes14:51
jungleboyjfungi: Other guys in the cinder channel weren't so sure.14:51
*** wenlock has joined #openstack-infra14:51
fungijungleboyj: cinder unit tests are currently failing in the check pipeline, yes?14:51
jungleboyjfungi: Everything in check is failing that way right now.14:51
jungleboyjfungi: Yes.14:51
fungijungleboyj: yeah, the ones in the gate ahead of that version pin would jave done as well14:51
jungleboyjfungi: Ok, that is what I thought.14:52
fungicoolsvap: you could also browse and see what jumps out at you14:52
*** armax has joined #openstack-infra14:52
tristanCfungi: jungleboyj: I think we also need that fixing change for stable/havana:
fungicoolsvap: but i don't see anything cinder-specific in our configuration there14:52
jungleboyjfungi: I think you are right.14:53
jungleboyjfungi: Let me approve that now.14:53
*** reed_ has joined #openstack-infra14:53
*** morganfainberg is now known as morgan14:53
*** rwsu has joined #openstack-infra14:53
fungitristanC: if you want to play the vmt card, say that you have a security advisory which needs to go out and i'll happily promote that stable branch fix and whatever security-related change together14:54
coolsvapfungi, yeah, I'm looking at it14:55
*** bhuvan has joined #openstack-infra14:55
openstackgerritAidan McGinley proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Powershell plugin.
jeblairtime to start another day of moving things from one list to another list!14:56
jungleboyjfungi: is +A now.14:56
*** zz_dimtruck is now known as dimtruck14:56
*** jp_at_hp has joined #openstack-infra14:57
*** hdd has quit IRC14:57
jeblairalso, i think the old puppetmaster is stuck, since i'm still seeing hosts that want to do the ssh key update14:57
jeblairfungi, clarkb, mordred: fyi ^ (but maybe we don't need to fix)14:57
jeblair(depending on how far we get today)14:58
openstackgerritAidan McGinley proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Powershell plugin.
*** hdd has joined #openstack-infra14:58
jungleboyjfungi: Never mind on the stable ones.  Looks like the global requirements haven't been updated.14:59
*** emagana has quit IRC15:00
*** reed_ has quit IRC15:00
*** emagana has joined #openstack-infra15:00
*** reed_ has joined #openstack-infra15:00
*** vigneshvar has quit IRC15:00
*** emagana has quit IRC15:00
*** emagana has joined #openstack-infra15:01
*** yfried|afk is now known as yfried_15:02
jeblairon wiki.o.o, i'm getting this error: err: /Stage[main]/Elasticsearch/Package[elasticsearch]: Could not evaluate: Could not get latest version: Execution of '/usr/bin/dpkg-deb --show /tmp/elasticsearch-1.3.2.deb' returned 2: dpkg-deb: error: failed to read archive `/tmp/elasticsearch-1.3.2.deb': No such file or directory15:04
jeblairi'm assuming that's because it's just something that it can't do in noop mode15:05
jeblairso i think i'll just run it for real since i don't see any other errors15:05
*** flaviof_zzz is now known as flaviof15:05
fungithat would be my guess. --noop is going to refuse to download files which it will then proceed to want to pretend to do things with... catch-2215:05
jeblairyep, and it's happy after running for real15:06
*** kgiusti has quit IRC15:08
*** dannywilson has joined #openstack-infra15:08
*** Longgeek has joined #openstack-infra15:08
*** marun has joined #openstack-infra15:08
clarkbya that is one of the known limitations with noop and execs15:09
*** dannywilson has quit IRC15:09
clarkband side effects you expect from the execs will probably fail15:09
*** dannywilson has joined #openstack-infra15:09
*** jogo is now known as flashgordon15:11
*** mestery has quit IRC15:12
*** kgiusti has joined #openstack-infra15:12
*** kgiusti has quit IRC15:12
*** mestery has joined #openstack-infra15:12
*** kgiusti has joined #openstack-infra15:13
*** ramashri has quit IRC15:14
*** vigneshvar has joined #openstack-infra15:17
*** yfried_ is now known as yfried|afk15:20
*** ramashri has joined #openstack-infra15:20
*** reed_ is now known as reed15:21
openstackgerritMatthew Treinish proposed a change to openstack-infra/infra-specs: Add spec for creating a test metric database
*** reed has quit IRC15:21
*** reed has joined #openstack-infra15:21
*** kgiusti has quit IRC15:21
clarkbnote I think that logstash is finally digging into those very large nova unittest logs from yesterday which is why the queue exploded15:22
*** e0ne has quit IRC15:22
*** dustins has joined #openstack-infra15:22
*** pabelanger has joined #openstack-infra15:23
openstackgerritMatthew Treinish proposed a change to openstack-infra/infra-specs: Add spec for creating a test metric database
clarkbjeblair: fungi: for the puppet changes are sufficient to end up with a diff in the haproxy config which will trigger an haproxy restart15:23
*** armax has quit IRC15:24
*** r-daneel_ has joined #openstack-infra15:24
jeblairclarkb: that will be a momentary outage, right?15:24
clarkbdo we want to try and sneak that in today? or I can probably give it a go on sunday? I feel bad doing disruptive stuff while everyone is trying to get FFE stuff in15:24
*** kgiusti has joined #openstack-infra15:24
clarkbjeblair: yup any git connections that are live when we restart will be closed and those clients will error15:24
*** Longgeek has quit IRC15:24
clarkbsomething else we can try is turning off puppet, updating the config manually, performning a reload, then running puppet. But I don't think that has worked well in the past15:25
*** r-daneel has quit IRC15:25
jeblairclarkb: i think we should just take the hit.  try not to do it right when the top of the gate is starting jobs :)15:25
fungiclarkb: well, we *could* replace the git.o.o a rr with 5 a rrs for the ip addresses of the backends... oh, except that they don't listen on the same ports15:26
sdaguehow much fail will it create?15:26
*** r-daneel__ has joined #openstack-infra15:26
clarkbfungi: ya that was my bad >_>15:26
clarkbfungi: lesson learned and I have a note somewhere to fix that at some point15:26
fungino, it just dawned on me that wasn't going to work15:26
clarkbfungi: yeah, it should be made to work... but I derped15:27
clarkbsdague: if we time it right not much at all. It will only affect git fetches to git.o.o15:27
fungiperhaps we can neuter puppet's ability to restart that daemon?15:27
fungijust temporarily15:27
clarkbsdague: so if we time it when we have no nodes ready to go and other jobs are running we can mostly get away with it15:28
*** ndipanoff has quit IRC15:28
clarkbfungi: yeah I have tried in the past and it is hit and miss. I think the concat module "refreshes" in some cases even when the diff is nil15:28
clarkbfungi: as if it uses a cache or something15:28
sdagueok, just... you know, 25hr deep gate queue15:28
clarkboh cool we caught up since yesterday15:29
clarkbit was ~32 hours last night15:29
fungiyeah, we've gained a few hours15:29
*** koolhead17 has joined #openstack-infra15:29
fungii think it's powering through15:29
*** rcarrillocruz has quit IRC15:29
fungibut also having nova, cinder, et cetera broken for a good chunk of yesterday and today after the sqlam release probably also kept the queue from growing on the tail end as much as it would have otherwise15:30
*** r-daneel_ has quit IRC15:30
*** emagana has quit IRC15:30
fungisince fewer changes could get into the gate15:30
*** koolhead_ has joined #openstack-infra15:30
flashgordonsdague: so it looks like david-lyle patch isn't ready yet15:31
*** koolhead17 has quit IRC15:31
flashgordonsdague: so we have the backup
*** ildikov has quit IRC15:31
*** ramashri has quit IRC15:31
*** emagana_ has joined #openstack-infra15:33
david-lyleflashgordon: I think the skip patch is best for now, still a ways from a proper solution15:33
*** amotoki_ has joined #openstack-infra15:33
flashgordondavid-lyle: kk15:33
flashgordonmtreinish: ^ tempest review15:33
flashgordonclarkb: btw thanks for the weechat help, its pretty nice15:34
sdaguedavid-lyle: what's up with the patch?15:34
clarkbflashgordon: no problem15:34
flashgordonclarkb: the spell check support is better so that should be a big win for me :)15:34
*** salv-orlando has joined #openstack-infra15:35
fungiit *looks* like cacti.o.o is testing clean against the new master, but with all the execs it does, it's hard to be absolutely sure. no errors at least15:35
*** mriedem has quit IRC15:35
*** rcarrillocruz has joined #openstack-infra15:35
fungiany objections to me pressing forward with that one?15:35
clarkbfungi: jeblair: review.o.o and review-dev will need that patch from nibalizer to use the non deprecated variable access in erbs15:36
clarkbfungi: none here15:36
*** johnthetubaguy is now known as zz_johnthetubagu15:36
sdaguedavid-lyle: so it looks like it's only touching a couple of files15:36
openstackgerritA change was merged to openstack-infra/config: Fixing deprecation warnings in gerrit module
openstackgerritA change was merged to openstack-infra/infra-specs: Add RSS feed
clarkboh there it is. y'all are way ahead of me15:37
*** mmaglana has joined #openstack-infra15:37
*** e0ne has joined #openstack-infra15:37
david-lylesdague: that patch is incomplete15:37
sdaguedavid-lyle: ok. What else needs to change?15:37
jeblairclarkb: yeah, approved it 1st thing :)15:38
david-lyleI believe there's a collectstatic step that needs to happen as well15:38
*** julim has quit IRC15:39
*** _nadya_ has joined #openstack-infra15:39
*** kmartin has joined #openstack-infra15:40
sdaguedavid-lyle: ok, do we know who the expert here might be?15:40
*** dtantsur is now known as dtantsur|afk15:40
david-lylemrunge does the packaging for fedora, pinging him15:41
*** julim has joined #openstack-infra15:42
*** dizquierdo has quit IRC15:42
*** dizquierdo has joined #openstack-infra15:43
*** lttrl has joined #openstack-infra15:44
*** _nadya_ has quit IRC15:44
flashgordonsdague david-lyle: is  now approved15:45
flashgordonso if we find a gate reset about to happen we can get that prioritized15:45
flashgordonsdague: thoughts?15:45
*** todd_dsm has joined #openstack-infra15:46
flashgordonsdague: in fact  from top of gate is about to fail15:46
*** todd_dsm has quit IRC15:46
fungiflashgordon: that's unfortunate, since it's the sqlam pin to unblock cinder's unit tests15:49
*** amotoki_ has quit IRC15:50
fungiflashgordon: i can promote 120938 ahead of it i guess, to give it a second chance15:50
flashgordonfungi: haha ouch15:50
flashgordonfungi: any way to re-promote the cinder patch back to the top as well15:51
*** rushiagr is now known as rushiagr_away15:52
fungiflashgordon: yeah, just did15:52
*** ZZelle has quit IRC15:53
*** mriedem has joined #openstack-infra15:53
*** harlowja_at_home has joined #openstack-infra15:53
*** ZZelle has joined #openstack-infra15:53
*** harlowja_home has joined #openstack-infra15:53
*** mriedem1 has joined #openstack-infra15:54
*** harlowja_home has quit IRC15:54
* krotscheck has a config naming conflict between a CLI parameter and a config setting :/15:55
* krotscheck finds this tedious and annoying.15:55
sdaguekrotscheck: you have old pyc files15:55
*** Guest1245 has quit IRC15:55
mordredkrotscheck: it's shocking that python does not have a normal library that does both CLI and config isn't it?15:56
*** amotoki_ has joined #openstack-infra15:56
flashgordonfungi: thank you15:56
krotschecksdague: Naah, it’s actually us using a similar word for a different context.15:56
flashgordonfungi: hopefully with those two patches we can help reduce the gate logjam15:56
krotscheckmordred: Oslo-config actually does that.15:56
jgriffithfungi: flashgordon :(15:56
mordredkrotscheck: oh - you know ... we're already using oslo.config in storyboard - you could just use that instead of argparse directly15:56
mordredkrotscheck: yes15:57
mordredkrotscheck: sorry, you're one step ahead of me15:57
* mordred shuts up and caffinates15:57
*** mriedem has quit IRC15:57
krotscheckmordred: Downside is that we’re already using the “project” config term for something else"15:57
*** arxcruz has quit IRC15:58
clarkbfungi: jeblair: etherpad* are seeing the same ServerAdmin undefined diff that one of you saw yesterday. did we treat that as harmless or does a patch need writing?15:58
*** dimtruck is now known as zz_dimtruck15:59
*** emagana_ has quit IRC15:59
jeblairmordred: ^?15:59
fungii don't remember encountering it15:59
mordredclarkb: the thought was a) it's harmless b) we should fix it anyway c) I don't know how to fix it15:59
*** emagana has joined #openstack-infra15:59
clarkbmordred: so I can go ahead and run it as not --noop?15:59
*** zz_dimtruck is now known as dimtruck15:59
*** ashaeron has quit IRC16:00
jeblairmordred: what's the deal with ipa.o.o?16:00
clarkboh I see <%= scope.lookupvar("etherpad_lite::apache::serveradmin") %> is the lookup16:00
clarkband that is coming back as empty string now but was 'undefined' before /me works out a patch16:01
*** emagana has quit IRC16:01
*** emagana_ has joined #openstack-infra16:01
mordredjeblair: sigh. I suck. it should diaf16:02
mordredI _keep_ meaning to delete it and keep forgetting16:02
*** datsun180b has quit IRC16:02
*** morgan is now known as morganfainberg16:02
*** datsun180b has joined #openstack-infra16:02
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add etherpadlite server admin variable
clarkbok ^ that should fix it16:03
*** hemna_ is now known as hemna16:04
krotscheckmordred: Looking at this script of yours, all it does is dump a project from launchpad into a text file, yes?16:04
jeblairerr: Could not retrieve catalog from remote server: Error 400 on SERVER: invalid byte sequence in US-ASCII at /opt/config/production/modules/openstack_project/manifests/lists.pp:1 on node lists.openstack.org16:04
*** e0ne has quit IRC16:04
clarkbdid we put a snowman in that file?16:04
jeblairhrm.  not on line 1, but i'm pretty sure there's utf8 in there16:05
jeblairor at least latin116:05
*** marun has quit IRC16:05
mordredkrotscheck: no?16:05
*** sabeen has quit IRC16:05
*** vigneshvar has quit IRC16:06
mordredkrotscheck: you're looking at launchpad_loader, right?16:06
*** mriedem has joined #openstack-infra16:06
*** mriedem1 has quit IRC16:06
clarkbyup 'español'16:06
krotscheckRight. I see main(), and do_load_models does the whole shebang, but I don’t see do_load_models getting invoked from anywhere.16:06
ttxhmm, the ratio of 'in use' vs. 'deleting' in nodepool is now suspicious16:07
clarkbjeblair: puppet bugs say that puppet will honor LANG of the env16:07
jeblairclarkb: i think that's latin116:07
jeblairclarkb: i think that character is latin116:07
ttxjeblair, clarkb: ^16:07
jeblairclarkb: it says it's a server error, so i'm assuming that's the lang env that matters in this case?16:08
clarkbvim says '<ñ> 241, Hex 00f1, Octal 361'16:08
mordredkrotscheck: oh, well that might just be me sucking16:08
clarkbjeblair: yup I think server side lang env is what matters16:08
mordredkrotscheck: i was using the json dump to eyeball what it was pulling down16:09
*** tty57 has quit IRC16:09
clarkbjeblair: my lang is en_US.UTF-8 and I see the ñ properly16:09
clarkbthough that may be vim being smart16:09
krotscheckmordred: Got it16:10
jeblairttx: let's see if stays like that; i think there was a gate reset recently16:10
*** pballand has joined #openstack-infra16:10
ttxjeblair: ack16:10
jeblairclarkb: so we need to find out the lang of the apache process?16:11
ttxfwiw I witnessed 8 changes in the gate going at once, this morning16:11
clarkbjeblair: ya  Ithink so. that error message implies it is US-ASCII16:11
jeblairttx: woo16:12
ttxI almost cried16:12
clarkbjeblair: su - www-date then echo $LANG?16:12
ttxit was a beautiful thing to see16:12
clarkbjeblair: as an alternative we can probably use the utf8 codepoint for that character then set to utf8 instead of latin116:12
*** rushiagr_away is now known as rushiagr16:12
jeblairclarkb: i'm still in data-gathering mode, but the little voice in my head is already starting to say this is insane16:13
clarkbya :/16:13
mordredjeblair: the little voice in my head agrees with you16:13
jeblairit's not easy to su to that account16:13
*** kgiusti has left #openstack-infra16:13
*** dmsimard is now known as dmsimard_away16:13
*** kgiusti has joined #openstack-infra16:13
*** mestery has quit IRC16:13
*** berendt has quit IRC16:14
clarkb is the bug with the note btw16:14
*** mestery has joined #openstack-infra16:14
sdaguehmmm... sphinx now always talks to the internet? -
jeblairthat'll be the intersphinx link module16:15
mordredsdague: you know - we should turn off the intersphinx stuff16:15
mordredI don't think we really use it, do we?16:15
*** sdague is now known as daguegoneit16:15
mordredat least, we especially probably don't use the python reference16:16
daguegoneitwould work for me16:16
*** r-daneel__ has quit IRC16:16
daguegoneitalso, do you have an opinion on the pyc thing ?16:16
daguegoneitbecause I thought the last time it came up in irc turning it off didn't seem like a big deal16:16
daguegoneitbut now folks are freaking out that I proposed it16:17
clarkbI think has always deleted those files16:17
jeblairdhellmann: are you aware of any systemic intersphinx usage? (see ^)16:17
clarkbso at least in the past everyone was cool with it16:17
uvirtbotDebian bug 723188 in puppet-common "puppetmaster: Non-ASCII characters in manifests break catalogs" [Important,Open]16:18
mordreddaguegoneit: what's the suggestion? just turning off pyc generation in the first place?16:18
daguegoneitmordred: yes16:18
daguegoneitin tox16:18
mordreddaguegoneit: 100% in support of it16:18
jeblairclarkb: hrm.  there's a suggested workaround and an indication it doesn't work :)16:18
daguegoneitmordred: then get in on the email thread :)16:18
*** markmcclain has joined #openstack-infra16:19
clarkbjeblair: could be they edited the wrong file16:19
clarkbjeblair: apache envvars vs /etc/default/puppet16:19
*** shardy is now known as shardy_afk16:19
jeblairseems like apache envars is what we'd want, yeah?16:19
*** markmcclain has quit IRC16:20
clarkbjeblair: yup16:20
*** markmcclain has joined #openstack-infra16:20
clarkband restart apache16:20
openstackgerritMark Vanderwiel proposed a change to openstack-infra/config: Fix missing sudo on gem install
*** patrickeast has joined #openstack-infra16:22
*** kgiusti has quit IRC16:22
*** koolhead_ has quit IRC16:23
*** kgiusti has joined #openstack-infra16:23
*** zz_johnthetubagu is now known as johnthetubaguy16:23
*** AJaeger has joined #openstack-infra16:24
jeblairclarkb: okay i will give that a shot now16:24
*** mestery has quit IRC16:24
AJaegerfungi, clarkb: The translation changes from yesterday worked great and gave a few projects a chance to get merged!16:25
*** mestery has joined #openstack-infra16:25
*** vigneshvar has joined #openstack-infra16:25
jeblairclarkb: export LANG=en_US.UTF-816:25
jeblairmade it happy16:25
*** Longgeek has joined #openstack-infra16:25
clarkbjeblair: awesome16:25
jeblairso... to puppet that16:25
clarkbjeblair: we should probably puppet that ;)16:25
clarkbdoes apache accept an envvars.d/ dir? that would probably be easiest16:26
clarkbthen we don't have to worry about other things16:26
*** jistr has quit IRC16:26
jeblairclarkb: nope :/16:27
*** dhellman_ has joined #openstack-infra16:27
mordredisn't there that apache mod_setenv ?16:27
jeblairmordred: that may be worth a try16:28
*** r-daneel__ has joined #openstack-infra16:28
funginodepool building ratio seems to only be getting worse not better... i found a bunch of nodes in a building state in dfw for more than an hour and have set them to delete successfully but there's probably something going on. i'll check the logs16:29
*** marcoemorais has joined #openstack-infra16:29
jeblairmordred: though probably as complicated, since the config file comes from the package16:29
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Install libxslt-devel on Fedora nodepool templates
jeblairthe vhost config file16:29
clarkbfungi: I was just looking at that too. our error rate graphs aren't very bad looking though16:29
*** Dafna has quit IRC16:29
* clarkb gets out of fungi's way16:29
*** Longgeek has quit IRC16:30
clarkbwe could puppet the entire envvars file16:30
fungiyeah, most of rax-dfw is in building state right now16:31
*** MaxV has quit IRC16:32
fungiwe are still getting usable nodes out of dfw, just not particularly quickly16:33
*** emagana_ has quit IRC16:33
*** ramashri has joined #openstack-infra16:34
jeblairclarkb, mordred: did we not add apache2 to the puppetmaster manifest?16:34
fungiand there are plenty hanging out for an hour without completing boot, then get deleted by nodepool16:34
mordredjeblair: puppetmaster-passenger pulls it in16:34
clarkbfungi: that explains why they don't show up in error stats in the graphs. they go straight from building to delete16:35
jeblairmordred: i don't see that in the manifest16:35
fungiclarkb: yeah, if you look for LaunchStatusException in /var/log/nodepool/nodepool.log there are plenty16:36
*** zehicle_at_dell has quit IRC16:36
*** tonytan4ever has quit IRC16:37
*** Ryan_Lane has joined #openstack-infra16:37
fungigrep "NodeLauncher: LaunchStatusException" /var/log/nodepool/nodepool.log|cut -d' ' -f12|sort|uniq -c16:38
*** salv-orlando has quit IRC16:38
*** derekh has quit IRC16:38
fungiwe've had 1561 launch exceptions today since the log rotated, and >99% were in rax-dfw16:39
clarkbI wonder if the thing roaert was talking about has gotten worse16:39
*** k4n0 has quit IRC16:40
clarkbmaybe we should reduce max nodes in dfw for a bit?16:40
*** dhellman_ has quit IRC16:40
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add apache envvars file for puppetmaster
clarkbjeblair: I think a reasonable soluton is to splat a new complete envvars file over the top of what apache2 package installs with puppet16:40
jeblairclarkb: done ^ :)16:40
fungii suspect that wreaks havoc with the nodepool demand calculations too, since it counts building as the same as ready, but a majority of them are dead weight which hang around for an hour and then get deleted without ever actually reaching a state where they can be put to use16:41
clarkbjeblair: reviewing16:41
jeblairfungi: well, they still count against quota16:41
jeblairso it's the right thing to do16:41
jeblairwould reducing our usage help?16:41
fungii'm just wondering whether we'd get better throughput by removing dfw until the problem there gets addressed16:42
*** emagana has joined #openstack-infra16:42
*** tonytan4ever has joined #openstack-infra16:42
jeblairwhat's the tldr on the thing roaert described?16:42
clarkbjeblair: they cannot recycle ipv4 addresses quickly enough for us16:42
clarkbjeblair: which is why trying to use ipv6 only was a potential workaround16:42
fungisomething about we might see fewer errors if we only requested ipv6 addresses, except that openstack sort of assumes you will have an ipv4 address forced on you anyway16:43
*** pkoniszewski has quit IRC16:43
clarkbwithout knowing the entire situation reducing our churn in that region may help with ipv4 address recycling16:43
fungiand so there may be no good way to do that even if we did support v6-only providers16:43
*** emagana has quit IRC16:43
*** dannywilson has quit IRC16:44
fungiwithout some further modification to neutron and clients16:44
jeblairas long as we're over capacity, reducing usage in dfw won't significantly affect usage in other regions that are working; but if reducing usage of dfw makes it more successful, that's probably a net win16:44
*** dannywilson has joined #openstack-infra16:44
clarkbthis is another thing rebuild vs delete may end up helping with16:45
clarkbshould look into that as well as the ipv6 support16:45
fungijust wondering if having less-frequently-used node types tied up in building dead weight might be detrimentally impacting our ability to run some sorts of jobs16:45
*** bhuvan has quit IRC16:45
jeblairfungi: i think things will mostly average out as long as demand is high enough16:46
jeblair(unless dfw suddenly has a lot of capacity and the round-robin allocator puts everything in there)16:46
*** dannywilson has quit IRC16:47
fungiwe have more bare-precise nodes in rax-dfw than all other providers combined, for example16:47
*** dannywilson has joined #openstack-infra16:48
fungibasically all in building and delete16:48
jeblairso we're currently at dfw=184/ord=55/iad=8516:48
jeblairyay round robin allocations16:48
fungithough it did just manage to provide two bare-precise nodes into a used state now, so i guess it's not totally dead for those16:49
jeblairi'm trying to work out out actual quotas16:49
fungipy3k-precise seems to be fairly evenly distributed too, so i guess while dfw isn't really helping much, it's not especially hurting us more than not having it would16:49
*** Hal has joined #openstack-infra16:50
clarkbthat is good news16:50
*** harlowja_at_home has quit IRC16:50
*** Hal is now known as Guest4552316:50
clarkbso I think this boils down to whether or not dfw would be in a happier place with us requesting fewer things at once16:50
clarkb(for tweaking max server value)16:50
openstackgerritA change was merged to openstack-infra/config: Install puppetmaster-passenger on puppetmasters
jeblaircurrent: dfw=184/ord=55/iad=8516:51
jeblairquota: dfw=192/ord=62/iad=9316:51
jeblairso we don't actually have room to increase the other regions, we'd just be reducing16:52
fungiright, so the ratios probably mostly work out16:52
jeblairfungi: what do you mean?16:52
jeblairfungi: oh, about node type allocation.  ok16:52
fungiseeing more nodes of a given type in dfw than ord and iad isn't entirely unexpected, as the quota is also much higher there16:52
fungithough hpcloud should still trump that by a wide margin16:53
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Drop 90 servers from rax-dfw
jeblairfungi: so most of dfw is build/delete, right?16:54
dhellmannjeblair: oslosphinx provides an intersphinx mapping for some openstack projects, but afaict it doesn't turn on the extension by itself16:54
*** jlk has joined #openstack-infra16:55
*** vigneshvar has quit IRC16:55
jeblairfungi: can you ?16:56
fungijeblair: 63 used, 63 building, 43 delete, 5 ready16:56
*** dannywilson has quit IRC16:56
fungijeblair: yeah, was already looking at it. is the envvars file one you lifted from the current server?16:56
*** dannywilson has joined #openstack-infra16:56
clarkbyup I checked the current file contents on the puppetmaster16:56
fungiagreed. sent it in now16:57
jeblairfungi: okay, so we're probably not going to loose anything by setting the max to 9416:57
fungijeblair: agreed16:57
jeblairin current state16:57
fungiyep, i believe that is probably the case16:57
*** jp_at_hp has quit IRC16:58
fungiif this means it can turn building servers over faster, then it may even help16:58
jeblairwhat's the deal with  111782,35 ?16:58
jeblairin the gate16:58
jeblairnova change with what looks like a deterministic py26 unit test failure that's been in queue for 1 hour16:59
*** daya_k has quit IRC16:59
*** marcoemorais has quit IRC16:59
jeblairis the time just wrong due to a bug in promote or something?16:59
*** Hefeweizen has joined #openstack-infra17:00
*** marcoemorais has joined #openstack-infra17:00
jeblair(since it's not at the top)17:00
clarkbjeblair: it was approved ~3 hours ago17:00
clarkboh wait that aws yesterday nevermind17:00
*** marcoemorais has quit IRC17:00
fungijeblair: yeah, there's been at least one promote since it entered the gate, and it was pulled in by a parent change which seems to cause zuul to not preserve the enqueued time17:00
jeblairok, makes sense17:01
fungiwe normally saw this back when we allowed approved parents to enqueue approved child changes17:01
*** dannywilson has quit IRC17:01
fungiand it's returned since we reenabled that behavior17:01
*** dannywilson has joined #openstack-infra17:01
*** marcoemorais has joined #openstack-infra17:01
fungibasically time in queue for the child changes gets reset17:02
*** emagana has joined #openstack-infra17:02
fungii suspect there may also be a potential for a queue-hopping exploit (rebase your change on one near the top of the gate, get it approved, then wait for someone to promote an unrelated change while they're both still in the gate) but i doubt anyone would actually take advantage of that behavior17:03
clarkbrebasing like that means you need approval so it requires some oversight17:04
fungiyep, absolutely17:05
*** ihrachyshka has quit IRC17:07
clarkbfungi: when you are done debugging nodepool review on would be great17:07
*** sabeen has joined #openstack-infra17:08
clarkbok /me does morning things back in a bit17:09
*** marun has joined #openstack-infra17:09
*** e0ne has joined #openstack-infra17:10
*** bhuvan has joined #openstack-infra17:11
*** Adri2000 has quit IRC17:13
krotscheckmordred: Did you ever decide what you were going to do with users that don’t have a valid openid provider?17:14
*** tty57 has joined #openstack-infra17:14
fungijungleboyj: jgriffith: looks like the cinder pin made it into master now17:14
mordredkrotscheck: can you say those same words but slightly differently?17:15
krotscheckmordred: DiscoveryFailure: No usable OpenID services found for
mordredkrotscheck: yet there is a bug assigned to that person?17:15
mordredthat's frustrating17:15
mordredkrotscheck: so - for assignee, we can just leave those unassigned ...17:16
mordredkrotscheck: for owner, maybe we allow owner/creator to be null?17:16
fungikrotscheck: mordred: might consider special-casing that one. ~janitor is a pseudo-account in lp17:16
*** praneshp has joined #openstack-infra17:17
mordredfungi: excellent point17:17
fungiit's not an actual user at all17:17
clarkbI should tell my custodian uncles that janitors are not actual people17:17
*** melwitt has joined #openstack-infra17:17
clarkblp said so17:17
krotscheckOf course they’re not. They’re sanitary technicians17:18
mordredclarkb: we should make them a greeting card that says that and have all of openstack sign it17:18
*** aviau has quit IRC17:18
krotscheckI think we can allow nulls in all those fields.17:18
*** grantbow has quit IRC17:18
*** torandu has quit IRC17:18
krotscheckAre there any other pseudo accounts I need to know about?17:19
mordredkrotscheck: what if ... (and bear with me for this being a STUPID idea)...17:19
*** Guest55717 has quit IRC17:19
jeblairwe assign them all to mordred?17:19
*** grantbow has joined #openstack-infra17:19
*** grantbow has joined #openstack-infra17:19
*** torandu has joined #openstack-infra17:19
fungikrotscheck: you might ask lifeless or StevenK when they're around... i just happened to be aware of that one in particular17:19
mordredkrotscheck: what if we had the import script alter the tables to allow nulls, then alter them back to put the constraint on when it's done17:19
*** aviau has joined #openstack-infra17:19
* jeblair throws an unmaintainable database at mordred17:20
*** rhsu has joined #openstack-infra17:20
mordredjeblair: yay! unmaintainable databases keep me in a job!17:20
mordredkrotscheck: I retract my statement17:20
fungiflashgordon: daguegoneit: the tempest skip for that horizon compression concurrency issue is in now too17:20
fungihopefully our gate stability will climb a bit17:20
* krotscheck isn’t quite certain what just happened.17:20
* krotscheck goes back to reading about generators.17:20
flashgordonfungi: yeah horizon compression was a big one so hoping that helps17:21
jeblair  Detail: Could not find value for 'vhost_extra' at /opt/config/production/modules/meetbot/templates/vhost.erb:2517:21
*** dtantsur|afk has quit IRC17:22
jungleboyjfungi: Thanks!17:22
jeblairclarkb: do i need to sprinkle '@' symbols everywhere in modules/meetbot/templates/vhost.erb ?17:22
jungleboyjYeah, I had been watching that.17:22
flashgordonfungi: there is another gate bug fix in the gate queue btw:
flashgordonfungi: although that one is much more infrequent17:23
*** tty57 has quit IRC17:23
clarkbjeblair: yup that should fix it17:23
*** luqas has quit IRC17:23
fungiflashgordon: the one you linked only impacts stable/icehouse anyway, so probably relatively infrequent given the percentage of changes it sees17:24
flashgordonfungi: it hits grenade actually17:24
fungiflashgordon: oh!17:24
fungiflashgordon: good point, then that would affect master changes as well17:24
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add more @ signs to meetbot vhost
flashgordonbut yes, 2 hits in 24 hours17:24
*** sdake has joined #openstack-infra17:24
fungiflashgordon: in the gate or overall?17:24
flashgordonfungi: overall17:24
*** sdake has quit IRC17:24
*** sdake has joined #openstack-infra17:24
*** Adri2000 has joined #openstack-infra17:25
mordredjeblair: @ symbols make it more like perl, which is what ruby wants to be anyway17:25
fungiahh, if it gets worse i guess we could consider promoting it, otherwise it'll probably be applied in the next 24 hours anyway17:25
jeblairmordred: that i can understand17:25
clarkbjeblair: my only quetion about that is if we need @'s in the #{} string interpolation too17:25
*** Adri2000 is now known as Guest2433317:25
flashgordonfungi: yeah no gate hits in over 24 hours for that one.17:25
clarkbjeblair: that might be a question for nibalizer17:25
clarkbnibalizer: ^17:25
*** lttrl has quit IRC17:25
jeblairclarkb: i looked at modules/drupal/templates/drupal.vhost.erb17:26
*** tty57 has joined #openstack-infra17:26
jeblairclarkb: and now that i look closer... i see we did not do it in one place and did it in another17:26
clarkbsdake: good morning17:27
ekarlso-phschwartz: sorry to nag, goes with the gh stuff ?17:27
*** craigbr has joined #openstack-infra17:28
mordredphschwartz: we want your codes!!! and also all of your chickens!!! and, perhaps, do you have a milk cow we could borrow for a while?17:30
jeblairclarkb: i'm testing that change in a local env17:30
nibalizer@thing in templates?17:30
nibalizeryea if you #{@foo}17:31
nibalizerdo the @ thing17:31
openstackgerritA change was merged to openstack-infra/config: Add etherpadlite server admin variable
jeblair#{@name} vs #{name} doesn't seem to make a difference according to noop.  but i'll do the @ thing.17:31
nibalizerit makes it a 'global' variable in ruby i guess and the way the templating work is i guess the variables get injected as globals17:31
crinkle*instance variable17:32
nibalizerclarkb: ya something ruby mumble mumble17:32
nibalizerer crinkle ^17:32
crinkleya that17:33
fungiout of curiosity, was the "puppet agent --server" command (after removing from old list and before adding to new list) a typo? should that have had --test in it? doesn't seem to return, just ends up running as a daemon17:33
*** harlowja_away is now known as harlowja17:34
*** jlibosva has quit IRC17:34
jeblairnibalizer: wait a sec17:35
fungihrm, looks like maybe it finally exited on jenkins-dev where i ran it last night, so maybe just takes time17:35
jeblairnibalizer: look at line 9 of
nibalizerfungi: yea you likely wanted --test and --noop17:35
nibalizerjeblair: that looks right17:36
funginibalizer: the --test --noop was for testing it first i thought, but then after we takt it out of the old list we had noted to run without either of those17:36
*** penguinRaider has joined #openstack-infra17:36
nibalizerfungi: oh, hrm, yea you wanted --test on both sides i think17:36
nibalizersince we dont run puppet daemon anymore17:37
fungiokay, i wonder why that eventually exited on jenkins-dev then17:37
jeblairnibalizer: okay, we're getting conflicting information about #{name}17:37
fungii assume everyone else was typing it from memory rather than cutting and pasting from the etherpad17:37
jesusaurusyeah --test doesnt actually mean 'test', it means 'no-daemon'17:37
jeblairnibalizer: name in that case is not passed into the template, it's a local var from the for loop17:37
jeblairnibalizer: in that case, should it have @ or not?17:37
nibalizerjeblair: in that case name should not have a @17:38
nibalizersince its not being passed in from puppet, it was declared inside the template17:38
jeblairnibalizer: okay.  but on line 11, we should put an @ in because it's passed in, right?17:38
nibalizerjeblair: yes line 11 should have @17:39
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add more @ signs to meetbot vhost
*** ZZelle_ has joined #openstack-infra17:39
sandywalshcraigbr: thanks for the eyeballs!17:40
*** emagana has quit IRC17:41
*** emagana has joined #openstack-infra17:41
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add more @ signs to logs-dev and status vhosts
jeblairclarkb: can you apply your serveradmin solution to storyboard.o.o?17:42
jeblairmordred: i'm stealing your hosts since you're busy :)17:43
mordredjeblair: please do17:43
jeblairclarkb: also, the etherpad changes have merged; do you want to pick those back up?17:44
*** torandu has quit IRC17:44
jeblairokay, i'll do the serveradmin thing for storyboard17:45
*** rushiagr is now known as rushiagr_away17:45
craigbrsandywalsh: np. Only one left to do. You left the repo name out of the commit for
*** emagana has quit IRC17:46
*** penguinRaider has quit IRC17:46
sandywalshcraigbr: whoops, fixing17:46
*** torandu has joined #openstack-infra17:46
nibalizerjeblair: i put a review up for the serveradmin thing on storyboard17:46
jeblairnibalizer: link?17:46
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Winchester - A notification event processing library based on persistant streams.
jeblairnibalizer: i will paste it into the etherpad17:47
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Shoebox: Binary data archiving library - supports uploading to object storage.
*** SumitNaiksatam has quit IRC17:48
jeblairnibalizer: can you explain why it's bad to compare to undefined?17:48
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Shoebox: Binary data archiving library - with object storage upload.
*** dizquierdo has quit IRC17:48
fungireview-dev is way, way, way behind in puppetingness as puppet agent has been disabled on it for no idea how long17:48
nibalizerjeblair: :undef is a fine thing to compare to, but :undefined is a literally undefined variable, i.e. you could just as well compare to :lazors17:49
fungizaro: this will presumably need to be reenabled for us to migrate it to the new puppet master. fair warning any local changes will probably be reverted17:49
jeblairnibalizer: but testing truthiness of a string is better?17:49
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Shoebox: Binary data archiving library.
nibalizerchanging it the way i did gives you the same resuluts on 2.7 and on 3 that you want, which is nothing to be printed out for serveradmin17:49
jeblairkrotscheck: what's your preference for spelling server admin?  "serveradmin" or "server_admin"?17:49
*** marcoemorais has quit IRC17:49
sandywalshcraigbr: sorry, had to do a few patches to get the subject line length to fit.17:50
*** marcoemorais has joined #openstack-infra17:50
*** marcoemorais has quit IRC17:50
*** marcoemorais has joined #openstack-infra17:50
*** marcoemorais has quit IRC17:51
*** marcoemorais has joined #openstack-infra17:51
openstackgerritJames E. Blair proposed a change to openstack-infra/puppet-storyboard: Spell 'server_admin' consistently
jeblairkrotscheck: nibalizer: let's do ^ too17:51
jeblairsince that would also cause the error17:51
krotscheckYay consistency!17:52
jeblairkrotscheck: lgty too?17:52
jeblairkrotscheck: oops, meant to ask about
clarkbok +2 on the storyboard server admin change17:53
krotscheckjeblair: Hrmmm… see, that one bit me a few weeks back, but I don’t recall enough about the circumstances.17:54
jeblairkrotscheck: i can test out the stack real quick in noop and see if there's a diff17:54
krotscheckI think I was trying to make it all work on precise and trusty at the same time.17:54
krotscheckWithout using the puppet ppa17:55
jeblairoh, we use the puppet ppa17:55
krotscheckSo that should be good.17:55
openstackgerritA change was merged to openstack-infra/config: Add apache envvars file for puppetmaster
jeblairfungi: can you aprv and ?17:56
jeblairnibalizer: if i want local modules for a test env, should i reverse the modulepath list order?17:57
jeblairnibalizer: eg, should it be /opt/config/corvus/modules:modules:$basemodulepath ?17:57
*** shashankhegde has quit IRC17:57
rcarrillocruzhi there17:58
rcarrillocruzcould i get some more reviews on ?17:58
rcarrillocruzpls :-)17:58
nibalizerjeblair: im not sure, will require some fiddling i think17:58
nibalizerif you make a /etc/puppet/environments/corvus/environment.conf you can do anything you want17:59
*** amotoki has quit IRC17:59
jeblairnibalizer: yeah, that's what i'm talking about :)17:59
nibalizerso you might need to reverse it like you said, or just remove basemoudle path and manage all of corvus/modules by hand17:59
jeblairnibalizer: i'm just starting with the modulepath we're using for production, and it seems backwards for supporting local overrides17:59
jeblairnibalizer: however, it's probably good for production17:59
nibalizerim not sure if puppetmodules use the same kind of first-find-bail that other things do with PATH variables18:00
jeblairnibalizer: because it would mean that as we create new separate modules, they will override what we have in the config tree18:00
jeblairnibalizer: oh jeez.18:00
nibalizeri mean they might, but we shouldn't assume the puppet devs have used sanity18:00
fungii certainly will never let myself be accused of assuming that18:00
jeblairnibalizer: you are right.  it's probably a binary search starting from the middle because moustache.18:00
*** SumitNaiksatam has joined #openstack-infra18:01
*** shashankhegde has joined #openstack-infra18:02
clarkbjeblair: :{18:02
*** johnthetubaguy is now known as zz_johnthetubagu18:02
openstackgerritA change was merged to openstack-infra/config: Drop 90 servers from rax-dfw
openstackgerritA change was merged to openstack-infra/puppet-storyboard: Template was comparing to :undefined
jeblair+    ServerAdmin undef18:03
clarkbjeblair: fwiw that nova change you were looking at with the python26 failure just passed python26 after a gate reset... so not very deterministic it seems18:04
jeblairclarkb: wow, that's bad.18:04
*** emagana has joined #openstack-infra18:05
*** e0ne has quit IRC18:05
*** julim has quit IRC18:07
*** armax has joined #openstack-infra18:07
*** cdent has quit IRC18:07
jeblairnibalizer, krotscheck: i think we want != :undef18:08
krotscheckFall down go boom?18:08
jeblair+    ServerAdmin undef18:08
jeblairthe current state with those patches does that ^18:08
jeblairi tested with != :undef and there's no diff from current18:09
jeblairwith != :undefined it also adds ServerAdmin undef18:09
nibalizerjeblair: okay != :undef works for me18:09
nibalizershall i update my review?18:09
jeblairmaybe we could do ='' instead of =undef in the assignment18:10
jeblairnibalizer: no, your change has merged18:10
*** e0ne has joined #openstack-infra18:10
openstackgerritIan Cordasco proposed a change to openstack-infra/elastic-recheck: Add elastic-recheck query for bug 1368782
uvirtbotLaunchpad bug 1368782 in openstack-ci "Fixed IPs Test Build Errors Out" [Undecided,New]
jeblairnope, undef it is18:11
*** sdake_ has joined #openstack-infra18:11
fungireview-dev completed the transition to the new puppetmaster cleanly and gerrit is still running on it... should i go ahead and do review.o.o or save that for a quieter period?18:12
*** sdake has quit IRC18:12
openstackgerritSpencer Krum proposed a change to openstack-infra/puppet-storyboard: Compare values to :undef to test for existence
nibalizerjeblair: like that?18:13
jeblairfungi: let's do it now18:13
*** marun has quit IRC18:13
fungisounds good. starting it now18:13
jeblairnibalizer: please base that on my change18:14
clarkbI am going to do etherpad.o.o this will bounce apache (so you may need to refresh the puppet3 switch pad18:14
nibalizerjeblair: okay18:15
*** Longgeek has joined #openstack-infra18:15
*** pelix has quit IRC18:15
fungiokay, review.o.o is done18:16
clarkbetherpad is done18:16
fungigerrit on review.o.o is still running and responding normally18:17
fungietherpad seems fine as well18:17
*** flaviof is now known as flaviof_zzz18:17
openstackgerritSpencer Krum proposed a change to openstack-infra/puppet-storyboard: Compare values to :undef to test for existence
*** dimtruck is now known as zz_dimtruck18:18
clarkbnibalizer: see comment on that change18:18
*** tonytan4ever has quit IRC18:18
nibalizergah! every time!18:19
*** Longgeek has quit IRC18:19
* nibalizer ponders writing a pre-commit hook to test for that kind of thing18:19
*** shashankhegde has quit IRC18:19
openstackgerritSpencer Krum proposed a change to openstack-infra/puppet-storyboard: Compare values to :undef to test for existence
*** ramashri has quit IRC18:20
*** _nadya_ has joined #openstack-infra18:21
*** jungleboyj has quit IRC18:23
jeblairokay, those are all approved; the current storyboard state isn't any better on the old puppetmaster at this point, so i've gone ahead and moved it18:24
*** tty57 has quit IRC18:24
clarkbwhat about eavesdrop?18:24
clarkbnevermind I see the change18:24
* krotscheck blinks18:24
clarkbI missed hte consensus on @'s18:24
krotscheckWait a sec, storyboard is down? Here I was eyeballs deep in the migration script.18:25
clarkbbut sb has filled me in18:25
clarkbkrotscheck: no, I don't think so18:25
fungithat list got plowed through surprisingly quickly18:25
krotscheckOh, good18:25
*** yamahata has quit IRC18:25
clarkbkrotscheck: just that hte puppet was equally broken with both masters18:25
jeblairit's actually slightly less broken on the new master right now, due to change sequencing18:25
fungionce we get these last few stragglers done, i suppose next wee we start going through and upgrading puppet on all the servers?18:25
jeblairthough it may get worse then better again :)18:25
clarkbfungi: yup18:26
clarkbthough we might be able to start on that today?18:26
jeblairi was planning on it18:26
fungiwe could probably make puppet upgrade puppet, but that strikes me as marginally dangerous18:26
*** e0ne has quit IRC18:26
clarkbjeblair: fungi: looking at the gate now do you think it is a good time to puppet git.o.o?18:26
jeblairclarkb: i think so18:26
jeblair(was just checking that myself)18:26
clarkbok I will do that now18:26
fungiseems safe18:26
clarkbdoes someone else want to edit the hostfiles?18:26
fungias long as it doesn't end up taking too long18:27
fungisure, i'll edit18:27
fungiit's out of the old file now18:27
clarkbpuppet is running now18:27
fungiassuming it's clear, i'll toss it into the new file18:28
clarkball done18:28
* clarkb tests git.o.o18:28
fungiadded to the new list18:28
fungiremoved from the etherpad18:28
*** AJaeger has quit IRC18:28
*** AJaeger has joined #openstack-infra18:29
clarkball three protocol worked cloning swift18:29
clarkbso I think we are good18:29
fungiso as soon as we have our liberal sprinkling of @ signs for eavesdrop...18:29
jeblaireta in 0 mins18:29
openstackgerritA change was merged to openstack-infra/config: Add more @ signs to meetbot vhost
fungimoar @ signs18:30
nibalizer@fungi yes18:31
jeblairworking on eavesdrop now18:31
* fungi cringes18:31
*** ramashri has joined #openstack-infra18:31
fungii guess next i can add a copy of the new hostfile into the etherpad and we can divvy up pin edits/package upgrades?18:31
jeblairfungi: ++18:32
jeblaireavesdrop is done18:32
clarkbfungi: sounds like a perfect idea18:32
*** amotoki_ has quit IRC18:33
fungilist is in there. grab what you like18:33
jeblairsomeone should write a sed command for the pin and paste it in there18:34
jeblairso we're all doing the same thing there18:34
clarkbdo we haev an example for what it should look like?18:34
clarkb++ to sed that is even better18:34
fungiadding now18:34
*** IAX has quit IRC18:36
*** iax7 has quit IRC18:37
clarkbfungi: jeblair is that sed sufficient? we pin other packages related to puppet too18:37
clarkblike facter18:37
fungilooks like new puppet is going to pull in "hiera libjson-ruby puppet-common ruby-json"18:37
fungiyeah, so what newer versions of facter et all do we want to be using?18:38
funginibalizer: ^ ?18:38
nibalizeroh hrm18:38
nibalizeris there a puppet-hiera package?18:38
clarkbI think facter should be 2.* right?18:38
nibalizerinstalled now?18:38
nibalizerneed to nix that18:38
fungipuppetdb-terminus and facter are pinned to 1.*18:38
nibalizeryou want puppetdb-terminus and facter to go to 2.*18:39
nibalizeryou know18:39
nibalizerwhat you could do18:39
nibalizerwell nevermind18:39
nibalizeri was gonna say comment out the whole file and let apt decide what to do18:39
clarkband do we need to pin puppet to 3.6.*?18:39
nibalizerclarkb: i'd like to18:39
*** shashankhegde has joined #openstack-infra18:40
dvorakhrm, is there any way to have a default for a macro parameter in jenkins job builder?18:40
nibalizertechnically we can go 3.7 but our master isn't 3.7 so thats not really advisable18:40
clarkbnibalizer: ya so we should pin to 3.6*18:40
fungiso i've added that and the terminus/facter pin to 2.*18:41
nibalizeryou might, instaed of complex sed, just heredoc in a new file18:41
jeblairwhat what?18:41
jeblairi thought we were pinning to 3.x18:42
jeblairwhy are we changing our minds?18:42
*** ling-yun-1 has quit IRC18:42
*** ling-yun has joined #openstack-infra18:42
fungigroupthink? but yes i explicitly remember yesterday the plan was 3.*18:42
jeblairthese are the pins we agreed on earlier:
clarkbso the thing to be aware of is you don't want your master to be older than your agents18:43
*** Sincler has quit IRC18:43
nibalizerI remember that too(pinning to 3.x)18:43
clarkbif we are ok with the puppetmaster having that pin too so that it moves along with the clients then +18:43
clarkber +118:43
jeblairclarkb: i'm pretty sure it will18:43
jeblairit's in base18:44
nibalizerdo we manage the puppet package in puppet right now?18:44
jeblairoh, we can set it explicitly, but it matches in the proposed change.18:44
nibalizerwhat i can see happening is the puppetmaster never actually getting upgraded then a new node comes online with 3.718:44
jeblairnibalizer: that's an error18:44
clarkbwe do autoupgrades with package pins18:44
jeblairnibalizer: the puppemaster should run nightly updates just like all the other machines18:45
clarkbso it should auto update both master and agent18:45
clarkbjeblair: yup18:45
jeblairand i _think_ we put puppetlabs in the list of repos to do that for18:45
nibalizerdo we have to restart the puppetmaster service to pick up a new version taht was installed by apt?18:45
clarkbhrm actually it doesn't look like we d18:45
jeblairclarkb: i agree18:45
flashgordonwe are starting to see some errors like this
clarkbI am good with adding ti as long as we have the pin to 3 and don't randomly get 4 :)18:46
fungipassenger needs apache restarted if it's upgraded i think, but can't recall if it does that on its own18:46
*** Sincler has joined #openstack-infra18:46
flashgordongetting a 502 from
*** dannywilson has quit IRC18:46
jeblairflashgordon: stop using intersphinx18:46
flashgordonjeblair: what does intersphinx do anyway?18:46
*** zz_dimtruck is now known as dimtruck18:47
notmynameFYI I jsut added a -2 to something in the gate queue. it's been there for >24 hours, but I wanted to say something since the gate is so long and I'd hate for it to cause a gate flush because it has a -2 now. The patch is 12042418:47
*** dannywilson has joined #openstack-infra18:47
clarkbnotmyname: it won't flush. it will continue to be processed and if your -2 is still there when it should merge it won't merge18:47
clarkbnotmyname: so should be fine and thank you18:47
*** vigneshvar has joined #openstack-infra18:47
notmynameoh ok18:47
*** dannywilson has quit IRC18:47
*** dannywilson has joined #openstack-infra18:48
jeblairflashgordon: links to other sphinx documents out on the internet18:48
clarkbjeblair: fungi nibalizer so should we write a change to allow auto updates of puppet?18:48
jeblairclarkb: probably so18:49
* clarkb learns him an unattended upgrade config18:49
nibalizerclarkb: you could just package { 'puppet': ensure => latest, }18:50
clarkbnibalizer: no18:50
clarkbwe don't do that for any apt packages because we want unattended upgrades to deal with it18:50
clarkbbetter to have consistency here18:50
nibalizerokay im with you on that18:50
nibalizeryou can always win me over with a consistency argument18:50
jeblairfungi: the sed still needs something for puppetdb-terminus18:51
jeblairand facter18:51
flashgordonjeblair: thanks,
jeblairdhellmann: ^18:52
*** stevemar has quit IRC18:52
fungijeblair: should be okay now18:53
*** ling-yun has quit IRC18:53
jeblairfungi: lgtm18:53
*** stevemar has joined #openstack-infra18:53
*** ling-yun has joined #openstack-infra18:53
fungialso strangely, i see some hosts pinning puppetdb-terminus to 1.* in its own section, and others lumping it in with the puppet 2.7.* pin18:53
*** MarkAtwood has joined #openstack-infra18:54
*** MarkAtwood has quit IRC18:54
jeblairoh 3.7 is out now18:54
fungicompare paste.o.o and pypi.dfw.o.o18:54
jeblairso we'll be putting 3.7 on all the clients18:54
*** nelsnelson has quit IRC18:54
fungiperhaps we should upgrade the master again first?18:55
*** amcrn has joined #openstack-infra18:55
markvanfungi: clarkb: sorry for same song & dance on ruby patch, but I missed a sudo in there:  thx!18:55
jeblairfungi: probably so18:55
jeblairnibalizer, clarkb: upgrade master to 3.7?18:55
*** dmsimard_away is now known as dmsimard18:57
jeblairfungi: maybe we should switch to the heredoc for consistency18:57
clarkbjeblair: we will need to have 3.7 master for 3.7 clients to reliably work18:57
jeblairfungi: the 3 tier one is what's in install_puppet18:57
fungijeblair: probably18:57
clarkbjeblair: fungi: what do yo uthink about pinning to 3.6 today and working on 3.7 next week? (I don'treally care either way but that may reduce the number of variables)18:57
fungii'll do that18:57
fungi(the heredoc thing)18:58
fungido we think 3.7 is likely substantially different from 3.6?18:58
jeblairclarkb: i'm getting pretty tired of working on puppet.  if you want to do that, i'll probably just switch to working on replacing puppet with ansible.18:58
*** ling-yun has quit IRC18:59
jeblairat that point, using puppet will have become _FAR_ more work than almost any other way i can think to manage systems, which is kind of the opposite of what it's supposed to be.18:59
clarkbfungi: no puppet is semver starting with version 3.018:59
jeblairso let's take them at their word and upgrade to 3.719:00
fungii'm in favor19:00
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Add 'task_origin' field to tasks table.
jeblairupgrading master now19:00
*** emagana has quit IRC19:00
jeblairupgrading puppetmaster-passenger did not restart apache19:01
jeblairi don't know if that's required or not.19:01
jeblairis there a way to ask the puppetmaster what version it's currently running?19:01
jeblairnibalizer: ^ any idea?19:02
openstackgerritClark Boylan proposed a change to openstack-infra/config: Allow puppet to be automatically upgraded
*** tonytan4ever has joined #openstack-infra19:02
clarkbI think that will do it for debuntu nodes19:02
clarkblooking at how we do autoupgrades on centos now19:02
*** ling-yun has joined #openstack-infra19:03
*** ociuhandu has quit IRC19:03
jeblairnothing in the apache error log that makes me thing it did anything19:03
fungigah paste from etherpad is teh sux. i'll put it in a paste.o.o19:03
*** ChuckC_ has joined #openstack-infra19:03
*** dkranz has quit IRC19:04
jeblairrestarted apache manually19:05
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.2 bug 1368391
uvirtbotLaunchpad bug 1368391 in glance "sqlalchemy-migrate 0.9.2 is breaking nova unit tests" [Undecided,In progress]
jeblairi consider that a bug in the puppet-passenger package19:05
*** emagana has joined #openstack-infra19:06
nibalizerjeblair: puppet --versio19:07
jeblairnibalizer: run that where?19:07
nibalizeron the master19:07
openstackgerritA change was merged to openstack-infra/puppet-storyboard: Spell 'server_admin' consistently
jeblairnibalizer: and that will communicate with the master via apache to ask it the running version?19:08
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1368910
uvirtbotLaunchpad bug 1368910 in nova "intersphinx requires network access  which sometimes fails" [Undecided,In progress]
nibalizerjeblair: no, i see what you're getting at now19:08
*** ling-yun has quit IRC19:08
clarkbhrm I don't see anything that indicates packagekit won't already update puppet for us19:08
openstackgerritA change was merged to openstack-infra/devstack-gate: Add option for building Ironic deploy ramdisks
openstackgerritA change was merged to openstack-infra/puppet-storyboard: Compare values to :undef to test for existence
*** coolsvap has quit IRC19:09
jeblairour manifests are going to cause some harmless comment churn on /etc/default/puppet.  we should update that, but it's not critical19:09
jeblairi'm going to grab lunch now, but i think we're all set to proceed19:10
fungiokay, i'll start tackling a handful and raise in here if i find any exceptions19:10
clarkbwe do indeed have latest puppet 2.7 on the git nodes19:10
clarkbso I think we may be ok there once we update the excludes in /etc/yum.repos.d/puppetlabs.repo19:11
clarkbI am going to grab lunch now back in a bit19:11
fungiyeah, unattended-upgrades does some magical whitelisting of specific repositories and won't upgrade packages from others by default without additional convincing19:11
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Update comments in puppet default
fungihopefully 121202 passes in the right magic19:12
superdanSo can we get 121200 promoted?19:12
fungii honestly can't remember how to identify exactly what it needs without digging in apt repo release files to find the encoded suite name and all that19:12
nibalizerjeblair: this might work for getting puppetmaster status
*** mjturek has left #openstack-infra19:13
superdanfungi: ^19:13
clarkboh crud I just relized the archive of trusty may not be correcr in my auto updates change19:14
clarkbfungi ^19:14
*** e0ne has joined #openstack-infra19:15
nibalizerya so this, with paths replaced to be more openstacky, works to give version of the running puppetmaster19:15
nibalizercurl -k --cert /var/lib/puppet/ssl/certs/ --key /var/lib/puppet/ssl/private_keys/ -H "Accept: pson" https://localhost:8140/production/status/puppetclient19:15
*** vigneshvar has quit IRC19:15
clarkbjeblair: nibalizee subscribe an apache service resource to the passenger package19:16
*** koolhead17 has joined #openstack-infra19:17
*** koolhead17 has joined #openstack-infra19:17
*** koolhead17 has quit IRC19:17
*** koolhead17 has joined #openstack-infra19:17
fungisuperdan: the bug linked from that change has a logstash query which indicates that this happened to four changes in the past week, most recently on tuesday19:18
superdanflashgordon: ?19:19
superdanfungi: thought it just took out like a ten-patch set of numa bits19:19
fungiunless it's got additional manifestations not covered by the query in the bug19:19
*** homeless_ has joined #openstack-infra19:20
*** balles has quit IRC19:21
*** penguinRaider has joined #openstack-infra19:21
flashgordonfungi: it happened moments ago in the gate19:21
flashgordonas well19:22
funginibalizer: what was the plan for dealing with the in puppet.conf on all the servers?19:22
flashgordonfungi: if we are doing another batch of promotions here are some other patches that are ready19:22
funginibalizer: is that getting puppet-managed once we're done?19:22
flashgordonneither are very high on the gate bug list19:22
flashgordonbug they do hit occassionally19:23
mtreinishflashgordon: oh, good call on the second one. I had a patch bump on that one...19:23
*** homeless has quit IRC19:23
superdando it do it!19:24
flashgordonfigure for the next promote we do can do all the outstanding gate bug fixes that are approved19:24
flashgordonalso it looks like logstash is pretty backed up19:25
flashgordonso we are running blind19:25
*** cnesa9 has quit IRC19:26
flashgordonnot seeing many entries past 8:00 and its now  19:2619:26
fungiyeah, its job queue is ~110k now19:27
fungiseems to have spiked up shortly after the job volume did19:27
fungiwhich makes sense19:27
*** dannywilson has quit IRC19:28
flashgordonanyway time to find some food19:28
*** dannywilson has joined #openstack-infra19:28
funginibalizer: do we need to be worried about any of these new warnings?
*** e0ne has quit IRC19:29
fungipuppet agent likes to make them a nice bold red color19:29
fungiseems to think we should stop setting modulepath and manifest in puppet.conf, and manifestdir and templatedir at all19:30
*** homeless_ has quit IRC19:31
fungihrm... i'm unconvinced our automated puppet runs are working... the change we merged to nodepool.yaml still hasn't been installed to the server, more than an hour later19:31
*** Guest24333 is now known as Adri200019:32
*** Adri2000 has quit IRC19:32
*** Adri2000 has joined #openstack-infra19:32
*** chmouel- has joined #openstack-infra19:32
fungihowever, whatever the problem was in dfw seems to have lightened up again, so maybe we should revert that max-servers patch anyway19:32
*** chmouel- has quit IRC19:33
*** dannywilson has quit IRC19:33
*** bhuvan_ has joined #openstack-infra19:33
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove query for bug 1347692
uvirtbotLaunchpad bug 1347692 in glance "gate failure with tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image (dup-of: 1345955)" [Undecided,Confirmed]
uvirtbotLaunchpad bug 1345955 in horizon "KeyError: <SassRule <RuleAncestry ()>, 0 props> Failure" [Critical,Confirmed]
*** dannywilson has joined #openstack-infra19:34
*** vigneshvar has joined #openstack-infra19:34
*** bhuvan has quit IRC19:36
*** sdake_ has quit IRC19:37
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Switch sample sqlalchemy url to use mysql
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add get_failing_from_run db api method
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add subunit2sql-db-manage utility
*** e0ne has joined #openstack-infra19:39
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Revert "Drop 90 servers from rax-dfw"
*** kgiusti has quit IRC19:39
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.2 bug 1368391
uvirtbotLaunchpad bug 1368391 in glance "sqlalchemy-migrate 0.9.2 is breaking nova unit tests" [Undecided,In progress]
*** e0ne has quit IRC19:41
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1368910
uvirtbotLaunchpad bug 1368910 in nova "intersphinx requires network access  which sometimes fails" [Medium,In progress]
*** mriedem has quit IRC19:43
*** nelsnelson has joined #openstack-infra19:43
*** koolhead17 has quit IRC19:43
*** mriedem has joined #openstack-infra19:43
*** kgiusti has joined #openstack-infra19:44
*** sdake has joined #openstack-infra19:44
clarkbflashgordon: fungi: it spiked due to nova console logs exploding yesterday19:45
fungiclarkb: oh, one other possible gotcha. i have a vague recollection that unattended-upgrades fails to interpret apt-cache policy and so doesn't work with pinned packages. looking now for confirmation19:46
*** mriedem1 has joined #openstack-infra19:47
*** andreaf has quit IRC19:48
*** mriedem has quit IRC19:48
*** andreaf has joined #openstack-infra19:49
*** MaxV has joined #openstack-infra19:49
clarkbfungi: oh fun19:51
fungioh, right, also i think i remember looking into this before... the release files in the puppetlabs apt repo lack a suite field19:51
clarkbfungi: is there a fallback value like 'main' or similar that apt will not blow up on?19:53
clarkbI am guessing not as this seems to be very conservative19:53
fungioh, nevermind...
fungiso "suite" is precise on precise, trusty on trusty19:53
clarkbgah ok so I need to use powers of facter to get that value19:53
fungiand origin is Puppetlabs (with a capital P)19:54
clarkbyup I got that bit19:54
openstackgerritClark Boylan proposed a change to openstack-infra/config: Allow puppet to be automatically upgraded
clarkbfungi: ^ I think that should pass jenkins testing and set the right suite19:55
clarkbbut whether or not pins are respected is an open question19:56
fungii did find surprisingly spot-on bug 1251228 but it's not yet been triaged so may be based on a fallacy19:56
uvirtbotLaunchpad bug 1251228 in unattended-upgrades "unattended-upgrade doesn't upgrade packages pinned to version" [Undecided,New]
funginevertheless, if true, more dangerous than not upgrading pinned packages19:57
clarkboh I see the issue with puppet I am going to fix it now19:57
*** alexpilotti has quit IRC19:58
*** salv-orlando has joined #openstack-infra19:58
*** amcrn has quit IRC19:59
clarkbI apparently missed that we had to manually run puppet on all the nodes for the ssh key thing19:59
*** JayJ__ has joined #openstack-infra19:59
clarkbI am swinging around and doing that on hosts that I skipped because they were like others... my bad19:59
*** sdake has quit IRC20:00
*** nelsnelson has quit IRC20:00
fungii'm taking a break to cook dinner20:01
*** dannywilson has quit IRC20:01
*** dannywilson has joined #openstack-infra20:01
*** emagana has quit IRC20:02
*** alexpilotti has joined #openstack-infra20:02
dhellmannjeblair: it looks like mriedem1 took care of approving that nova/intersphinx change20:03
*** marcoemorais1 has joined #openstack-infra20:04
*** marcoemorais1 has quit IRC20:04
*** marcoemorais1 has joined #openstack-infra20:05
*** dannywilson has quit IRC20:05
*** dannywilson has joined #openstack-infra20:05
*** andreykurilin_ has joined #openstack-infra20:05
*** dmsimard is now known as dmsimard_away20:06
*** marcoemorais has quit IRC20:07
*** kgiusti has quit IRC20:08
clarkbfungi: jeblair: did we decide to continue pinning facter to 1.*?20:09
*** MaxV has quit IRC20:10
*** ramashri has quit IRC20:11
*** kgiusti has joined #openstack-infra20:11
*** nelsnelson has joined #openstack-infra20:15
nibalizerfungi: yea we should stopdoing that20:16
clarkbjudging by facter and puppetdb-termins should both be pinned to 2*20:17
jeblairclarkb: we did not decide to pin to 1.  and this is the third time we've had this conversation.  :)20:18
clarkbjeblair: I ask because the heredoc pasted on the etherpad pins that way...20:19
jeblairis an updated version20:19
daguegoneitfungi: we just took a gate reset, any chance we could promote this ?20:19
daguegoneitshould help on the nova docs thing killing FFE patches20:19
jeblairsigh, our centos pins are not equivalent to our ubuntu ones20:23
*** Sincler has quit IRC20:24
*** andreaf has quit IRC20:24
*** rmcall has quit IRC20:25
*** andreaf has joined #openstack-infra20:25
clarkbjeblair: ya they are inverted20:25
clarkbI am getting dpkg was interrupted errors20:25
clarkband apparmore package corruption20:25
* clarkb hacks with hacksaw20:25
flashgordondaguegoneit: if we do a promote we should do and as well20:26
superdanoh, both of those?20:26
superdanI'd rather just 12084320:26
*** flaviof_zzz is now known as flaviof20:26
flashgordonsuperdan: those are other fixes for known gate bugs20:26
superdannot 120843 though, just 12084320:27
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Also stop managing yum puppet pins
flashgordonoh woops20:27
* superdan is being a friday-inspired smartass20:27
flashgordondaguegoneit: looks like fungi is AFK For a bit20:29
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Ensure redis and memcached are on slaves (and running)
*** lcheng_ has joined #openstack-infra20:29
*** r-daneel__ has quit IRC20:29
AJaegerdaguegoneit: I see that intersphinx is used by many other projects as well - should it be disabled everywhere?20:30
daguegoneitAJaeger: probably20:30
flashgordonAJaeger: at least on any project that fails on sphinx warnings20:31
*** kgiusti has left #openstack-infra20:31
flashgordonAJaeger: but I think everywhere makes sense20:31
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Revert "Stop managing puppet apt pins"
AJaegerflashgordon, daguegoneit is either of you doing some patches or shall I have a go?20:32
daguegoneitAJaeger: go for it20:32
*** dustins has quit IRC20:32
flashgordonAJaeger: go for it20:33
*** emagana has joined #openstack-infra20:33
*** gyee has joined #openstack-infra20:34
*** leakypipes has quit IRC20:34
*** andreaf has quit IRC20:34
*** andreaf has joined #openstack-infra20:35
AJaegerok, will do some patches...20:35
*** _nadya_ has quit IRC20:35
*** nelsnelson has quit IRC20:36
flashgordonjeblair: oh I think I was supposed to remind you to look into why I can't do without being authenticated20:37
*** nelsnelson has joined #openstack-infra20:37
*** emagana has quit IRC20:37
clarkboh I just remembered that the facter metadata stuff failing should be better in puppet3 iirc20:38
clarkbso this may fix a class of failure we hit \o/20:38
jeblairflashgordon: i'm not seeing anything obvious; next step may involve examining the gerrit source to see if there's a concept of anonymous viewing of group members at all20:40
jeblairclarkb, fungi: etherpad links to pastes have been updated for ubuntu and centos20:40
clarkbjeblair: thanks20:41
*** mmedvede has quit IRC20:41
*** ChuckC_ has quit IRC20:41
jeblairi approved centos nodes will need to wait on that20:41
*** nelsnelson has quit IRC20:42
flashgordonjeblair: cool20:42
*** radez is now known as radez_g0n320:42
AJaegerflashgordon, daguegoneit: I'm reusing the change-id and appreciate reviews of,n,z20:43
AJaegerwill do some more patches tomorrow20:43
superdanAJaeger: that just unapproved the one we had20:43
superdanwas that intentional?20:43
flashgordonAJaeger: why did you upload a version of the nova one?20:44
AJaegersuperdan, flashgordon: I screwed up, sorry ;((((20:44
AJaegerI mixed up neutron and nova ;(20:45
flashgordonsuperdan: want to re-approve20:45
superdanis fungi the only one that can promote?20:45
*** dimtruck is now known as zz_dimtruck20:46
flashgordonsuperdan: no, clarkb can as well20:46
flashgordonsuperdan: we need to wait for gate reset before doing it though20:46
jeblairany infra-root can20:46
superdanflashgordon: oh?20:46
superdando we queue it for the next reset, or do we have to notice that there's a reset?20:47
jeblairwe can do it at any time, it's just instantaneous, so it depends on what you're trying to accomplish20:47
superdanyeah, I figured that for the set of things we have, we'd just do it and take the obligatory reset20:47
superdanthat was my assumption I mean20:48
flashgordonsuperdan: I think we should wait for a reset20:48
flashgordonas a few patches are close to landing20:48
flashgordonor failing20:48
superdansucks if you're the guy that gets kicked out because of this fail :)20:48
*** mbacchi has quit IRC20:49
*** tsufiev_ is now known as tsufiev20:50
*** fifieldt_ has joined #openstack-infra20:50
*** craigbr has quit IRC20:50
AJaeger73 hits in various projects for the last 7 days searching with logstash...20:51
*** marcoemorais1 has quit IRC20:52
*** marcoemorais has joined #openstack-infra20:53
flashgordonAJaeger: not all are failures right?20:53
AJaegerflashgordon: correct, most don't have warnerrors set.20:53
*** marcoemorais has quit IRC20:53
*** marcoemorais has joined #openstack-infra20:54
* AJaeger will update the commit message for the next patches20:54
*** fifieldt has quit IRC20:54
flashgordonAJaeger: just be careful to stay away from the nova one :)20:54
*** sdake has joined #openstack-infra20:54
*** dprince has quit IRC20:55
AJaegerflashgordon: I'm calling it a day now, you should be safe from me now ;)20:55
*** marcoemorais has quit IRC20:55
AJaegergood night!20:55
*** marcoemorais has joined #openstack-infra20:55
*** AJaeger has quit IRC20:55
*** ramashri has joined #openstack-infra20:56
*** dmsimard_away has quit IRC20:57
*** cnesa9 has joined #openstack-infra20:57
*** craigbr has joined #openstack-infra20:57
jeblairfound an error on wiki20:57
jeblairError: Failed to apply catalog: Parameter name failed on Package[apache_dev_package]: Name must be a String not Array at /etc/puppet/modules/apache/manifests/dev.pp:2020:57
adam_gdaguegoneit, any chance of a peek at
*** mfer has quit IRC20:57
clarkbI am just about done with logstash nodes and can help look at that in a few minutes20:58
*** shashankhegde has quit IRC20:58
*** ChuckC_ has joined #openstack-infra20:58
daguegoneitadam_g: looks fine20:58
jeblairthat's the only manifest that requires 'apache::dev'; i wonder if we need to update apache module for puppet320:59
*** dmsimard_away has joined #openstack-infra20:59
*** dmsimard_away is now known as dmsimard20:59
*** salv-orlando has quit IRC21:00
adam_gdaguegoneit, danke21:00
jeblairyeah, i think so21:01
jeblairnibalizer: thoughts on upgrading puppetlabs-apache?21:02
jeblairour version is like the oldest there is.21:03
*** shashankhegde has joined #openstack-infra21:03
clarkbjeblair: it requires a lot of work21:03
jeblairclarkb: more or less work than changing wiki.o.o to ansible?21:03
clarkbjeblair: because they made a lot of changes that were not backwrad compat and removed features but apparently they are back in now21:03
jeblairthis is my new yardstick21:03
clarkbjeblair: well the tricky thing is that you can't update it just for wiki you have to do it for everything at once. so potentially a similar amount of work21:04
mgagneclarkb: except the custom vhost template =)21:04
clarkbmgagne: is that still missing?21:04
mgagneclarkb: yep21:04
* clarkb looks into what apache::dev does21:04
mgagneclarkb: support matrix I did back then:
jeblairso for now we could drop using apache::dev and replace it with what that currently explicitly does21:04
nibalizerjeblair: i dont think we can upgrade our apache version without quite a bit of work21:05
jeblairbecause that is probably less work than ansibling things21:05
clarkbjeblair: yup21:05
jeblairclarkb: i think it's literally just package installs.  i'll write the change.21:05
clarkbjeblair: ok21:05
mgagneclarkb: template support was dropped in 0.7.021:05
*** _nadya_ has joined #openstack-infra21:06
nibalizeryou can custom_fragment, which is very close to a custom vhost21:06
clarkbjeblair: ya its trying to install all of in one go but that breaks on puppet321:06
clarkbjeblair: so that is a quick and easy work around to do what you are doing21:06
nibalizerit could also be possible for us to make openstack_apache::vhost that does the same vhost behavior as the 0.0.4 module and use a modern apache module for everything but the vhosts21:07
*** pcm_ has quit IRC21:07
nibalizerthat would be a way forward i think21:07
*** emagana has joined #openstack-infra21:07
flashgordondaguegoneit: is there a devstack-gate patch for ?21:07
daguegoneitnot yet21:08
devanandadaguegoneit: if you have another spare minute, I think would be great to land soon (consolidate redundant ironic tests)21:08
daguegoneitwell, I don't have landing power in infra21:08
devanandagah, right21:08
clarkbmgagne: that is a handy matrix21:08
flashgordondaguegoneit: in that case I'll post one so once that lands we can try it out in d-g21:09
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Drop apache::dev in wiki
jeblairclarkb, nibalizer: ^21:10
jeblairclarkb: can you go ahead and aprv it?21:10
jeblairit's just us right now and it's going to take a while21:10
*** marun has joined #openstack-infra21:11
*** ling-yun has joined #openstack-infra21:11
nibalizeri found a tab so -121:11
clarkbnibalizer: will that fail lint?21:11
devanandaclarkb: seems to need you to +2 it again21:11
jeblairlint should diea21:12
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Drop apache::dev in wiki
*** coolsvap has joined #openstack-infra21:12
*** zz_dimtruck is now known as dimtruck21:14
devanandajeblair, daguegoneit: assuming ironic graduates, at what point do we flip the switch and make an ironic test voting on nova (and, given the ongoing layering discussion, should we even do that?)21:14
*** vigneshvar has quit IRC21:14
devanandaclarkb: ty21:14
clarkbdevananda: oh I reapproved jeblair's thing21:14
devanandaclarkb: oh lol21:15
jeblairdevananda: whenever you and nova are ready21:15
clarkbdevananda: will try to get to that other thing later but currently in puppet3 hackery mode21:15
jeblairdevananda: and yeah, i think we should do it regardless21:15
*** sabeen has quit IRC21:15
devanandajeblair: cool21:16
devanandai like it when my questions have simple answers21:16
jeblairme too!21:16
*** rfolco has quit IRC21:18
*** adalbas has quit IRC21:19
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Limit number of API workers used
*** tonytan4ever has quit IRC21:21
*** dhellmann is now known as dhellmann_21:22
*** jungleboyj has joined #openstack-infra21:22
harlowjaqq, how does one resolve a cache inconsistency between pypi.iad and pypi.ord ?21:23
*** nelsnelson has joined #openstack-infra21:23
clarkbharlowja: you don't...21:24
clarkbits not perfect21:24
clarkband until we have afs is a hard thing to fix21:24
jeblairwe might have a stuck bandersnatch on one though, right?21:24
harlowjaclarkb will it eventually resync?21:24
clarkbjeblair: good point, we could have one that is stuck21:25
clarkbjeblair: usually I would expect all to get stuck though21:25
jeblairclarkb: i don't think bandersnatch works like that :(21:25
jeblairi think we need to run it in a timeout or something :(21:25
clarkboh I see ya thats a good idea21:25
clarkbmaybe a half hour timeout in the cron21:25
clarkbas updates shouldnt'21:26
clarkb take that long21:26
*** andreykurilin_ has quit IRC21:26
*** shardy_afk is now known as shardy21:26
jeblairclarkb: don't do review21:26
jeblairclarkb: please ack21:26
*** vigneshvar has joined #openstack-infra21:26
*** bswartz has quit IRC21:26
jeblairclarkb: we should start with review-dev first21:27
clarkboh good point /me goes over to that node first21:27
clarkboh you have it21:27
* clarkb gets out of the way21:27
jeblairyeah, just starting on that21:27
flashgordonsuperdan: btw if passes we merge 3 patches21:28
flashgordonsuperdan: which is why I want to wait for a reset21:28
superdanflashgordon: crossing all crossable appendages ..... now.21:28
flashgordonsuperdan: hehe21:28
clarkbjeblair: no reason to avoid proposal.slave.o.o right?21:29
jeblairclarkb: nope; but check out the output from the noop run on review-dev21:29
jeblairthe gerrit scripts things are interesting21:29
clarkbwe want those scripts21:30
jeblair  file { '/usr/local/gerrit/scripts':21:30
jeblair    ensure  => absent,21:30
jeblairclarkb: we seem conflicted on the point.21:30
clarkboh I was thinking these are the things in review_site/hooks21:30
*** LuckyVega has quit IRC21:30
clarkbjeblair: and the github scripts have been replaced by jeepyb?21:31
jeblairclarkb: i think maybe both of those dirs have21:31
clarkbthat makes sense. odd that it hasn't been cleaned up already though21:31
clarkbmaybe it isn't recursive and thus failing?21:31
*** ramashri has quit IRC21:31
*** emagana has quit IRC21:31
jeblairclarkb: i'm guessing so.  those dirs do not exist on prod21:32
jeblairclarkb: so i think i'll just rm -rf them on dev and try again21:32
*** emagana has joined #openstack-infra21:32
jeblairsound good?21:32
*** mwagner_lap has quit IRC21:32
clarkbjeblair: yup21:32
*** ramashri has joined #openstack-infra21:33
*** emagana has quit IRC21:33
openstackgerritA change was merged to openstack-infra/config: Also stop managing yum puppet pins
flashgordonsuperdan: your appendage crossing worked21:33
clarkbjeblair: doing pypi.slave now21:33
jeblairclarkb: looks good; did you want to do review.o.o?21:33
clarkbjeblair: you can do it21:33
jeblairwill do21:33
*** emagana has joined #openstack-infra21:33
superdanflashgordon: os i nac ssorcna won ?21:34
*** andreaf has quit IRC21:34
*** andreaf has joined #openstack-infra21:34
fungioh crappity. back now, fixing that heredoc21:35
clarkbfungi: its fine21:35
clarkbfungi: I think jeblair updated21:35
clarkband we are almost done21:35
fungiahh, somebody else already did. cool21:35
*** flaper87 is now known as flaper87|afk21:36
*** dimsum_ has quit IRC21:36
*** dimsum_ has joined #openstack-infra21:36
*** sballe__ has joined #openstack-infra21:37
*** stevemar2 has joined #openstack-infra21:37
*** stevemar has quit IRC21:38
jeblairi'm pretty sure i'm hitting the environment cache21:39
jeblairwe should probably turn that off21:39
*** jcooley_ has joined #openstack-infra21:39
clarkbjeblair: ++21:39
*** jcooley has quit IRC21:39
*** jcooley_ is now known as jcooley21:39
clarkbfungi: you getting the pypi mirrors?21:39
* clarkb does hpcloud mirror21:39
*** jamielennox has quit IRC21:39
*** yfried has joined #openstack-infra21:40
*** jamielenz has joined #openstack-infra21:40
*** jamielenz is now known as jamielennox21:40
*** dimsum_ has quit IRC21:40
*** dmellado has quit IRC21:41
*** gnuoy has quit IRC21:41
*** sweston has quit IRC21:41
jeblair [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"21:41
jeblairthis is confusing me21:41
*** winston-1 has joined #openstack-infra21:41
*** aysyd has quit IRC21:41
*** cnesa10 has joined #openstack-infra21:41
*** torandu_ has joined #openstack-infra21:41
jeblairit should be trying on level down under x86_6421:41
*** aysyd has joined #openstack-infra21:41
*** Alex_Gaynor has quit IRC21:42
*** gnuoy has joined #openstack-infra21:42
*** sweston has joined #openstack-infra21:42
jeblair$ escaping in bash21:42
*** Mmike has quit IRC21:42
*** ZZelle has quit IRC21:43
*** cnesa9 has quit IRC21:43
*** torandu has quit IRC21:43
*** sballe_ has quit IRC21:43
*** winston-d has quit IRC21:43
*** yfried|afk has quit IRC21:43
*** michchap has quit IRC21:43
*** kmartin has quit IRC21:43
*** markvan has quit IRC21:43
jeblairupdated paste21:43
*** ZZelle has joined #openstack-infra21:43
*** vladan has quit IRC21:43
*** stevemar2 has quit IRC21:43
fungiokay, caught up with scrollback. i'll get the pypi mirrors, sure21:43
*** markvan has joined #openstack-infra21:43
clarkbfungi: I am going bottom to top you go top to bottom and we will meet21:43
*** lcheng_ has quit IRC21:43
*** kmartin has joined #openstack-infra21:43
*** nimrodsun_ has quit IRC21:43
*** skraynev_ has joined #openstack-infra21:44
*** ubuntu has joined #openstack-infra21:44
*** ubuntu is now known as Guest2052221:44
*** Alex_Gaynor has joined #openstack-infra21:44
*** vladan has joined #openstack-infra21:45
fungitesting after puppet upgrade still involves passing --server it seems, as puppet.conf still sets server to the old master21:45
*** ZZelle_ has quit IRC21:45
*** ubuntu3 has quit IRC21:45
*** skraynev has quit IRC21:45
fungipresumably this is known21:45
clarkbfungi: yup21:45
*** nimrodsun_ has joined #openstack-infra21:45
clarkbwe should do a final day change to remove that completely or update the value21:45
*** simonmcc___ has quit IRC21:45
*** michchap has joined #openstack-infra21:45
clarkbs/day/everything upgraded/21:45
jeblairfungi: yeah, ansible supplies the arg explicitly21:46
*** Mmike has joined #openstack-infra21:46
jeblairbut also what clarkb said21:46
*** simonmcc____ has joined #openstack-infra21:47
fungiare we continying to be paranoid about git.o.o?21:47
clarkbfungi: slightly. I was trying to get centos pasty to come up so I can do it21:47
clarkbbut go ahead21:47
*** dmellado has joined #openstack-infra21:47
fungion it21:47
funginevermind--jeblair claimed it while i was gabbing21:47
jeblairya; i was holding until we all agreed i had that lock though; looks like we do21:48
fungigo for it21:48
fungiif nobody has done a pass checking for hung bandersnatch processes yet, i can do that now21:48
flashgordonmordred daguegoneit:
flashgordonmordred daguegoneit: run daguegoneit's script for openstack/openstack21:49
flashgordonhmm it didn't work21:49
*** mriedem1 has quit IRC21:50
*** datsun180b has quit IRC21:51
fungiright-o... bandersnatch is hung on ord since 00:50 utc today21:51
*** Longgeek has joined #openstack-infra21:52
clarkbjeblair: do we want to give the wiki.o.o fix any special treatment?21:52
clarkbjeblair: or just swiing around on monday and get it done21:52
flashgordondaguegoneit: fixed landing that should make our openstack/openstack numbers more accurate21:53
jeblairclarkb: i'm trying to figure out a problem with git01 that ansible sees that i did not see when running manually21:53
fungiseems bandersnatch hung on ord trying to sync
clarkbjeblair: kk21:54
jeblairError: Could not set 'present' on ensure: Could not locate the pip command. at 74:/opt/config/production/modules/openstack_project/manifests/base.pp21:54
* clarkb tries to reproduce on 0521:54
flashgordonfungi: ok just hit a gate reset21:54
flashgordonsuperdan: ^21:54
superdando eeet!21:54
fungiflashgordon: superdan: 120843,2 120793,1 121200,3 ?21:55
jeblairclarkb: ansible does not see the problem on 0521:56
flashgordonfungi: yup!21:56
clarkbjeblair: oh right it runs them in parallel21:56
clarkbjeblair: looks like 03 may have the problem?21:56
* clarkb looks at 0321:56
daguegoneitflashgordon: +221:56
fungiflashgordon: superdan: nevermind, seems to be 120843,1 120793,1 121200,4 but that last one needs an enqueue first since it got inadvertently bumped with a rebase21:56
jeblairclarkb: oh? i just ran it and only saw it on 0121:56
*** Longgeek has quit IRC21:56
clarkbjeblair: ansible said 03 failed. could be something different21:56
jeblairclarkb: no, git03 was a run already in progress21:57
flashgordonfungi: sigh, it was a super trivial rebase21:57
jeblairi believe i collided with it21:57
clarkbjeblair: that explains it21:57
flashgordonfungi: and a mistaken one21:57
fungiyep, saw in scrollback21:57
flashgordonfungi: so we have to wait for it to get through check first?21:57
*** carl_baldwin has joined #openstack-infra21:57
carl_baldwinHi, question on
*** hdd has quit IRC21:58
jeblairclarkb: okay, noop does not show the error, but running it for real it's reproducible21:58
carl_baldwinIt passed in the Jenkins check queue and went to the gate queue.21:58
clarkbjeblair: weird. and `which pip` shows me what I expect21:58
fungiflashgordon: no, i took care of it21:58
carl_baldwinThen, Kyle added recheck-opendaylight in a comment and the Jenkins check queue picked it up again.21:59
flashgordonfungi: awesome21:59
flashgordonfungi: btw want to review (its trivial)21:59
clarkbcarl_baldwin: we match recheck* for jenkins21:59
clarkbcarl_baldwin: so that will enqueue it into check21:59
flashgordonfungi: thanks21:59
fungithough this is an odd failure... seems to have failed to get refs from zm01 at 10:41 utc...
flashgordonsuperdan: they are all in :)21:59
carl_baldwinclarkb: So, it seems there is not a way to retrigger 3rd party without retriggering Jenkins22:00
fungier, 20:4122:00
fungiwas apache maybe restarting on zm01 at 20:41?22:00
superdanflashgordon: woot22:00
clarkbfungi: or maybe they restarted ti again?22:00
clarkbcarl_baldwin: depends on the third party. however you shouldn't be rechecking changes that have bene approved I don't think that accomplishes much22:01
clarkbor vice versa don't approve changes that need rechecking22:01
*** marcusvrn has quit IRC22:01
*** JayJ__ has quit IRC22:02
carl_baldwinclarkb: I can’t always help that.22:02
*** IAX has joined #openstack-infra22:02
carl_baldwinIf you look at the patch, you’ll see that I approved it, then opendaylight failed it, and then Kyle added recheck.  We were both surprised to see it reenter the check queue.22:02
*** david-lyle has quit IRC22:03
*** signed8bit has quit IRC22:03
anteayacarl_baldwin: understood, but jenkins will recheck patches with recheck in the comment22:03
fungii actually didn't realize zuul would enqueue a patchset into check which was already in the gate pipeline, but regardless it shouldn't hurt anything22:03
jeblairclarkb, nibalizer, fungi: any ideas on the pip thing?22:03
*** signed8bit has joined #openstack-infra22:04
carl_baldwinfungi: *shouldn’t*  :)22:04
clarkb looks like the current proposed syntax and jenkins will recheck on that22:04
clarkbjeblair: no, but let me look at it harder22:04
jeblairclarkb, fungi, nibalizer: pasted full error in etherpad22:04
nibalizerjeblair: at the bottom of infra-puppet3?22:05
fungijeblair: my kneejerk reaction would be to kiwf (rpm and pip-installed versions) and manually redance the get-pip magic and hope it sorted itself out22:05
clarkbit is trying to install virtualenv wiht the puppet pip provider22:05
clarkbiirc pip provider tries to find pip22:05
nibalizeri assume there is a which pip ya?22:05
clarkbnibalizer: yup22:05
clarkband pip --version exits 022:06
* clarkb reads puppet pip provider code22:06
fungiit looks in a fairly arbitrary list of explicit paths22:06
jeblairvirtualenv --version22:06
fungithe provider i mean22:06
jeblairpip freeze|grep virtualenv22:06
*** signed8bit has quit IRC22:07
clarkbI think that is the problem. now i don't know yet why only one node is affected22:07
jeblair/usr/bin/which: no pip-python in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)22:07
clarkb03 has pip-python22:07
jeblairexists on 0222:07
clarkbiirc we did a symlink somewhere? so this may be an order of operations thing22:07
jeblairlrwxrwxrwx. 1 root root 3 Jan 30  2014 /usr/bin/pip-python -> pip22:07
clarkbwe do it in install_puppet.sh22:08
clarkbmaybe we should add it to the pip manifest too22:08
clarkbso that when you include pip you get the symlink22:08
jeblairhow did git01 get into this state then?22:08
clarkb(still an order of ops thing though)22:08
clarkbjeblair: not sure22:08
fungithose are backward-compat symlinks because at one point rhel provided a pip-python (or wast it python-pip?) command and then a later fedora release switched that around and rhel (and thus centos) epel followed suit22:08
fungiand there was a point at which the pip provider and the rpm conflicted and set up a symlink to a mutually-recursive pair of symlinks pointed at each other and no executable22:09
nibalizeris this because we're using the git-pip script instead of the package?22:10
jeblairi don't see any recent package installs that would have affected it22:10
jeblairnibalizer: yep22:10
fungiand so if you tried to run pip, you basically ran an infinite symlink loop causing your shell to barf22:10
*** Guest45523 has quit IRC22:10
*** dalgaaf__ has joined #openstack-infra22:11
jeblairfungi, clarkb: should we write this off as cosmic rays and shall i create the symlink?22:11
clarkbjeblair: I think so22:11
fungiyeah, i think so too22:11
*** eglynn-regus has joined #openstack-infra22:11
fungii believe we manually dealt with this on all our centos servers at one point, but maybe missed one or reintroduced it somehow22:12
*** eglynn-officeafk has quit IRC22:12
jeblairi believe this is what has prevented puppet runs recently; perhaps the thing that changed is puppet's behavior around this.  maybe this was a nonfatal error before.22:12
fungii think we shouldn't run into it on new centos hosts we bootstrap, it was a relatively unresolvable upgrade issue22:13
*** jgrimm_ has joined #openstack-infra22:13
*** aysyanne has joined #openstack-infra22:13
daguegoneitclarkb: hey, so I dug into this -
uvirtbotLaunchpad bug 1093812 in openstack-ci "nova API version doc should point to Nova developer doc" [High,Confirmed]22:13
daguegoneitthe urls in question aren't coming from nova docs tree22:13
daguegoneitthey might have in the past22:13
*** dmsimard_away has joined #openstack-infra22:14
daguegoneitbut they are all stale files that infra needs to delete22:14
fungiit cropped up when epel changed the package on the fly (which seems odd for a stable distro, but maybe epel is not assumed to be especially stable?)22:14
*** cnesa11 has joined #openstack-infra22:14
daguegoneitI have commented as such in the bug22:14
nibalizerso i think its sensible for puppet to try to use python-pip on redhat but fall back to pip on redhat if that file doesn't exist22:14
jeblairok this time we made it past git*22:14
*** Hefeweiz1n has joined #openstack-infra22:14
nibalizerso i think I'm going to PR puppet and try to add that feature22:14
nibalizerwhich doesn't fix us in the here and now22:14
clarkbdaguegoneit: the urls are in stale files?22:14
jeblairi accepted keys for dfw and ord mirrors22:14
funginibalizer: well, we also use a custom pip provider22:14
*** ctracey has quit IRC22:15
*** willroberts has quit IRC22:15
daguegoneit ... is not something that's currently generated22:15
nibalizerfungi: the error we're getting now though seems to come from the default provider22:15
*** asadoughi has joined #openstack-infra22:15
clarkbfungi: only for pip322:15
*** serverascode has quit IRC22:15
nibalizerwe only have a custom provider for python 3 stuff22:15
fungioh, doubly fun22:15
daguegoneitnote the lack of theme consistency22:15
*** jgrimm_ is now known as zz_jgrimm_22:15
daguegoneitit looks like there is a bunch of content there from an old docs process22:15
funginibalizer: ahh, maybe so. for some reason i thought we ended up replacing the actual pip provider too22:15
clarkbdaguegoneit: so this is something we will have to fix with new docs hosting platform22:15
*** unicell has quit IRC22:15
daguegoneitthat google still finds22:15
clarkbdaguegoneit: because currently there is no safe way of deleting old stuff22:15
*** tsufiev_ has joined #openstack-infra22:15
*** tsufiev has quit IRC22:15
daguegoneitclarkb: ok22:15
*** ctracey has joined #openstack-infra22:15
nibalizerfungi: there is a long lived review to have our own custom pip2 and pip3 providers (from morderd i believe)22:16
daguegoneitwell, anyway, I have commented on the bug22:16
nibalizeri dont think it ever landed22:16
fungidaguegoneit: current docs publication automation only adds or replaces files, it doesn't delete22:16
daguegoneitand marked it invalid for nova22:16
*** jraim___ has joined #openstack-infra22:16
daguegoneitright, that seems... suboptimal22:16
*** sdake has quit IRC22:16
annegentle_daguegoneit: nice nick. Want me to just delete the object.model.html file on the server?22:16
fungidaguegoneit: it's an upshot of using rackspace cloudsites and the jenkins ftp publisher22:16
*** StevenK_ has joined #openstack-infra22:16
daguegoneitannegentle_: sure, I don't know what else needs to be purged22:16
*** electrichead has joined #openstack-infra22:17
funginibalizer: oh, right, i'm confusing that review with the production code. that's a danger when you're a sysadmin doing code review ;)22:17
*** willroberts_ has joined #openstack-infra22:17
*** serverascode has joined #openstack-infra22:17
clarkboh wow we don't have a lot of nodes available?22:17
*** winston-d has joined #openstack-infra22:17
clarkbthat is a giant blob of purple we seem to be getting past22:18
*** dstufft has quit IRC22:18
anteayagoodness no22:18
*** kmartin has quit IRC22:18
*** torandu_ has quit IRC22:18
*** cnesa10 has quit IRC22:18
*** winston-1 has quit IRC22:18
*** Adri2000 has quit IRC22:18
*** weshay has quit IRC22:18
*** dalgaaf_ has quit IRC22:18
*** yamahata__ has quit IRC22:18
*** asadough1 has quit IRC22:18
*** StevenK has quit IRC22:18
*** rkukura_ has quit IRC22:18
*** rakhmerov has quit IRC22:18
*** jgrimm has quit IRC22:18
*** gpocentek has quit IRC22:18
*** Svedrin has quit IRC22:18
*** leifmadsen has quit IRC22:18
*** afazekas has quit IRC22:18
*** dhellmann_ has quit IRC22:18
*** jesusaurus has quit IRC22:18
*** msuriar has quit IRC22:18
*** jraim__ has quit IRC22:18
*** saper has quit IRC22:18
*** dstufft has joined #openstack-infra22:18
clarkbdfw ord and iad are all having errors22:19
*** dalgaaf__ is now known as dalgaaf_22:19
clarkbhpcloud is carrying us right now22:19
*** s0nea_ has joined #openstack-infra22:19
*** mikap has joined #openstack-infra22:19
fungifor a little while dfw was back to happiness22:19
anteayafor that thin skim of blue that are running nodes22:19
*** devanand1 has joined #openstack-infra22:19
JayFclarkb: Rackspace currently has an incident that is likely impacted y'all22:20
*** jraim___ is now known as jraim__22:20
clarkbJayF: thanks /me checks status page22:20
*** enikanorov__ has joined #openstack-infra22:20
fungiJayF: i think i can confirm that, yeah22:20
JayFclarkb: off and on all day22:20
*** leifmadsen has joined #openstack-infra22:20
JayFfungi: the "likely" was more or less wishful thinking :)22:20
jeblairi wish we were less "lucky"22:21
*** gpocentek has joined #openstack-infra22:21
fungihrm, status.r.c actually doesn't show any issues i would expect to be related22:21
anteayacloudsites looks impacted too22:21
*** nelsnels_ has joined #openstack-infra22:21
fungimay be lagging behind reality though22:21
clarkbya just cloudsites and panels22:21
*** torandu has joined #openstack-infra22:21
*** rakhmerov has joined #openstack-infra22:21
*** boris-42_ has joined #openstack-infra22:22
fungithough the writeup says "errors within the Control Panels as well as API's"22:22
clarkboh that may explain it if it is api laer22:22
fungiso maybe they count "panels" to include api access to control instances22:22
jeblairminor detail22:22
fungiJayF: ^ any translation for rackspace terminology there?22:23
*** schang_ has joined #openstack-infra22:23
*** MIDENN_ has joined #openstack-infra22:23
clarkbI am going to take this as an opporunity to run home and pack for the wekend22:23
clarkbback in a bit22:23
fungii would expect api access to control next-gen cloud servers to show up as impacting next-gen cloud servers. maybe this is a compromise for not having an "apis" impact line22:23
* clarkb is barcamping then getting keys to new place hopefully22:24
fungiclarkb: oh! have fun at barcamp. those are always a blast22:24
jeblairi thought mordred fixed the cacti puppet updates?22:24
JayFfungi: clarkb: It's control plane for most cloud things, I think your interpretation about panels == control plane is accurate22:24
*** weshay has joined #openstack-infra22:24
jeblairit's running and is very slow :/22:25
mordredI thought I did too22:25
*** Clabbe has quit IRC22:25
*** gondoi is now known as zz_gondoi22:25
anteayaclarkb: safe travels22:25
*** Adri2000 has joined #openstack-infra22:25
*** dannywil_ has joined #openstack-infra22:26
*** Adri2000 is now known as Guest5435822:26
*** andreaf has quit IRC22:26
*** yamahata__ has joined #openstack-infra22:26
*** patrickeast has quit IRC22:27
*** stevemar has joined #openstack-infra22:27
*** patrickeast has joined #openstack-infra22:27
*** openstack has joined #openstack-infra23:37
*** bookwar is now known as 16WAAF8A523:42
*** 6JTAAFWQC has joined #openstack-infra23:42
*** YorikSar has joined #openstack-infra23:42
*** Mmike has joined #openstack-infra23:42
*** dkliban has joined #openstack-infra23:42
*** tristanC has joined #openstack-infra23:42
*** Svedrin has joined #openstack-infra23:42
*** coolsvap has joined #openstack-infra23:42
*** ianw has joined #openstack-infra23:42
*** NobodyCam has joined #openstack-infra23:42
*** skraynev has joined #openstack-infra23:42
*** bookwar has joined #openstack-infra23:42
*** dangers has joined #openstack-infra23:42
*** thingee has joined #openstack-infra23:42
*** yfried|afk has joined #openstack-infra23:42
*** pballand has joined #openstack-infra23:42
*** dimsum_ has joined #openstack-infra23:42
*** masayukig has joined #openstack-infra23:42
*** jgriffith has joined #openstack-infra23:42
*** marcoemorais has joined #openstack-infra23:42
*** Clabbe has joined #openstack-infra23:42
*** afazekas has joined #openstack-infra23:42
*** cnesa11 has joined #openstack-infra23:42
*** rakhmerov has joined #openstack-infra23:42
*** katyafervent_awa has joined #openstack-infra23:42
*** Hefeweizen has joined #openstack-infra23:42
*** jlk has joined #openstack-infra23:42
*** eglynn__ has joined #openstack-infra23:42
*** sweston has joined #openstack-infra23:42
*** zul has joined #openstack-infra23:42
*** dmellado has joined #openstack-infra23:42
*** ZZelle has joined #openstack-infra23:42
*** ling-yun-1 has joined #openstack-infra23:42
*** ekarlso- has joined #openstack-infra23:42
*** nikhil_k has joined #openstack-infra23:42
*** pcrews has joined #openstack-infra23:42
*** dkehnx1 has joined #openstack-infra23:42
*** mrodden has joined #openstack-infra23:42
*** dhellmann has joined #openstack-infra23:42
*** Daviey has joined #openstack-infra23:42
*** openstackgerrit has joined #openstack-infra23:42
*** weshay has joined #openstack-infra23:42
*** krotscheck has joined #openstack-infra23:42
*** zaro has joined #openstack-infra23:42
*** tchaypo has joined #openstack-infra23:42
*** reed has joined #openstack-infra23:42
*** saper_ has joined #openstack-infra23:42
*** schang has joined #openstack-infra23:42
*** mkoderer has joined #openstack-infra23:42
*** dmsimard has joined #openstack-infra23:42
*** dteselkin has joined #openstack-infra23:42
*** m1dev has joined #openstack-infra23:42
*** slagle has joined #openstack-infra23:42
*** jmeridth has joined #openstack-infra23:42
*** vipul has joined #openstack-infra23:42
*** asadoughi has joined #openstack-infra23:42
*** dstufft has joined #openstack-infra23:42
*** spiffxp_ has joined #openstack-infra23:42
*** msuriar has joined #openstack-infra23:42
*** pabelanger has joined #openstack-infra23:42
*** gnuoy` has joined #openstack-infra23:42
*** che-arne has joined #openstack-infra23:42
*** SpamapS has joined #openstack-infra23:42
*** mika has joined #openstack-infra23:42
*** winston-d has joined #openstack-infra23:42
*** AaronGr has joined #openstack-infra23:42
*** HenryG has joined #openstack-infra23:42
*** russellb has joined #openstack-infra23:42
*** blkperl has joined #openstack-infra23:42
*** _nadya__ has joined #openstack-infra23:42
*** Shrews has joined #openstack-infra23:42
*** gus has joined #openstack-infra23:42
*** pbelanyi has joined #openstack-infra23:42
*** iax7 has joined #openstack-infra23:42
*** mkerrin has joined #openstack-infra23:42
*** joa has joined #openstack-infra23:42
*** stevebaker has joined #openstack-infra23:42
*** davidlenwell has joined #openstack-infra23:42
*** medieval1_ has joined #openstack-infra23:42
*** marzif_ has joined #openstack-infra23:42
*** aysyanne has joined #openstack-infra23:42
*** markmc has joined #openstack-infra23:42
*** rhsu has joined #openstack-infra23:42
*** Guest74618 has joined #openstack-infra23:42
*** greghaynes has joined #openstack-infra23:42
*** chuckC has joined #openstack-infra23:42
*** jyuso012 has joined #openstack-infra23:42
*** doug-fish1 has joined #openstack-infra23:42
*** yamamoto has joined #openstack-infra23:42
*** Sukhdev has joined #openstack-infra23:42
*** baoli_ has joined #openstack-infra23:42
*** lbragstad has joined #openstack-infra23:42
*** jroll has joined #openstack-infra23:42
*** d34dh0r53 has joined #openstack-infra23:42
*** gondoi has joined #openstack-infra23:42
*** blitzrage has joined #openstack-infra23:42
*** adam_g has joined #openstack-infra23:42
*** gokrokve has joined #openstack-infra23:42
*** sc68cal has joined #openstack-infra23:42
*** dimtruck has joined #openstack-infra23:42
*** dolphm has joined #openstack-infra23:42
*** boris-42 has joined #openstack-infra23:42
*** nibalizer has joined #openstack-infra23:42
*** StevenK has joined #openstack-infra23:42
*** HeOS__ has joined #openstack-infra23:42
*** gpocente1 has joined #openstack-infra23:42
*** fungi has joined #openstack-infra23:42
*** BadCub_ has joined #openstack-infra23:42
*** persia__ has joined #openstack-infra23:42
*** QuinnyPi1 has joined #openstack-infra23:42
*** garyh has joined #openstack-infra23:42
*** kevinbenton has joined #openstack-infra23:42
*** harlowja_ has joined #openstack-infra23:42
*** juice has joined #openstack-infra23:42
*** gargola_ has joined #openstack-infra23:42
*** hemna has joined #openstack-infra23:42
*** patrickeast has joined #openstack-infra23:42
*** Guest54358 has joined #openstack-infra23:42
*** torandu has joined #openstack-infra23:42
*** enikanorov__ has joined #openstack-infra23:42
*** devananda has joined #openstack-infra23:42
*** s0nea has joined #openstack-infra23:42
*** serverascode has joined #openstack-infra23:42
*** willroberts_ has joined #openstack-infra23:42
*** electrichead has joined #openstack-infra23:42
*** jraim__ has joined #openstack-infra23:42
*** ctracey has joined #openstack-infra23:42
*** tsufiev_ has joined #openstack-infra23:42
*** zz_jgrimm_ has joined #openstack-infra23:42
*** simonmcc____ has joined #openstack-infra23:42
*** michchap has joined #openstack-infra23:42
*** nimrodsun_ has joined #openstack-infra23:42
*** vladan has joined #openstack-infra23:42
*** Alex_Gaynor has joined #openstack-infra23:42
*** Guest20522 has joined #openstack-infra23:42
*** markvan has joined #openstack-infra23:42
*** jamielennox has joined #openstack-infra23:42
*** jcooley has joined #openstack-infra23:42
*** sballe__ has joined #openstack-infra23:42
*** ramashri has joined #openstack-infra23:42
*** vigneshvar has joined #openstack-infra23:42
*** marun has joined #openstack-infra23:42
*** gyee has joined #openstack-infra23:42
*** bhuvan_ has joined #openstack-infra23:42
*** penguinRaider has joined #openstack-infra23:42
*** praneshp has joined #openstack-infra23:42
*** Ryan_Lane has joined #openstack-infra23:42
*** mestery has joined #openstack-infra23:42
*** markmcclain has joined #openstack-infra23:42
*** mmaglana has joined #openstack-infra23:42
*** rcarrillocruz has joined #openstack-infra23:42
*** rwsu has joined #openstack-infra23:42
*** wenlock has joined #openstack-infra23:42
*** dtroyer has joined #openstack-infra23:42
*** jpeeler has joined #openstack-infra23:42
*** niska has joined #openstack-infra23:42
*** flashgordon has joined #openstack-infra23:42
*** asettle has joined #openstack-infra23:42
*** ryanpetre has joined #openstack-infra23:42
*** LinuxJedi has joined #openstack-infra23:42
*** gothicmindfood has joined #openstack-infra23:42
*** JayF has joined #openstack-infra23:42
*** portante has joined #openstack-infra23:42
*** sandywalsh has joined #openstack-infra23:42
*** miqui has joined #openstack-infra23:42
*** sileht has joined #openstack-infra23:42
*** krtaylor has joined #openstack-infra23:42
*** dkehn has joined #openstack-infra23:42
*** _shaps_ has joined #openstack-infra23:42
*** asselin has joined #openstack-infra23:42
*** j^2 has joined #openstack-infra23:42
*** mattoliverau has joined #openstack-infra23:42
*** wendar has joined #openstack-infra23:42
*** dosaboy has joined #openstack-infra23:42
*** comstud has joined #openstack-infra23:42
*** odyi has joined #openstack-infra23:42
*** ttx has joined #openstack-infra23:42
*** flaper87|afk has joined #openstack-infra23:42
*** bodepd_ has joined #openstack-infra23:42
*** vponomaryov has joined #openstack-infra23:42
*** bogdando has joined #openstack-infra23:42
*** yongli has joined #openstack-infra23:42
*** sbadia has joined #openstack-infra23:42
*** morgan_remote has joined #openstack-infra23:42
*** pfallenop has joined #openstack-infra23:42
*** ilyashakhat has joined #openstack-infra23:42
*** Mithrandir has joined #openstack-infra23:42
*** imcsk8 has joined #openstack-infra23:42
*** toabctl has joined #openstack-infra23:42
*** bmwiedemann has joined #openstack-infra23:42
*** GheRivero_pto has joined #openstack-infra23:42
*** enikanorov has joined #openstack-infra23:42
*** shardy has joined #openstack-infra23:42
*** psedlak has joined #openstack-infra23:42
*** primeministerp has joined #openstack-infra23:42
*** jhesketh has joined #openstack-infra23:42
*** paul-- has joined #openstack-infra23:42
*** morgan has joined #openstack-infra23:42
*** crinkle has joined #openstack-infra23:42
*** mrda-vacation has joined #openstack-infra23:42
*** mmitchell has joined #openstack-infra23:42
*** Ng has joined #openstack-infra23:42
*** mflobo has joined #openstack-infra23:42
*** dougwig has joined #openstack-infra23:42
*** esmute has joined #openstack-infra23:42
*** dmitryme has joined #openstack-infra23:42
*** therve has joined #openstack-infra23:42
*** vishy has joined #openstack-infra23:42
*** zz_johnthetubagu has joined #openstack-infra23:42
*** gema has joined #openstack-infra23:42
*** rushiagr_away has joined #openstack-infra23:42
*** erw_ has joined #openstack-infra23:42
*** maurosr has joined #openstack-infra23:42
*** rossella_s has joined #openstack-infra23:42
*** cyeoh has joined #openstack-infra23:42
*** nijaba has joined #openstack-infra23:42
*** charz has joined #openstack-infra23:42
*** lxsli has joined #openstack-infra23:42
*** fabo has joined #openstack-infra23:42
*** Hunner has joined #openstack-infra23:42
*** EmilienM has joined #openstack-infra23:42
*** abramley has joined #openstack-infra23:42
*** niedbalski_ has joined #openstack-infra23:42
*** pleia2 has joined #openstack-infra23:42
*** jamespd_ has joined #openstack-infra23:42
*** zhiyan has joined #openstack-infra23:42
*** isviridov has joined #openstack-infra23:42
*** timrc has joined #openstack-infra23:42
*** Vek_ has joined #openstack-infra23:42
*** radez_g0n3 has joined #openstack-infra23:42
*** DuncanT-away has joined #openstack-infra23:42
*** mnaser has joined #openstack-infra23:42
*** phschwartz has joined #openstack-infra23:42
*** Vivek has joined #openstack-infra23:42
*** chmouel has joined #openstack-infra23:42
*** notmyname has joined #openstack-infra23:42
*** funzo has joined #openstack-infra23:42
*** soren has joined #openstack-infra23:42
*** bradm has joined #openstack-infra23:42
*** JoshNang has joined #openstack-infra23:42
*** kashyap has joined #openstack-infra23:42
*** timfreund has joined #openstack-infra23:42
*** gaelL has joined #openstack-infra23:42
*** SergeyLukjanov has joined #openstack-infra23:42
*** zz_naotok has joined #openstack-infra23:42
*** jvrbanac has joined #openstack-infra23:42
*** spiffxp has joined #openstack-infra23:42
*** sdague has joined #openstack-infra23:42
*** plars has joined #openstack-infra23:42
*** dvorak has joined #openstack-infra23:42
*** mikal has joined #openstack-infra23:42
*** mugsie has joined #openstack-infra23:42
*** uvirtbot has joined #openstack-infra23:42
*** wolsen has joined #openstack-infra23:42
*** DinaBelova has joined #openstack-infra23:42
*** gmurphy has joined #openstack-infra23:42
*** tteggel has joined #openstack-infra23:42
*** proffalk1n has joined #openstack-infra23:42
*** TheJulia has joined #openstack-infra23:42
*** jd__ has joined #openstack-infra23:42
*** NikitaKonovalov has joined #openstack-infra23:42
*** superdan has joined #openstack-infra23:42
*** ruhe has joined #openstack-infra23:42
*** dguerri has joined #openstack-infra23:42
*** froek has joined #openstack-infra23:42
*** clarkb has joined #openstack-infra23:42
*** mtreinish has joined #openstack-infra23:42
*** BobBall_AWOL has joined #openstack-infra23:42
*** clayg has joined #openstack-infra23:42
*** Kiall has joined #openstack-infra23:42
*** lifeless has joined #openstack-infra23:42
*** tcammann_ has joined #openstack-infra23:42
*** jeblair has joined #openstack-infra23:42
*** anteaya has joined #openstack-infra23:42
*** jbjohnso__ has joined #openstack-infra23:42
*** a2hill has joined #openstack-infra23:42
*** loquacities has joined #openstack-infra23:42
*** zigo has joined #openstack-infra23:42
*** gilliard has joined #openstack-infra23:42
*** cinerama has joined #openstack-infra23:42
*** mordred has joined #openstack-infra23:42
*** SlickNik has joined #openstack-infra23:42
*** jdsn has joined #openstack-infra23:42
*** morgabra has joined #openstack-infra23:42
*** d0ugal has joined #openstack-infra23:42
*** ChanServ has joined #openstack-infra23:42
*** dstanek has joined #openstack-infra23:42
*** sets mode: +o ChanServ23:42
*** Steap_ has joined #openstack-infra23:42
funginibalizer: curl: (60) SSL certificate problem: unable to get local issuer certificate23:42
*** dimsum_ has quit IRC23:43
nibalizerfungi: either add -k(insecure) or --cacert `puppet config print cacert`23:44
*** hemna is now known as 18VAAXY4623:44
fungi(also the url needed quoting because of the &)23:44
nibalizeri would reccomend -k since it doesnt matter and im pretty sure the second idea is syntactically incorrect23:44
*** grantbow has joined #openstack-infra23:44
*** bcrochet_ has joined #openstack-infra23:44
*** alaski has joined #openstack-infra23:44
*** harlowja has joined #openstack-infra23:44
*** annegentle_ has joined #openstack-infra23:44
*** hemna has joined #openstack-infra23:44
*** bcrochet has joined #openstack-infra23:44
*** harlowja has quit IRC23:44
*** annegentle_ has quit IRC23:44
*** hemna has quit IRC23:44
*** bcrochet has quit IRC23:44
*** bcrochet_ is now known as bcrochet23:44
*** annegentle has joined #openstack-infra23:45
*** aviau has joined #openstack-infra23:45
*** 18VAAXY46 is now known as hemna23:45
fungi-k(insecure) also seems to be incorrect syntax, even after escaping the parens23:45
*** gokrokve has quit IRC23:46
fungithe --cacert option worked though23:46
*** dimtruck is now known as zz_dimtruck23:46
nibalizerno -k on its own23:47
nibalizersorry the parens was my syntax23:47
nibalizercool though23:47
fungior generated a crapton of json as a response anyway23:47
nibalizeryou got a response of some kind?23:47
nibalizerokay that says to me that the master has no cert problems interacting with puppetdb23:47
fungier, maybe yaml23:47
fungianyway some serialized response23:47
nibalizeryea if it didn't send you on your way you're probably fine23:47
nibalizeralso sounds like puppetdb has made its api a little more sane23:48
fungiit's info on other machines which have reported in, looks like23:48
nibalizeryea, that was just a hello world test to check for certificate functionality23:48
nibalizeri probably should have said that at the beginning23:48
fungii figured23:48
*** wenlock has quit IRC23:49
nibalizerif you apt-cache policy puppetdb on the puppetdb server, is it at latest?23:49
funginot installed on the new puppet master23:49
fungioh, on the puppetdb server23:49
funginope, we never upgraded it. d'oh!23:50
fungiInstalled: 1.6.3-1puppetlabs1 Candidate: 2.2.0-1puppetlabs123:50
fungiit's upgraded now23:50
fungilooks like that restarted the jvm it uses too23:51
funginow we wait and drink?23:51
nibalizerwell we definitely do the second23:52
nibalizerwant to run puppet on the puppet3master again23:52
*** pcrews has quit IRC23:52
nibalizerthat should tell us immedeatly if things are working23:52
nibalizeror worse, if we broke everything23:52
fungicompleted with no errors23:52
fungibut then again, it was doing that before23:53
nibalizerand report is in puppetdb!23:53
fungichecking logs on the db23:53
clarkb looks like it worked23:53
*** bswartz has joined #openstack-infra23:53
fungioh, it was just lagging23:53
clarkbfungi: nibalizer: so what was the problem?23:53
nibalizeri think you need equal versions of puppetdb and puppetdb-terminus23:54
clarkboh I see and us not explicitly updating puppetdb borked it23:54
*** dimsum_ has joined #openstack-infra23:54
*** fifieldt has joined #openstack-infra23:54
*** lyxus has joined #openstack-infra23:54
*** andreaf has joined #openstack-infra23:54
*** changbl has joined #openstack-infra23:54
*** cody-somerville has joined #openstack-infra23:54
*** bnemec has joined #openstack-infra23:54
fungiwe might should pin puppetdb too then?23:54
fungisince it allowed us to upgrade puppetdb-terminus without forcing a puppetdb upgrade23:54
nibalizerim a bit leery of this 'auto upgrade puppet' business23:54
*** dimsum_ has quit IRC23:54
nibalizeri know we agreed to 3.x pin but pinning to a known version and upgrading on our terms is somtehing i would prefer23:55
*** dimsum_ has joined #openstack-infra23:55
nibalizerbut thats a topic for next week or the meeting or the summit23:55
clarkbnibalizer: well jeblair said this or ansible :P23:55
fungiyeah, the down side to "upgrading on our terms" is that if it's every minor release bump... that's potentially lots of these "log into all the servers" fire drills23:56
nibalizernah, cus we could update the pin, and puppet + automatic upgrades would gently move us up23:56
nibalizerfungi: want to run one more check for me?23:56
*** flaper87|afk has quit IRC23:56
*** flaper87|afk has joined #openstack-infra23:56
nibalizerwill you run '' manually and see if its all working good23:56
nibalizerthere was a puppet 3.7 bug where they broke 'puppet module install'23:57
funginibalizer: on a random machine, or the master?23:57
nibalizeryou can read roots crontab23:57
nibalizerto see what to do23:57
*** pcrews has joined #openstack-infra23:57
*** zz_jgrimm_ has quit IRC23:58
*** zz_jgrimm_ has joined #openstack-infra23:58
*** ctracey has quit IRC23:58
*** ctracey has joined #openstack-infra23:58
*** jraim__ has quit IRC23:58
*** jraim__ has joined #openstack-infra23:58
*** serverascode has quit IRC23:58
*** serverascode has joined #openstack-infra23:58
*** willroberts_ has quit IRC23:58
*** willroberts_ has joined #openstack-infra23:58
*** hemna has quit IRC23:58
*** hemna has joined #openstack-infra23:58
*** gargola_ has quit IRC23:58
*** gargola_ has joined #openstack-infra23:58
*** harlowja_ has quit IRC23:58
*** harlowja_ has joined #openstack-infra23:58
*** baoli_ has quit IRC23:58
*** baoli_ has joined #openstack-infra23:58
*** marzif_ has quit IRC23:59
*** marzif_ has joined #openstack-infra23:59
nibalizeruh thats fine23:59
fungiand 'echo $?' says 023:59
*** mkerrin has quit IRC23:59
*** mkerrin has joined #openstack-infra23:59
*** iax7 has quit IRC23:59
*** iax7 has joined #openstack-infra23:59
*** pbelanyi has quit IRC23:59
*** pbelanyi has joined #openstack-infra23:59
nibalizerweird that its trying to puppet module install prostgres23:59
*** che-arne has quit IRC23:59
*** che-arne has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!