Monday, 2014-02-10

*** bknudson has joined #openstack-infra00:02
*** atiwari has quit IRC00:03
mordredok. restarted00:03
*** mattoliverau has quit IRC00:04
*** yamahata has quit IRC00:04
*** dizquierdo has quit IRC00:10
*** mattoliverau has joined #openstack-infra00:12
openstackgerritA change was merged to openstack-infra/config: Move o.o user creation to it's own manifest.
mordredfungi: I can't log in to graphite.o.o00:21
anteayafungi: shall I remove WIP on the reversion patch? 7217800:24
*** sarob has joined #openstack-infra00:24
anteayaand morning from a few hours ago mattoliverau00:24
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix path to graphite db
anteayahow are you this fine Monday morning?00:24
anteayamordred fungi so what is the direction for manage-projects? smaller pieces? still under the ministration of puppet if we can manage it?00:25
*** CaptTofu has joined #openstack-infra00:26
*** AaronGr_Zzz is now known as AaronGr00:26
*** sarob has quit IRC00:28
mattoliverauGood morning anteaya :)00:32
*** AaronGr is now known as AaronGr_Zzz00:32
mattoliverauanteaya: I'm doing well, been having another heat wave in Melbourne, today is a bit of a cool change, a great start to the week :)00:36
mattoliverauanteaya: how's you weekend going?00:36
*** markmcclain has joined #openstack-infra00:38
anteayawell not bad00:39
anteayaspent most of it reading and listening to classical music00:39
anteayahow hot for you?00:39
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better
anteayasomebody from nova is awake00:40
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better
*** markmcclain has quit IRC00:43
fungianteaya: i didn't want to un-wip the revert until we had some consensus on where we want to go next. but i can confirm that the logoutput option at least seems to work for manual invocations of puppet agent --test there00:44
fungimordred: i suspect puppet may be broken/off on graphite.o.o... checking00:45
fungiyeah, not running00:45
fungistarting it now00:46
fungimordred: your keys are updated now00:48
anteayafungi: okay, great let's let it steep for a bit and see what others have to offer00:48
mordredfungi: also,
anteayabtw I won't be around for the meeting on Tuesday, I have a conflict00:49
anteayaso the only items I have are the manage-projects item and the proposal to change the meeting time to make it easier to include mattoliverau00:49
mordredfungi: I can now confirm that the sync_db runs on every puppet invocation00:50
*** gokrokve has joined #openstack-infra00:50
anteayamattoliverau: if you can make it to next week's meeting and speak up about the proposal to change the meeting time that would be great00:50
mattoliverauBy next week you mean this weeks I assume ;) I'll make sure I am available and there.00:51
*** yamahata has joined #openstack-infra00:54
*** yongli has joined #openstack-infra00:57
*** mwagner_lap has joined #openstack-infra01:00
anteayathis week's, yes01:00
anteayaand thank you01:00
anteayafungi: I am brewing up an email reply to:
anteayaand was wondering if I could have your input:
fungianteaya: do you have a link to any discussion which took place when we blocked their access? i honestly don't recall which account it was or the circumstances surrounding it, so would like to refresh my memory01:04
anteayafungi: let me dig up some logs01:04
fungianteaya: thanks!01:04
openstackgerritA change was merged to openstack-infra/jeepyb: Correct create-cgitrepos projects.yaml versioning
anteayawell Dec 31 I sent this: and then Jan 9 I sent this:
anteayaboth in reference to tail-F but I don't think we had mentioned them by name yet01:09
anteayastill digging01:09
anteayahere is the service account request:
*** nosnos has joined #openstack-infra01:12
*** idella4 has joined #openstack-infra01:13
*** gothicmindfood has quit IRC01:13
*** gothicmindfood has joined #openstack-infra01:14
anteayagot something, timestamp 2014-01-07T12:55:5701:16
anteayausername kruskakli01:16
idella4hello there == g'day01:17
*** e0ne has quit IRC01:17
idella4guessing anteaya is busy; how was the conf.?01:21
*** dims has joined #openstack-infra01:21
*** nosnos_ has joined #openstack-infra01:21
*** nosnos_ has quit IRC01:21
*** nosnos has quit IRC01:22
*** nosnos_ has joined #openstack-infra01:22
*** CaptTofu has quit IRC01:24
anteayafungi: part two: 2014-01-10T14:28:3601:24
anteayahello idella401:24
*** dkehn__ has quit IRC01:24
*** locke105 has quit IRC01:24
*** sarob has joined #openstack-infra01:25
*** dkehn__ has joined #openstack-infra01:25
anteayaif you are interested in contributing to openstack, please do respond to the comments left on at least one of the bug reports you filed01:25
anteayaif you have attached yourself to me, don't01:25
*** locke105 has joined #openstack-infra01:25
anteayaI am not in the mood and am more than likely to publicly embarass you01:25
anteayasimply as a mechanism to ensure I have the space I need to work01:26
*** jasondotstar has joined #openstack-infra01:26
anteayafungi: so let me know if you have any thoughts about the email response so far01:27
idella4thought as much01:27
anteayaokay, I think that was in an email thread that was jammed in with some other stuff01:28
anteayaI will continue to dig01:28
*** CaptTofu has joined #openstack-infra01:30
*** sarob has quit IRC01:30
*** yamahata has quit IRC01:33
mattoliverauHi idella4, but your g'day reference, I take it your an Aussie?01:33
idella4\yes matt I am indeed01:33
idella4I tend to throw the odd au term in mostly as a fun stir01:35
mattoliverauMe too, you'll find a few of us on here. Monday's aren't as busy as it's stil Sunday in the US. Welcome to the channel :)01:35
idella4ah thanks01:35
anteayafungi: part of the issue is that there seem to be two players: here is an email from luke gorrie:
mattoliverausome American's and Canadians on the channel, just don't like taking weekends :P01:36
anteayaJan 15 from tobbe:
fungii think i just found it01:37
idella4well au is the land of the long weekend and you're here anyway?01:37
idella4except it's in fact now Monday01:37
fungianteaya: thread seems to start at
mattoliverauidella4: lol, yeah, well although it is actually a wekeend in some parts of Oz today, it isn't here in Melbourne :(01:38
mattoliverau*long weekend01:38
anteayafungi: yes01:38
mattoliverauidella4: where are you based?01:39
anteayaso while Luke acknowledged the email, there was no sense of ownership, for me01:39
fungianteaya: though the quoted content goes back further... finding the original thread now01:39
mattoliverauAhh nice, I was there for LCA, lovely city :)01:39
idella4the only gentoo dev this side of Melbourne I believe01:39
anteayafungi: k01:39
idella4oh you were at the conf.01:40
mattoliverauCool, I once co-created a distro based of gentoo, many years back. it was called kororaa :) But haven't used gentoo in a while now days.01:41
*** derekh has quit IRC01:41
idella4It's a water oriented city with the Swan river as it is but the state is spooked by a recent history of sharks grabbing the odd swimmer01:42
idella4mostly down the south-west area of the state01:42
idella4kororaa vaguely rings a bell01:42
mattoliveraulol, well Vicoria is currently on fire, so I think combined we can scare off all non aussies in the channel :)01:42
idella4hmm yep01:43
idella4still like livin' where I live01:43
mattoliverauWe released the first XGL live CD when wiggly windows and the 3d desktop cube was brand spakin new.01:43
mattoliverauMy best mate, has created kororaa, called it the korora project and it's now a fedora remix.. some for one Linux distra extreme to the next :P01:44
idella4did you do this is a private venture or was it work related in some wau?01:45
*** zehicle_at_dell has quit IRC01:46
mattoliverauThis as in openstack or this as in kororaa. The first is fun and cause I get paid to, the latter was just for fun.01:46
*** zehicle_at_dell has joined #openstack-infra01:46
idella4hmm that makes for very keen wither way01:47
openstackgerritRohan Kanade proposed a change to openstack/requirements: Adds python-seamicroclient dependency for Ironic
fungianteaya: so that thread started out on december 22 with which was focused on the mellanox account01:47
idella4either way01:47
*** yamahata has joined #openstack-infra01:48
anteayafungi: yes and then expanded01:50
anteayaI am trying to find if I emailed directly01:50
anteayaso many emails to select from01:51
fungii know how you feel :/01:51
idella4anyway nice to make yr acquaintance.  think I'll check with the innovator of openstack @ gentoo whether he's perused those bugs I files months ago01:51
*** yamahata has quit IRC01:53
mattoliverauidella4: cool, well nice to meet you too :)01:55
*** lttrl has quit IRC01:56
*** morganfainberg_Z is now known as morganfainberg01:56
mattoliverauOn that note I think I'll go find some lunch :)01:56
idella4good move that01:56
idella4meat pie with tomato sauce plus a vegemite sandwich01:57
anteayafungi: that's what it was sean collins had emailed mid december:
*** lttrl has joined #openstack-infra01:57
anteayafungi: he did so privately and forwarded me the email01:57
fungianteaya: oh wow... anyway, i think your text accurately portrays the situation01:57
anteayafungi: stand by I will dig up the email to confirm and will forward it to you01:58
*** yaguang has joined #openstack-infra02:02
anteayaokay so back to the etherpad02:09
anteayaI'm open as to a next steps or thoughts for redirection as well as a closing02:09
anteayaline 10 is basically you and I comparing notes, I'll let you edit out any part that you think should not be in the final message, fungi02:10
*** e0ne has joined #openstack-infra02:10
fungianteaya: yeah, that was inline commentary. deleting02:12
fungianteaya: while perhaps not relevant to the account at hand, note that we're now adding new accounts as members of the Third-Party CI group initially, which can comment on any project's changes but can only cast verify votes on the openstack-dev/sandbox changes02:13
anteayaI love that02:13
anteayaand yes, caught that patch merge earlier today02:14
anteayathe shiny penny that distracted me from remove manage-projects automation02:14
fungiit's still a little unclear to me how we should deal with figuring out when they're ready to be moved to the Voting Third-Party CI group (a meetbot vote during the weekly irc meeting for the project(s) on which they intend to start voting maybe?02:14
anteayathat sounds fair to me02:14
anteayahaving meetbot record the votes ensures we have a record with a url02:15
fungibut as far as accounts getting voting access disabled now, it can be as simple as moving them from the full voting group to the one which only votes on sandbox02:15
fungiwhich gives them the ability to continue retesting their voting problems on an actual repository at least02:16
anteayaspeaking of the sandbox, are their any repos they can track on the sandbox to test the triggers for their systems?02:16
anteayathe sandbox is a repo?02:16
fungithey can always add a change to the openstack-dev/sandbox repository on gerrit, yes02:16
anteayaso it is:
fungiit's just a normal project repository like any other, except with some special acl rules02:17
*** ryanpetrello has joined #openstack-infra02:17
anteayawell perhaps that might be something we can mention in the wrap up?02:17
anteayaor should that be a separate email that is reference by this one?02:17
fungii'm not sure it directly addresses the concerns raised by the initial e-mail, unfortunately02:17
anteayadon't want to clog the discussion with too many issues02:17
anteayayes okay02:17
anteayaso I won't mention it in my wrap up02:18
anteayatrying to find something to offer02:18
anteayaunfortunately the tone of the last irc conversation didn't fill me with hope02:18
fungiand we still need to figure out which, if any, of the 40 existing voting accounts get grandfathered in or demoted, and when/how that will happen02:18
anteayaah the grandfathering02:19
*** ryanpetrello has quit IRC02:22
anteayafungi: my ending feels weak but that is all I have right now02:25
anteayacan you think of anything better?02:25
*** ryanpetrello has joined #openstack-infra02:27
*** sarob has joined #openstack-infra02:28
*** yongli has quit IRC02:31
*** zhiwei has joined #openstack-infra02:31
*** sarob has quit IRC02:32
*** CaptTofu has quit IRC02:32
zhiweifungi: hi, I'am back. we now can pick a date to rename the stackforge project.02:33
anteayafungi: I like that, can we confirm that tail-f can still provide comments?02:33
fungianteaya: checking02:33
fungizhiwei: great! it's still on our meeting agenda, so we'll discuss it again on tuesday. you just missed a great opportunity for us to do it last friday since we needed to rename a bunch of oslo projects and could have done yours in the same window02:34
zhiweioh, a pity.02:34
fungizhiwei: there is a rename slated for savanna's projects... depending on how soon they figure out what they're changing their name to we could potentially batch your rename up with all theirs02:35
zhiweiok, thank you very much. I will pay attention on this.02:36
fungizhiwei: but if you can't show up for the 19:00 utc infrastructure team meeting on tuesday in #openstack-meeting, just keep your eyes on the meeting log at for details afterward02:36
*** yamahata has joined #openstack-infra02:36
anteayazhiwei: here is the savanna rename email thread:
*** ryanpetrello has quit IRC02:38
anteayafungi: yeah, I like what you have, after you have confirmed permissions on that account02:39
*** ryanpetrello has joined #openstack-infra02:40
fungianteaya: their account appears to be intact in the db (we didn't clear their ssh keys or set their account inactive). they're not presently in the Voting Third-Party CI group but i can add them to the probationary Third-Party CI group if there's no consensus from the neutron community that they're ready to reinstate voting access02:41
anteayaso Third-Party CI allows them to vote on the sandbox repo as well as comment on any other repo in gerrit, correct?02:42
fungiyep, just as described. i've gone ahead and stuck them into that group so we don't lose track of their account (so i won't need to look it up again)02:43
fungiin part because i'm lazy02:43
anteayayay for laziness, and we needed some account to be in that group02:43
anteayaokay let me add some words for that to the etherpad02:44
fungiwell, jaypipes has a testing account in there as of a few hours ago too02:44
jaypipesty fungi02:44
fungiyw as always, jaypipes!02:44
*** ryanpetrello has quit IRC02:45
*** dkehn__ is now known as dkehn_02:45
anteayahey jaypipes02:47
jaypipeshey all :)02:48
jaypipeshow are ya, Anita?02:48
anteayafungi: okay any final comments on the etherpad?02:48
anteayadoing okay, had a concussion after code sprint, didn't know until last Wednesday02:48
anteayaturns out classical music, silk, amber and a good novel is good for a concussion02:49
anteayafinally able to collect a thought and get some work done today02:49
*** weshay has quit IRC02:49
anteayaand I was on stove duty this weekend, thought about letting you know, then figured, naw, he'll just laugh at me02:49
anteayaand thou?02:49
fungianteaya: i made one more minor clarification in the first paragraph, and also added a note in the second which you may or may not be able to do anything to address02:52
* anteaya reads02:52
openstackgerritJeremy Stanley proposed a change to openstack-dev/hacking: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-dev/pbr: Remove tox locale overrides
*** dstanek has joined #openstack-infra02:59
openstackgerritJeremy Stanley proposed a change to openstack-infra/elastic-recheck: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/gear: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/os-loganalyze: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/release-tools: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/reviewstats: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-infra/storyboard-webclient: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack-dev/cookiecutter: Remove tox locale overrides
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Remove tox locale overrides
anteayafungi: thanks for the help, I posted it:
fungianteaya: you're welcome03:16
StevenKfungi: Are you done with the tox spam? :-)03:17
StevenKfungi: And let me say *damn* for the amount of reviews you've pushed up in a short time03:17
anteayagood night03:17
*** morganfainberg is now known as morganfainberg_Z03:19
*** morganfainberg_Z is now known as morganfainberg03:20
fungianteaya: good night!03:20
fungiStevenK: almost ;)03:22
StevenKfungi: Heh03:22
fungiStevenK: *now* i am03:24
fungithere's nothing like a patch blast to 50 projects to round out a weekend03:25
*** dkranz has quit IRC03:25
fungiand now, i'm off to enjoy the rest of my sunday03:25
mordredfungi: you COULD have put a topic on those so that I could easily search and script votes03:27
fungimordred: yeah, i noticed that *after* i submitted them :/03:27
fungitotally meant to add a -t footopic and didn't03:28
*** sarob has joined #openstack-infra03:28
mordredfungi: it's ok- I think I've got a search03:29
fungii figure people will merge them in their own time... some faster than others if they're blocked on getting py33 unit tests to pass and mine's the only change getting a +1 from jenkins ;)03:29
*** harlowja is now known as harlowja_away03:30
*** markmcclain has joined #openstack-infra03:31
mordredfungi: I'm +1-ing all of them03:32
fungilooks like gear and release-tools have some other bitrot issues to work out first03:32
mordredmainly because I'm low on review stats right now03:32
fungifair enough. i was low of commit stats ;)03:33
fungier, low on03:33
mordredeven at this point with mostly being awol I don't think I'm low on commit stats03:33
fungii think you've stocked up enough commits to survive through a nuclear winter03:34
*** zhiyan has joined #openstack-infra03:34
*** sarob has quit IRC03:34
fungiokay, i swear. this time i'm really stepping away from the computer. catch up with everybody in the morning03:34
mordredyou'll be back03:35
fungii will, but only after a short nap ;)03:35
*** idella4 has left #openstack-infra03:35
StevenKfungi: Do you just watch for a few minutes after saying you're stepping away? :-P03:35
fungioh, and i guess i have expense reports to file too03:35
fungiStevenK: no, i just keep failing at the task at hand03:36
* fungi -> up, up and /away!03:36
StevenKfungi: Funnily enough, I'm just looking at that too03:36
*** sarob has joined #openstack-infra03:38
*** banix has quit IRC03:40
mordredStevenK: you'll love it03:42
mordredStevenK: and by that - I do mean that you WILL love it03:42
*** e0ne has quit IRC03:42
* StevenK is looking at the training for it03:43
*** sarob has quit IRC03:43
markmcclainmordred: thanks for abandoning that review for me03:44
mordredmarkmcclain: I do what I can to be useful03:44
mordredmarkmcclain: which is less and less...03:44
markmcclainmordred: hardly… less is more right?03:48
openstackgerritYAMAMOTO Takashi proposed a change to openstack-dev/hacking: HACKING.rst: Fix odd indentation in an example code
*** mriedem has joined #openstack-infra04:10
*** e0ne has joined #openstack-infra04:10
*** jasondotstar has quit IRC04:21
*** sarob has joined #openstack-infra04:32
*** CaptTofu has joined #openstack-infra04:33
*** sarob has quit IRC04:36
*** CaptTofu has quit IRC04:38
*** jasondotstar has joined #openstack-infra04:53
*** talluri has joined #openstack-infra04:56
*** talluri has quit IRC05:08
*** talluri has joined #openstack-infra05:08
*** talluri has quit IRC05:12
*** chandankumar_ has joined #openstack-infra05:17
*** david_lyle_ has joined #openstack-infra05:20
*** nicedice has quit IRC05:20
*** david_lyle_ is now known as david_lyle05:22
*** david-lyle has quit IRC05:23
*** VijayT has joined #openstack-infra05:23
*** e0ne has quit IRC05:24
*** sarob has joined #openstack-infra05:32
*** coolsvap has joined #openstack-infra05:38
*** sarob has quit IRC05:39
*** mriedem has quit IRC05:42
*** sarob has joined #openstack-infra05:42
*** markwash has joined #openstack-infra05:44
*** mriedem has joined #openstack-infra05:45
*** AaronGr_Zzz is now known as AaronGr05:46
*** mriedem has quit IRC05:47
*** sarob has quit IRC05:47
*** markwash has quit IRC05:48
*** AaronGr is now known as AaronGr_Zzz05:48
*** markwash has joined #openstack-infra05:49
*** nosnos has joined #openstack-infra05:56
*** e0ne has joined #openstack-infra05:56
*** nosnos has quit IRC05:57
*** nosnos has joined #openstack-infra05:58
*** nosnos has quit IRC05:58
*** boris-42_ has quit IRC05:58
*** nosnos has joined #openstack-infra05:59
*** Ryan_Lane has quit IRC05:59
*** nosnos_ has quit IRC05:59
*** e0ne has quit IRC06:00
*** jasondotstar has quit IRC06:02
*** david_lyle_ has joined #openstack-infra06:04
*** amotoki has joined #openstack-infra06:05
*** david_lyle has quit IRC06:05
*** Ryan_Lane has joined #openstack-infra06:09
*** markmcclain has quit IRC06:13
*** markmcclain has joined #openstack-infra06:16
*** markmcclain1 has joined #openstack-infra06:18
*** markmcclain has quit IRC06:21
*** oubiwann_ has quit IRC06:27
jgriffithAnybody else looked at this:
jgriffithThis just started showing up today...06:32
jgriffithfunny thing is it seems "bogus"06:32
jgriffithas in cinder doesn't return that response for that call....06:32
jgriffithinfact AFAICT it doesn't return that response for anything06:32
jgriffithand I don't see that error in the Cinder logs anywhere either06:33
*** CaptTofu has joined #openstack-infra06:34
*** sarob has joined #openstack-infra06:36
*** CaptTofu has quit IRC06:39
*** sarob has quit IRC06:40
*** markmcclain1 has quit IRC06:42
*** nosnos has quit IRC06:47
*** nosnos_ has joined #openstack-infra06:48
*** NikitaKonovalov_ is now known as NikitaKonovalov06:52
*** saju_m has joined #openstack-infra06:53
*** NikitaKonovalov is now known as NikitaKonovalov_06:55
*** saju_m has quit IRC07:00
*** david_lyle_ has quit IRC07:02
*** chandankumar_ has quit IRC07:07
*** chandan_kumar has joined #openstack-infra07:07
*** david-lyle has joined #openstack-infra07:12
*** saju_m has joined #openstack-infra07:16
*** igor has joined #openstack-infra07:16
*** dims has quit IRC07:18
*** dims has joined #openstack-infra07:19
*** igor has quit IRC07:20
*** igor has joined #openstack-infra07:21
*** pblaho has joined #openstack-infra07:23
*** yolanda has joined #openstack-infra07:30
*** rlandy has joined #openstack-infra07:34
*** sarob has joined #openstack-infra07:36
*** DeeJay1 has joined #openstack-infra07:37
*** plomakin_ has quit IRC07:39
*** plomakin has joined #openstack-infra07:39
*** sarob has quit IRC07:43
*** sarob has joined #openstack-infra07:46
*** HenryG has quit IRC07:47
*** HenryG has joined #openstack-infra07:48
*** rockstar has quit IRC07:48
*** gokrokve has quit IRC07:48
*** rockstar has joined #openstack-infra07:48
*** rockstar has quit IRC07:48
*** rockstar has joined #openstack-infra07:48
*** sarob has quit IRC07:51
*** boris-42_ has joined #openstack-infra07:57
*** salv-orlando has joined #openstack-infra07:59
*** markwash has quit IRC07:59
*** VijayT has quit IRC08:00
*** nosnos_ has quit IRC08:02
*** nosnos has joined #openstack-infra08:02
*** ekarlso has quit IRC08:04
*** ekarlso has joined #openstack-infra08:05
*** ekarlso has quit IRC08:12
*** coolsvap1 has joined #openstack-infra08:13
*** ekarlso has joined #openstack-infra08:13
*** NikitaKonovalov_ is now known as NikitaKonovalov08:16
*** coolsvap has quit IRC08:17
SergeyLukjanovfungi, great job with proposing locale overrides removal, +1s from me to all of them except
*** jcoufal has joined #openstack-infra08:18
*** gokrokve has joined #openstack-infra08:18
*** e0ne has joined #openstack-infra08:19
*** gokrokve_ has joined #openstack-infra08:20
*** pleia2 has quit IRC08:21
*** morganfainberg has quit IRC08:22
*** pleia2 has joined #openstack-infra08:23
*** morganfainberg has joined #openstack-infra08:23
*** gokrokve has quit IRC08:23
*** gokrokve_ has quit IRC08:26
*** hashar has joined #openstack-infra08:27
*** e0ne has quit IRC08:28
openstackgerritLukasz Jernas proposed a change to openstack-infra/config: Add fonts-takao package on the Debian build slave
*** Ryan_Lane has quit IRC08:30
*** coolsvap1 has quit IRC08:33
*** CaptTofu has joined #openstack-infra08:35
*** CaptTofu has quit IRC08:39
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Added base for functional tests
*** flaper87|afk is now known as flaper8708:49
*** mlipchuk has joined #openstack-infra08:51
DeeJay1shouldn't post changes to the associated bug ?08:54
*** jooools has joined #openstack-infra08:56
*** matrohon has joined #openstack-infra08:58
*** abhishek has joined #openstack-infra08:59
*** zhiwei has quit IRC08:59
*** yolanda has quit IRC09:04
*** jpich has joined #openstack-infra09:04
*** yolanda has joined #openstack-infra09:05
openstackgerritFlavio Percoco proposed a change to openstack-infra/devstack-gate: Archive config files along with logs
*** yolanda has quit IRC09:06
*** e0ne has joined #openstack-infra09:07
*** yolanda has joined #openstack-infra09:07
*** yolanda has quit IRC09:08
*** coolsvap has joined #openstack-infra09:08
*** derekh has joined #openstack-infra09:08
*** yolanda has joined #openstack-infra09:12
*** yassine has joined #openstack-infra09:12
*** yolanda has quit IRC09:14
*** yolanda has joined #openstack-infra09:16
*** yolanda has quit IRC09:16
*** yolanda has joined #openstack-infra09:17
*** yolanda has quit IRC09:17
*** yolanda has joined #openstack-infra09:18
*** yolanda has quit IRC09:19
*** yolanda has joined #openstack-infra09:19
*** gokrokve has joined #openstack-infra09:21
*** mkerrin has joined #openstack-infra09:23
*** yolanda has quit IRC09:23
*** yolanda has joined #openstack-infra09:23
*** yolanda has quit IRC09:25
*** gokrokve has quit IRC09:26
*** yolanda has joined #openstack-infra09:26
*** zhiwei has joined #openstack-infra09:27
*** morganfainberg is now known as morganfainberg_Z09:28
*** afazekas has joined #openstack-infra09:29
*** coolsvap1 has joined #openstack-infra09:32
*** krtaylor has quit IRC09:33
*** krtaylor has joined #openstack-infra09:34
*** yolanda has quit IRC09:34
*** yolanda has joined #openstack-infra09:35
*** coolsvap has quit IRC09:35
*** jcoufal has quit IRC09:36
*** jcoufal has joined #openstack-infra09:36
*** e0ne_ has joined #openstack-infra09:38
*** e0ne has quit IRC09:38
*** sarob has joined #openstack-infra09:40
*** fbo_away is now known as fbo09:42
*** sarob has quit IRC09:47
*** sarob has joined #openstack-infra09:50
abhishekHi, Can any one help me regarding this error09:55
*** sarob has quit IRC09:55
abhishek+ git clone git:// 2014-02-10 08:37:46.680 | fatal: unable to connect to 2014-02-10 08:37:46.680 | Temporary failure in name resolution09:55
*** jasondotstar has joined #openstack-infra10:00
*** jp_at_hp has joined #openstack-infra10:01
*** markmc has joined #openstack-infra10:10
openstackgerritAkihiro Motoki proposed a change to openstack-infra/zuul: Correct the example of zuul enqueue command
*** dizquierdo has joined #openstack-infra10:17
*** mrmartin has joined #openstack-infra10:18
*** gokrokve has joined #openstack-infra10:21
*** DinaBelova_ is now known as DinaBelova10:21
*** andre__ has joined #openstack-infra10:23
abhishekhi can any one help me regarding the error fatal: unable to connect to git.openstack.org10:25
abhishekI am getting this error on patch set
abhishekhow can I resolve this error>10:25
*** gokrokve has quit IRC10:26
*** zhiyan has quit IRC10:27
*** zhiyan has joined #openstack-infra10:30
*** jcoufal has quit IRC10:32
*** zhiyan has quit IRC10:34
*** CaptTofu has joined #openstack-infra10:36
*** amotoki has quit IRC10:38
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Heavy Job Plugin support added
*** jcoufal has joined #openstack-infra10:39
*** CaptTofu has quit IRC10:41
*** yamahata has quit IRC10:41
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Heavy Job Plugin support added
*** coolsvap_ has joined #openstack-infra10:43
SergeyLukjanovDeeJay1, issue is under the i18n project and CR is to infra10:44
*** sarob has joined #openstack-infra10:44
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Heavy Job Plugin support added
SergeyLukjanovabhishek, hi, is it the local error or in jenkins jobs?10:44
*** yaguang has quit IRC10:45
*** coolsvap1 has quit IRC10:46
*** coolsvap1 has joined #openstack-infra10:48
*** sarob has quit IRC10:48
*** zhiwei has left #openstack-infra10:51
*** coolsvap_ has quit IRC10:51
*** CaptTofu has joined #openstack-infra10:52
*** coolsvap has joined #openstack-infra10:52
*** yassine has quit IRC10:53
*** coolsvap1 has quit IRC10:53
DeeJay1SergeyLukjanov: yeah, I couldn't find the infra project somehow, and issue affects both i18n and infra10:57
SergeyLukjanovDeeJay1, it's named openstack-ci at launchpad10:57
abhishek@SergeyLukjanov: its in jenkins job10:58
* DeeJay1 facepalms 10:58
uvirtbotLaunchpad bug 1277751 in openstack-ci "fatal: unable to connect to" [Undecided,New]10:59
SergeyLukjanovabhishek, as you can see, 1277751 is on top of
*** jasondotstar has quit IRC11:00
*** ArxCruz has joined #openstack-infra11:01
abhishek@SergeyLukjanov:so recheck bug 1277751 will work for me right?11:01
uvirtbotLaunchpad bug 1277751 in openstack-ci "fatal: unable to connect to" [Undecided,New]
SergeyLukjanovabhishek, yup, but 1270382 is more relevant IMO11:02
SergeyLukjanovabhishek, due to the " Temporary failure in name resolution"11:02
abhishekyes, thanks a lot....11:03
*** jasondotstar has joined #openstack-infra11:06
*** mrmartin has quit IRC11:16
*** gokrokve has joined #openstack-infra11:21
*** gokrokve has quit IRC11:25
jpichIs the linking to ChangeId currently broken (ChangeId urls show as,$1,n,z ) a known issue? I can't quite tell if is meant to fix it after a restart or if it's something else11:28
uvirtbotLaunchpad bug 1265646 in openstack-ci "Links to ChangeIds from Gerrit comments are broken" [Undecided,Fix released]11:28
*** CaptTofu has quit IRC11:29
jpiche.g. in the commit message or any review really11:29
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client
*** jasondotstar has quit IRC11:35
markmcjpich, yeah, I noticed that earlier too ... file a bug?11:36
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client
*** DinaBelova is now known as DinaBelova_11:39
jpichmarkmc: I supposed it can simply be marked as duplicate if it is one. Opened
uvirtbotLaunchpad bug 1278412 in openstack-ci "ChangeId linking broken in commit message" [Undecided,New]11:40
*** sarob has joined #openstack-infra11:44
*** sarob has quit IRC11:51
*** rfolco has joined #openstack-infra11:54
*** HenryG has quit IRC11:58
*** CaptTofu has joined #openstack-infra11:59
*** pleia2 has quit IRC12:01
*** pleia2 has joined #openstack-infra12:02
*** CaptTofu has quit IRC12:04
*** DinaBelova_ is now known as DinaBelova12:05
*** pleia2 has quit IRC12:07
*** pafuent has joined #openstack-infra12:07
*** pleia2 has joined #openstack-infra12:07
*** pafuent has left #openstack-infra12:08
*** pafuent has joined #openstack-infra12:09
openstackgerritZang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP
*** HenryG has joined #openstack-infra12:14
openstackgerritZang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Heavy Job Plugin support added
*** CaptTofu has joined #openstack-infra12:21
*** gokrokve has joined #openstack-infra12:21
chmouelis jenkins verifying everytime we add whatever comments now?12:21
*** yamahata has joined #openstack-infra12:22
kiallchmouel: I hope not, because if it is, things are probably pretty close to melting right now ;) Seriously though - I haven't noticed any extra test runs on the last few reviews I've done12:22
chmouelkiall: ii get all those emails saying "Patch Set 8: No score' everytime i add a new comment now12:23
chmouel(or whatever number)12:23
chmouelso i don't know if it's just testing or just add a new comment12:23
kiallFrom Jenkins, or yourself?12:24
*** masayukig has joined #openstack-infra12:24
kiallOn - is "CC Me On Comments I Write" ticked?12:24
chmoueli just add a +2 without approve and he started the check jobs12:25
*** gokrokve has quit IRC12:25
*** CaptTofu has quit IRC12:26
kiallAh - That does look a little off ..12:26
kiallNothing I can do to fix it though ;)12:27
tristanCoh, it even restart check jobs (when comment are added)12:28
kiall <-- Just +2'd this without approval, Jenkins doesn't seem to be re-running12:28
openstackgerritAndreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin
tristanCkiall: jenkins job are running in zuul for that one
chmouelmaybe that depend by proj who knows :)12:29
kialltristanC: wrong link ;)12:30
*** CaptTofu has joined #openstack-infra12:31
openstackgerritMark McLoughlin proposed a change to openstack-infra/elastic-recheck: Add query for cinder bug #1278416
uvirtbotopenstackgerrit: Error: Could not parse data returned by Launchpad: The read operation timed out12:31
*** CaptTofu has quit IRC12:32
openstackgerritMark McLoughlin proposed a change to openstack-infra/elastic-recheck: Add query for cinder bug #1278416
uvirtbotLaunchpad bug 1278416 in cinder "remove_iscsi_target() fails with ENOENT trying to unlink file" [High,New]
tristanCkiall: well chmouel posted a comments on that review, and jenkins is now restarting the tests12:32
*** CaptTofu has joined #openstack-infra12:32
*** pafuent has quit IRC12:32
mordredfungi: morning12:32
kiallOh, yea... I saw that and tried the same thing (leaving a +2, with no approval) on another review.. it didn't seem to kick Jenkins off again12:33
*** CaptTofu_ has joined #openstack-infra12:35
*** CaptTofu has quit IRC12:35
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Oslo program has adopted cliff, pycadf, stevedore, and taskflow
*** bookwar has joined #openstack-infra12:36
openstackgerritAndreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin
*** david_lyle_ has joined #openstack-infra12:38
*** bhuvan has joined #openstack-infra12:38
*** smarcet has joined #openstack-infra12:39
*** abhishek has quit IRC12:40
*** david-lyle has quit IRC12:42
openstackgerritAndreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin
*** banix has joined #openstack-infra12:45
*** sarob has joined #openstack-infra12:47
openstackgerritAndreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin
*** coolsvap has quit IRC12:50
*** banix has quit IRC12:50
*** pafuent has joined #openstack-infra12:51
*** sarob has quit IRC12:52
*** masayukig has quit IRC12:53
*** masayukig has joined #openstack-infra12:53
*** nosnos_ has joined #openstack-infra12:58
openstackgerritSascha Peilicke proposed a change to openstack/requirements: Bump lesscpy to >=0.10.1
*** nosnos has quit IRC13:01
jd__chmouel: yep it seems there's something like that, I just saw that pattern too – someone comment and Jenkins start checking against :)13:02
*** max_lobur_afk is now known as max_lobur13:03
*** nosnos_ has quit IRC13:03
*** pafuent has quit IRC13:04
*** CaptTofu_ has quit IRC13:05
*** nosnos has joined #openstack-infra13:07
mordredjd__, chmouel: in zuul, zuul reads all comments, but it has a filter to tell if it should care about the comment or not13:10
*** freyes has joined #openstack-infra13:10
mordredwait - are you saying OpenStack's jenkins is running on every comment?13:11
*** saju_m has quit IRC13:12
*** ilyashakhat has quit IRC13:14
SergeyLukjanovmordred, jd__, chmouel, I think that it's because the latest jenkins run was >48h ago13:14
SergeyLukjanovit was about
SergeyLukjanovas you can see the prev. jenkins run was Feb 613:15
*** yassine has joined #openstack-infra13:16
*** pdmars has joined #openstack-infra13:17
*** gokrokve has joined #openstack-infra13:21
*** bhuvan has quit IRC13:24
mordredSergeyLukjanov: grumble. that's not good13:24
SergeyLukjanovmordred, why? we're going with it for about a week ;)13:24
mordredgotcha. I totally understand now13:25
mordredthis is the re-check-stale-patches code13:25
SergeyLukjanovmordred, exactly13:26
SergeyLukjanovmordred, the only cons is that now jenkins post "+0, I'm running jobs" each time ;)13:26
mordredyeah - I remember this now13:27
*** gokrokve has quit IRC13:27
chmouelhopeflly the mail server scales because that's a lot of emails being sent :)13:28
mordredof all of the things I'm worried about scaling - I have 100% confidence in jeblair's ability to scale a mail server13:28
mordred(you should have seen the mail cluster he ran before he worked on openstack)13:29
jd__hm I just had 3 python26 jobs lost13:29
jd__and dvsm13:30
*** bhuvan has joined #openstack-infra13:31
jd__ah I guess it's because new patchset has been uploaded in the meantime13:31
*** mrmartin has joined #openstack-infra13:31
jd__I was not used to that :)13:31
*** ilyashakhat has joined #openstack-infra13:32
*** masayukig has quit IRC13:32
*** masayukig has joined #openstack-infra13:33
SergeyLukjanovmordred, are you sure that zuul could work ok with two identical jobs in one pipeline? (
*** mrodden1 has quit IRC13:35
mordredSergeyLukjanov: I believe we've been doing it for a while in other places, yeah13:35
*** masayukig has quit IRC13:37
anteayacan someone else confirm that zuul has almost every patch in check and gate queued?13:37
SergeyLukjanovmordred, yup, I've checked this on zuul that handles savanna-ci and it looks like it works correctly.13:39
SergeyLukjanovanteaya, mm, I'm not sure that I understand what does you mean :(13:40
anteayathere are very few tests running13:41
anteayatests are queued, not running13:41
anteayathen timing out13:41
SergeyLukjanovanteaya, it looks like there are no free build nodes, so, it shoud be ok13:42
sdaguefyi - - looks like gerrit is no longer linking ids correctly13:44
anteayamorning sdague13:44
anteayaalso looks like there is a git.o.o issue:
uvirtbotLaunchpad bug 1277751 in openstack-ci "fatal: unable to connect to" [Undecided,New]13:45
sdagueanteaya: yeh, that seems to come and go13:45
anteayathough I can get there via the gui and clone a repo13:45
SergeyLukjanovsdague, ack, link is,$1,n,z atm13:45
anteayagreat, a random failure Monday13:45
anteayawe organized it just for you, sdague, as a welcome home13:46
chmouelmordred: i think it would be nice to have an email describing the latest update on the new things hapenning between LOST and the recheck for the  48  hours change13:46
*** max_lobur is now known as max_lobur_afk13:47
anteayasdague: sorry I am not seeing the failure to link ids correctly in the 71947 patch13:47
*** dims has quit IRC13:47
*** sarob has joined #openstack-infra13:48
*** mrodden has joined #openstack-infra13:49
*** dims has joined #openstack-infra13:50
*** ryanpetrello has joined #openstack-infra13:54
hughsaundershey all, just noticed a "LOST" status for jenkins jobs: "gate-rally-pep8 LOST in 2m 36s"13:54
*** sarob has quit IRC13:54
*** yamahata has quit IRC13:56
*** eharney has joined #openstack-infra13:56
*** oanufriev has joined #openstack-infra13:57
anteayahughsaunders: yes, zuul seems to be having some issues this morning though we haven't figured out why yet13:57
anteayasome neutron patches are also reporting lost test logs13:57
hughsaundersanteaya: thanks for looking into it :)13:58
*** dklyle has joined #openstack-infra13:58
anteayathanks for reporting your findings13:58
*** sarob has joined #openstack-infra13:58
*** tru_tru has joined #openstack-infra13:59
*** thomasem has joined #openstack-infra13:59
*** yamahata has joined #openstack-infra13:59
anteayathe /srv/static/log storage still has what appears to be about 2T worth of space:
*** mfer has joined #openstack-infra14:01
*** david_lyle_ has quit IRC14:01
sdagueanteaya: try clicking on any of those links in the commit message14:01
anteayasdague: ah ha14:02
anteayano worky14:02
*** sarob has quit IRC14:03
anteayathe cpu usage graph for static.o.o spent an aweful lot of time in IOwait for many hours a while ago:
jpichsdague, anteaya: I reported the ChangeId issue this morning, fwiw ->
uvirtbotLaunchpad bug 1278412 in openstack-ci "ChangeId linking broken in commit message" [Undecided,New]14:03
*** oubiwann_ has joined #openstack-infra14:03
anteayaalmost like some sort of scheduled job was running, or something was unavailable, for a period of time and then it went away14:04
anteayathanks jpich14:04
anteayajpich: so the first time you noticed it was about 2 hours ago?14:05
*** weshay has joined #openstack-infra14:05
jpichanteaya: Give or take, yup14:06
anteayadevice reads really spiked for a while there, in the IO operations14:06
anteayajpich: k, trying to track down a time line, something spiked a while ago, might be related14:06
anteayaspiked or disappeared14:07
*** esker has joined #openstack-infra14:07
anteayaanyone know what time zone the cacti graphs use?14:07
anteayaif it is utc something kicked off about 8 hours ago14:08
*** dprince has joined #openstack-infra14:09
*** oubiwan__ has joined #openstack-infra14:09
*** e0ne has joined #openstack-infra14:10
anteayawell I looked at all the cacti graphs for static.o.o all I can confirm is that we have space for logs and something happened a few hours ago14:11
anteayanot sure where to look next14:11
*** bhuvan has quit IRC14:12
*** e0ne_ has quit IRC14:12
fungimordred: good morning... how broken are we on this fine monday?14:13
anteayastart by looking at zuul I think14:14
*** oubiwann_ has quit IRC14:14
anteayaI see most patches in both check and gate with queued tests14:14
anteayafrom gerrit has broken links to change ids:
uvirtbotLaunchpad bug 1278412 in openstack-ci "ChangeId linking broken in commit message" [Undecided,New]14:14
anteayajobs have lost logs14:15
anteayaand some failures connecting to git.o.o:
uvirtbotLaunchpad bug 1277751 in openstack-ci "fatal: unable to connect to" [Undecided,New]14:15
anteayatake your pick where you want to begin14:15
anteayaI checked cacti for static.o.o, we have about 2T left for writing logs14:16
anteayaand something happened about 8 hours ago that included much IOwait time14:16
mordredI understand what  "both check and gate with queued tests" means now - I was reading it wrong14:16
*** AJaeger has joined #openstack-infra14:16
anteayahow could I have been clearer the first time?14:17
mordrednope. you were fine. my brain processed it weirdly14:17
anteayamorning brain14:17
mordredthe sentence seems quite obvious now that I grok14:17
mordredI thought you were teling me that the problem was the changes were putting things into both check and gate simultaneously and it was happening for most of the changes in gerrit14:18
mordredwhich I did not see happening :)14:18
anteayaah okay, yes, I can understand the non-grokness then if that was your takeaway from my posting14:18
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Extracted DB API layer
mordred71393 is running14:19
anteayaalso chmouel was mentioning something about jenkins triggering check jobs on comments14:19
anteayachmouel: did you have a url or a bug for that?14:19
fungiDeeJay1: we've been wrestling with some recent problems in the bug update hook. i'll see if i can spot what happened with that one14:19
mordredor, is running things for three things14:19
anteayasome tests are running on patches, yes14:19
chmouelanteaya: this is what SergeyLukjanov and mordred were talkking about the re-check-stale-patches code14:20
*** e0ne_ has joined #openstack-infra14:20
anteayabut I am seeing much queueness14:20
mordredwe seem to be awful low on nodes14:20
anteayadid a host fall over?14:20
anteayaor a zone?14:20
mordredssh-ing to nodepool.o.o is currently taking a non-trivial amount of time14:20
DeeJay1fungi: AFAIK this was my issue, as the bug is in another project in launchpad than openstack-ci14:21
*** gokrokve has joined #openstack-infra14:21
mordredfungi: it seems we may be having some sort of fail with nodepool14:21
mordredSystem load:  3.7214:21
mordredSwap usage:   85%14:22
*** oanufriev has quit IRC14:22
mordredpuppet has been running since the 6th14:22
*** oanufriev has joined #openstack-infra14:22
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Refactor REST API to work on top of DB API
*** e0ne has quit IRC14:23
fungiDeeJay1: oh, i see SergeyLukjanov already answered your question14:24
mordredfungi: nodepool image-list was very slow in returning - doesn't it just talk to mysql/14:24
*** rossella_s has joined #openstack-infra14:25
*** gokrokve has quit IRC14:26
*** max_lobur_afk is now known as max_lobur14:27
fungi1277751 probably shouldn't be on top of the recheck list. i strongly suspect people are hitting bug 1270382 but the title of the other it too vague so they're selecting it because other people are selecting it (since it's remaining so visible on the rechecks list)14:27
uvirtbotLaunchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Incomplete]
*** nicedice has joined #openstack-infra14:27
*** blitzrage is now known as leifmadsen14:28
*** smarcet has quit IRC14:29
*** pblaho has quit IRC14:31
fungiretitled 1277751 so as to be less vague and marked it as a duplicate of 126661614:31
*** smarcet has joined #openstack-infra14:34
fungiman it's taking me a while to get through all the scrollback. problems with zuul?14:34
mordredfungi: looks like nodepool14:34
fungilooking okay to me at the moment, but i've only just glanced at it14:34
*** pblaho has joined #openstack-infra14:35
fungianteaya: the iowait on static.o.o is the weekly compression job which looks through the filesystem for uncompressed job logs and fixes those14:35
*** sirushti has left #openstack-infra14:35
anteayaah thank you14:36
*** boris-42_ has quit IRC14:36
openstackgerritPhilippe Godin proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.
*** ewindisch has joined #openstack-infra14:37
*** prad has joined #openstack-infra14:38
fungiokay, caught up. i assume from the lack of mention in scrollback that chmouel and others completely forgot what we discussed last week about zuul rerunning check jobs for any change which receives activity (comments, et cetera) if it hasn't seen a valid check result in the past 48 hours14:38
*** chris_johnson has joined #openstack-infra14:39
fungiso that core reviewers don't go blindly approving changes into the gate pipeline without fresh check results indicating it has some chance of success14:39
chmouelfungi: yep missed that, probably an email on the list may be useful ;)14:39
anteayaI forgot or never knew that discussion took place14:39
fungichmouel: i could have sworn you asked about it in here last week14:39
fungianteaya: you asked about it too14:39
anteayaI believe you14:40
chmouelfungi: ah wasn't me :) or maybe there is an another chmouel which i highly doubt :)14:40
fungianyway, i'll take a poke at nodepool14:40
*** mriedem has joined #openstack-infra14:43
fungichmouel: ahh, yep. i don't see in my channel log where you were asking about it. i was thinking of jd__ (he did notice the new zuul recheck behavior last week)14:43
anteayafungi: can you remember about when it merged? I just checked the log for config for that last 10 days and don't see a commit title that rings a bell for me14:44
jd__yeah, the most troubling thing is the LOST jobs I see a lot today :)14:44
*** pdmars has quit IRC14:45
openstackgerritMatthew Treinish proposed a change to openstack-infra/devstack-gate: Add support for running tempest serially without tenant isolation
*** burt1 has joined #openstack-infra14:45
*** malini_afk is now known as malini14:49
fungimordred: did you say you were rebooting the nodepool server? when i try to log into it i get "The system is going down for reboot NOW! Control-Alt-Delete pressed. Connection to nodepool closed by remote host."14:49
fungilooks like it just came back up (1 minute uptime)14:49
fungii'll get to work cleaning up the mess14:49
*** matsuhashi has joined #openstack-infra14:49
*** shortstop has joined #openstack-infra14:49
anteayafungi: thanks, okay, that title did strike me but yes I did pass over it14:50
mordredfungi: woah. I did NOT do that14:50
mordredbut fascinating14:50
fungimordred: okay, then rackspace probably left us a ticket saying "we noticed your vm was very busy, so we figured it was hung and rebooted it for you. kthx!"14:51
mordredit would be great if we could positively mark things as "we expect this to run hot"14:51
*** sarob has joined #openstack-infra14:51
fungimordred: now you know that actually using the resources you're allotted is not at all conducive to a heavy oversubscription model ;)14:52
chmouelthere is a "support  instruction" text box for the RAX support staff that they should check if this get added before rebooting14:52
jaypipeshi all, I see some Jenkins jobs reported by Zuul as LOST... just want to check to see if this is a known thing and if I should just hold off on running a recheck.14:52
fungi"This message is to inform you that your Cloud Server '' was found to be swapping as of 8:45 CST Today. The extent of disk IO usage had begun to have a negative impact on other customers on your host machine. As such, we had to reboot your cloud server machine to improve overall performance."14:53
anteayajaypipes: hi, yes, please hold off14:53
jaypipesanteaya: no problemo. that's why I checked here :) thx!14:53
mtreinishfungi: heh nice14:53
anteayafungi: would that be due to the file compression job?14:53
fungianteaya: not at all. nodepool != static14:53
fungithose are two completely separate servers14:53
anteayacan we swap nodepool hosts?14:54
anteayaafter the fire fighting14:55
fungianteaya: we could take an outage to resize the nodepool server, but i really think we're looking at a memory leak judging from the overall ramp of the memory and swap graphs14:55
*** sarob has quit IRC14:56
anteayasome newly upgraded tool perhaps14:56
*** oubiwan__ has quit IRC14:56
fungiso i have doubts that a bigger server would do anything other than allow us to go longer between restarts14:56
fungianteaya: i don't know that it's anything new. looking at the graph it looks like it ramps pretty continually and then drops sharply (presumably corresponding to restarts of the server)14:57
mrmartinfungi: I want to take-over a patch that was sent to review, but was not accepted. Do you know a guide that describe the steps?14:57
fungier, to restarts of the service14:57
anteayafungi: hmmm14:57
fungimrmartin: not accepted/take over how?14:57
fungimrmartin: is it just abandoned?14:57
mrmartinfungi: this one:
fungi(as in the gerrit status for the cahnge says "abandoned"?)14:58
mrmartinit was posted by smarcet, but I want to continue his work14:58
mrmartinwithout loosing the previous comments.14:58
fungimrmartin: oh, by not accepted you just mean it hasn't been approved yet14:58
fungimrmartin: git review -d 6963614:59
mrmartinyep, not approved is the exact phrase :)14:59
anteayamrmartin: you can just download the latest patcheset and edit it, submitting a new patchset14:59
anteayausing the command fungi gave you14:59
fungimrmartin: that will download the latest patchset for change 69636 and create a local topic branch from it14:59
*** yamahata has quit IRC14:59
mrmartinoh great, thank you.14:59
anteayaif you want patchset 4 do, git review -d 69636,414:59
fungimrmartin: the change-id header in the commit message (along with everything else) will be preserved, so when you fir commit -a --amend after making changes and then git review it will continue with subsequent patchsets on the same change15:00
*** zhiyan has joined #openstack-infra15:00
*** zehicle_at_dell has quit IRC15:00
fungier, s/fir/git/ (fingers were in the wrong place)15:00
*** banix has joined #openstack-infra15:00
*** saju_m has joined #openstack-infra15:02
ArxCruzfungi: anteaya hey, do you guys have an sample about how to inherit from openstack_project recipes in puppet? I'm trying to create the recipes for my CI inheriting from openstack_projects:: but mostly classes has parameters, and these parameters doesn't work well with heritage15:02
ArxCruzI try to use the include, but inside the classes also have other classes that I can't change the parameters15:02
*** bhuvan has joined #openstack-infra15:02
hughsaundersre nodepool instance: performance instances don't have swap by default, so you could move to a performance instance and not get rebooted for "excessive swapping"15:02
fungiArxCruz: we try to extract anything general people might want to reuse into separate modules outside of the openstack_project module15:02
*** yamahata has joined #openstack-infra15:03
fungihughsaunders: we set up swap partitions on performance nodes. running the linus kernel with no swap device or file at all is a very bad idea15:03
*** damnsmith is now known as dansmith15:03
*** dkliban has joined #openstack-infra15:03
*** saju_m has quit IRC15:03
*** saju_m has joined #openstack-infra15:04
ArxCruzfungi: so, I should have my own arx_openstack_project modules inherits from ::service_name ?15:05
ArxCruzthat's the way it is right now :) okay ,thanks, just wondering if inheriting was a better idea :)15:05
*** CaptTofu has joined #openstack-infra15:05
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add non-isolated serial periodic tempest job
*** pdmars has joined #openstack-infra15:06
fungiArxCruz: well, my point was that the openstack_project module is supposed to be an example of how to make use of our other modules in a deployment. that said, there are probably still lots of reusable classes in there which could stand to get mostly extracted out to separate top-level modules15:06
ArxCruzI see15:07
*** lcostantino has joined #openstack-infra15:07
*** max_lobur is now known as max_lobur_afk15:07
*** markwash has joined #openstack-infra15:07
fungiArxCruz: so i don't really have an example of inheriting bits and pieces from openstack_project::.*15:08
mordredyes. we're working on that at hp too15:08
mordredI wouldn't inherit - I'd copy and edit where needed, tbh15:08
mordredit's a bit of a mess15:08
ArxCruzmordred: that's what I was trying to avoid15:08
ArxCruzinheriting would be more easy to maintain15:08
mordredAaronGr_Zzz and I are working throuh getting more of opensatck_project re-usable15:08
ArxCruzbut puppet itself have some limitations about inherit15:08
ArxCruzmordred: count with me, do you have a action plan ?15:09
fungiyes, best effort would be to help untangling the interdependencies between openstack_project classes so that it's more modular15:09
mordredArxCruz: AaronGr_Zzz does15:09
fungiultimately openstack_project should be a very, very light shim layer to provide us with specific behaviors nobody else should really want15:10
*** rcleere has quit IRC15:10
*** CaptTofu has quit IRC15:10
cody-somervilleMaybe we should start using juju charms? ;)15:11
mordredit also turns out there is more in openstack_project that people want to reuse than I thinkw e originally thought15:11
* mordred punches cody-somerville in the face15:11
ArxCruzmordred: indeed, our CI is all based on openstack_project puppet recipes, with some modifications15:12
* cody-somerville runs to HR with a bloody nose and is told by HR he deserved it. :P15:12
*** lcostantino has quit IRC15:12
ArxCruzbut as you said, it's a mess maintain that15:12
*** sdague has quit IRC15:12
mordredwell, some of it only comes up when we try to re-consume15:13
mordredfor instance - the devstack gate prep scripts in nodepool all assume that you want to clone from review.openstack.org15:13
*** lcostantino has joined #openstack-infra15:13
mordredwhich is fine -they were written for openstack15:13
ArxCruzmordred: for this case, what I'm doing right now is overwriting the gate yaml file15:14
mordredbut trying to track them is hard, because you have to replace that string in a bunch of places15:14
ArxCruzso, I need to get all the files from openstack, run jjb script15:14
*** bhuvan has quit IRC15:14
ArxCruzthen I copy my own yaml file, and reran the jjb15:14
fungideleting all the stale ready and building nodes from before the nodepool server got rebooted15:15
dripton (Nova) seems to have been Approved and passed gate jobs last night, but is not in Git.  Is this a gate problem or am I missing something?15:15
*** dklyle is now known as david-lyle15:15
*** smarcet has quit IRC15:15
*** jgrimm has joined #openstack-infra15:16
fungi#status the gate is experiencing delays due to nodepool resource issues (fix in progress, eta 16:00 utc)15:16
fungi#status alert the gate is experiencing delays due to nodepool resource issues (fix in progress, eta 16:00 utc)15:16
openstackstatusNOTICE: the gate is experiencing delays due to nodepool resource issues (fix in progress, eta 16:00 utc)15:16
*** ChanServ changes topic to "the gate is experiencing delays due to nodepool resource issues (fix in progress, eta 16:00 utc)"15:16
fungidripton: we're working on it15:16
driptonthanks fungi15:16
driptonI failed to read the status line15:17
*** rwsu has joined #openstack-infra15:17
fungidripton: i only just set it15:17
fungi(you reminded me!)15:17
driptonaha, then I feel less bad.  Thanks for the update.15:17
*** max_lobur_afk is now known as max_lobur15:18
*** lcostantino has quit IRC15:19
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Implements:  blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts
*** mrodden has quit IRC15:20
*** AaronGr_Zzz is now known as AaronGr15:20
*** chandan_kumar has quit IRC15:20
ArxCruzfungi: clarkb mordred I'm having problems to publish my logs outside my network, is it possible I start report success only for gerrit, with a message saying logs will be available in approximately 1 hour at http://my.outside.server ?15:21
*** smarcet has joined #openstack-infra15:21
*** gokrokve has joined #openstack-infra15:21
ArxCruzor I need to have the logs ready when I post the result ?15:21
*** oubiwann_ has joined #openstack-infra15:22
krtaylorArxCruz, we could make it a bit faster than that, maybe 30 mins for starters15:23
ArxCruzif and goes upstream... :)15:24
*** gokrokve has quit IRC15:25
*** mrodden has joined #openstack-infra15:26
*** gokrokve has joined #openstack-infra15:28
*** sdague has joined #openstack-infra15:29
*** mriedem has quit IRC15:30
*** oubiwann_ has quit IRC15:31
fungiso, nodepool's running and i'm clearing out the stale ready/building/delete nodes from earlier, but even though we're now well below quota it doesn't seem to be building new nodes15:31
fungiaha! "OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'UPDATE node SET external_id=%s WHERE = %s' (4519309, 1332924L)"15:32
fungiweird... it's definitely running15:32
*** markwash has quit IRC15:32
*** jnoller has joined #openstack-infra15:33
zhiyanfungi: Hello, can you pls give a hand on ? it pending on merger a while, not sure why..15:33
fungizhiyan: see the channel topic15:33
zhiyanfungi: wo, got it. sorry15:33
*** lcostantino has joined #openstack-infra15:34
fungizhiyan: though that doesn't look necessarily related. i think gerrit may have done what we occasionally see where it doesn't notice that the change is merged in the repository15:34
fungizhiyan: check the tip of glance master... it's probably merged15:35
zhiyanfungi: seems it's better than i think...will pull15:35
fungithe mysql error in the nodepool log might be a red herring... it's from almost an hour ago (and nodepool hasn't written anything to its log since even though it's running)15:36
*** bhuvan has joined #openstack-infra15:36
*** lttrl has quit IRC15:36
fungiyeah, that's from the time of the reboot... i think it logged that because mysqld got stopped before nodepoold15:37
*** matsuhashi has quit IRC15:37
*** nosnos has quit IRC15:37
*** yamahata has quit IRC15:37
*** nosnos has joined #openstack-infra15:37
*** dkranz has joined #openstack-infra15:37
*** matsuhashi has joined #openstack-infra15:37
*** saju_m has quit IRC15:37
*** lttrl has joined #openstack-infra15:38
fungirestarted nodepoold and it's launching a buuuunch of nodes now15:38
*** freyes has quit IRC15:39
*** oubiwann_ has joined #openstack-infra15:39
*** CaptTofu has joined #openstack-infra15:39
*** saju_m has joined #openstack-infra15:39
DeeJay1fungi: didn't reconnect to the DB after the restart?15:39
*** boris-42_ has joined #openstack-infra15:40
fungiDeeJay1: it's quite possible we lack proper lsb dependency ordering headers in its initscript or maybe it doesn't wait correctly for the db to be ready when starting. hard to say since it was causing it not to log anything15:40
zhiyanfungi: thanks. it's there, already been merged to trank.15:40
zhiyanfungi: good luck. brb15:41
fungiDeeJay1: i should have attached an strace to its pid before i stopped and restarted it15:41
*** dims has quit IRC15:41
*** zhiyan has left #openstack-infra15:41
DeeJay1fungi: oh well, next time ;)15:41
*** rcleere has joined #openstack-infra15:41
*** pcrews has joined #openstack-infra15:42
*** markwash has joined #openstack-infra15:42
fungiyep. my hindsight is always 20/2015:43
*** dims has joined #openstack-infra15:45
*** markwash has quit IRC15:46
*** afazekas has quit IRC15:47
DeeJay1fungi: that error is straight from libmysql so I think there's a 90% chance it was it ;)15:48
fungiDeeJay1: right, but that was from prior to shutdown as i surmised. roughly 3 minutes before last says the system booted up15:49
*** wenlock has joined #openstack-infra15:49
*** bhuvan has quit IRC15:50
*** sarob has joined #openstack-infra15:52
*** NikitaKonovalov is now known as NikitaKonovalov_15:53
*** dkliban has quit IRC15:55
openstackgerritAaron Greengrass proposed a change to openstack-infra/config: Extend user creation with more granularity
*** DinaBelova is now known as DinaBelova_15:57
*** sarob has quit IRC15:57
*** AlexF has joined #openstack-infra15:59
*** pblaho has quit IRC16:00
*** chris_johnson is now known as wchrisj|away16:01
*** AlexF has quit IRC16:01
*** wchrisj|away is now known as chris_johnson16:02
*** w_ has joined #openstack-infra16:02
jeblairfungi: good morning.  anything i can help with?16:02
*** atiwari has joined #openstack-infra16:02
fungijeblair: unravelling the latest nodepool mystery16:03
*** olaph has quit IRC16:04
jeblairi see it's been rebooted.  that's nice.16:04
fungiit starts out that we have a fairly severe memory leak, to the point where we finally left it running so long without a nodepoold restart that this morning rackspace "rebooted" it for us. the swap activity was apparently impacting other customers on the same compute node16:04
jeblair(i find that practice annoying, btw)16:05
*** zul has joined #openstack-infra16:05
*** mriedem has joined #openstack-infra16:06
fungiit came up with, as best i can tell, nodepoold not finding the mysqld socket and waiting indefinitely. i'm still digging in the debug log to confirm16:06
*** mlipchuk has quit IRC16:06
*** nosnos has quit IRC16:06
fungiafter i manually restarted nodepool and deleted a bunch of stale nodes from before the server reboot, it's been building and running a bunch of jobs16:06
fungihowever, zuul had already marked many of them lost, even though jenkins seems to be running the jobs for those lost resutls now16:07
fungiwhich i think is why we don't yet see any movement in the zuul pipelines16:07
*** matsuhashi has quit IRC16:07
*** AlexF has joined #openstack-infra16:08
*** dkliban has joined #openstack-infra16:08
fungioh, it's finally started getting some nodes to the oldest pending changes. good16:09
jnoller@fungi @jeblair We rebooted the server(s)?16:10
fungijnoller: yes, ticket 140210-1822033-216:11
*** mgagne has joined #openstack-infra16:11
jeblairjnoller: yeah; we have a habit of not caring if our servers swap occasionally, and rax has a habit of caring enough about that to reboot them.16:11
jeblairjnoller: which is usually our cue to move to a larger server16:11
jnollerYeah, a bigger server helps16:12
fungijnoller: jeblair: happened back on feb 1st as well (ticket 140201-1811175-2)16:12
jnollerDid we notify you *before hand*?16:12
fungipart of the problem is, i think, that e-mail for the openstackci account goes to monty@inaugust.com16:12
jnollerwestmaas ^^^16:13
fungiis there somewhere better we should have ticket updates going?16:13
jeblairfungi: i could create an alias we could put all of infra-root on...16:14
fungibut yeah, those two tickets i mentioned are "hi there, we've opened this ticket to let you know we rebooted your server" so not really any advance notice no16:14
fungi"This server was experiencing out of memory errors and using the swap space partition for a significant period of time."16:15
fungii'm not sure using swap is necessarily an "out of memory error"16:15
westmaaspoor wording16:15
westmaasthis isn't a perf flavor though, right?16:16
*** mrmartin has quit IRC16:16
westmaas(catching up)16:16
* fungi checks to see if nodepool.o.o was really logging logging an actual oom condition16:16
sc68calI'm seeing jenkins doing rechecks when someone posts a comment on a review that doesn't have "recheck" in it16:16
fungiwestmaas: no, i think it's not a performance node. i believe we built it before you had those generally available16:16
westmaasfungi: makes sense16:16
*** david_lyle_ has joined #openstack-infra16:16
fungiwestmaas: yeah, seems to have been built in august16:17
*** KurtMartin is now known as kmartin16:17
sc68calalso this review at 8:36AM EST
fungisc68cal: that's because there was no recent check result on the change (within the last 48 hours) so it's being automatically rechecked to make sure it doesn't merge conflict ot have any new issues with the current branch tip code16:18
*** reed has joined #openstack-infra16:18
sc68caloooh - that's nice, especially for alembic migrations16:18
sc68calsince those can get stale real quick16:18
*** thedodd has joined #openstack-infra16:19
fungisc68cal: this prevents core reviewers from blindly approving stale changes without some recent assurance that it will have a chance of passing gating jobs16:19
sc68calThat's awesome!16:19
fungisc68cal: new behavior as of 5 days ago. jeblair implemented it. i agree, very neat stuff16:19
*** david-lyle has quit IRC16:20
*** sdake has quit IRC16:20
*** sdake has joined #openstack-infra16:21
*** sdake has quit IRC16:21
*** sdake has joined #openstack-infra16:21
fungi#status ok jobs are running for changes again, but there's a bit of a backlog so it will still probably take a few hours for everything to catch up16:21
openstackstatusNOTICE: jobs are running for changes again, but there's a bit of a backlog so it will still probably take a few hours for everything to catch up16:21
*** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs | Bugs | Code"16:21
*** DeeJay1 has quit IRC16:21
*** mrda_away is now known as mrda16:22
*** virmitio has joined #openstack-infra16:22
*** david_lyle_ has quit IRC16:24
*** johnthetubaguy has joined #openstack-infra16:25
openstackgerritA change was merged to openstack-infra/reviewstats: Remove tox locale overrides
*** dprince_ has joined #openstack-infra16:28
*** w_ is now known as olaph16:29
*** dprince has quit IRC16:29
*** beekneemech is now known as bnemec16:30
fungiall active changes in the gate pipeline finally have nodes assigned for all their jobs, and zuul is working through the check backlog16:31
*** reed has quit IRC16:31
*** reed_ has joined #openstack-infra16:31
*** CaptTofu has quit IRC16:35
*** reed_ has quit IRC16:36
*** bhuvan has joined #openstack-infra16:37
*** thuc has joined #openstack-infra16:38
*** thuc has quit IRC16:39
*** dizquierdo has quit IRC16:39
*** thuc has joined #openstack-infra16:39
*** oanufriev has quit IRC16:42
*** lcostantino has quit IRC16:43
*** markmcclain has joined #openstack-infra16:43
*** lcostantino has joined #openstack-infra16:44
*** markmcclain1 has joined #openstack-infra16:46
*** igor has quit IRC16:47
*** AlexF has quit IRC16:47
*** markmcclain has quit IRC16:47
*** igor has joined #openstack-infra16:47
*** lcostantino has quit IRC16:48
*** talluri has joined #openstack-infra16:48
*** lcostantino has joined #openstack-infra16:48
*** jooools has quit IRC16:50
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Remove tox locale overrides
*** jooools has joined #openstack-infra16:50
*** igor has quit IRC16:52
*** gyee has joined #openstack-infra16:53
zarofungi: any concerns with my last comment here:
uvirtbotLaunchpad bug 979227 in openstack-ci "convert gerrit's mysql tables to utf8" [Medium,Triaged]16:54
*** sarob has joined #openstack-infra16:55
*** lcostantino has quit IRC16:55
*** lcostantino has joined #openstack-infra16:56
lcostantinoHi there, qq: i am getting -1 on some reviews because it has a depdency. Does gerrit take care of that and won't let jenkins attempt to merge that until the dependency lands? Or will try to do it and just fail.16:56
*** dprince_ has quit IRC16:57
jeblairlcostantino: it won't merge until the dependency merges.  zuul won't even start running merge tests until they are both approved16:57
*** gokrokve has quit IRC16:57
lcostantinojeblair, thanks that i wanted to know.16:57
*** smarcet has quit IRC16:58
*** saju_m has quit IRC16:59
*** sarob has quit IRC17:00
*** sarob has joined #openstack-infra17:00
zarojeblair: would like your feedback on bug 97922717:01
uvirtbotLaunchpad bug 979227 in openstack-ci "convert gerrit's mysql tables to utf8" [Medium,Triaged]
*** lcostantino has quit IRC17:02
*** jcoufal has quit IRC17:02
fungizaro: it may be a good question for #gerrit... i doubt gerrit relies on string collation order for anything of consequence, but it would be good to find out from them17:02
*** cadenzajon has joined #openstack-infra17:02
*** gokrokve has joined #openstack-infra17:03
fungicase (in)sensitivity collisions are a bigger concern for me17:03
*** saju_m has joined #openstack-infra17:04
*** CaptTofu has joined #openstack-infra17:04
*** yamahata has joined #openstack-infra17:04
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Cancel obsolete builds on reconfiguration
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Remove double accounting of builds
*** markwash has joined #openstack-infra17:05
fungijeblair: i assume 72042 solves what we saw with pending tripleo jobs after they were removed from the layout?17:05
*** DinaBelova_ is now known as DinaBelova17:06
jeblairfungi: yes, then 72043 solves it better17:06
jeblairwell, more cleanup and simplification really17:06
jeblairzaro: looking17:07
*** david-lyle has joined #openstack-infra17:08
jeblairzaro: what kind of duplication do you see?17:08
*** yamahata has quit IRC17:08
jeblairzaro: (when going from cs to ci)17:08
jeblairzaro: (which tables/what kind of data in the rows)17:08
openstackgerritPhilippe Godin proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.
zarofungi: good idea. i'll do that.17:09
*** yamahata has joined #openstack-infra17:09
*** andreaf has joined #openstack-infra17:09
*** markmcclain1 has quit IRC17:09
*** markmc has quit IRC17:09
*** smarcet has joined #openstack-infra17:10
*** markmcclain has joined #openstack-infra17:10
*** e0ne_ has quit IRC17:10
*** dprince has joined #openstack-infra17:10
zarojeblair: there's quite a few of them, sorry i didn't record every dup error.  but i do remember that there were duplicate entries for emails and commit messages.17:11
jeblairwhy would commit messages have a unique constraint?17:12
zarosorry, i could be wrong.  let me try to get a history and be more accurate.17:13
*** rossella_s has quit IRC17:13
jeblairemails, however, are case-sensitive17:13
*** coolsvap has joined #openstack-infra17:14
fungiat least the local part (the domain part is case insensitive, but the string overall needs case sensitivity because of the local part)17:14
*** dangers_away is now known as dangers17:14
ArxCruzAaronGr: around?17:15
*** hashar has quit IRC17:15
AaronGryes, good morning17:15
fungiwow, i love receiving out-of-office autoreplies for code review changes i've submitted17:15
fungimakes my day17:16
dhellmanngood morning/afternoon17:16
*** reed has joined #openstack-infra17:16
dhellmannis the process for automatically importing git repos still borked?17:17
dhellmannI'd like to see about moving ahead with oslo-cookiecutter if possible:
fungidhellmann: semi-borked. it works when triggered by hand, and i identified some of the ways in which it's breaking under puppet yesterday17:18
*** gokrokve has quit IRC17:18
fungiand anteaya has a patch up to get it to log its error messages from stdout/stderr when exec'd under puppet17:18
jeblairdhellmann: we'll run it by hand for openstack projects; i don't intend to let it impact the project17:19
fungiwhich i have tested and seems to at least work for manual puppet agent --test invocations17:19
jeblairdhellmann: (stackforge projects may be slower to be created though)17:19
dhellmannok, thanks17:19
zarojeblair, jeblair : so the other dups i saw were for userid and change messages17:19
fungidhellmann: reviewing now17:19
*** saju_m has quit IRC17:19
malinidtroyer: ping17:19
dhellmannI don't have any stackforge projects to add for now, but that one is a semi-blocker for automating some of the graduation steps for oslo libs17:19
dhellmannI could just use the github repo, but you know17:20
jeblairdhellmann: please don't, just let us know if something is blocking (as you are doing) :)17:20
jeblairzaro: why would a change message have a unique constraint?17:20
*** dkliban has quit IRC17:20
dhellmannjeblair: yep, not blocked yet, but that's why I was asking :-)17:21
jeblairdhellmann: so i guess we'll just want to try to keep these in sync for the parts that are common?17:21
dhellmannjeblair: yeah, I don't know a good way to do that17:22
dhellmannthe other repo is already somewhat out of date, but I don't remember the details about that17:22
dhellmannit seemed like I had to make a few changes to make it work17:22
fungidhellmann: mordred: jeblair: can openstack-dev/cookiecutter have multiple templates maybe?17:22
dhellmannfungi: that would be ideal, but cookiecutter takes a git url as an arg17:22
fungiahh, poop17:23
dhellmannit *can* take a directory, but I'm afraid it won't be used that way17:23
dhellmannbecause of the extra step of checking out the repo17:23
dhellmannI'm sure Audrey would be open to suggestions if we can come up with a way to say "this directory inside this git repo"17:23
dhellmannmaybe we can make the root of the repo not valid, and put each template in a subdir, for example17:24
dimsjeblair, could use your feedback on oslo.vmware ( - there are a few reviews against oslo-incubator that are stuck and will need to be rebased to oslo.vmware repo when we get it ready17:25
*** whoops has joined #openstack-infra17:25
dtroyermalini: yo17:26
fungidhellmann: looking at your github repo, you'll probably want to also add something like
malinidtroyer: thanks for the review17:26
*** jpich has quit IRC17:26
malinido you want me to update with a new patch set or add a dependent patch ?17:27
*** apevec has joined #openstack-infra17:27
jeblairdims: lgtm17:27
*** salv-orlando has quit IRC17:27
dhellmannfungi: ok, I can do that now if you haven't already approved the import17:27
jeblairfungi, anteaya: how close are we to having m-p logging?17:27
malinidtroyer: not sure how this is usually done17:28
anteayajeblair: well I think fungi said that logging works17:28
dimsjeblair, thanks!17:28
dtroyermalini: You can actually just change to add the is_marconi_enabled() function, it need not be dependent as nothing changes until is merged.  No need to make it dependent.17:28
mordredfungi: /me just reading - why do yuo want more than one template?17:28
anteayawith the patch I offered which he tried by hand17:28
*** sarob has quit IRC17:28
anteayajeblair: seems the m-p script is hitting a github request limit, which is set at 5000/hour17:29
jeblairanteaya: awesome, thx17:29
*** yassine has quit IRC17:29
fungimordred: oslo projects are apparently special and need to differ from other cookiecutter projects, per
dhellmannfungi: pushed17:29
mordredah. well, yeah - I think we need different repos17:29
*** sandywalsh has joined #openstack-infra17:30
malinidtroyer: thanks! Do you expect 69497 to be merged soon? We are trying to get through the marconi graduation requirements & would like to know sooner if we need something else17:30
*** salv-orlando has joined #openstack-infra17:30
*** lcostantino has joined #openstack-infra17:30
fungijeblair: so logged in the puppet output when applied from a dev environment and run under puppet agent --test manually yesterday. not sure whether it will also log to syslog with other agent messages when run under puppet agent daemon17:31
jeblairfungi: oh i see, neat.17:31
fungijeblair: but the easiest way to tell is to reenable the exec for it in puppet and let it run, i suppose17:31
dtroyermalini: I want it to be…I'll prod chmouel and sdague to look at it again...17:32
malinithanks dtroyer!!17:32
jeblairfungi, anteaya: it depends on a WIP patch17:32
fungijeblair: also, i'm quite positive that 10 minutes is no longer nearly enough for a full manage-projects pass17:32
fungijeblair: the wip patch is the one to turn the exec back on17:32
anteayajeblair: fungi suggested I leave the reversion as WIP until further notice17:32
*** saju_m has joined #openstack-infra17:32
jeblairfungi: it looks like the timeout is 900 secs17:32
jeblairfungi: is that enough17:32
anteayaI can un-WIP if you wish17:32
jeblairanteaya: where did you see that we are hitting a rate limit?17:33
fungijeblair: ahh, yes 900 will just make it (we were hovering around 700-800 when i was testing)17:33
anteayafungi saw it17:33
fungijeblair: i was getting it in the logs with a link to github's faq about authenticated queries being limited to 5000 per hour17:33
fungiwhen rerunning manage-projects multiple times within the span of an hour17:34
anteayafungi: I had thought you had put that in a bug report comment, had you not?17:34
jeblairthat's a lot of queries17:34
fungianteaya: i mentioned it in my last review comment on 7215817:34
zarojeblair: i don't know the answer to your question, but can take a look at the tables to see if i can find the unique contraint17:35
anteayathere it is17:35
fungibut the github rate limit was running it 4 or 5 times in the span of an hour17:35
jeblairfungi: ok, so not an immediate problem17:35
*** elasticio__ has joined #openstack-infra17:36
fungiso i'm guessing we're running somehwere in excess of 1000 github queries for a full run17:36
*** elasticio__ is now known as Guest4451417:36
fungijeblair: no, not an immediate concern, but certainly something we need to keep in mind as we continue to grow our project base. we're within sight of the 15-minute timeout and the 5000 github request limits17:36
*** lcostantino has quit IRC17:37
fungibut yeah, next step i think is to merge 72158 and the revert in 72178 and then see if it logs when it breaks17:37
*** fbo is now known as fbo_away17:37
*** lcostantino has joined #openstack-infra17:38
fungibut wanted to discuss it before recommending to anteaya to take that revert out of wip17:38
jeblairfungi, anteaya: both lgtm17:38
anteayaso take revert out of wip?17:38
fungianteaya: already did17:38
fungiand approved17:39
fungilet's see what transpires17:39
anteayaokay thanks17:39
anteayaI have to have something to eat and then a walk17:39
anteayaback in a bit17:39
*** AlexF has joined #openstack-infra17:40
*** mkoderer has quit IRC17:41
*** johnthetubaguy has quit IRC17:41
*** markmcclain has quit IRC17:42
*** andre__ has quit IRC17:42
*** mkoderer has joined #openstack-infra17:43
*** dkliban has joined #openstack-infra17:44
*** moted has joined #openstack-infra17:45
apevecis launchpad having issues or is just me?17:47
apeveceven openid login via launchpad hangs17:47
jeblairfungi, clarkb, mordred:
openstackgerritA change was merged to openstack-dev/hacking: Remove tox locale overrides
openstackgerritA change was merged to openstack-dev/hacking: Updated from global requirements
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs
fungijeblair: thanks!17:52
openstackgerritA change was merged to openstack-dev/hacking: Clarify H403 message
mordredjeblair: looking17:54
*** smurugesan has joined #openstack-infra17:55
mordredjeblair: +2 (I'm away from ability to respond to that via email, but I am strongly +2)17:55
jeblairmordred: what happened to your irc-to-email gateway?17:56
clarkbjeblair: did you catch the logstash crm114 problem in scroll back? I am about to catch up on my sb too17:56
jeblairclarkb: no17:56
mordredjeblair: god I need one17:57
clarkbjeblair: I submitted a bug which I will dig up, but tl;dr is that logstash claims the select timeout is being hit without data to read so we don't index the remaining data for that file17:57
clarkbjeblair: I did a quick debug of the problem, but didn't really sort anything out (probably need to run crm by hand against the files that timeout)17:57
*** Steap_ has joined #openstack-infra17:57
uvirtbotLaunchpad bug 1278210 in openstack-ci "CRM114 timeouts preventing logs from being indexed" [High,Triaged]17:58
openstackgerritA change was merged to openstack-infra/config: Revert "Stop running manage-projects"
*** igor_ has joined #openstack-infra17:58
openstackgerritA change was merged to openstack-infra/config: Add logging to manage-projects
jeblairclarkb: we don't _index_ or we don't add error_pr values?17:58
*** harlowja_away is now known as harlowja17:58
clarkbjeblair: we don't index because of where the exception is caght17:59
openstackgerritJames E. Blair proposed a change to openstack-infra/jeepyb: Allow welcome-message to configure logging
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Enable verbose output for welcome message
jeblairclarkb: whoops.  yeah, we should catch that, log, and proceed18:00
*** amcrn has joined #openstack-infra18:01
clarkbjeblair: ok, I can work on that patch18:01
*** derekh has quit IRC18:02
jeblair(then secondarily, check into why it timed out)18:02
*** igor_ has quit IRC18:02
*** markmcclain has joined #openstack-infra18:04
*** johnthetubaguy has joined #openstack-infra18:05
openstackgerritA change was merged to openstack-infra/storyboard: Added base for functional tests
jeblairclarkb: with the active window, i wonder if we should consider switching gate back to high gearman precedence18:08
clarkbjeblair: oh, good point18:09
clarkbin theory that prevents unnecesary starvation18:09
fungijeblair: clarkb: i think it's worth revisiting. we no longer consume enough nodes in a gate reset to starve the check pipeline18:09
jeblairclarkb, fungi: also, this is a bad test since we had the nodepool fubar, but i have a suspicion that monday mornings will have a high check load because of the freshness requirement.18:10
fungilikely so18:10
*** e0ne has joined #openstack-infra18:11
*** saju_m has quit IRC18:12
clarkbjust like freshly squeezed OJ18:15
*** jp_at_hp has quit IRC18:16
fungiour check queue is high in vitamin c too18:16
*** oanufriev has joined #openstack-infra18:20
*** e0ne_ has joined #openstack-infra18:21
*** e0ne has quit IRC18:21
openstackgerritClark Boylan proposed a change to openstack-infra/config: Handle log filter exceptions more gracefully.
clarkbjeblair: fungi ^ going to apply that on logstash-worker0118:21
clarkb:( forgot to set related-bug in the commit message, will update18:22
*** CaptTofu has quit IRC18:22
jeblairclarkb: one worry: will that cause a 20 second timeout on every subsequent line?18:22
jeblairclarkb: oh heh18:23
jeblairclarkb: i see you handle it with new_filters18:23
*** max_lobur is now known as max_lobur_afk18:23
clarkbyeah :)18:23
openstackgerritBhuvan Arumugam proposed a change to openstack-infra/config: log analyzer for openstack IRC logs
jeblairttx: are you ready for to merge?18:24
*** gokrokve has joined #openstack-infra18:27
openstackgerritA change was merged to openstack-infra/config: Import oslo-cookiecutter repository
*** apevec has quit IRC18:27
fungiwatching the puppet logs on review.o.o now as that ^ gets applied18:28
jeblairSergeyLukjanov: ping18:28
*** saju_m has joined #openstack-infra18:29
*** e0ne_ has quit IRC18:31
*** e0ne has joined #openstack-infra18:31
*** alexpilotti has joined #openstack-infra18:31
*** CaptTofu has joined #openstack-infra18:34
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Support filtering by review id(s)
*** sarob has joined #openstack-infra18:34
*** rwsu has quit IRC18:36
*** freyes has joined #openstack-infra18:38
sdaguejeblair: question on the recheck bits, where does clean check for gate stand?18:39
openstackgerritBhuvan Arumugam proposed a change to openstack-infra/config: log analyzer for openstack IRC logs
jeblairsdague: just un-wip'd the change18:39
*** gokrokve has quit IRC18:40
*** Sukhdev has joined #openstack-infra18:43
sdaguejeblair: what does the middle comment-added block mean?18:43
jeblairfungi: what do you think of climate using "openstack-climate" as the rtfd project name?  that seems premature to me.18:43
*** VijayT has joined #openstack-infra18:43
sdaguerequire approval: verified: 1 on approval: verified: 1 ?18:43
fungijeblair: somewhat agree--i hadn't see that change yet18:44
fungijeblair: if "climate" were a descriptive term, then i'd be more okay with it18:44
*** Guest44514 has quit IRC18:45
fungijeblair: but this isn't "the climate you want when using openstack" because climate is itself already a fairly nondescriptive codename18:45
jeblairsdague: so that when jenkins votes +1 at the completion of a check run it will be enqueued into the gate.18:45
*** zigo has quit IRC18:45
jeblairsdague: (strictly speaking, any +1 vote (including from 3rd party ci) would trigger this too; we could probably narrow that down)18:47
jeblairsdague: in all cases, the normal requirements to be enqueued (can merge, has approved vote, has +2 code review and no -2 code review) still apply18:47
sdaguejeblair: ok, I guess it's not obvious to me from the dsl18:47
*** gokrokve has joined #openstack-infra18:49
*** zigo has joined #openstack-infra18:49
jeblairsdague: "a comment-added event with an approval vote of verified=1 will trigger the pipeline iff there is also an approval of verified=1 from jenkins within the last 48h"18:50
*** pafuent has joined #openstack-infra18:50
*** rwsu has joined #openstack-infra18:51
*** saju_m has quit IRC18:51
clarkbjeblair: my logstash-worker change seems to work, I am going to self approve the change and get that running everywhere18:51
jeblairclarkb: cool.  i did +2 it i think.  :)18:52
*** saju_m has joined #openstack-infra18:52
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Require a recent (48h) positive check result
jeblairsdague: there, that's more specific: "a comment-added event with an approval vote of verified=1 from jenkins will trigger the pipeline"18:53
*** morganfainberg_Z is now known as morganfainberg18:53
*** burt1 has quit IRC18:53
sdagueok, I get it now.18:55
sdagueit still has to handle sufficiency separately18:55
*** exz has quit IRC18:55
jeblairthat's magic that gets invoked if 'submit' is set on a pipeline18:55
*** saju_m has quit IRC18:58
*** talluri has quit IRC18:59
*** talluri has joined #openstack-infra18:59
*** johnthetubaguy has quit IRC19:00
clarkbbtw is a thing19:01
clarkbI figured hg out enough to get that far19:01
*** sarob has quit IRC19:01
clarkbdhellmann: mordred jeblair ^19:01
jeblairclarkb: from what i hear from mordred, hopefully you won't have to update that.  :)19:01
*** markmcclain has quit IRC19:01
clarkbjeblair: I had to update it once already, its basically the github PR model where you do a new commit and add it to the PR, the difference is that hg won't let you squash them best I can tell19:02
*** markmcclain has joined #openstack-infra19:02
reedjeblair, fungi: can you please give your opinion to ?19:02
jeblairoh, ew.19:02
clarkband if that PR gets rejected on silly grounds I will probably be forking with that as the tip of master :)19:03
jeblairreed: yep, it's about 10 changes down my list; though one thing i can see before i've reviewed it is we need to point them at19:03
*** talluri has quit IRC19:03
*** AlexF has quit IRC19:04
*** johnthetubaguy has joined #openstack-infra19:04
*** dcramer_ has joined #openstack-infra19:04
*** pafuent1 has joined #openstack-infra19:06
*** pafuent has quit IRC19:06
*** pafuent1 has quit IRC19:07
reedjeblair, agreed, it's not a good commit message... keep in mind we need that patch when you review it, if you can, avoid -1/2ing just because of the commit message19:07
*** lcostantino has quit IRC19:09
*** lcostantino has joined #openstack-infra19:10
jeblairreed: i'm the last person to needlessly -1 something on style grounds, but that really does need a better commit message and i will probably -1 it for that.  it's more than mere style, it's functionally deficient.  but it's easy to fix so it shouldn't be a big deal.19:10
fungidoes <VirtualHost *:443> work these days?19:11
reedjeblair, the big deal is that any extra step requires 2+ days to be evaluated and that is awful19:11
jeblairreed: they work every other day?19:12
clarkbfungi: yes I believe it does19:12
reedjeblair, marton took over the change, he's in EU and infra review queue is just long19:12
fungii need to fix up my apache https vhost configs for that eventually19:13
*** sarob has joined #openstack-infra19:13
jeblairreed: yeah, i know it's long.  that's why i'm reviewing changes now.  or at least i was.  but we're not going to just merge stuff that's broken and not maintainable.  we do review for a reason.19:14
lcostantinoanother qq for this gerrit newbie. Is the normal behavior to get all dependencies review updated after pushing change? ( i cannot find anything explicit on the wiki ). For instance A->B->C , i pushed C and A/B cleared all the +1. Is that right?19:14
reedjeblair, one request to edit anything and the change goes back to the bottom of the queue19:14
jeblairreed: that's not how i run my queue.19:14
reedjeblair, is the commit message the only thing?19:14
jeblairreed: i don't know, i haven't reviewed it yet.  it's still 10 changes away.19:15
clarkblcostantino: short answer is it depends :) any time the git commit sha1 updates, the gerrit changes will get new patchsets. So if you had to rebase the stack A,B,C to resolve a merge conflict all will update19:15
jeblairrussellb: but i will review the whole thing and leave comments on everything i see in order to minimize the number of round trips.19:15
reedjeblair, thanks, see what you can do19:15
jeblairreed: ^19:15
clarkblcostantino: if there was no rebase and merge conflict then it shouldn't update (ensure you are using the latest release of git review to get that behavior, old releases always rebased)19:15
openstackgerritA change was merged to openstack-infra/config: Handle log filter exceptions more gracefully.
reedjeblair, I'll ping marto, let him know that he may need to edit the commit and hope he can do it before going to bed19:16
jeblairreed: i don't like re-reviewing things any more than anyone likes revising them.19:16
lcostantinoclarkb,  thanks. This happened when i just created the new C review.  So i think i am getting hit but the gerrit version then19:16
lcostantinoclarkb, which version should be the right one?19:16
clarkblcostantino: 1.23 is the latest19:16
lcostantinoclarkb, thanks19:16
fungigrr... so openstack-dev/oslo-cookiecutter got created and its acl applied in gerrit, but says "Repository seems to be empty" and the puppet agent log didn't mention anything between "(/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]) Triggered 'refresh' from 2 events" and "Finished catalog run in 684.92 secon19:17
fungiso either manage-projects didn't actually output an error, or the exec logoutput option in puppet only works when you puppet apply --test but doesn't get copied to the syslog stream19:18
fungiAaronGr: nibalizer: ^ do you know much about exec logoutput behavior?19:18
*** johnthetubaguy has quit IRC19:19
fungiif it's the latter, then i think manage-projects needs python logging configuration so it can log on its own19:19
*** mburned has joined #openstack-infra19:19
* fungi gets to work cleaning up the oslo-cookiecutter git repo mess19:20
nibalizerhrm, never used that parameter, gimmie a sec19:20
nibalizerfungi: where is this code?19:21
funginibalizer: thanks!
*** smarcet has quit IRC19:22
*** smarcet has joined #openstack-infra19:23
*** bhuvan has quit IRC19:23
fungithough in retrospect, i may have been too hasty to suspect breakage from manage-projects itself... the git repositories on review and also on zuul look correct19:24
fungimaybe create-cgitrepos is failing on the git farm19:25
nibalizerfrom the puppet docs, its unclear if true means on_failure or if false means on_failure19:25
*** melwitt has joined #openstack-infra19:25
nibalizerbut i imagine that true means log the output19:25
jeblairfungi: or is it just that replication happened out of sequence?19:25
*** beagles is now known as beagles_brb19:26
nibalizerya, logoutput dumps the entire output of the exec19:27
fungijeblair: seems possible. create-cgitrepos runs clean on all of the git farm. i manually did a gerrit replicate openstack-dev/oslo-cookiecutter and is okay now19:28
*** NikitaKonovalov_ is now known as NikitaKonovalov19:28
*** NikitaKonovalov is now known as NikitaKonovalov_19:28
funginibalizer: thanks for double-checking! i'll continue to keep an eye on it. maybe we want manage-projects to spit something on stdout rather than silence on success, to exercise that19:28
fungidhellmann: oslo-cookiecutter is hopefully open for business19:29
* fungi needs to step away for a bit and run a couple errands. back soon19:29
nibalizerso are ya'll syslogging your puppet output then?19:29
nibalizerI generally reccomend against that for security19:30
jeblairnibalizer: can you elaborate?19:30
*** thedodd has quit IRC19:30
nibalizeroften you have an exec somewhere, mysql module used to do this (not sure if it still does), that shells out to create the user/password19:31
nibalizerif that fails for whatever reason, the shell command with password expanded will be dumped to puppet output, then to syslog19:31
nibalizernow if you have encrypted logs and all that, no worries19:31
jeblairnibalizer: ah.  well we would need to avoid things like that then because moreso than syslog, we actually want all the puppet logs available on the web.19:32
nibalizeryea, thats why i brought it up19:32
nibalizerits possible you dont have this problem19:32
jeblairnibalizer: they used to be until puppet-dashboard broke; there are efforts to replace that though.19:32
jeblairnibalizer: or it's possible we do and didn't know about it.  :)19:32
nibalizerdon't they go to clarkb's logstash?19:32
*** ruhe_ has joined #openstack-infra19:32
jeblairnibalizer: no, that's only test runs19:33
*** jamesmcarthur has joined #openstack-infra19:33
nibalizeri think AaronGr is scheming to bring up puppetboard, which i totally +119:33
jeblairyes, Hunner was working on that as well...19:34
jeblairHunner and AaronGr should probably connect on that.  anteaya too.19:34
*** chris_johnson is now known as wchrisj|away19:35
SergeyLukjanovjeblair, pong, thank you :)19:36
jeblairSergeyLukjanov: no, thank you.  :)19:36
nibalizerpuppetboard can help you identify failing nodes and failing resources on those nodes, which is really all we need from a puppet dashborad19:36
nibalizerplus i have a half-dozen commits in it, so i guess im biased19:36
jeblairnibalizer: it can display logs too, right?19:37
*** nicedice_ has joined #openstack-infra19:37
SergeyLukjanovjeblair, AFAIK yes19:37
nibalizeri dont think it has a page with just the raw text of the puppet output19:37
jeblairnibalizer: basically i want AaronGr to be able to do whatever he wants to the system without an infra-root needing to go and look at logs19:38
SergeyLukjanovruhe, ping, I remember that you was talking about puppetboard19:38
jeblairnibalizer: oh, er, how do you use it to diagnose failures then?19:38
*** markmcclain has quit IRC19:38
nibalizerpuppetboard will get him most of the way there19:38
openstackgerritMatt Ray proposed a change to openstack-infra/config: Create new Chef cookbook-openstack-integration-test for Tempest support.
*** saju_m has joined #openstack-infra19:38
ruhe_SergeyLukjanov: i've deployed puppetboard manually, and i haven't seen any logs on the dashboard19:39
nibalizerwell because you see the resource thats failing and the text of its failure, you have all the same information, just tabled and clickable19:39
*** mrmartin has joined #openstack-infra19:39
jeblairnibalizer: ok, good, the text of the failure; that should be sufficient.19:39
ruhe_SergeyLukjanov: but it can tell you that some deployment went wrong. that's what deployer/admin actually needs19:40
*** pafuent has joined #openstack-infra19:40
*** nicedice has quit IRC19:40
SergeyLukjanovruhe_, could it share info about props, env, etc/19:41
nibalizeralso if you all want to review that would be awesome19:41
*** wchrisj|away is now known as chris_johnson19:42
nibalizerand here is the basics of the plan AaronGr cooked up to improve the openstack puppet19:42
ruhe_SergeyLukjanov: probably yes, because it has a query interface for puppetdb. unfortunatelly VM with puppetboard went down and i can't check it now19:42
pafuentHi. Can someone take a look at this review
clarkblogstash workers 01-06 are happy now. logstash itself is about half an hour behind. Going to spin up workers on 07 and 08 to help pick up the backlog19:43
openstackgerritValeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job
pafuent, will enable gate jobs for climate project19:45
*** thuc has quit IRC19:46
jeblairSergeyLukjanov: i had a question about
*** thuc has joined #openstack-infra19:46
jeblairSergeyLukjanov: i'm pretty sure i've asked this before, but i forgot; sorry...19:46
jeblairSergeyLukjanov: why do you need to run that as a 3rd-party test instead of running it in openstack-infra?19:47
*** DinaBelova is now known as DinaBelova_19:47
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra
SergeyLukjanovjeblair, tl;dr - we need nested virt and up to 7 2vcpu 2GB RAM vms for running one of 5 scenarious19:48
*** zigo has quit IRC19:48
SergeyLukjanovjeblair, I've checked hp and rackspace vms that are used for gating, Hadoop could not start on nested vms on both of them19:49
*** thuc has quit IRC19:50
jeblairSergeyLukjanov: ok.  if we can run a job on multiple vms in the future, we could probably do this upstream then, right?19:50
jeblair(that's something I'd like to look into after we stop using jenkins)19:51
*** smarcet has quit IRC19:52
*** exz has joined #openstack-infra19:52
SergeyLukjanovjeblair, if such vms will support nested virt acceleration19:52
SergeyLukjanovjeblair, the another approach is to use TripleO19:52
*** johnthetubaguy has joined #openstack-infra19:53
*** smarcet has joined #openstack-infra19:53
jeblairSergeyLukjanov: oh, you don't need nested virt just because you need lots of vms, you need it because you need to deploy openstack and then within openstack delpoy hadoop vms; so the devstack-deployed version needs to be something can can nest a vm19:54
jeblairSergeyLukjanov: in that case, yes, i agree tripleo might be the best bet in the future19:54
*** DinaBelova_ is now known as DinaBelova19:54
SergeyLukjanovjeblair, yeah19:54
jeblairSergeyLukjanov: okay cool.  thanks, and sorry i forgot.  :(19:55
SergeyLukjanovjeblair, in fact currently we're running all of it on one 24 CPU, 128 GB RAM server with devstack :)19:55
SergeyLukjanovjeblair, np, I need to recheck and document all issues some day ;)19:55
*** markmcclain has joined #openstack-infra19:58
*** hashar has joined #openstack-infra19:58
*** saju_m has quit IRC20:01
openstackgerritMatt Ray proposed a change to openstack-infra/config: Create new Chef cookbook-openstack-integration-test for Tempest support.
*** openstack has joined #openstack-infra21:33
jeblairoh look it's retrying21:33
jeblairopenstack: welcome back21:33
fungigood to know it re-resolves names21:33
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add non-isolated serial periodic tempest job
openstackgerritDoug Hellmann proposed a change to openstack-infra/devstack-gate: Add cliff, pycadf, stevefore & taskflow
*** mattray has joined #openstack-infra21:36
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Set periodic tempest jobs to run with master
openstackgerritA change was merged to openstack/requirements: Adds python-seamicroclient dependency for Ironic
anteayafungi: I am drawing together my thoughts on what I need to mock a run of manage-projects:
anteayado drop by if you feel so inclined21:42
*** amcrn_ has joined #openstack-infra21:44
*** amcrn has quit IRC21:45
*** gokrokve has quit IRC21:46
*** gokrokve has joined #openstack-infra21:46
anteayafungi: thanks21:46
anteayado I need the vm to be a minumum size?21:47
anteayaor will a small or medium flavour suffice21:47
*** amcrn_ is now known as amcrn21:48
fungianteaya: a 2gb vm will almost certainly work for basic gerrit/jeepyb testing21:48
*** eharney has quit IRC21:49
anteayagreat thank you21:49
fungi1gb or smaller would probably even be fine (i ran it at home for a while in a vm with 256mb ram and 512mb swap and a very slow nfs root filesystem with reasonable performance)21:49
*** chris_johnson is now known as wchrisj|away21:50
openstackgerritA change was merged to openstack-infra/config: Make files in reviewday docroot accessible
fungipresumably the goal is to profile the various github api calls made when running manage-projectys for one single project, and then multiply that by the couple hundred projects we have to get a rough count21:50
*** gokrokve has quit IRC21:50
fungimordred suggested possibly splitting manage-projects into a couple of separate moving parts to allow the github and similar interactions to occur asynchronously and at a different frequency and speed to avoid running into rate limiting issues21:51
openstackgerritA change was merged to openstack/requirements: Update README with rationale and internal mirror
*** hashar has joined #openstack-infra21:52
anteayamakes sense21:52
anteayaand yes, I would like to see what happens where, so that when the splitting happens the calls are distributed21:52
anteayanot still left in one component21:53
openstackgerritSteve Baker proposed a change to openstack-infra/config: tempest-dsvm-neutron-heat-slow graduates from experimental
stevebakerjeblair, fungi, I fixed the commit message ^21:55
fungistevebaker: awesome. lgtm21:56
openstackgerritA change was merged to openstack/requirements: Remove tox locale overrides
jeblairjog0: i think could be done with only one web page fairly easily, and just a js function call when selecting a different pipeline... did you consider that?21:57
jog0jeblair: I did but my JS is bad21:58
jog0I tried21:58
jog0and this was easier21:58
jog0not bette per se21:58
jog0it also opens us up for other links21:58
jog0sdague: ^21:58
sdaguejeblair: in general we've been going down the path of multi pages in e-r, and trying to move logic out of javascript21:58
*** mfer has quit IRC21:59
sdagueespecially because our contributor set is more python familiar than js21:59
jeblairsdague: okay, but in this case you're literally only changing the url from which the json blob is fetched; a duplication of the wrapping html seems overkill21:59
*** hashar has quit IRC22:00
sdaguejeblair: so honestly we probably want to get to a template model here22:00
sdaguehonestly, I'd be fine on this as incremental change as it will get useful UI out there right now, and we can refactor the back end later22:02
jeblairjog0: thanks, i just wanted to make sure the approach wasn't discarded for some reason; i would prefer to avoid duplication, but i don't immediately have time to work it up myself and i won't ask you to either22:02
jeblairsdague: wfm22:02
*** pdmars has quit IRC22:03
*** smurugesan has joined #openstack-infra22:04
jeblairclarkb: yay!22:05
*** apevec has joined #openstack-infra22:06
*** jhesketh has joined #openstack-infra22:06
*** gokrokve has joined #openstack-infra22:07
jeblairjhesketh: hello there22:07
*** johnthetubaguy has joined #openstack-infra22:07
*** ywu has joined #openstack-infra22:08
stevemarquestion for the infra folks: how could we go about gating against warnings/errors in docs for keystone ?22:08
jeblairstevebaker: errors should already be blocked by the gate-keystone-docs job...22:09
jeblairstevemar: ^22:09
wenlockmordred, clarkb, was updating the review and I came across storyboard::init section where story board is installed with pip... any reason why we wouldn't make that a package install with provider pip with reference to the clone directory?22:09
* stevebaker goes back to sleep22:09
dolphmjeblair: for openstack/keystone/docs ?22:09
jeblairstevemar: about warnings; i'm not sure if the needed bits were added to pbr....22:10
*** salv-orlando_ has joined #openstack-infra22:10
jeblairdolphm: like this job
stevemarjeblair, specifically warnings for improper RST syntax for docstrings22:11
dolphmjeblair: the log is full of non-fatal ERRORs and WARNINGs that we'd like to gate against22:11
stevemarjeblair, yes, what dolphm said22:11
dolphm"build succeeded, 6 warnings." <-- 3 of those are actually logged as "ERRORs"22:11
stevemarjeblair, and hopefully make oslo play nice too :)22:12
jeblairi wonder if sphinx counts them all as warnings, even the errors22:12
dolphmstevemar: ++ might just be able to exclude keystone/openstack/common from the doc job22:12
clarkbjeblair: dolphm stevemar I thought pbr already did that for us and it not doing so is a bug22:13
stevemardolphm, oh good point22:13
jeblairclarkb: yeah, i see "warningiserror=True" in pbr22:14
jeblairso i'm trying to find out what's up with that22:14
*** salv-orlando has quit IRC22:14
*** salv-orlando_ is now known as salv-orlando22:14
anteayahey jhesketh22:14
*** rlandy has quit IRC22:15
openstackgerritClark Boylan proposed a change to openstack-infra/config: Give test mysql user create and drop perms.
clarkbnow with resolved merge conflicts ^22:17
jeblairoh i think 'warnerrors' is needed to enable that function22:17
jeblairwhat in the world is this?22:20
jeblairdolphm, stevemar: ^22:20
*** dangers is now known as dangers_away22:20
jeblairmordred: ^22:20
jeblairdolphm: if i'm reading that right, rather than submitting a two line patch to pbr, the author added a new private sphinx extension to keystone?22:21
dolphmdstanek: ^22:22
jeblairdolphm: please tell me i'm missing something22:22
dolphmjeblair: i don't think so22:22
*** dangers_away has quit IRC22:23
mordredjeblair: looking22:23
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better
dstanekjeblair: i did submit a patch, but it wasn't that well received and i haven't gotten back to it22:24
jeblair is the bug22:24
uvirtbotLaunchpad bug 1260495 in keystone "Setting autodoc_tree_index_modules makes documentation builds fail" [High,Confirmed]22:24
dstanekjeblair: my attempt at a pbr patch
jeblairstevemar, dolphm: before we go too far down the rabbit hole, this should address your original question:
* fungi needs to step away and cook dinner, but will be back in a while22:26
dolphmjeblair: ah, thanks!22:26
jeblairstevemar, dolphm: i would say that it should fail tests because of the current errors, but i don't know the impact of the current pbr/apidoc ext issue on that22:26
dolphmjeblair: dstanek: so, i'm comparing the extension in keystone to the pbr patch; jeblair- what's the ~2 line approach?22:26
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better
jeblairstevemar, dolphm: eg, it might not be run at all right now22:27
jeblairdolphm: oh, it was the last comment on the bug22:27
*** dangers_away has joined #openstack-infra22:27
*** thomasem has quit IRC22:27
dolphmjeblair: ah... any suggestion on where they should live?22:28
*** jergerber has joined #openstack-infra22:28
dolphmjeblair: assuming you're in favor22:28
apevecanteaya, markmcclain - can we declare stable/havana gate as stable as master? Last few rechecks I saw only bug 1254890 and bug 1253896 which aren't solved on master either22:28
uvirtbotLaunchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [Critical,Confirmed]
uvirtbotLaunchpad bug 1253896 in tempest "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress]
jeblairdolphm: since mordred is around, i'd like him to take the lead on that decision22:29
*** masayukig has joined #openstack-infra22:29
markmcclainapevec: maybe.. I had run rechecks and needed to dig through the results22:29
apevecmarkmcclain, ok, I'd like we push what was pulled out so I can send 2013.2.2 call for testing22:30
*** dkranz has quit IRC22:32
*** gokrokve has quit IRC22:32
*** gokrokve has joined #openstack-infra22:32
*** smurugesan1 has joined #openstack-infra22:36
*** smurugesan has quit IRC22:36
*** gokrokve has quit IRC22:36
clarkbjeblair: any chance you can poke at the crm 114 thing this week?22:37
jeblairclarkb: to try to identify why it times out?22:37
clarkbjeblair: ya22:38
*** salv-orlando_ has joined #openstack-infra22:39
jeblairclarkb: not for a while, i don't think (i don't think it's very high priority now that it's not blocking the rest of the log stuff)22:40
jeblairclarkb: and the review queue is quite big :(22:40
*** mattray has quit IRC22:41
clarkbjeblair: ya, probably not a big priority. the index backlog is pretty big right now though, but that is more to do with the number of tests that have been running all day22:43
*** salv-orlando has quit IRC22:43
*** salv-orlando_ is now known as salv-orlando22:43
* clarkb context switches to reviews22:44
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: fix some small syntax issues in puppet manifest.
*** wchrisj|away is now known as chris_johnson22:47
*** dcramer_ has quit IRC22:49
*** Sukhdev has quit IRC22:54
anteayaapevec: the only move I will make is if a patch is stuck in the gate I will remove it to keep the gate moving, but if markmcclain agrees, I won't pull them out if they don't cause issues22:54
apevecanteaya, all patches themselves were innocent :)22:55
anteayadoesn't matter, to me if something is failing in the gate, it needs to be removed22:56
anteayaI don't know the code in question22:56
anteayaotherwise the gate gets blocked22:56
anteayabut if you have markmcclain's confidence, that is fine for me to let them in22:57
*** eharney has joined #openstack-infra22:58
*** nati_uen_ has quit IRC23:01
jeblairi think we need to get bare centos nodes up soon.23:01
*** shardy is now known as shardy_afk23:01
clarkbjeblair: ++, pleia2 is working on fedora which should be very similar23:01
clarkbthough fedora has new puppet problems so maybe we tackle them more independently23:01
*** packet has quit IRC23:02
pleia2I think puppet is ok now, working through some nodepool networking stuff (we still hard code /etc/network/interfaces, which doesn't exist on fedora)23:03
*** gokrokve has joined #openstack-infra23:04
pleia2it does pull in puppet 3, but the corrections I made in template variables should make it work for simple nodes23:04
*** banix has quit IRC23:04
*** jamesmcarthur has quit IRC23:05
pleia2that said, this shouldn't be an issue in centos because it can actually pull in a 2.7 version of puppet (fedora20's version of ruby is too new)23:05
*** malini_afk has quit IRC23:05
*** stevemar has quit IRC23:06
*** malini_afk has joined #openstack-infra23:07
*** malini_afk is now known as malini23:07
*** VijayT has joined #openstack-infra23:07
jeblairSergeyLukjanov is now a member of openstack-ci-core and openstack-ci-admins in gerrit.  congrats!23:08
jeblair(and we really should finish consolidating those)23:09
anteayacongratulations SergeyLukjanov23:10
anteayaget ready for the deluge while the rest of us are sleeping23:10
*** prad has quit IRC23:11
anteayaI am making myself shut down now that it is dark23:11
clarkbjog0: are you still around? I am going to approve those changes for you, but hoping you cna help untangle them if anything goes sideways23:12
jog0clarkb:  Iam23:12
clarkbjog0: great, approved23:13
*** thedodd has quit IRC23:13
*** alexpilotti has quit IRC23:13
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add tab to only plot gate failures to status.o.o/elastic-recheck/
*** dstanek is now known as dstanek_afk23:16
*** nati_ueno has joined #openstack-infra23:17
mordredcongrats SergeyLukjanov !23:18
clarkbjog0: I did the config change too, but it is taking longer to merge :(23:19
*** jgrimm has quit IRC23:19
*** VijayT has quit IRC23:19
openstackgerritA change was merged to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only
jog0that was pretty quick23:21
jog0clarkb: how long until the web page updates23:22
*** nati_ueno has quit IRC23:23
clarkbjog0: after the :30 puppet updates up to 10 minutes beyond that23:23
*** VijayT has joined #openstack-infra23:25
*** wenlock has quit IRC23:26
jog0clarkb: thanks23:26
*** esker has quit IRC23:29
*** esker has joined #openstack-infra23:29
jog0it just updated23:29
jog0the data isn't there yet (I think)23:30
jog0sdague: is quasi live23:30
*** sarob has quit IRC23:30
sdaguejog0: cool23:30
sdaguejog0: as in neither work now?23:30
*** sarob has joined #openstack-infra23:31
*** jergerber has quit IRC23:32
jog0sdague: the data has to be generated23:32
*** pdmars has joined #openstack-infra23:33
sdaguebut the base page is not working either23:33
sdagueand that data should have been there23:33
*** sarob has quit IRC23:33
*** esker has quit IRC23:34
clarkbsdague: didn't the base data file change its name too?23:34
*** mtreinish has quit IRC23:34
sdagueoh, I guess so23:34
sdagueok, so the real issue is this was supposed to wait until that all happened23:35
zarojeblair, fungi : I have attached a file to bug 979227 containing the case insensitive duplicates that i found when attempting to convert the gerrit db23:35
uvirtbotLaunchpad bug 979227 in openstack-ci "convert gerrit's mysql tables to utf8" [Medium,Triaged]
*** denis_makogon has quit IRC23:36
*** homeless has quit IRC23:36
*** VijayT has quit IRC23:36
*** dkranz has joined #openstack-infra23:38
*** chris_johnson is now known as wchrisj|away23:38
*** mtreinish has joined #openstack-infra23:39
*** cadenzajon has quit IRC23:40
*** virmitio has quit IRC23:40
*** masayukig has quit IRC23:41
*** CaptTofu has quit IRC23:41
*** jnoller has quit IRC23:42
*** e0ne has quit IRC23:42
jog0clarkb: so I don't see any data on yet23:43
*** e0ne has joined #openstack-infra23:44
clarkbjog0: its running now23:46
jog0clarkb: thanks23:46
*** cadenzajon has joined #openstack-infra23:47
*** mgagne has quit IRC23:47
*** dstanek_afk is now known as dstanek23:49
*** johnthetubaguy has quit IRC23:49
*** smurugesan1 has quit IRC23:50
*** johnthetubaguy has joined #openstack-infra23:50
*** mrda is now known as mrda_away23:50
*** johnthetubaguy has quit IRC23:50
*** markwash has quit IRC23:51
jog0sdague:  check it out
*** sarob has joined #openstack-infra23:52
jog0dims: ^23:52
*** mriedem has quit IRC23:52
*** markmcclain has quit IRC23:53
*** sandywalsh has quit IRC23:53
*** zul has quit IRC23:54
*** wenlock has joined #openstack-infra23:56
*** wenlock has quit IRC23:56
*** wenlock has joined #openstack-infra23:57
clarkbjog0: sdague: looking at es graphs, we may want to only run those jobs once an hour instead of 4x an hour23:57
clarkbes goes a bit crazy and I think it may contribute to the backlog being so large23:58
clarkbbut we can watch it this week and see23:58
sdagueclarkb: sure, hourly should be completely sufficient23:58
*** whoops has quit IRC23:59
openstackgerritJames E. Blair proposed a change to openstack-infra/elastic-recheck: Fix alignment of tabs
jeblairjog0: ^ prettification23:59

Generated by 2.14.0 by Marius Gedminas - find it at!