Thursday, 2013-07-25

*** ^demon|away has quit IRC00:05
openstackgerritSean Dague proposed a change to openstack-infra/config: updated htmlifier for safety and features
sdagueclarkb: you want to take a spin on that00:06
sdagueI got rid of the glaring security hole, plus cleaned up other things00:07
openstackgerritSean Dague proposed a change to openstack-infra/config: updated htmlifier for safety and features
clarkbya reviewing now00:16
*** ewindisch has joined #openstack-infra00:17
*** mriedem1 has joined #openstack-infra00:24
*** mriedem has quit IRC00:25
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Conditionally install Python 3 development libs.
clarkbfungi: jeblair ^ want to review sdague's change really quick? if not I am going to approve it00:28
fungii'll take a quick peek00:28
*** erfanian has joined #openstack-infra00:29
fungisdague: "Pull out a save path from a url." was that a typo of "safe?" i'm going to assume so, otherwise i'm not sure what the word means in that context00:32
clarkbfungi: it is a typo00:33
sdaguebut I need to go cook dinner now, so won't be able to fix that until later00:33
clarkbsdague: thats fine I am happy to merge as is00:33
clarkbsdague: I left some other improvement comments but nothing not worth merging over00:34
fungii don't mind the typo, just asking so i understand whati'm revewing00:34
*** anteaya has quit IRC00:36
fungisdague: in safe_path() you're passing in root='/srv/static/logs/' from application() but then building newpath = os.path.abspath("%s/%s" % (root, raw))00:37
fungiso you get abspath('/srv/static/logs//something') there00:37
fungithe double-slash should get cleaned up by abspath, but want to make sure that's intentional00:38
clarkbfungi: abspath does clean it up. I am sure it is not intentional. Its probably worht a comment00:38
fungiif it's there to allow root to be optionally non-terminated then that's probably okay, but it's safest to always terminate the root path with a / so the find provides a more explicit result00:40
clarkbfungi: if you refresh I have left a comment that os.path.join would be beneficial there00:41
clarkbfungi: I think using os.path.join will make you happy and prevent double slashes00:41
sdagueit's actually a little intentional00:42
*** mjfork has quit IRC00:42
fungii think passing in /srv/static/logs (without the slash) is probably a bad idea though, so catering to the possibility is questionable00:42
fungithough in most cases it's unlikely to lead to anything exploitable00:43
sdagueyeh, that's why I passed it with the /, and ran it through abspath to clean00:43
clarkbfungi: correct I think root should have a trailing slash.00:43
sdaguefirst thing went on the grill... so able to chat for a minute :)00:43
fungithe risk is a bit pathological, but to quote futurama out of context, "technically correct is the best kind of correct"00:44
sdaguefuturama is one of my favorite shows ever00:45
fungimine as well00:45
clarkbif I could buy slurm and man chow I probably would00:45
fungiBACHELOR chow00:45
clarkbthats what it was called00:45
fungii preferred soylent cola (made from the best stuff on earth: people!)00:46
fungianyway, dropping the modem again for more conferencing00:46
Ryan_LaneI'm upgrading mediawiki on wiki.openstack.org00:46
clarkbfungi: you ok with me approving?00:46
fungiclarkb: yeah, should be safe enough00:46
* clarkb hits the go button00:47
*** ianw has quit IRC00:47
sdagueI'm pretty sure - is basically bachelor chow00:47
clarkbsdague: approved00:47
clarkbRyan_Lane: gl hf00:47
sdagueat least one guy in the office treated it as such for about 2 months00:47
*** ianw has joined #openstack-infra00:48
openstackgerritA change was merged to openstack-infra/config: updated htmlifier for safety and features
* clarkb runs home between before puppet can apply that00:51
*** rfolco has quit IRC00:54
*** nati_ueno has quit IRC00:54
*** alexpilotti has quit IRC01:00
*** sgviking has quit IRC01:00
*** sgviking has joined #openstack-infra01:01
*** ewindisch has quit IRC01:02
*** gyee has quit IRC01:02
*** yamahata has quit IRC01:04
*** nati_ueno has joined #openstack-infra01:07
*** markmcclain has quit IRC01:08
*** ewindisch has joined #openstack-infra01:15
*** nati_ueno has quit IRC01:18
*** yamahata has joined #openstack-infra01:20
*** ewindisch has quit IRC01:29
*** UtahDave has quit IRC01:31
*** lbragstad has quit IRC01:52
*** lbragstad has joined #openstack-infra01:53
*** pcrews has quit IRC01:56
*** ewindisch has joined #openstack-infra02:06
*** olaph has quit IRC02:12
*** olaph has joined #openstack-infra02:14
clarkbjust as I sit down for dinner docs jobs go unstable again. /me looks into it02:16
*** mriedem1 has quit IRC02:21
clarkbumounted, fscked clean, mounted again seems to be ok02:21
clarkbit was a different error this time around02:21
clarkbjeblair: fungi ^ you may want to look at dmesg on static.o.o02:22
clarkband now time for dinner02:22
*** ewindisch has quit IRC02:29
*** HenryG has quit IRC03:02
*** yaguang has joined #openstack-infra03:07
*** yaguang has quit IRC03:16
*** cody-somerville has joined #openstack-infra03:18
*** dkliban has quit IRC03:26
*** yaguang has joined #openstack-infra03:30
*** changbl has quit IRC03:30
*** jjmb has joined #openstack-infra03:34
*** jjmb1 has joined #openstack-infra03:34
*** jjmb has quit IRC03:38
*** pcrews has joined #openstack-infra03:39
*** dkliban has joined #openstack-infra03:40
*** dkliban has quit IRC03:56
*** erfanian has quit IRC03:56
*** changbl has joined #openstack-infra04:14
*** yaguang has quit IRC04:16
*** SergeyLukjanov has joined #openstack-infra04:22
*** changbl has quit IRC04:22
*** changbl has joined #openstack-infra04:24
*** yaguang has joined #openstack-infra04:29
*** cody-somerville has quit IRC04:32
*** nati_ueno_2 has quit IRC04:34
*** nati_ueno has joined #openstack-infra04:35
*** afazekas has joined #openstack-infra04:35
*** changbl has quit IRC04:37
*** chuckieb|2 has joined #openstack-infra04:40
*** koolhead17 has joined #openstack-infra04:42
*** chuckieb has quit IRC04:43
*** lbragstad1 has joined #openstack-infra05:08
*** lbragstad has quit IRC05:08
* mordred throws a goat at clarkb05:25
clarkbmordred: the openstack party has goats? if I had known that05:26
mordredclarkb: sOOO many goats05:26
* Alex_Gaynor has linkedin endorsements for goats05:27
jesusaurusi hear you can teach goats to openstack05:28
mordred 7mmm05:28
*** nati_ueno_2 has joined #openstack-infra05:28
* mordred throws goats indiscriminaantly into the air05:29
clarkbthat many goats?05:29
mordredALL the goats05:29
clarkbmordred: does this mean you IRC on your phone now?05:29
mordredI'm so done with the places with the alcohol05:29
clarkbmordred: also if you see fungi and/or jeblair it would be super duper if you could mention to them about the docs-draft derpage05:30
mordrednow you've got to deal with me on IRC  on a laptop05:30
clarkbmordred: oh and if you see spearce you should talk to him about our WIP needs05:30
clarkbmordred: and perhaps ask for ideas on making that upstreamable05:30
mordrednot tonight05:30
mordredis spearce here?05:30
clarkbmordred: yes, he and Chris Dibona give the google open source update05:30
clarkbwhich at this point must be "We do GSoC and Gerrit"05:31
clarkb"oh and android is open source every 9 months or so"05:31
* clarkb is cranky about not having a 4.3 image for his phone because verizon05:31
*** nati_ueno_2 has quit IRC05:32
*** nati_ueno has quit IRC05:32
*** nati_ueno has joined #openstack-infra05:32
* mordred lobs a goat at verizon05:34
* mordred hefts a scimitar at clarkb05:34
clarkbmordred: when is the seattle thing with tripleo folks?05:37
mordredclarkb: btw, jeblair and fungi and I had a good talk in person today about dib in d-g05:37
mordredclarkb: which was largely what was in that etherpad05:38
clarkboh good. is the use case starting to gel?05:38
clarkbI should check if they gave us beta access05:39
clarkbmordred: also tomorrow you should comp that account :)05:39
mordredclarkb: I've also got a half-done dib element for us05:39
mordredclarkb: bah05:39
mordredclarkb: tomorrow I have a talk05:39
clarkbhave you written the talk yet?05:39
mordredclarkb: yup05:39
clarkboh if you see anyone from mirantis you might want to mention that their stackalytics thing seems to ignore us despite being hosted on stackforge05:40
mordredclarkb: I saw everyone who works at mirantis05:41
SergeyLukjanovhey guys, I'm from Mirantis05:41
SergeyLukjanovI'll address your comments to stackalytics team05:42
clarkbSergeyLukjanov: hi there. I was looking at stackalytics earlier this evening and the openstack-infra/* repos seem to be missing05:42
clarkbgranted we have a lot of them maybe it was too much work or something. In any case openstack-infra/config should probably be included05:42
SergeyLukjanovclarkb, mm, yep, see it too05:42
* mordred hands SergeyLukjanov a cake05:42
clarkbmordred: run out of goats?05:43
* mordred knows that mirantis doesnt' like being handed goats05:43
* mordred thinks mirantis might be his favorite OS company05:43
SergeyLukjanovthe same story with openstack-dev projects05:44
SergeyLukjanovI'll talk today with Ilya Shakhat that runs the stackalytics project about extending number of monitored projects05:45
clarkbSergeyLukjanov: awesome, thanks05:46
SergeyLukjanovbtw I have a small question about jobs that we can use in savanna)05:47
SergeyLukjanovi've proposed the change
SergeyLukjanovit includes -docs jobs05:47
SergeyLukjanovis it ok to use them for savanna project?05:48
clarkbSergeyLukjanov: it is ok to use the gate-*-docs jobs in the check and gate queue. Typically it is not ok to use them in the post, pre-release and release queues as the *-docs jobs want to publish to whcih we are trying to keep for openstack projects05:48
clarkbSergeyLukjanov: however, We do have read the docs jobs that you can setup05:49
clarkbSergeyLukjanov: the read the docs jobs tell read the docs to update your docs whenever code merges. You shoudl eb able to see them in the layout file if you search for rtfd05:49
SergeyLukjanovclarkb, we are already using rtfd jobs05:49
clarkboh I see that now, yeah. Today rtfd is probably the best options for stackforge projects05:50
SergeyLukjanovclarkb, I'll remove gate-*-docs from post/release05:50
SergeyLukjanovso, it'll be able to see on docs w/o merging or manually building them for patches to docs05:51
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add more jobs for Savanna projects
SergeyLukjanovI've removed gate-savanna-docs has been removed from post/pre-release/release, could you please take a look at this patch set?05:58
*** boris-42 has joined #openstack-infra06:01
*** SergeyLukjanov has quit IRC06:03
*** melwitt has quit IRC06:10
clarkbyup plan to tomorrow morning but now sleep06:20
*** nati_ueno_2 has joined #openstack-infra06:26
*** afazekas is now known as afazekas__06:27
*** nati_ueno has quit IRC06:29
*** odyssey4me has joined #openstack-infra06:34
Vivekmordred: Hi06:34
Vivekmordred: I fell asleep last night.06:35
*** dansmith has quit IRC06:38
*** dansmith has joined #openstack-infra06:39
*** koolhead17 has quit IRC06:42
*** boris-42 has quit IRC07:01
*** dkehn_ has joined #openstack-infra07:02
*** dkehn has quit IRC07:03
*** hashar has joined #openstack-infra07:08
*** EmilienM has quit IRC07:11
*** SergeyLukjanov has joined #openstack-infra07:12
*** EmilienM has joined #openstack-infra07:16
jeblairclarkb, mordred, fungi: it appears dm-1 (docs-draft) has failed yet again (just about 20 minutes ago); i see clarkb fixed it a few hours ago07:18
jeblairi think that's three times that the same lv has failed; lvm says that it is on a single pv, so i'm going to add a new cinder volume and move it, deprecating the current pv that it's on07:18
jeblairlvs --segments -o +pe_ranges07:24
jeblairthat shows which pvs the lv is on07:24
jeblaircurrently running pvmove -n docs-draft /dev/xvdh1 /dev/xvdi107:24
jeblairalso unmounted and fscking the partition07:25
*** hashar_ has joined #openstack-infra07:29
openstackgerritA change was merged to openstack-infra/config: Don't remove lost+found from static.o.o.
jeblairreading the lost+found directory triggers the error07:32
*** hashar has quit IRC07:33
*** hashar_ is now known as hashar07:33
*** Ryan_Lane has quit IRC07:33
*** nayward has joined #openstack-infra07:35
jeblairnm, different error.07:39
*** jpich has joined #openstack-infra07:39
*** rpodolyaka has left #openstack-infra07:42
jeblairrepeated invocations of 'fsck -f' without mounting continue to try to correct similar errors; i think we may need a new filesystem as well07:44
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status
jeblairokay, i ran it 3 times and the most recent time it corrected no errors07:49
jeblairso i'm going to remount it and not touch lost+found07:49
jeblairafter the pvmove is done (in a few hours) and we remove the pv that encountered timeouts, we may want to consider making a new filesystem on a new lv and starting an rsync07:50
jeblairdocs-draft is remounted07:51
*** SergeyLukjanov has quit IRC07:52
*** SergeyLukjanov has joined #openstack-infra07:54
*** johnthetubaguy has joined #openstack-infra08:00
*** mjura has joined #openstack-infra08:01
*** mjura has left #openstack-infra08:03
*** mjura_ has joined #openstack-infra08:03
*** mjura_ has quit IRC08:04
*** boris-42 has joined #openstack-infra08:06
*** SergeyLukjanov has quit IRC08:32
*** SergeyLukjanov has joined #openstack-infra08:36
*** derekh has joined #openstack-infra08:39
*** nati_ueno_2 has quit IRC09:01
*** amotoki has joined #openstack-infra09:28
*** afazekas__ has quit IRC09:51
*** markmc has joined #openstack-infra10:01
markmcis the wiki in read-only mode or something?10:02
* markmc sees10:02
markmc2013-07-25T00:46:46  <Ryan_Lane> I'm upgrading mediawiki on wiki.openstack.org10:02
markmcI don't see a login button today, basically10:02
jpichmarkmc: Hm... I can't see the button either but accessing directly works10:03
markmcjpich, super, thanks10:03
* markmc filed
uvirtbotLaunchpad bug 1204841 in openstack-ci "No login button on wiki" [Undecided,New]10:05
*** jhesketh__ has quit IRC10:06
*** jhesketh__ has joined #openstack-infra10:06
openstackgerritThomas Goirand proposed a change to openstack/requirements: Adds oslo.sphinx
*** hashar has quit IRC10:22
*** fbo_away is now known as fbo10:25
*** yaguang has quit IRC10:29
*** yaguang has joined #openstack-infra10:41
sdaguebummer that the wiki isn't under puppet control10:43
*** yaguang has quit IRC10:46
*** afazekas has joined #openstack-infra10:51
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Pathignore plugin support added
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: HTML Publisher plugin support added
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: TextFinder plugin support added
*** afazekas has quit IRC10:57
*** yaguang has joined #openstack-infra10:58
*** yaguang has quit IRC10:58
*** weshay has joined #openstack-infra11:00
*** dprince has joined #openstack-infra11:01
*** afazekas has joined #openstack-infra11:12
*** johnthetubaguy has quit IRC11:13
*** hashar has joined #openstack-infra11:27
sdaguejeblair: did you have any further thoughts on the openstack-qa mail alias?11:31
*** alexpilotti has joined #openstack-infra11:32
*** sungju_ has joined #openstack-infra11:34
*** alexpilotti has quit IRC11:42
*** jjmb1 has quit IRC11:43
*** pabelanger has quit IRC11:44
*** rfolco has joined #openstack-infra11:58
*** boris-42 has quit IRC12:01
*** boris-42 has joined #openstack-infra12:03
*** bnemec has quit IRC12:04
*** sungju_ has quit IRC12:06
*** SergeyLukjanov has quit IRC12:07
*** dprince has quit IRC12:07
*** bnemec has joined #openstack-infra12:07
*** prad has quit IRC12:10
*** dhellman_ has joined #openstack-infra12:14
*** dkranz has quit IRC12:15
*** prad has joined #openstack-infra12:17
*** novel1 is now known as mirrorbox12:20
*** mirrorbox has joined #openstack-infra12:20
*** dkliban has joined #openstack-infra12:23
*** woodspa has joined #openstack-infra12:25
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding the Sauce OnDemand Plugin
*** derekh has quit IRC12:37
*** jergerber has joined #openstack-infra12:37
*** johnthetubaguy has joined #openstack-infra12:43
jd__does that error raises a bell ? (mordred maybe?)12:44
*** derekh has joined #openstack-infra12:47
*** HenryG has joined #openstack-infra12:47
*** cppcabrera has joined #openstack-infra12:48
*** cppcabrera has left #openstack-infra12:48
*** alexpilotti has joined #openstack-infra12:49
*** mriedem has joined #openstack-infra12:49
*** anteaya has joined #openstack-infra12:50
*** jergerber has quit IRC12:51
*** mgagne has joined #openstack-infra12:53
*** pabelanger has joined #openstack-infra12:59
*** prad has quit IRC13:05
*** dkliban has quit IRC13:06
*** changbl has joined #openstack-infra13:08
*** ewindisch has joined #openstack-infra13:10
*** dprince has joined #openstack-infra13:11
*** boris-42 has quit IRC13:13
*** alexpilotti_ has joined #openstack-infra13:17
*** dims has joined #openstack-infra13:19
*** alexpilotti has quit IRC13:20
*** alexpilotti_ is now known as alexpilotti13:20
*** dkehn_ is now known as dkehn13:25
*** pentameter has joined #openstack-infra13:28
*** amotoki has quit IRC13:29
*** vogxn has joined #openstack-infra13:32
*** markmcclain has joined #openstack-infra13:46
*** vijendar has joined #openstack-infra13:48
*** burt has joined #openstack-infra13:49
*** chuckieb|2 has quit IRC13:49
*** prad has joined #openstack-infra13:50
*** leifmadsen has quit IRC13:51
*** changbl has quit IRC13:52
*** michchap has quit IRC13:53
openstackgerritA change was merged to openstack/requirements: Updated minimal version of pymongo to 2.4.
*** dims has quit IRC13:58
*** leifmadsen has joined #openstack-infra14:02
*** _TheDodd_ has joined #openstack-infra14:05
jd__dhellmann: do you happen to know why failed? I'm having the same issues on a pbr patchset I sent14:06
*** vogxn has quit IRC14:08
*** dkliban has joined #openstack-infra14:08
*** xchu has joined #openstack-infra14:08
*** HenryG has quit IRC14:09
*** datsun180b has joined #openstack-infra14:15
*** datsun180b has joined #openstack-infra14:15
*** avtar has joined #openstack-infra14:17
jpichjd__: FWIW today I had issues with setuptools locally as well and only managed to "fix" them by using the openstack pypi mirror in order to avoid the 404. I'm surprised this job is calling to upstream pypi14:18
jd__jpich: equally :)14:18
*** datsun180b_ has joined #openstack-infra14:20
*** HenryG has joined #openstack-infra14:21
*** datsun180b has quit IRC14:24
*** datsun180b_ is now known as datsun180b14:24
*** jjmb has joined #openstack-infra14:28
ShrewsAs did I. I think a new version is out that hasn't yet propogated to all of the mirrors. Although it may have by now.14:28
Shrewsre: setuptools, that is14:29
*** dkranz has joined #openstack-infra14:29
*** cp16net is now known as cp16net|away14:31
mordredjd__: that's failing building the mirror that it uses to test that building a mirror works14:43
jd__mordred: why is that part of pbr tests?14:44
mordredscuse me, misspoke. it's building the mirror that it uses to test that installing all of the projects works.14:45
*** mjfork has joined #openstack-infra14:45
mordredwhich is does so that it can inject the currently tested version of pbr into that mirror14:45
mordredso that we can test that different install scenarios haven't gone bad14:46
mordredwe don't normally have that many problems in the pbr test - also because pbr doesn't get nearly as many patches as the rest of the project... but if it stays a problem, we might have to get cleverer14:46
jd__ok, that makes more sense :)14:49
*** SergeyLukjanov has joined #openstack-infra14:50
*** changbl has joined #openstack-infra14:50
mordredactually - I see a bug in the integration script now that pip 1.4 is out...14:50
*** jjmb has quit IRC14:55
*** xchu has quit IRC14:58
*** _TheDodd_ has quit IRC15:03
*** _TheDodd_ has joined #openstack-infra15:04
*** changbl has quit IRC15:05
*** mkoderer has joined #openstack-infra15:08
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR
mkodererHi folks, I am not able to login to the OpenStack Wiki15:08
mkodererseems the login button is gone ;) could someone have a look15:09
Shrewsmordred: so does 38647 mean you must upgrade to pip 1.4 if you use pbr?15:10
*** mrodden has quit IRC15:10
sdaguemordred: what's your thinking about unit test on python scripts in the config repo?15:11
sdagueor if something is getting that complicated, it should come out into another repo?15:11
mkoderersdague: I am not able to change the QA meeting wiki .. seems the wiki is somehow broken15:12
mordredShrews: no. it means that that integration script installs things via pip for testing which is now 1.4 by default from upstream15:12
uvirtbotLaunchpad bug 1204841 in openstack-ci "No login button on wiki" [Undecided,Confirmed]15:12
mkoderersdague: great thanks!15:13
mordredShrews: we DO want to get everyone to upgrade to 1.4 though, because it fixes a bunch of issues, and also allows us to safely upload pre-release software to pypi without breaking peope15:13
mordredbut we have not gotten to that point yet15:13
*** nayward has quit IRC15:13
mordredsdague: yes. if they've gotten that complex, they should probably have their own home15:13
*** CaptTofu has quit IRC15:13
mordredsdague: although I could certainly be argued off of that stance - it's not a strong one15:14
*** CaptTofu has joined #openstack-infra15:14
mordredjd__: should fix the pbr integration test gate15:14
sdaguemordred: I think it's a fine stance15:14
*** gyee has joined #openstack-infra15:15
sdagueI wonder if all the htmlifier filters in config should become a project together15:15
sdaguesubunit2html is in there as well, and not really unit tested15:15
sdagueand I get scared when I make changes to that15:15
mordredsdague: so, subunit2html wants to move to subunit itself I believe15:15
mordredbut clarkb wanted to clean it up before submitting it there15:15
sdagueis subunit in git?15:16
jd__mordred: cool I'll recheck once it's in then15:16
*** changbl has joined #openstack-infra15:16
sdagueor does that mean having to fight with bzr to make submissions15:16
mordredhtmlifier probably could be its own thing15:16
mordredsdague: fight with bzr15:16
mordredsdague: or, fight with lifeless to get him to move it to git15:16
sdaguehmph. I actually had some more prettinesses I wanted to add to subunit2html15:17
mordredoh - well please do!15:17
mordredalso, we could make it its own thing15:17
sdagueso it's a little easier if it lives in a gerrit tree, honestly15:17
mordredit's certainly a generally useful program15:17
mordredsdague: I fully support making it its own gerrit tree with its own pypi releases15:18
*** mrodden has joined #openstack-infra15:18
*** cp16net|away is now known as cp16net15:18
sdagueit seems a little weird to push something like the openstack log htmlifier to pypi, as it's so specific though15:22
mordredis it? couldn't it be used to htmlify other logs?15:23
sdagueit's pretty specific to the log format in OpenStack15:23
sdagueI'm not sure I have a strong opinion realistically, I just don't want to be seen poluting with non generic stuff.15:24
sdaguemy real goal is just to get unit testing around it though, because I borked a format thing on my last update (not terribly, but it should be different than it is)15:25
*** danger_fo_away is now known as danger_fo15:26
dansmiththere was something else about the postgres gate job that was different, but I can't remember what it is15:31
dansmithjeblair said something last week I think15:31
jeblairsdague: i think it might become the first part of a log submission, processing, and retrieval system, if that helps you think about how to structure the repo/project.15:31
dansmithI'm trying to reproduce a fail that only happens there, but just moving to postgres isn't enough, apparently15:31
jeblairdansmith: zeromq15:32
dansmithoh, hmm.15:32
sdaguepostgresl is running on zmq now?15:32
sdagueif so, that's news to me15:32
jeblairer, no that's the gate-tempest-devstack-vm-postgres-zeromq-full job15:32
jeblairdansmith: i was wrong, gate-tempest-devstack-vm-postgres is only DEVSTACK_GATE_TEMPEST=1 and DEVSTACK_GATE_POSTGRES=115:33
dansmithhmm, okay15:33
*** gyee has quit IRC15:33
jeblairdansmith: DEVSTACK_GATE_TEMPEST_FULL=1 is not set on gate-tempest-devstack-vm-postgres15:34
*** gyee has joined #openstack-infra15:34
dansmithjeblair: wait, what? meaning it's not a full tempest run? it takes long enough that it seems like it's a full run15:34
dansmithgate-tempest-devstack-vm-postgres-full  is the test that fails for me15:34
dansmithwhich should be full and I don't see anything in the localrc about zmq15:35
*** krtaylor has quit IRC15:35
dansmithwell, then I can't reproduce it here so jenkins is clearly wrong :P15:35
jeblairdansmith: sorry if i've lead you astray :(15:35
dansmithjeblair: it's all good :)15:36
clarkbgood morning15:38
clarkbjeblair: is the docs-draft pv situation good now? (eg you have migrated to a new pv and removed the old one?)15:38
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Add a CDR log file
clarkbreducing the d-g pool size does seem to have made the slave situation much better. I haven't caught any more occurences of the double test run and premature node delete15:41
jeblairclarkb: yes, i have removed the pv from the system and renamed it in the cinder interface to "this-has-timeout-errors"15:41
clarkbjeblair: re I think that will help us debug the problem as it adds some logging and doesn't make the situation so painful (hopefully meaning we can observe it without making ttx angry :) )15:43
jeblairclarkb: next steps i think are to handle the (unconfirmed) possibility of a permanently bad FS: create a new lv, rsync docs-draft to it, poke around the old one a bit (including the l+f folder) and then consider switching to the new volume.15:43
jeblairclarkb: we have sufficient logging15:44
*** johnthetubaguy has quit IRC15:44
jeblairclarkb: can you point to nodes with problems that i can look into?15:44
*** johnthetubaguy has joined #openstack-infra15:44
clarkbjeblair: yeah there is a bunch of captured logs in the bug I opened. let me find that15:45
jeblairclarkb: is this just to address the timeout problem?15:45
clarkbjeblair: has a paste full of relavent logs15:46
uvirtbotLaunchpad bug 1204625 in openstack-ci "Devstack gate launch jobs timeout confusing Jenkins and devstack gate" [High,Triaged]15:46
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Add a CDR log file
clarkbjeblair: the timeouts were the inspiration yes15:46
annegentleuh guys, the Wiki Signin link is gone? Mac OSX Chrome 2715:47
clarkbannegentle: yes, that bug has a nasty workaround15:47
uvirtbotLaunchpad bug 1204841 in openstack-ci "No login button on wiki" [Critical,Confirmed]15:47
clarkbI intend on asking Ryan_Lane to share a screen session with me when he fixes it so that I don't feel so useless when it comes to the wiki15:48
*** adam_g has quit IRC15:48
jeblairclarkb: ok, so that's a known problem.  the best fix for that would be to detect the case where we are trying to add a node that is already in jenkins and set its state to ready if the state is building.15:48
*** adam_g has joined #openstack-infra15:48
*** adam_g has joined #openstack-infra15:48
clarkband I will poke at it in a bit on my own probably15:48
annegentleheh we thought we were goign crazy15:48
jeblairclarkb: +1 wiki15:48
jeblairclarkb: (i feel like adding another race condition to fix a race condition isn't going to make us any happier)15:49
*** johnthetubaguy has quit IRC15:49
anteayatalking about buttons15:49
anteayacan we get an easier to discover button for signing up/logging into the foundation website?15:50
clarkbjeblair: the way around the race is to set the node offline first, then check if it is idle. I just realized this. I can update the patch if you think making it correct is useful15:50
anteayawho do I talk to about this button?15:50
sdaguedansmith: you have a failing run on postgresql that you can point me at logs?15:50
dansmithI'm doing a full tempest run locally to see if it's a symptom of residue or something15:51
dansmithbut running that test alone or with the rest of its class doesn't poke it for me15:51
jeblairclarkb: i still feel like we should just fix the bug in the system we have, which is supposed to prevent this.15:51
ttxsdague: I'm a bit disconnected from reality while at OSCON -- what's the situation on the Neutron testsuite re-enablement ?15:51
dansmithI'm sure it's related to my change, I just need to add some stuff to figure out why and need to be able to repro it of course15:51
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports
clarkbjeblair: ok, I will work on detecting jenkins node state and handling that properly15:53
clarkbttx: nachi thought he had a fix, but once the fix was subjected to the rigors of the gate it wasn't reliable15:53
clarkbttx: so we are still not voting on that test15:53
ttxclarkb: ok. didn't know the fix was not enough[tm]15:53
jeblairclarkb: also think about it this way -- it's always an error to attempt to delete a node running a job; that's not something we want to mask (in this case it's benign, but in other cases (running a test twice on a node) it's not)15:54
ttxclarkb: why aren't you with us ? HP being cheap on Seatlle-PDX shuttle tickets or what15:54
clarkbttx: looking at the most recent several hours of logstash data I have the neutron test failed ~40% of the time in the gate queue15:55
jeblairi need to run; i have just rolled out of bed, worked 1.5 hours and not done any of the things one does in the morning, including eating.15:55
clarkbttx: I have driven a couple thousand miles this month and didn't want to do more. And I don't think they would've given me admission15:55
ttxclarkb: so close and yet so far :)15:56
annegentlettx: ping if you've got a sec15:56
ttxannegentle: need to go to plenary room, should be back soon if wifi is not busted15:56
annegentlettx: ok good luck15:57
mordredsdague: also, re: ssh tests of nodes in tempest not working - do you know what the issue is and is it something easy to describe to someone so that they might work on it?15:57
sdaguettx: on the neutron tests, last time I looked we were still seeing the race15:58
* mordred agrees with anteaya about foundation login button/link15:58
sdaguenatchi now thinks it's a timeout issue, so there is proposed tempest fix to bump the timeout15:58
sdaguewe'll see if that makes it better15:58
sdaguemordred: on the ssh tests, I think the crux of it is the ssh success detection isn't good15:58
pabelangerany infra-core mind approving ?15:58
clarkbsdague: it is. logstash indicates a ~40% failure rate in the gate (which is much much higher than our typical breakage)15:59
clarkbpabelanger: done. I am guessing jeblair didn't do it because traveling + OSCON15:59
mordredsdague: ah, as in it's tough to know if the ssh command to the node was actually good?15:59
sdagueso a more reliable way of testing success on ssh is probably all that's need. I think right now it's looking for cirros log in text, shuold instead run some basic comment15:59
sdaguemordred: right15:59
sdagueand I think the check bit rotted16:00
pabelangerclarkb: np, thanks16:00
sdaguebecause the tests were off16:00
mordredyeah. they all fail at this point16:00
pabelangeradding some asterisk patchsets now :)16:00
mordredbut you expect that the setup should be there in as much as users/keys should be done properly16:00
ttxannegentle: back16:00
ttxso far16:00
sdaguemordred: there may be some subtlety in getting paramiko to deal with something too16:00
openstackgerritA change was merged to openstack-infra/config: Add sshkey for pabelanger
mordredmmm. my favorite16:01
*** changbl has quit IRC16:01
sdagueI'll be honest, last time I looked in detail the ssh actually was working, but the check it was running to know if it worked was bad16:01
clarkbmordred: failed16:01
sdagueand I haven't looked deeper16:01
* mordred shoots self in face16:02
clarkbmordred: don't do that, we need you to fix it :)16:02
mordredis a 40416:02
mordreddstufft: ^^^16:03
mordreddstufft: danger will robinson16:03
mordred0.9.7 is there16:04
*** changbl has joined #openstack-infra16:06
dstufftmordred: try now16:07
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7
mordreddstufft: yay!16:07
openstackgerritA change was merged to openstack-infra/config: Added OpenStack docs jobs for Trove.
dstufftmordred: the Fastly Purge Status API (which tells us if a purge completed or not) is down right now (or was yesterday) so I had to remove the status checks temporarily16:08
dstufftso some purges might not happen successfully :/16:08
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7
mordreddstufft: loverly16:08
*** pcrews has quit IRC16:09
ttxannegentle: I have high lag but I can probably try to talk16:10
clarkbhow logstash feels this week but elasticsearch hasn't crashed yet so progress!16:10
zaroclarkb: in jjb, is it possible to wrap multiple job groups from one job-group?16:10
clarkbzaro: I don't think so16:11
clarkbI am trying to remember how job-groups expand and I think they are not recursive16:11
*** markmc has quit IRC16:12
mordreddstufft: also, amusingly, there are no things that match 'setuptools<0.7' for pip 1.416:12
mordreddstufft: because, you know, they all had letters16:12
*** SergeyLukjanov has quit IRC16:12
openstackgerritA change was merged to openstack-infra/config: Use the new oslo.sphinx library for the doc theme
dstufftmordred: yea lol16:12
mordredclarkb: have I mentioned that I think a neat thing to track/report/graph would be number of gate failures for things that passed in check queue16:14
mordredclarkb: and which percentage of those were things kicked out for merge errors16:14
*** dkliban is now known as dkliban_lunch16:15
clarkbkeeping track of merge failures would definitely be interesting16:15
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add an option for the buildChooser
clarkbmordred: still around?16:19
mordredclarkb: yup16:19
clarkbmordred: opinions on logging more IRC channels? Do we just want to approve those puppet changes or do we need to get some additional level of approval16:19
jeblairclarkb: i haven't looked at specific changes but i think we should require the approval of the relevant ptl16:21
*** pcrews has joined #openstack-infra16:21
mordredthat seems reasonable to me16:21
openstackgerritA change was merged to openstack-infra/config: Document volume maintenance on static
* jeblair goes to get breakfast and then to the convention center16:22
clarkbhave fun16:23
*** bclifford has joined #openstack-infra16:24
clarkbpabelanger: if you think I shouldn't approve based on your +1 just let me know16:24
openstackgerritA change was merged to openstack-infra/config: Support
*** Ryan_Lane has joined #openstack-infra16:26
openstackgerritKhai Do proposed a change to openstack-infra/config: run bitrot jobs after pypi mirror update
pabelangerclarkb: sounds good16:29
dansmithsdague: so, in that log link I sent,16:35
dansmithsdague: the tempest console logs go many minutes longer than the compute and api logs...16:35
clarkb_TheDodd_: re I have a feeling that one jeblair and fungi have a chance to review it they will be happy with it16:35
clarkb_TheDodd_: hopefully sometime early next week. I also think that this is the sort of change we probably want to announce to the openstack-dev mailing list. Do you want to draft a short announcement in an etherpad?16:36
clarkb_TheDodd_: then when we merge 36886 you can send that out16:36
_TheDodd_clarkb: I would be happy to.16:40
_TheDodd_clarkb: Hopefully I will be able to get around to it tomorrow.16:41
clarkb_TheDodd_: no rush. jeblair and fungi are at OSCON so I don't expect their reviews until next week16:41
*** UtahDave has joined #openstack-infra16:41
clarkbRyan_Lane: for the wiki fixing16:42
clarkbRyan_Lane: sounds like you have already fixed the problem \o/ any chance you can share a screen session later today and show me what you have done? If you are at OSCON and that is just clunky maybe we can do something like that next week16:43
clarkbRyan_Lane: I just feel fairly useless when it comes to the wiki and think that should change16:43
_TheDodd_clarkb: What do you think would be most beneficial for me to mention in the etherpad? I will go ahead and reference this file ( ) so that everyone knows that we are pushing for header style bug references, etc.16:43
clarkb_TheDodd_: I am thinking that a description of the new workflow support would be good16:44
*** afrittoli has quit IRC16:46
*** andreaf has joined #openstack-infra16:46
clarkbok now I really need to AFK for a bit so that I can make my way to the office16:47
Ryan_Laneclarkb: I'm not as OSCON, but I can definitely share a screen session with you16:47
Ryan_Lanethe problem was with the skin. I need to upstream a change16:47
Ryan_Lanethe skin doesn't handle openid login links properly16:47
*** krtaylor has joined #openstack-infra16:50
*** mjfork has quit IRC16:50
*** mjfork_ has joined #openstack-infra16:55
*** hashar has quit IRC16:56
*** jpich has quit IRC16:56
*** koolhead17 has joined #openstack-infra16:56
*** nati_ueno has joined #openstack-infra16:57
*** mjfork has joined #openstack-infra16:57
*** dkliban_lunch is now known as dkliban16:58
*** rnirmal has joined #openstack-infra17:00
*** mjfork_ has quit IRC17:00
*** mjfork_ has joined #openstack-infra17:01
*** Ryan_Lane has quit IRC17:02
*** SergeyLukjanov has joined #openstack-infra17:02
*** mjfork has quit IRC17:02
*** mjfork_ is now known as mjfork17:02
*** mjfork has quit IRC17:07
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Stop trying to install old pip and setuptools
* mordred cries17:09
clarkbmordred: more trouble?17:09
clarkbalso I am only halfway through my semi AFK. So many distractions17:09
*** derekh has quit IRC17:10
*** mjfork has joined #openstack-infra17:10
*** zul has quit IRC17:11
*** avtar has quit IRC17:12
mordredclarkb: yup.17:14
clarkbok back to AFK almost done17:15
*** Shrews has quit IRC17:21
*** ewindisch has quit IRC17:23
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Invert modules.conf logic
*** Shrews has joined #openstack-infra17:24
*** SergeyLukjanov has quit IRC17:27
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk
*** SergeyLukjanov has joined #openstack-infra17:28
*** ewindisch has joined #openstack-infra17:29
*** SergeyLukjanov has quit IRC17:30
*** SergeyLukjanov has joined #openstack-infra17:31
*** HenryG has quit IRC17:31
*** cody-somerville has joined #openstack-infra17:31
*** SergeyLukjanov has quit IRC17:34
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status
*** datsun180b has quit IRC17:34
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk
*** pcrews has quit IRC17:41
*** sarob has joined #openstack-infra17:42
jeblairi will create a new docs-draft now and start an rsync17:42
*** cody-somerville has quit IRC17:42
*** datsun180b has joined #openstack-infra17:46
fungithat sounds safest17:46
clarkbjeblair: did it break again? I just got to my desk17:48
*** sarob has quit IRC17:50
jeblairclarkb: no; this is the next steps i outlined earlier17:50
*** Ryan_Lane has joined #openstack-infra17:50
jeblairclarkb: basically i don't trust the filesystem at this point; it was behaving too weirdly.17:51
*** sarob has joined #openstack-infra17:51
jeblairclarkb: (2 fsck's in a row on an unmounted fs fixed errors.  3rd one was okay, but still)17:51
*** bclifford has quit IRC17:52
clarkbI am writing the d-g change to double check jenkins and d-g state beofre erroring now17:52
jeblairi have an rsync running now.  if there _is_ a problem with the current filesystem, that rsync may trigger an error, so we should be alert for that.17:52
*** melwitt has joined #openstack-infra17:53
jeblairclarkb: cool, thanks17:53 seems to be working now that I reload the dialplan17:53
jeblairclarkb: (also, increasing the job timeout is another quick thing we can do to mitigate that if we want to increase the pool again)17:53
clarkbjeblair: increases the timeout if you want to do that17:53
clarkbI -1'd it because I wanted your opinion on it17:54
clarkbI think we are ok at 45 node right now17:54
*** SergeyLukjanov has joined #openstack-infra17:55
pabelangerrussellb: already getting scanned on pbx.openstack.org17:55
pabelangermaybe should do fail2ban or something17:55
pabelangeror changing the default SIP port will help too17:55
pabelangersince we are doing SIP URI dialing17:55
*** jpeeler has quit IRC17:55
pabelangerwe can just update the DNS SRV records17:56
*** nati_ueno has quit IRC17:57
*** nati_ueno has joined #openstack-infra17:58
*** bnemec has quit IRC17:58
*** jpeeler has joined #openstack-infra18:00
*** bnemec has joined #openstack-infra18:01
*** jpeeler has quit IRC18:02
*** avtar has joined #openstack-infra18:02
*** jpeeler has joined #openstack-infra18:02
*** jpeeler has joined #openstack-infra18:02
*** vijendar has quit IRC18:05
fungipabelanger: just give me a heads up if you want that changed, though it will probably require another trouble ticket with rackspace to change the second one18:05
*** vijendar has joined #openstack-infra18:07
zaroclarkb: the gearman-plugin is no longer listed on the jenkins plugin manager.  it might be because of how we released ver
*** vijendar has quit IRC18:07
zaroclarkb: can you make a release of the zmq plugin to see if it will appear?18:08
*** pabelanger has quit IRC18:08
*** odyssey4me has quit IRC18:09
*** cp16net is now known as cp16net|away18:09
clarkbzaro: ya I can do that later today18:10
*** Ryan_Lane has quit IRC18:14
*** Ryan_Lane has joined #openstack-infra18:14
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Handle Jenkins node already being added to Jenkins
*** cp16net|away is now known as cp16net18:16
clarkbjeblair: ^ I think that should do it18:16
*** gyee has quit IRC18:19
*** sarob has quit IRC18:22
*** sarob has joined #openstack-infra18:22
*** chuckieb has joined #openstack-infra18:26
*** jpeeler1 has joined #openstack-infra18:30
jeblairi see one unstable docs job, but the filesystem, block devices and server seem okay18:31
*** jpeeler has quit IRC18:31
jeblairi am assuming it's a jenkins scp plugin fluke for now18:31
jeblair(possibly it timed out because of the additional load on the fs from the rsync)18:32
*** bclifford has joined #openstack-infra18:32
jeblairclarkb: i am at a talk by james turnbull on logstash18:32
jeblair(but am sadly missing pleia2's talk)18:33
*** sarob has quit IRC18:33
*** jpeeler1 is now known as jpeeler18:33
*** afazekas has quit IRC18:33
*** jpeeler has quit IRC18:33
*** jpeeler has joined #openstack-infra18:33
dhellmannjd__: I left a +2 on mordred's change to the pbr tests; I think that's what's breaking things for both of us?18:33
jd__dhellmann: I think he fast approved the one needed18:34
dhellmannoh, not
jd__dhellmann: this one, but he did fast approved before you +218:35
jd__Look at 6:11 PM18:35
mordreddhellmann: yes18:35
jd__now I keep having
mordredand thank you18:35
jd__dunnow why18:35
mordredwhile you're at it:
jd__too soon recheck?18:36
mordredyeah. it hasn't landed yet. there was another thing18:36
*** bclifford has left #openstack-infra18:36
dhellmannjd__, mordred : ah, it wasn't approved when I got there :-)18:36
jeblairturnbull: "anyone here who works for a distro please choose an an init system; i'm getting sick of it"  :)18:37
mordredjeblair: :)18:37
dhellmannmordred: your commit messages are like something out of lewis carroll18:37
jeblair"actual actual lifecycle of a log: record -> delete"18:38
clarkbjeblair: learn all the logstash things18:38
* clarkb just had an idea18:39
clarkbwe should do "internal" learn the system things. Ryan_Lane can show us the wiki, I can show logstash, jeblair can do d-g and so on18:39
clarkband use that to try and keep the team up to date on the broad list of things and be focused on the team and not external conference audiences. Kind of like refining the bootcamp topics18:40
annegentlejeblair: ping18:40
jeblairannegentle: pong but i'm supposed to be paying attention to a conference talk :(18:41
clarkbannegentle: I am happy to help if this isn't something that requires jeblair18:41
annegentlejeblair: ok no worries just wanted to see how it went with Todd re:repos18:41
clarkb(I secretly want him to become a logstash ninja)18:41
annegentlejeblair: holler when you can chat18:42
*** jpeeler has quit IRC18:42
jeblairannegentle: short version; went well, expecting progress by next week18:42
clarkbsdague: got a minute to talk about the number of tempest check tests? specifically can we stop running neutron full if the non full job doesn't even pass?18:42
clarkbsdague: or vice versa, put the effort into making the full job pass and forget the non full job18:42
clarkbsdague: I mean changes to neutron itself don't even trigger the full neutron job...18:43
clarkbsdague: the other one I wanted to talk about was the cells job. I am tempted to make that nova only until it is passing at least smoketests18:45
openstackgerritSean Dague proposed a change to openstack-infra/config: change date link to something more understandable
*** dina_belova has joined #openstack-infra18:45
sdagueclarkb: I thought neutron ran both?18:46
sdagueclarkb: so neutron full is probably a long way from getting fixed18:47
clarkbsdague: tempest and nova run both neutron jobs, neutron does not18:48
sdagueok, so we should change that18:48
clarkbmaybe we make neutron and tempest run both and remove it from nova?18:48
*** GheRivero has quit IRC18:48
sdagueno, nova needs to run both as well18:48
sdaguebecause most of the fixes are actually on the nova side18:49
*** GheRivero has joined #openstack-infra18:49
sdaguethe fails are all basically interactions between nova and neutron18:49
sdagueI think restricting cells to nova, devstack, and tempest is fine18:50
sdaguethose are the projects that will need changes to make it work, as far as I know18:50
sdaguethough I thought alaski said he was working on it18:50
clarkbsdague: he said possibly late this week18:50
sdaguewell, it's thursday :)18:50
sdaguethat's pretty late18:50
clarkbI bring this up because some of the pain with testng right now is d-g resource related, we think we have a fix lined up but until then the use of 7 d-g slaves per patchset is pretty painful particularly if we aren't get useful testing out of it18:51
clarkbso trying to clean up where possible (if the tests are needed we should keep them, just checking that we actually need to be running a test 500 times a day that we know will fail)18:52
alaskiI'm currently sidetracked at the moment, though I have started looking into cells failures.18:53
alaskiIf it's causing issues I say turn it off for now18:53
sdagueclarkb: if cells gets restricted to nova, devstack, and tempest, is that helpful?18:55
*** krtaylor has quit IRC18:55
*** pabelanger has joined #openstack-infra18:55
sdaguebecause my guess is 95% of the required fixes will be in those 3 projects, so that should let cells make progress, but reduce d-g usage18:55
*** afazekas has joined #openstack-infra18:56
clarkba little, but probably not worth it is nova is the majority of the test load. I think we will be ok, just wanted to double check that cleanup wasn't necessary (the neutron stuff bothered me more because it is inconsistent and hasn't passed for much much longer)18:56
*** dina_belova is now known as DinaBelova18:56
*** HenryG has joined #openstack-infra18:57
*** alexpilotti has quit IRC18:57
sdagueclarkb: so once this merges -, it might be the root issue on the neutron smoke side18:57
clarkbcool, I will keep an eye on it18:58
jeblairclarkb: what's our logstash processing rate?19:00
clarkbjeblair: ~5k events per second19:00
clarkbjeblair: and the bottleneck appears to be in logstash filtering19:01
jeblairapparently very large sites process millions of events/sec.  but i don't have an idea of what kind of resources are behind that19:02
clarkbI think it probably helps a lot to put the filtering on the same node as the service19:02
clarkbwhich is something that jenkins slaves make tricky for us19:03
dkranzclarkb: The changes to add #openstack-qa and #savanna to irc logging are both pending.19:03
dkranzclarkb: I didn't understand how to do what you suggested in the review comment so perhaps you could just push one through19:03
dkranzclarkb: The other can then resubmit?19:03
clarkbdkranz: you can rebase one of the changes on the other, eg git review -d 37945 && git review -d 38152 && git rebase branch_for_3794519:04
clarkbdkranz: they don't have to be merged to do that, just create a normal gerrit dependency. Or we can approve one and let the other sort it out19:05
*** vijendar has joined #openstack-infra19:05
clarkbdkranz: I was also hoping that sdague would give his +1 to the qa logging19:05
HenryGdtroyer: ping19:05
*** jpeeler has joined #openstack-infra19:05
ekarlso-anyone here wanna help me understand why fails ?19:05
dkranzclarkb: He did in IRC but I can ask him to put it in the ticket if you prefer.19:06
clarkbekarlso-: ya, its a missing dependency that was masked by older versions of testrepository. Add 'discover' to your test-requirements19:06
clarkbdkranz: a +1 in the review would be awesome19:06
sdagueclarkb: done19:09
sdagueclarkb: also one more round of changes on the htmlifier out there, which I think is about where I'm good before we live it19:10
clarkbsdague: dkranz logging inbound19:10
clarkbsdague: will review19:11
sdaguemostly because my fun jquery stuff on the front to make all the debug lines disappear is going to have to wait, as getting testr functioning in the tempest gate really needs attension19:11
clarkbtestr in the gate is going to be baller19:12
*** HenryG has quit IRC19:12
sdagueyeh, so we're probably going to need some help to get to the bottom of things19:12
sdaguethere are at least 6 races right now19:12
*** HenryG has joined #openstack-infra19:12
*** chuckieb has quit IRC19:14
*** rahmu has quit IRC19:17
*** BobBall has quit IRC19:17
*** alaski has quit IRC19:17
*** adarazs has quit IRC19:17
*** comstud has quit IRC19:17
*** rockstar has quit IRC19:17
*** avtar has quit IRC19:17
*** SergeyLukjanov has quit IRC19:17
*** datsun180b has quit IRC19:17
*** rnirmal has quit IRC19:17
*** UtahDave has quit IRC19:17
*** dprince has quit IRC19:17
*** rfolco has quit IRC19:17
*** yamahata has quit IRC19:17
*** nibalizer has quit IRC19:17
*** westmaas has quit IRC19:17
*** ogelbukh has quit IRC19:17
*** mordred has quit IRC19:17
*** sshturm__ has quit IRC19:17
*** nijaba has quit IRC19:17
*** gaelL has quit IRC19:17
*** sgviking has quit IRC19:17
*** koolhead17 has quit IRC19:17
*** andreaf has quit IRC19:17
*** CaptTofu has quit IRC19:17
*** leifmadsen has quit IRC19:17
*** mgagne has quit IRC19:17
*** locke105 has quit IRC19:17
*** akscram has quit IRC19:17
*** sdake_ has quit IRC19:17
*** Ryan_Lane has quit IRC19:17
*** bnemec has quit IRC19:17
*** Shrews has quit IRC19:17
*** mrodden has quit IRC19:17
*** dkehn has quit IRC19:17
*** maurosr has quit IRC19:17
*** plomakin has quit IRC19:17
*** primeministerp has quit IRC19:17
*** bodepd has quit IRC19:17
*** Mithrandir has quit IRC19:17
*** ladquin has quit IRC19:17
*** jd__ has quit IRC19:17
*** vijendar has quit IRC19:17
*** dkliban has quit IRC19:17
*** prad has quit IRC19:17
*** pentameter has quit IRC19:17
*** anteaya has quit IRC19:17
*** mriedem has quit IRC19:17
*** jeblair has quit IRC19:17
*** thingee has quit IRC19:17
*** saper has quit IRC19:17
*** markmcclain has quit IRC19:17
*** mestery has quit IRC19:18
*** tian has quit IRC19:18
*** mikal has quit IRC19:18
*** odyi has quit IRC19:18
*** sdake has quit IRC19:18
*** jeremyb has quit IRC19:18
*** HenryG has quit IRC19:18
*** GheRivero has quit IRC19:18
*** DinaBelova has quit IRC19:18
*** melwitt has quit IRC19:18
*** uvirtbot has quit IRC19:18
*** dripton has quit IRC19:18
*** retr0h has quit IRC19:18
*** blamar has quit IRC19:18
*** shanewang has quit IRC19:18
*** jhesketh__ has quit IRC19:18
*** lbragstad1 has quit IRC19:18
*** chmouel has quit IRC19:18
*** jpeeler has quit IRC19:18
*** _TheDodd_ has quit IRC19:18
*** burt has quit IRC19:18
*** woodspa has quit IRC19:18
*** weshay has quit IRC19:18
*** pleia2 has quit IRC19:18
*** dhellmann has quit IRC19:18
*** jog0 has quit IRC19:18
*** devananda has quit IRC19:18
*** koobs has quit IRC19:18
*** afazekas has quit IRC19:18
*** pabelanger has quit IRC19:18
*** nati_ueno has quit IRC19:18
*** olaph has quit IRC19:18
*** ianw has quit IRC19:18
*** openstackgerrit has quit IRC19:18
*** SlickNik has quit IRC19:18
*** hughsaunders has quit IRC19:18
*** briancline has quit IRC19:18
*** Adri2000 has quit IRC19:18
*** sdague has quit IRC19:18
*** cyeoh has quit IRC19:18
*** ChanServ has quit IRC19:18
*** lifeless has quit IRC19:18
*** zaro has quit IRC19:18
*** davidhadas has quit IRC19:18
*** soren has quit IRC19:18
*** SpamapS has quit IRC19:18
*** fbo has quit IRC19:18
*** EmilienM has quit IRC19:18
*** Guest63475 has quit IRC19:18
*** jcooley has quit IRC19:18
*** sileht has quit IRC19:18
*** jgriffith has quit IRC19:18
*** cp16net has quit IRC19:18
*** NobodyCam has quit IRC19:18
*** vipul has quit IRC19:18
*** Daviey has quit IRC19:18
*** juice has quit IRC19:18
*** mirrorbox has quit IRC19:18
*** antonym has quit IRC19:18
*** spiffxp has quit IRC19:18
*** kiall has quit IRC19:18
*** oubiwann has quit IRC19:18
*** adam_g has quit IRC19:18
*** jaypipes has quit IRC19:18
*** crank has quit IRC19:18
*** jbjohnso_ has quit IRC19:18
*** annegentle has quit IRC19:18
*** EntropyWorks has quit IRC19:18
*** ttx has quit IRC19:18
*** stevebaker has quit IRC19:18
*** dstufft has quit IRC19:18
*** wayneeseguin has quit IRC19:18
*** notmyname has quit IRC19:18
*** russellb has quit IRC19:18
*** jlk has quit IRC19:18
*** vishy has quit IRC19:18
*** fungi has quit IRC19:18
*** clarkb has quit IRC19:18
*** kpepple_ has quit IRC19:18
*** danger_fo has quit IRC19:18
*** Alex_Gaynor has quit IRC19:18
*** changbl has quit IRC19:18
*** dkranz has quit IRC19:18
*** shardy has quit IRC19:18
*** s1rp has quit IRC19:18
*** jesusaurus has quit IRC19:18
*** ekarlso- has quit IRC19:18
*** simonmcc has quit IRC19:18
*** Ng has quit IRC19:18
*** lyz has quit IRC19:18
*** LinuxJedi has quit IRC19:18
*** dtroyer has quit IRC19:18
*** Vivek has quit IRC19:18
*** mancdaz has quit IRC19:18
*** mtreinish has quit IRC19:18
*** lcheng has quit IRC19:18
*** davidlenwell has quit IRC19:18
*** jkt has quit IRC19:18
*** harlowja has quit IRC19:18
*** dhouck has quit IRC19:18
*** mkoderer has quit IRC19:18
*** ewindisch has quit IRC19:18
*** mjfork has quit IRC19:18
*** arosen has quit IRC19:18
*** swifterdarrell has quit IRC19:18
*** larsks has quit IRC19:18
*** klrmn has quit IRC19:18
*** HenryG has joined #openstack-infra19:19
*** jpeeler has joined #openstack-infra19:19
*** vijendar has joined #openstack-infra19:19
*** afazekas has joined #openstack-infra19:19
*** pabelanger has joined #openstack-infra19:19
*** GheRivero has joined #openstack-infra19:19
*** DinaBelova has joined #openstack-infra19:19
*** Ryan_Lane has joined #openstack-infra19:19
*** avtar has joined #openstack-infra19:19
*** bnemec has joined #openstack-infra19:19
*** nati_ueno has joined #openstack-infra19:19
*** SergeyLukjanov has joined #openstack-infra19:19
*** melwitt has joined #openstack-infra19:19
*** datsun180b has joined #openstack-infra19:19
*** ewindisch has joined #openstack-infra19:19
*** Shrews has joined #openstack-infra19:19
*** mjfork has joined #openstack-infra19:19
*** rnirmal has joined #openstack-infra19:19
*** koolhead17 has joined #openstack-infra19:19
*** andreaf has joined #openstack-infra19:19
*** UtahDave has joined #openstack-infra19:19
*** changbl has joined #openstack-infra19:19
*** adam_g has joined #openstack-infra19:19
*** mrodden has joined #openstack-infra19:19
*** CaptTofu has joined #openstack-infra19:19
*** mkoderer has joined #openstack-infra19:19
*** _TheDodd_ has joined #openstack-infra19:19
*** dkranz has joined #openstack-infra19:19
*** dkliban has joined #openstack-infra19:19
*** leifmadsen has joined #openstack-infra19:19
*** prad has joined #openstack-infra19:19
*** burt has joined #openstack-infra19:19
*** markmcclain has joined #openstack-infra19:19
*** pentameter has joined #openstack-infra19:19
*** dprince has joined #openstack-infra19:19
*** mgagne has joined #openstack-infra19:19
*** anteaya has joined #openstack-infra19:19
*** mriedem has joined #openstack-infra19:19
*** woodspa has joined #openstack-infra19:19
*** rfolco has joined #openstack-infra19:19
*** weshay has joined #openstack-infra19:19
*** jhesketh__ has joined #openstack-infra19:19
*** EmilienM has joined #openstack-infra19:19
*** dkehn has joined #openstack-infra19:19
*** lbragstad1 has joined #openstack-infra19:19
*** olaph has joined #openstack-infra19:19
*** yamahata has joined #openstack-infra19:19
*** sgviking has joined #openstack-infra19:19
*** ianw has joined #openstack-infra19:19
*** lifeless has joined #openstack-infra19:19
*** zaro has joined #openstack-infra19:19
*** SlickNik has joined #openstack-infra19:19
*** nibalizer has joined #openstack-infra19:19
*** locke105 has joined #openstack-infra19:19
*** openstackgerrit has joined #openstack-infra19:19
*** akscram has joined #openstack-infra19:19
*** maurosr has joined #openstack-infra19:19
*** pleia2 has joined #openstack-infra19:19
*** jeblair has joined #openstack-infra19:19
*** dhellmann has joined #openstack-infra19:19
*** thingee has joined #openstack-infra19:19
*** plomakin has joined #openstack-infra19:19
*** uvirtbot has joined #openstack-infra19:19
*** westmaas has joined #openstack-infra19:19
*** mestery has joined #openstack-infra19:19
*** tian has joined #openstack-infra19:19
*** mirrorbox has joined #openstack-infra19:19
*** ladquin has joined #openstack-infra19:19
*** davidhadas has joined #openstack-infra19:19
*** soren has joined #openstack-infra19:19
*** ogelbukh has joined #openstack-infra19:19
*** jbjohnso_ has joined #openstack-infra19:19
*** mikal has joined #openstack-infra19:19
*** sdake_ has joined #openstack-infra19:19
*** mordred has joined #openstack-infra19:19
*** jog0 has joined #openstack-infra19:19
*** chmouel has joined #openstack-infra19:19
*** primeministerp has joined #openstack-infra19:19
*** bodepd has joined #openstack-infra19:19
*** Mithrandir has joined #openstack-infra19:19
*** sshturm__ has joined #openstack-infra19:19
*** dripton has joined #openstack-infra19:19
*** nijaba has joined #openstack-infra19:19
*** retr0h has joined #openstack-infra19:19
*** saper has joined #openstack-infra19:19
*** rahmu has joined #openstack-infra19:19
*** gaelL has joined #openstack-infra19:19
*** jd__ has joined #openstack-infra19:19
*** arosen has joined #openstack-infra19:19
*** odyi has joined #openstack-infra19:19
*** devananda has joined #openstack-infra19:19
*** larsks has joined #openstack-infra19:19
*** blamar has joined #openstack-infra19:19
*** sdake has joined #openstack-infra19:19
*** stevebaker has joined #openstack-infra19:19
*** koobs has joined #openstack-infra19:19
*** Guest63475 has joined #openstack-infra19:19
*** shanewang has joined #openstack-infra19:19
*** shardy has joined #openstack-infra19:19
*** s1rp has joined #openstack-infra19:19
*** jesusaurus has joined #openstack-infra19:19
*** jcooley has joined #openstack-infra19:19
*** jaypipes has joined #openstack-infra19:19
*** ekarlso- has joined #openstack-infra19:19
*** crank has joined #openstack-infra19:19
*** sileht has joined #openstack-infra19:19
*** jgriffith has joined #openstack-infra19:19
*** cp16net has joined #openstack-infra19:19
*** NobodyCam has joined #openstack-infra19:19
*** swifterdarrell has joined #openstack-infra19:19
*** Daviey has joined #openstack-infra19:19
*** SpamapS has joined #openstack-infra19:19
*** vipul has joined #openstack-infra19:19
*** klrmn has joined #openstack-infra19:19
*** juice has joined #openstack-infra19:19
*** annegentle has joined #openstack-infra19:19
*** EntropyWorks has joined #openstack-infra19:19
*** hughsaunders has joined #openstack-infra19:19
*** briancline has joined #openstack-infra19:19
*** ttx has joined #openstack-infra19:19
*** LinuxJedi has joined #openstack-infra19:19
*** dtroyer has joined #openstack-infra19:19
*** simonmcc has joined #openstack-infra19:19
*** Vivek has joined #openstack-infra19:19
*** jeremyb has joined #openstack-infra19:19
*** cyeoh has joined #openstack-infra19:19
*** sdague has joined #openstack-infra19:19
*** Adri2000 has joined #openstack-infra19:19
*** Ng has joined #openstack-infra19:19
*** mancdaz has joined #openstack-infra19:19
*** antonym has joined #openstack-infra19:19
*** fbo has joined #openstack-infra19:19
*** dstufft has joined #openstack-infra19:19
*** mtreinish has joined #openstack-infra19:19
*** wayneeseguin has joined #openstack-infra19:19
*** notmyname has joined #openstack-infra19:19
*** lcheng has joined #openstack-infra19:19
*** spiffxp has joined #openstack-infra19:19
*** BobBall has joined #openstack-infra19:19
*** russellb has joined #openstack-infra19:19
*** jlk has joined #openstack-infra19:19
*** vishy has joined #openstack-infra19:19
*** fungi has joined #openstack-infra19:19
*** ChanServ has joined #openstack-infra19:19
*** kiall has joined #openstack-infra19:19
*** clarkb has joined #openstack-infra19:19
*** oubiwann has joined #openstack-infra19:19
*** kpepple_ has joined #openstack-infra19:19
*** danger_fo has joined #openstack-infra19:19
*** Alex_Gaynor has joined #openstack-infra19:19
*** davidlenwell has joined #openstack-infra19:19
*** jkt has joined #openstack-infra19:19
*** lyz has joined #openstack-infra19:19
*** harlowja has joined #openstack-infra19:19
*** dhouck has joined #openstack-infra19:19
*** rockstar has joined #openstack-infra19:19
*** comstud has joined #openstack-infra19:19
*** adarazs has joined #openstack-infra19:19
*** alaski has joined #openstack-infra19:19
*** sets mode: +o ChanServ19:19
*** HenryG_ has joined #openstack-infra19:19
sdagueso, d-g forces ceilo on?19:19
russellbpabelanger: let's start with fail2ban, IMO19:19
*** _TheDodd_ has quit IRC19:19
*** _TheDodd_ has joined #openstack-infra19:19
*** avtar has quit IRC19:20
*** SergeyLukjanov has quit IRC19:20
*** datsun180b has quit IRC19:20
*** rnirmal has quit IRC19:20
*** UtahDave has quit IRC19:20
*** dprince has quit IRC19:20
*** rfolco has quit IRC19:20
*** yamahata has quit IRC19:20
*** nibalizer has quit IRC19:20
*** westmaas has quit IRC19:20
*** ogelbukh has quit IRC19:20
*** mordred has quit IRC19:20
*** sshturm__ has quit IRC19:20
*** nijaba has quit IRC19:20
*** gaelL has quit IRC19:20
*** sgviking has quit IRC19:20
russellbfungi: you mentioned changing the SRV records, i don't see them, should they be there?  $ dig -t SRV _sip._udp openstack.org19:20
dhellmannsdague: what's the deal with the keystone client issue? are you still waiting on us to do something with ceilometer?19:20
pabelangerrussellb, ya, testing it out now locally19:21
*** avtar has joined #openstack-infra19:21
*** SergeyLukjanov has joined #openstack-infra19:21
*** datsun180b has joined #openstack-infra19:21
*** rnirmal has joined #openstack-infra19:21
*** UtahDave has joined #openstack-infra19:21
*** dprince has joined #openstack-infra19:21
*** rfolco has joined #openstack-infra19:21
*** yamahata has joined #openstack-infra19:21
*** sgviking has joined #openstack-infra19:21
*** nibalizer has joined #openstack-infra19:21
*** westmaas has joined #openstack-infra19:21
*** ogelbukh has joined #openstack-infra19:21
*** mordred has joined #openstack-infra19:21
*** sshturm__ has joined #openstack-infra19:21
*** nijaba has joined #openstack-infra19:21
*** gaelL has joined #openstack-infra19:21
*** SergeyLukjanov has quit IRC19:21
*** zehicle_at_dell has joined #openstack-infra19:21
russellbpabelanger: cool ... we documented it in ATDG, but need to make sure all the patterns are still current19:21
*** SergeyLukjanov has joined #openstack-infra19:21
clarkbsdague: yes iirc19:22
sdaguedhellmann: yes, there is a unit test fail to land that19:22
*** HenryG has quit IRC19:22
* dhellmann mutters under his breath19:23
pabelangerrussellb, ya, I'm more concerned with the default filters that ship with the fail2ban package right now19:23
pabelangerbut will check it out19:23
dhellmannsdague: can you point me at a review or something?19:23
sdaguedhellmann: I did bring it up on the ceilo meeting today, jd__ said he'd look at it19:23
sdaguebut more eyes good19:23
russellbpabelanger: oh didn't know it included asterisk support19:23
dhellmannsdague: ok, if he's on it that's fine19:23
sdaguedhellmann: I don't know if he did19:23
pabelangerstill waiting for leifmadsen to drop of the book at my place, like he said the last time he was in town19:23
dhellmannI saw something in the backlog about it earlier, but didn't have a chance to look into it19:23
*** ewindisch has quit IRC19:23
*** mjfork has quit IRC19:23
*** arosen has quit IRC19:23
*** swifterdarrell has quit IRC19:23
*** larsks has quit IRC19:23
*** klrmn has quit IRC19:23
pabelangerrussellb, it doesn't just other apps19:23
sdagueclarkb: ok, so we should probably try to sync the d-g defaults back to devstack19:24
*** sarob has joined #openstack-infra19:24
dhellmannsdague: after we have this fixed, I would like to prioritize a gate test using that nova plugin we have19:24
sdaguedhellmann: sure19:24
sdague1.0.2 is where you guys uncapped, right?19:25
jd__dhellmann: sdague: if that can't wait Monday, go ahead, I won't do that before then19:26
dhellmannjd__: OK19:26
dhellmannsdague: someone else did that so I'd have to check19:26
*** zehicle_at_dell has quit IRC19:27
*** krtaylor has joined #openstack-infra19:28
openstackgerritSean Dague proposed a change to openstack/requirements: update global requires on clients for keystone fix
sdagueman, one day I'll stop chasing this requirements thread....19:28
sdaguecurrently preventing me from installing devstack19:29
*** larsks has joined #openstack-infra19:29
*** Ryan_Lane has quit IRC19:30
*** openstack has joined #openstack-infra19:32
*** Ryan_Lane has joined #openstack-infra19:32
*** mjfork has joined #openstack-infra19:35
*** arosen has joined #openstack-infra19:35
*** swifterdarrell has joined #openstack-infra19:35
*** klrmn has joined #openstack-infra19:35
*** dprince has quit IRC19:37
dansmithis grenade borken?19:38
sdaguesorry, no19:38
sdaguewrong window19:39
sdaguedansmith: you got a link to a fail?19:39
dansmithnot yet, I just see at least one of mine is failing on grenade and I don't think there's any way it could19:40
sdagueany config changes?19:40
*** dims has joined #openstack-infra19:40
sdaguewe're gating on grenade everywhere19:40
dansmithnope, no config changes19:40
*** HenryG_ has quit IRC19:41
dansmithand one of the ones after it passes grenade19:41
dansmithsee 3636319:41
sdaguewell, there is always the possibility of a race19:41
sdaguethat we didn't find19:41
Ryan_Laneclarkb: I'm also interested in learning logstash19:41
*** HenryG_ has joined #openstack-infra19:41
Ryan_LaneI'd like to use it at wikimedia19:41
*** dkliban has quit IRC19:41
dansmithyeah, all my review numbers are palindromes, cause I'm awesome19:41
*** afazekas is now known as afazekas_zz19:41
clarkbdansmith: I need that skill19:43
clarkbRyan_Lane: we are using it for our test logs, and it seems to be stable now. Current issue is it can't keep up with the load19:43
dansmithclarkb: :P19:43
sdagueyeh, but it means you can mess with him a lot by making things one off19:43
Ryan_Laneheh. that's what I worry about19:44
Ryan_Lanebecause we have about 1000 systems and about 400 instances and we want to send all syslog data to it19:45
clarkbRyan_Lane: and in our case it seems to be related to the weird way we squeeze everything through a funnel19:45
clarkbRyan_Lane: if you can distribute the work load to as near the log source as possible you should be fine19:45
clarkbRyan_Lane: we don't trust our jenkins slaves so we don't have that luxury19:45
sdaguedansmith: looks like a more fundamental fail -
* Ryan_Lane nods19:45
*** HenryG_ has quit IRC19:45
dansmithsdague: ah19:46
clarkbRyan_Lane: also it does a good job of scaling horizontally so throwing hardware at the problem is a possibility19:46
*** HenryG has joined #openstack-infra19:46
sdagueclarkb: any ideas about that last link19:46
clarkbsdague: looking19:46
Ryan_Lanewe're going to have an elastic search cluster anyway, so I may be able to reuse it for this as well19:46
clarkbsdague: ya one sec19:47
clarkbsdague: dansmith is the bug19:47
uvirtbotLaunchpad bug 1204625 in openstack-ci "Devstack gate launch jobs timeout confusing Jenkins and devstack gate" [High,Triaged]19:47
clarkbsdague: dansmith and are two potential things that will mitigate it, but I really wanted jeblair to weight in on those changes before I approve anything19:48
*** burt has quit IRC19:48
clarkbsdague: dansmith: the tl;dr is that jenkins and devstack-gate get out of sync because some of the d-g jobs timeout19:48
clarkband that causes weirdness19:48
maurosreasy review: porting this fix to api-v3 fixed-ips19:49
clarkbsdague: if you want to review 38674 that would be helpful19:49
maurosrsorry, wrong channel19:49
Ryan_Laneok. redeployed the updated wiki skin with a live hack for the login link19:50
sdagueclarkb: looking19:50
Ryan_LaneI'd generally not do so, but the old skin has an XSS vulnerability19:50
*** _TheDodd_ has quit IRC19:50
clarkbRyan_Lane: want to fire up a screen on wiki.o.o that I can screen -x to?19:51
Ryan_LaneI always forget how to do this19:51
Ryan_LaneI have a screen session running on it19:51
Ryan_Laneif you're also root, you can just screen -x, right?19:51
clarkbRyan_Lane: if you have a screen running I should be able to just attach to it. yup19:51
sdagueclarkb: no additional typing or fields in those jenkins exceptions?19:52
clarkbsdague: no :(19:52
sdaguethen seems fine to me19:52
clarkbsdague: you can see the exception in myjenkins.py19:52
clarkbRyan_Lane: I am attached19:52
Ryan_LaneI'll give you a walkthrough19:52
dansmithman I really need to write a recheck-all script or something, because ten patches needing rechecks sucks :/19:52
Ryan_Lane mediawiki is installed under /srv/mediawiki19:52
Ryan_LaneI have two versions always there, under slot0 and slot119:52
Ryan_Laneslot0 is currently running19:53
clarkbRyan_Lane: how can you tell which one is running?19:53
clarkbw symlink>19:53
Ryan_Lanethe apache alias is for /srv/mediawiki/w19:53
clarkbgot it19:53
Ryan_LaneI use the wikimedia foundation deployment branches19:54
*** sarob has quit IRC19:54
Ryan_Laneit's abnormal, but there's a reason for it19:54
*** nayward has joined #openstack-infra19:54
*** sarob has joined #openstack-infra19:54
Ryan_Laneif you notice, that branch has a large number of extension submodules19:54
Ryan_Laneall wikimedia foundation extensions that are deployed to wikimedia sites are there19:54
Ryan_Lanethey are known to work with this version of mediawiki19:55
Ryan_Lanewe have 5 extensions that don't come from wikimedia19:55
clarkbdansmith: there is a tool like that in zuul/tools/* iirc19:55
clarkbRyan_Lane: I see them19:55
openstackgerritDiane Fleming proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide
Ryan_Laneextensions/EmbedVideo/ extensions/OpenID/ extensions/ReplaceText/ extensions/googleAnalytics/ skins/strapping/19:55
Ryan_LaneOpenID will soon also be deployed on wikimedia sites19:55
Ryan_Laneand that'll be one less we have to manage manually19:56
dansmithclarkb: oh, nice19:56
Ryan_Laneso that we can easily switch between versions, I move some things outside of the mediawiki path19:56
Ryan_Laneimages and Settings.php19:56
*** pcrews has joined #openstack-infra19:57
Ryan_LaneLocalSettings.php has been modified to require the shared Settings.php19:57
clarkband that allows you to not need to configure the two slots independently19:57
Ryan_Laneand /w/images is an alias in apache pointing at /srv/mediawiki/images19:58
sarobanyone up for reviewing
Ryan_Lanehow to upgrade:19:58
Ryan_Lane1. figure out which slot is being used, prepare to upgrade the other one19:58
sarobits holding up publishing some content I need for a meetup tonight19:59
clarkbsarob: I can review when Ryan_Lane is done giving me the howto19:59
Ryan_Lanedon't ever use /srv/mediawiki/w, always use the slot (otherwise git will fuck things up)19:59
sarobokay, thanks!19:59
clarkbRyan_Lane: good to know19:59
Ryan_Laneit sets submodule config paths to the directory you are in19:59
Ryan_Laneso it'll reference w, which will cause problems later20:00
Ryan_Lane2: fetch the changes via git fetch20:01
Ryan_Lane3: find the newest wmf branch20:01
Ryan_Lane4: create a local branch that tracks the newest remote branch20:02
Ryan_Lanefunny enough, there's already a newer branch20:02
Ryan_Lanewe could upgrade again right now. let me see if this is already deployed on wikimedia20:02
clarkbyou guys go fast20:02
*** datsun180b has quit IRC20:03
*** HenryG has quit IRC20:03
Ryan_Lanewe do weekly major upgrades20:03
Ryan_Lane <--20:03
Ryan_Lanerunning this new branch20:03 always gets deployed to first20:03
Ryan_Laneit's lesser used and lets devs eat their dogfood20:03
Ryan_Laneso, let's actually do an upgrade20:04
Ryan_Lane5: checkout the new branch20:04
Ryan_Lane6: update submodules: git submodule update —init20:04
*** sdake has quit IRC20:05
clarkbdo you typically want to wait for the new version to be deployed on before upgrading wiki.o.o?20:06
Ryan_Laneyes. always :)20:06
*** dkliban has joined #openstack-infra20:06
Ryan_Laneotherwise they may not be ready to actually deploy it20:06
Ryan_Laneor may rollback because of something being broken20:06
clarkbI see20:07
Ryan_Lanea major downside to using the wmf branches is that they don't backport security fixes20:07
Ryan_Laneyou have to upgrade20:07
Ryan_Lanethe good thing is that we're using almost nothing bug wikimedia code and that will just work with upgrades20:07
* Ryan_Lane yawns20:09
clarkbis the alternative to the wmf branches using "stable" releases and hoping that the plugins all work?20:10
Ryan_Lanewhich sucks20:10
Ryan_Laneextension management has always been the worst part of using MW20:10
Ryan_Lanethere's some movement to fix that, but I'll believe it when I see it20:10
clarkbthis seems like a reasonable solution even if it does mean we sit and wait for all of the submodules to update :)20:11
Ryan_Laneif we keep these more up to date more often this takes less time20:11
Ryan_Lanewe could actually run a cron with: git fetch && git submodule fetch20:12
Ryan_Laneand this would take basically no time20:12
clarkbRyan_Lane: ++20:12
clarkball done20:12
*** hashar has joined #openstack-infra20:12
Ryan_Lanewikimedia occasionally deprecates extensions20:13
*** woodspa has quit IRC20:13
Ryan_Laneif we are using them, we should stop. if we aren't using them, we just delete them20:13
fungirussellb: did anyone get back to you on the srv rrs? your dig command was missing a .20:13
fungirussellb: dig -t SRV _sip._udp.openstack.org20:14
russellboh, oops20:14
Ryan_Lanethe other slot has a live-hacked strapping skin20:14
Ryan_Laneso I ensured it's copied20:14
russellbthanks.  d'oh20:14
fungirussellb: also the _tcp version is there as well20:14
Ryan_Laneotherwise I'd just do a git pull20:14
russellbfungi: yep see it now :-)20:14
fungino worries20:14
* fungi goes back to being all conferency20:15
clarkbRyan_Lane: so you track master with strapping?20:15
clarkbRyan_Lane: and the other extensions looks like. got it20:15
*** nayward has quit IRC20:15
Ryan_LaneI had pulled in some security fixes for opend20:15
Ryan_Lanethey are in master now20:16
clarkbRyan_Lane: and they are causing the merge conflict?20:16
Ryan_LaneI had cherry-picked them when they weren't yet merged20:16
clarkbso a git reset --hard origin/master is needed?20:16
* Ryan_Lane has early access20:16
Ryan_Lanethey had been modified before being merged20:16
Ryan_Laneso the changes conflicted20:16
Ryan_LaneI knew that was the case because I had done it20:17
Ryan_Laneit would be good to have a place to document these things20:17
clarkbRyan_Lane: perhaps at /srv/mediawiki/NOTES? or /root/NOTES?20:17
Ryan_Laneyeah, that'll work20:17
Ryan_Laneextension authors for mediawiki usually try to ensure backwards compatibility20:18
Ryan_Laneand master is meant to be usable20:18
Ryan_Laneso doing a git pull on the other extensions is usually fine20:18
*** hashar has quit IRC20:19
Ryan_Laneok. now all of them are updated20:19
Ryan_Lanelet me write a quick script that we should puppetize20:19
Ryan_Lanethe backup script using /srv/mediawiki/w is fine, since it doesn't do anything with git20:21
*** DinaBelova has quit IRC20:21
clarkbRyan_Lane: my paranoia wonders if the script will be runnable from a web browser20:21
Ryan_Laneand we want to reference the running version of MW20:21
Ryan_Lanewe alias w and images directly20:22
Ryan_Lane /srv/mediawiki is not accessible20:22
* Ryan_Lane always aliases things in and hates modifying doc roots20:22
*** hashar has joined #openstack-infra20:22
clarkbcool I see that in the vhost20:23
Ryan_Laneok… so, now to backup20:23
Ryan_Lanethere could be schema upgrades with the upgrade, so we want to ensure we have a current backup20:23
Ryan_Laneschema upgrades are backwards compatible20:23
Ryan_Lanebut it's still a good idea :)20:24
openstackgerritDiane Fleming proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide
clarkbRyan_Lane: are we at a point where we can change the symlink? do we have to stop apache before doing that?20:27
Ryan_Laneone sec. talking to EFF20:28
*** koolhead17 has quit IRC20:29
*** sdake has joined #openstack-infra20:29
*** sdake has quit IRC20:29
*** sdake has joined #openstack-infra20:29
*** sdake has quit IRC20:32
clarkbsarob: I reviewed that change. There is one tiny nit pick I would like to see cleaned up, I can approve it once that is done20:35
*** sarob has quit IRC20:35
*** sarob has joined #openstack-infra20:35
Ryan_Lanesorry. in person EFF ambush20:36
*** sdake has joined #openstack-infra20:37
*** sdake has quit IRC20:37
*** sdake has joined #openstack-infra20:37
clarkbsarob: let me know if you can't make that change because you are at a conference or something. I can clean it up and push it through20:38
clarkbRyan_Lane: no problem20:38
*** woodspa has joined #openstack-infra20:40
fungiRyan_Lane: that's one of the coolest excuses ever20:42
* fungi wants to get ambushed by the eff sometimwe20:42
*** SergeyLukjanov has quit IRC20:42
clarkbfungi: can you review sarob's change too?20:42
lifelesssdake: mordred: I accept regular patches, no need for bzr.20:42
Ryan_Lanethem: "Hi. When is https by default for all wikipedia users happening?"20:42
sdagueclarkb: badness?20:42
clarkbfungi: its fine if not, just a new jenkins job20:42
Ryan_Laneme: "NIce to meet you too."20:42
openstackgerritcolinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training
clarkboh colinmcnamara addressed my comment \o/20:43
fungiclarkb: listening to frencil from sparkfun talk about open hardware, but can multi-task for a bit20:43
clarkbsdague: that looks like the same bug dansmith ran into :/20:43
sdagueclarkb: ok, wonder what's making it worse today20:44
clarkbsdague: the key bit is the inprogress job failing at the beginning of the console outputs20:44
fungiRyan_Lane: ha. that's classic20:44
clarkbsdague: its related to test load, more tests = less d-g slaves = higher chance of d-g launch jobs timing out20:44
sdagueright, fair20:44
clarkbRyan_Lane: ready to go when you are even if it looks like I am busy multitasking :)20:44
Ryan_Lanemaybe 10 mins20:44
*** sdake has quit IRC20:46
openstackgerritThomas Goirand proposed a change to openstack/requirements: Removes the cap for SQLAlchemy
*** jjmb has joined #openstack-infra20:47
*** dkranz has quit IRC20:48
sdaguelifeless: working on and reworking patches outside of an scm it really manual. I'd rather just keep subunit2html over in gerrit if it means loosing git to work on it to move it elsewhere.20:49
*** sdake has joined #openstack-infra20:50
*** sdake has quit IRC20:50
*** sdake has joined #openstack-infra20:50
clarkbsdague: jeblair just approved my change that should hopefully make d-g nodes happier20:51
*** sdake has quit IRC20:51
*** Ryan_Lane has quit IRC20:51
*** sdake has joined #openstack-infra20:51
*** sdake has quit IRC20:51
*** sdake has joined #openstack-infra20:51
clarkbhopefully the gate clears up long enough for it to get through (there have been issues with some of the tests in there not related to the derpyness we are seeing20:51
sdagueclarkb: awesome20:51
clarkbif necessary I can shove it in with extra force20:51
sdagued-g kick to the groin!20:52
*** Ryan_Lane has joined #openstack-infra20:52
*** jjmb1 has joined #openstack-infra20:52
Ryan_Laneclarkb: ok. ready to keep going20:52
clarkbRyan_Lane: I am ready20:53
Ryan_Laneso, after doing a backup, it's necessary to run mediawiki's update script20:53
Ryan_Laneyou run it before you switch the software over20:53
Ryan_Lanesince the old software will work with the new schema20:53
Ryan_Lane(I wish OpenStack worked like this ;) )20:53
clarkbRyan_Lane: can you document that php command here in the channel?20:54
Ryan_Laneso, notice:
clarkb(the channel is logged)20:54
Ryan_Laneit's php maintenance/update.php --quick20:54
clarkbrun within the slot you are upgrading20:54
Ryan_Lanefrom inside of the slot you are upgrading20:54
*** jjmb has quit IRC20:54
Ryan_Lanenote that the wiki is still up and it running the older version20:54
Ryan_Lanethen you just switch the link20:55
Ryan_Laneand check again20:55
Ryan_Lanenew version is running. no downtime20:55
clarkbnice, so restarting apache is not necessary?20:56
Ryan_Lanebecause we just switch the link20:56
Ryan_Laneand the link is the alias20:56
*** jjmb has joined #openstack-infra20:57
Ryan_Laneassuming you don't make any live hacks or pull in pre-merge security fixes, upgrading is very straightforward20:57
clarkbperfect, I feel like I have a much better grasp of this now. Though if something goes wrong I think my abilities to debug and correct are still minimal but this is a start20:57
Ryan_Lanethere's a few things that should get puppetized20:58
Ryan_LaneI should open bugs20:58
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Increase d-g launch timeout to 20 min
Ryan_Lanewe should have backups running in a cron20:58
clarkbRyan_Lane: dtroyer just mentioned that he noticed much larger margins after yesterdays upgrade20:59
Ryan_Laneand we should puppetize the wiki config20:59
jeblairclarkb: gah you already have a patch for that don't you?20:59
clarkbRyan_Lane: page margins I guess so skin related maybe?20:59
*** jjmb1 has quit IRC20:59
clarkbjeblair: I do but I only did 15 minutes. we can useyours20:59
Ryan_Laneah, meaning the page content area is smaller?20:59
openstackgerritcolinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training
Ryan_Laneit's possible. I'm using an unpatched upstream version now. it's the same skin used by redhat for their openstack wiki21:00
clarkbRyan_Lane: that was how I grokked it21:00
clarkbRyan_Lane: I have detached from screen. feel free to close it anytime21:02
jeblairdocs-draft rsync is a 58% done21:02
clarkbRyan_Lane: and thank you for running me through that21:02
dtroyerRyan_Lane: right, the left and right margins are now together over 1/4 the page width21:02
Ryan_Laneclarkb: totally welcome21:02
Ryan_Lanedtroyer: hm. I'll need to take a look at the CSS to see why21:02
dtroyerRyan_Lane: ok, thanks.  my command tables are wrapping like crazy now… ;)21:03
Ryan_Laneoh wow. awesome.21:03
Ryan_LaneI forgot that this upgrade allows editing from mobile skin now21:04
dtroyerthat's a plus21:04
clarkbjeblair: fungi mordred I plan to go through the IRC logs from today and write up a doc on this process21:04
clarkbI figure hosting it not on the wiki is a good thing :)21:04
fungiclarkb: agreed. don't want a catch-22 outage to push me to read instructions through the wayback machine21:05
*** rfolco has quit IRC21:05
clarkbfungi: if you approve that I can babysit21:06
jeblairclarkb: thanks!21:07
*** krtaylor has quit IRC21:08
*** sarob has quit IRC21:08
fungiclarkb: lgtm. approved21:11
* Ryan_Lane grumbles21:11
*** sarob has joined #openstack-infra21:12
Ryan_Lanemobile editing doesn't work with openid21:12
Ryan_Laneso, yeah….21:12
clarkboh well I guess21:12
*** jjmb has quit IRC21:12
openstackgerritA change was merged to openstack-infra/config: Increase d-g launch timeout to 20 min
*** jjmb has joined #openstack-infra21:15
openstackgerritA change was merged to openstack-infra/zuul: Enable warnerrors for pbr to catch Sphinx warnings
clarkbjog0: why not make rootwrap a small !python thing?21:18
clarkbit might do weird things to dependencies I suppose21:18
jog0clarkb: I think that is an option21:18
jog0nova unit tests are borking left and right21:19
clarkbyeah I think someone logged an issue again openstack-ci about it and I haven't had time to swing around and reassign it to nova21:19
jog0clarkb: I cannot reproduce locally at least not yet21:20
jog0haven't done a tox -r yet21:20
clarkbjog0: you can try grabbing the subunit log from one of the failed tests then `testr load $fileyoudownloaded && testr run --analyze-isolation`21:20
clarkbjog0: that will use the test partitions that were used when it failed.21:21
jog0just ran tox -r .... so will have to wait21:21
clarkband it will try to tell you which tests if any conflict with it21:21
clarkbjog0: ok, no rush21:21
clarkb(I would have to build a new nova dev env to do it myself so you are ahead of me)21:21
jog0clarkb: heh21:21
*** DinaBelova has joined #openstack-infra21:22
openstackgerritcolinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training
clarkbjog0: the other thing I like to do is `testr run $test_name` a lot of tests fail when run on their own and if they do knowing that makes debugging eaiser21:22
*** jjmb has quit IRC21:22
jog0yeah I tried that no luck21:22
jog0I hate it when things don't fail21:22
openstackgerritcolinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training
*** DinaBelova has quit IRC21:27
jog0clarkb: I tried testr load and connot  reproduce21:29
fungiclarkb: jeblair: on i guess the real problem is that jeepyb needs its updated to require a newer (non-broken) pbr?21:30
jog0' testr run --analyze-isolation ' did nothing21:30
jog0ohh I bet this is a python version thing21:30
jog0saw one earlier21:30
clarkbjog0: analyze-isolation should run off and run the test alone, if that passes it bisects the test list until it can find a minimal set of tests that when run together result in the failure. It is possible that it is non deterministic beyond just test order21:31
jog0clarkb: it ran nothig not sure why21:32
*** DinaBelova has joined #openstack-infra21:32
clarkbjog0: did the testr load load anything?21:32
jog0clarkb: I think so21:33
clarkb--analyize-isolation runs based on the previous run. load is a way of changing out the previous run21:33
*** bclifford has joined #openstack-infra21:33
jog0let me paste my terminal output21:33
clarkbfungi: thats possible. do we need to be less restrictive in the
jog0testr load says Ran tests in 37.906s  ; then PASSED21:34
openstackgerritA change was merged to openstack-infra/config: Update CI systems to validate and publish section openstack-training
jog0i did this inside the tox py27 venv21:34
clarkbjog0: it should say failed if the job failed. you grabbed and gunzipped it?21:35
jog0anyway I gotta jet (boarding one soon)21:35
jog0I did│21:35
jog0and gunzipped yeah21:35
jog0kk out for real21:35
fungiclarkb: mmm, no. the problem seems to be that relying on to install dependencies doesn't work when it runs with an older version of that dependency which prevents from running... chicken and egg problem21:36
clarkbfungi: so it isn't downloading the newer version because the one it already has is sufficient?21:36
clarkbwe can probably do a git clean to fix that21:36
*** DinaBelova has quit IRC21:37
fungiclarkb: possibly, though i think it's more porobably that is throwing an exception due to the broken pbr before it gets far enough to upgrade pbr21:39
fungiyeesh, not enough internets21:39
clarkbfungi: yeah but by removing broken pbr first it should grab a new one21:40
clarkband hopefulyl bypass the brokeness21:40
fungiso change the jeepyb puppet module exec to pip uninstall pbr||true before running jeepyb's
*** rnirmal has quit IRC21:42
fungii know we can manually do that to break the deadlock, but not sure if i need to think about others who might also use this module (who aren't us), or other servers where we haven't noticed this is broken, or in the future if the same sort of issue reemerges21:44
clarkbfungi: yeah something like that may be necessary21:44
fungithe fact that jeepyb uses pbr before possibly upgrading pbr leaves it open to this sort of issue, in a way which may be particular to pbr21:46
clarkbyeah I think this is definitely a pbr type of problem, because it is required to bootstrap itself21:46
fungimaybe mordred has an opinion, if he's got an internet at the moment21:46
clarkbfungi: you should just hunt him down in the conference center :)21:47
*** changbl has quit IRC21:47
fungii will in fact see his talk in about an hour21:47
jeblairyes, let's beat on mordred because this is sposed to work imho21:47
fungiso i can merely abuse him with it then21:47
jeblair(if we have to do something yucky to fix this one deadlock case, ok, but we should avoid accepting this as a systemic problem)21:48
fungii agree21:48
clarkbjeblair: fungi: I intend on approving BobBall's meetbot change tomorrow when meetings are quiet21:50
jeblairclarkb: link?21:50
fungii should look it over then, if i haven't already21:50
clarkbjeblair: fungi
jeblairclarkb: thx; was not watching meetbot repo21:51
*** UtahDave has quit IRC21:53
openstackgerritAnne Gentle proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide
*** UtahDave has joined #openstack-infra21:54
openstackgerritA change was merged to openstack-dev/pbr: Stop trying to install old pip and setuptools
openstackgerritA change was merged to openstack-dev/pbr: Install jeepyb and requirements from REPODIR
*** alexpilotti has joined #openstack-infra21:57
clarkbsdague: I reviewed your htmlifier change21:58
*** dhellmann is now known as dhellmann_21:59
*** vipul is now known as vipul-away22:01
openstackgerritA change was merged to openstack-dev/pbr: Swap the order of configparser imports
openstackgerritA change was merged to openstack-infra/devstack-gate: Handle Jenkins node already being added to Jenkins
clarkb\o/ zuul is merging all the things22:04
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Report an unknown time if a change ahead has failed
openstackgerritA change was merged to openstack-infra/devstack-gate: Add missing projects to pre-cache list
clarkbjeblair: I think the recent zuul improvements are awesome btw. On the fly reconfigs, queue priority, kicking out potentially unmergable changes and so on22:07
clarkbdespite the ridiculous test load zuul itself seems to be cool and collected.22:07
openstackgerritSean Dague proposed a change to openstack-infra/config: change date link to something more understandable
sdagueclarkb: thanks22:07
sdaguecode review is so useful to prevent me from doing stupid things :)22:08
sdagueok, time to ride my bike home, see you guys tomorrow22:08
jeblairclarkb: thx; wait till you see the new status screen.  :)22:08
clarkbjeblair: are you going to draw the gate queue?22:09
lifelesssdague: so moving stuff to git is possible, need tuits22:10
jeblairclarkb: hehe, not yet.  :)  just this:
* clarkb is waiting for the next launch jobs to kick off now that the d-g chang merged22:10
* fungi has seen it, is excited22:10
clarkbjeblair: shiny22:10
lifelesssdague: doesn't git have a bzr translator though?22:10
*** vijendar has quit IRC22:11
lifelessall that said though, subunit2html is an infra thing; I don't particularly care where it lives - there's no 'consolidate all users of libfoo into projectfoo' meme22:11
lifelessI have a bunch of generic filters in subunit but am enthusiastic for others to live whereever they want to22:12
*** mrodden has quit IRC22:17
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status
openstackgerritEvgeny Fadeev proposed a change to openstack-infra/askbot-theme: changed css for the language nav
*** vipul-away is now known as vipul22:18
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make it easier to get to the cacti graphs
*** prad has quit IRC22:19
clarkbjeblair: comments on patchset two of 3859822:21
clarkbjeblair: whoops I accidentally approved that zuul change, I removed my +A so it won't actually merge22:24
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status
jeblairclarkb: thx22:26
*** pentameter has quit IRC22:29
*** jhesketh__ has quit IRC22:31
*** DinaBelova has joined #openstack-infra22:32
*** mrodden has joined #openstack-infra22:32
*** rahmu_ has joined #openstack-infra22:33
*** BobBall_ has joined #openstack-infra22:33
*** jhesketh__ has joined #openstack-infra22:33
*** alaski_ has joined #openstack-infra22:33
ttxjeblair: what about conway's law ?22:34
*** comstud_ has joined #openstack-infra22:34
*** rockstar` has joined #openstack-infra22:34
*** adarazs_ has joined #openstack-infra22:34
*** adarazs has quit IRC22:34
*** rockstar has quit IRC22:34
*** rahmu has quit IRC22:34
*** alaski has quit IRC22:34
*** BobBall has quit IRC22:34
*** comstud has quit IRC22:34
*** rahmu_ is now known as rahmu22:34
*** DinaBelova has quit IRC22:37
jeblairttx: nothing in particular, just found it interesting.  :)22:40
*** hashar has quit IRC22:40
*** alexpilotti has quit IRC22:41
openstackgerritA change was merged to openstack-infra/askbot-theme: changed css for the language nav
clarkbjeblair: the reset back to head doesn't break anything because the fake change is referred to via a different ref?22:42
clarkb(in this case r)22:42
*** comstud_ is now known as comstud22:43
*** danger_fo is now known as danger_fo_away22:44
*** pcrews has quit IRC22:46
*** vipul is now known as vipul-away22:49
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Delay initialization of local repos
clarkbthe launch jobs haven't exploded since my patch went in. I think we are good22:55
jeblairclarkb: yeah, r points to the fake commit22:56
jeblairclarkb: then the 'hexsha' attr of that gets used later22:57
*** vipul-away is now known as vipul22:57
*** zul has joined #openstack-infra22:58
*** CaptTofu has quit IRC22:59
*** pcrews has joined #openstack-infra23:00
*** sdake has quit IRC23:00
*** avtar has quit IRC23:00
lifelessso whats needed to get landed ?23:00
clarkblifeless: I want jeblair to review it23:04
clarkblifeless: if he is ok with it I can approve it23:04
clarkband do the babysitting and so on.23:04
clarkbjeblair: ^23:04
*** sdake has joined #openstack-infra23:05
*** changbl has joined #openstack-infra23:09
*** beagles has joined #openstack-infra23:10
jeblair clarkb, lifeless: the line above the find command is where we delete a list of files generated by this run of the mirror script23:11
jeblair clarkb, lifeless: (which is an intermediate step to removing the code that generates the list of files from this run of the mirror script)23:11
lifelessjeblair: the line above deletes the html indices that would collide with each other23:14
lifelessjeblair: the line I added generates a per-source list that doesn't conflict23:14
lifelessjeblair: I'd also be happy with alternatives like 'use rsync' if you wanted to enable anonymous rsync on the server23:15
lifelessjeblair: but spidering is super slow, really don't want to do that.23:15
*** bclifford has left #openstack-infra23:18
saroblast manuals build failed23:19
sarobsad panda23:20
sarobcan i bug someone to take a look?23:21
clarkbsarob: thats the translation update. is that the one you are wanting us to look into? looks like transifex has done something funny with their certs23:22
sarobyeah, i saw that. also failed on training-manuals23:22
sarobbut i cant find where in the output23:22
clarkbsarob: looks like it failed to parse something?23:23
clarkbsarob: SEVERE: Unparseable XPointer: controller-os23:25
sarobhmm, found it23:25
*** pcrews has quit IRC23:26
jeblairlifeless: *nod* this feels like it is possible that it is in conflict with the way we decided to handle the canonical mirror file list.  it may be okay, but it's giving me pause right now, and i'm listening to monty talk about tripleo atm.23:27
clarkbsarob: does that get you sorted?23:28
sarobno unfortunately23:28
clarkbsarob: does it continue to fail after fixing the controller-os thing?23:29
sarobmy local maven doesnt handle remote include statement, but the ci was23:29
sarobthe section exists23:29
sarobso the error is wierd23:29
clarkboh are the docs referring back to github to build themselves?23:30
sarobyeah, using <xi:include href="
sarobit was working23:31
clarkbin general you probably don't want to depend on github for anything build related. github isn't reliable enough23:31
clarkbI wonder if it tried fetching something from github and failed because github 404'd or 500'd23:32
sarobits the only way i know of bringing the manuals content23:32
sarobi dont think so23:32
clarkbisn't the content in the current repo?23:32
sarobit seems to have a problem with using include xpointer statement23:32
clarkbI am probably misunderstanding something fundamental23:33
sarobfor this one yes23:33
*** DinaBelova has joined #openstack-infra23:33
sarobalot of the content in the near future will be from different repos23:33
jeblairsarob: what repos?23:33
saroband some others23:34
sarobidea is to not recreate existing content for the training manuals23:34
sarobuse existing as much as possible23:34
jeblairclarkb: this actually sounds like it needs d-g style integration to do properly.23:35
clarkbjeblair: well, they don't run in the gate queue so they inherently have race conditions23:36
*** sgviking has quit IRC23:36
clarkbjeblair: so they can probably get away with referring to remote things that are not github. eg git.o.o23:37
clarkbsarob: by referring to other repos it is possible that you will get outdated content if we don't properly order things23:38
*** DinaBelova has quit IRC23:38
jeblairclarkb: yes, but check/gate tests/drafts should; and somewhat awkwardly the post queue for them would need to observe strict ordering.23:38
*** pcrews has joined #openstack-infra23:38
jeblairclarkb: i agree that a good first step is to ignore that problem and just refer to a reliable source23:38
saroboh, i get that23:38
sarobno other way to reuse content23:38
sarobwe will finalize right after release docs are completed and static23:39
jeblair(making a job that assembles multiple repos non-concurrent would probably be good enough for the post queue)23:41
*** mjfork_ has joined #openstack-infra23:44
*** mjfork has quit IRC23:44
*** mjfork_ is now known as mjfork23:44
jeblairsarob, clarkb: maybe the best solution for now would be to 'git clone --depth 1' other needed repos23:45
*** mjfork has quit IRC23:45
*** mjfork has joined #openstack-infra23:45
clarkbjeblair: maybe. I don't think that addressed the xpointer issue though. I am not convinced it was a reliability of github problme23:45
clarkbunless it got back a file without the controller-os id in it because github can be unreliable23:46
sarobat time of build, how would local repo help?23:46
jeblairclarkb: ok, but that is also an issue.23:46
clarkbsarob: because the files would already be present and maven won't have to retrive them manually which may fail23:46
clarkbsarob: particularly when talking to github23:46
sarobso i would bring in all the repos in the repo i am working on?23:47
*** lbragstad1 has quit IRC23:47
sarobsounds messier that using github23:47
clarkbsarob: yes, prior to running the maven build have a step that clones all of your dependencies23:47
clarkbit is a little messy but far more debuggable23:48
jeblairsarob: trust us -- github fails all the time at the rate we run jobs23:48
clarkband if you are only using openstack repos you can fetch them from far more reliable sources23:48
sarobi want to make it as reliable as possible23:48
sarobso if i have 10 repos cloned23:49
sarobim working on a branch for a few days23:49
sarobvery likely ill be behind some commits23:49
clarkbjeblair: do you think there is value in trying to run that post job again to see if that particular failure is consistent? might indicate if github is at fault23:50
jeblairclarkb: up to you; i don't have bandwidth to debug.  i'm just saying that regardless, we want to stop using github.23:50
clarkbjeblair: indeed. I will go ahead and trigger it using the gearman client in zuul/tools/23:51
*** zehicle_at_dell has joined #openstack-infra23:53
clarkbsarob: it is consistent23:54
sarobfor this debuging i have removed all the remote github href23:55

Generated by 2.14.0 by Marius Gedminas - find it at!