Tuesday, 2014-02-25

dhellmannjog0: did you open a bug?00:00
clarkbjeblair: queue is at 100k, I don't think that will catch up tonight00:00
jog0got distacted doing it now00:00
dhellmannjog0: yeah, we may have to move that somewhere else to deal with circular dependencies00:00
jeblairclarkb: how do you feel about losing the queue then?  especially since it's backed up, i'd really like to get the data visible to jog0 and sdague  (not to mention to help you and fungi as you bring the new cluster up)00:01
*** e0ne has quit IRC00:01
dhellmannfor now let's get this unjammed00:01
dhellmannif you opened a bug I can assign it to gordc00:01
*** e0ne has joined #openstack-infra00:01
clarkbjeblair: its fine by me00:01
jog0dhellmann: https://bugs.launchpad.net/oslo/+bug/128437900:02
jog0dhellmann: lets make py33 non-voting for now00:02
dhellmannjog0: I have a patch ready to fix the tests to they work00:03
dhellmannit's installing pycadf, but none of the tests are using it00:03
dhellmanngive me 30 seconds00:03
*** lyxus has quit IRC00:03
dhellmannjog0: https://review.openstack.org/7604700:03
reedclarkb, fungi: can you please check https://review.openstack.org/#/c/69636/ ?00:04
*** jhesketh__ has joined #openstack-infra00:04
jog0dhellmann: what about tests/unit/middleware/test_audit.py00:05
*** alexpilotti has quit IRC00:05
dhellmanngordc: those tests are not run as part of the py33 configuration00:05
dhellmannsorry, jog000:05
dhellmannwe have a separate subset of the tests that work under python 3, so the gate job is to keep out regressions00:06
jog0dhellmann: ohh sneaky00:06
jog0in that case LGTM00:06
*** e0ne has quit IRC00:06
dhellmannI'll self-approve after the tests actually pass00:06
jog0dhellmann: cool, thanks for the lightning quick response00:07
dhellmannjog0: thanks for bringing it to my attention!00:08
*** sdake_ has quit IRC00:08
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Install and run elasticsearch on new workers  https://review.openstack.org/7604800:09
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Create new 30g heap elasticsearch workers  https://review.openstack.org/7602600:09
fungiclarkb: ^ (i'll wip the second patch until we're configured, and it can be split into multiples if we prefer)00:09
*** mgagne has quit IRC00:11
clarkbfungi: lgtm00:11
jog0dhellmann: dependencies are fun00:13
sdagueclarkb: https://review.openstack.org/7599700:13
dhellmannjog0: -100:13
sdaguethat's the latest on the xtrace trim00:13
sdagueI'm calling it a night00:13
*** relaxdiego has quit IRC00:13
jog0dhellmann: ^_^00:14
fungiclarkb: i'll try to --noop 76026 on elasticsearch.o.o and elasticsearch1.o.o to make sure the delta is empty00:15
*** weshay has joined #openstack-infra00:16
dhellmannjog0: looks like that change merged; I'm going to sign off so please update the bug report if you spot any other related issues00:17
jog0dhellmann: thaks00:18
*** salv-orlando has quit IRC00:19
*** khyati has joined #openstack-infra00:19
*** salv-orlando has joined #openstack-infra00:19
reeddevananda, the joys of Launchpad UI :)00:19
fungiclarkb: notice: /Stage[main]/Openstack_project::Base/Package[virtualenv]/ensure: current_value 1.9.1, should be 1.10.1 (noop)00:20
fungiclarkb: oh, nevermind. that's virtualenv00:20
* fungi misread00:21
*** yamahata has quit IRC00:21
*** zhiwei has joined #openstack-infra00:22
*** khyati has quit IRC00:22
*** cody-somerville has joined #openstack-infra00:25
*** CaptTofu has quit IRC00:25
*** khyati has joined #openstack-infra00:25
*** matsuhashi has joined #openstack-infra00:25
*** cadenzajon has quit IRC00:27
fungiclarkb: looks clean. should be safe00:29
*** bhuvan_ has joined #openstack-infra00:29
*** lnxnut has joined #openstack-infra00:30
*** bhuvan has quit IRC00:31
*** flaper87 is now known as flaper87|afk00:31
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Install and run elasticsearch on new workers  https://review.openstack.org/7604800:31
clarkbfungi: great thanks for checking00:31
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers  https://review.openstack.org/7605100:33
*** sdake_ has joined #openstack-infra00:34
*** CaptTofu has joined #openstack-infra00:34
*** lnxnut has quit IRC00:34
*** lnxnut has joined #openstack-infra00:35
*** sdake_ has quit IRC00:38
fungiclarkb: and i removed the dummy dns entries since i don't suppose we'll need them. we can keep puppet stopped on the new servers and start the agent one-by-one00:38
*** sdake_ has joined #openstack-infra00:38
*** sdake_ has quit IRC00:38
*** sdake_ has joined #openstack-infra00:38
clarkbfungi: but the old servers need rules too00:39
openstackgerritA change was merged to openstack-infra/config: Use statsd in logstash client  https://review.openstack.org/7604200:39
clarkbfungi: not sure how that will work00:39
fungii don't understand...00:40
*** lnxnut has quit IRC00:40
clarkbfungi: the list of servers is used on the existing machines to update their firewalls too00:40
*** lcheng has joined #openstack-infra00:40
clarkbso we need the list to be valid when ever puppet runs on new or old servers00:40
fungiso we merge 76026 first and build 6 new empty servers00:40
clarkboh I see, we are controlling this at the elasticsearch level00:40
fungithen we stop puppet agent on the 6 new servers00:40
clarkbI grok now, that will work00:40
*** Sukhdev has joined #openstack-infra00:41
fungithen we merge 76048 which updates the firewall rules based on the dns entries for the new servers, which exist by then00:41
fungiand we add the new nodes one by one by applying/enabling puppet on them00:42
fungistopping services on the old servers along the way as the migration progresses00:42
clarkbyup I grok now, sounds good00:42
fungithen we merge 76051 at the end after deleting the old servers00:42
fungiokay, cool00:42
*** derekh has quit IRC00:43
*** david_lyle has quit IRC00:44
fungiso anyway, i'll go ahead approving 76026 and start spinning up the 6 new base nodes00:44
fungiand add cinder volumes... how big and where should i mount them?00:45
fungipresumably we'll pvcreate them and use lvm00:46
fungibased on the current es servers, i'm going to guess 1t00:46
clarkbya 1t00:47
* clarkb looks for mount point its something in /var/lib iirc00:47
jeblairrestarting jenkins log client00:48
clarkbfungi: #DATA_DIR=/var/lib/elasticsearch00:48
clarkbfungi: so that should be the mount point, or optionally we can /mnt/someplace it and change the elasticsearch DATA_DIR00:48
clarkbfungi: that can be changed in /etc/default/elasticsearch00:49
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev  https://review.openstack.org/6132100:49
zaroclarkb: in transit had a little time to do ^00:49
*** lnxnut has joined #openstack-infra00:50
*** lcheng has quit IRC00:51
*** hogepodge has quit IRC00:51
zarofungi: my change to set java alternatives to java 7 was working yesterday, but doesn't seem to work today. happen to know what's going on?00:51
zarofungi: this build just ran and it's trying to use java6.. https://jenkins04.openstack.org/job/check-gerrit-buck/3/console00:52
fungizaro: did the not-working run end up on an hpcloud-az2 node?00:52
*** fifieldt has joined #openstack-infra00:53
fungizaro: yep, it did00:53
fungizaro: if only you knew someone who works at hp ;)00:53
*** sabari has quit IRC00:53
fungiwe still can't reliably rebuild server images there00:53
fungi(in az2 only)00:54
jeblairfungi, zaro: i believe i saw a "we're still looking into it" eamil00:54
jeblairfrom hpcloud00:54
*** lnxnut has quit IRC00:54
fungijeblair: more than one00:54
fungithey've sent about one of those a day i think00:54
*** lnxnut has joined #openstack-infra00:54
zarofungi: argh! do i just have to wait?00:55
*** dangers_away has quit IRC00:55
fungizaro: or retry and hope you hit a node on one of the other providers/regions00:55
*** relaxdiego has joined #openstack-infra00:56
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add logstash job queue graph  https://review.openstack.org/7605600:56
zarook.  i guess i'm unlucky, tried 3 times, no dice.00:56
fungizaro: did all 3 run on hpcloud-az2?00:56
jeblairclarkb, fungi: there's a graph ^ it's neat00:56
clarkbjeblair: correct there was a we are wroking on it update email this morning00:57
*** dangers_away has joined #openstack-infra00:57
jeblairclarkb, fungi: it has a red line that is nearly, but not quite, vertical00:57
openstackgerritA change was merged to openstack-infra/devstack-gate: make xtrace stateful  https://review.openstack.org/7569000:58
openstackgerritA change was merged to openstack-infra/devstack-gate: add indent function  https://review.openstack.org/7569700:58
openstackgerritA change was merged to openstack-infra/devstack-gate: suppress devstack console log  https://review.openstack.org/7569800:58
*** relaxdiego has quit IRC00:58
*** esker has joined #openstack-infra00:59
*** lnxnut has quit IRC00:59
*** e0ne has joined #openstack-infra00:59
openstackgerritClark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location  https://review.openstack.org/7605701:01
*** wenlock has quit IRC01:01
clarkbjeblair: AaronGr ^ I think that is a WIP. It passes existing tests but does not have any new tests01:01
*** lnxnut has joined #openstack-infra01:01
zarofungi: yep, just tried again, came up az2 again.01:01
*** zhiwei has quit IRC01:02
*** nati_ueno has quit IRC01:03
*** wchrisj has quit IRC01:03
*** e0ne has quit IRC01:04
*** nati_ueno has joined #openstack-infra01:04
openstackgerritA change was merged to openstack-infra/config: Create new 30g heap elasticsearch workers  https://review.openstack.org/7602601:05
clarkbjeblair: I marked 76057 WIP, but your feedback is always valuable01:06
*** CaptTofu has quit IRC01:09
*** mgagne has joined #openstack-infra01:09
*** CaptTofu has joined #openstack-infra01:11
*** khyati has quit IRC01:12
*** unicell has quit IRC01:13
*** unicell has joined #openstack-infra01:13
*** Sukhdev has quit IRC01:14
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: calculate remaining timeout locally  https://review.openstack.org/7572601:14
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: annotate stages of script  https://review.openstack.org/7570501:14
*** salv-orlando has quit IRC01:15
*** salv-orlando has joined #openstack-infra01:15
*** atiwari has quit IRC01:18
*** ryanpetrello has quit IRC01:19
*** ryanpetrello_ has joined #openstack-infra01:19
*** rcleere has quit IRC01:19
*** sarob has joined #openstack-infra01:20
*** wchrisj has joined #openstack-infra01:22
*** relaxdiego has joined #openstack-infra01:22
*** relaxdiego has quit IRC01:23
*** pmathews has quit IRC01:27
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Roll up node stats  https://review.openstack.org/7606401:28
*** yamahata has joined #openstack-infra01:29
*** wchrisj has quit IRC01:31
*** mgagne has quit IRC01:31
*** talluri has quit IRC01:31
*** melwitt has quit IRC01:32
*** zhiwei has joined #openstack-infra01:34
*** mgagne has joined #openstack-infra01:35
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/reviewday: Add JSON link and fix header  https://review.openstack.org/7606501:37
*** lnxnut has quit IRC01:38
*** lnxnut has joined #openstack-infra01:39
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/reviewday: Add JSON link and fix header  https://review.openstack.org/7606501:39
*** gokrokve has quit IRC01:40
*** gokrokve has joined #openstack-infra01:41
openstackgerritTushar Gohad proposed a change to openstack/requirements: Add PyECLib as a requirement  https://review.openstack.org/7606801:42
*** gokrokve_ has joined #openstack-infra01:42
*** gokrokve_ has quit IRC01:43
*** lnxnut has quit IRC01:44
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters.  https://review.openstack.org/7063601:44
*** nosnos has joined #openstack-infra01:44
*** gokrokve has quit IRC01:46
*** mgagne has quit IRC01:47
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters.  https://review.openstack.org/7063601:49
*** pmathews has joined #openstack-infra01:50
lifelesstripleo overcloud controller failed again01:52
lifelesslooks like network card driver issue01:52
lifelessand old Bios01:52
*** bhuvan_ has quit IRC01:53
*** lnxnut has joined #openstack-infra01:57
kashyapclarkb, Heya, when you're here: I pinged the sudo maintainer in Fedora and there's an update on this bug (you notified on the Fedora-on-Gate etherpad that it's one of the issues that's blocking to get Fedora on Gate)-- https://bugzilla.redhat.com/show_bug.cgi?id=1020147#c101:57
*** chenxu has joined #openstack-infra01:58
openstackgerritdaisy-ycguo proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex  https://review.openstack.org/6804201:58
clarkbkashyap: awesome, we managed to work around it by allocating ttys for all our ssh connections which is a bit silly but works01:59
*** esker has quit IRC01:59
kashyapclarkb, Like that? -- ssh -t host 'sudo -n command'01:59
clarkbbasically, but we are using paramiko01:59
*** e0ne has joined #openstack-infra01:59
kashyapI see.01:59
clarkbso its get_pty=True in the arg list to connections01:59
kashyapI see. I'm afraid, the sudo maintainer is not yet convinced about the default option there in the bug (comment #4)02:00
*** pmathews has quit IRC02:00
clarkbkashyap: ya02:01
clarkbfwiw fedora seems to break a lot of cloudy assumptions02:01
clarkbwhich is annoying02:01
kashyapclarkb, If you/infra folks think it'd be more elegant to have 'requiretty' disabled by default, then I can raise it with FESCO (Fedora Engineering Steering Comittee)02:01
clarkbmeh I don't care super strongly about it02:02
kashyapclarkb, Agree that's annoying. Yep, I can see  - you don't have to care too :-)02:02
clarkbkashyap: another issue we have run into is that the fedora user which you are supposed to use instead of root on fedora images isn't created by the time sshd starts02:04
kashyapThanks for the quick response. clarkb Is there anything else that's blocking?  /me just looking at this -- https://etherpad.openstack.org/p/fedora-on-gate .02:04
clarkbso you can't ssh in immediately02:04
clarkbkashyap: pleia2 would be the person to ask02:04
*** e0ne has quit IRC02:04
kashyapclarkb,  IIRC, it's created _after_ sshd is started, right?02:04
clarkbwhen it should just be baked into the image imo02:05
kashyapOkay, I think I noticed it (fedora user related)  in some gerrit review that I was going through some time ago.02:06
lifelessrebooted the node, we now are backish, going to give it a few minutes to stablise02:06
lifelessthen start thinking how I can move this to a less FITH machine02:07
kashyapclarkb, That issue seems to be worked around by retrying SSH on auth failure patch -- https://review.openstack.org/#/c/74825/02:07
*** zhiyan_ is now known as zhiyan02:07
*** MarkAtwood has quit IRC02:07
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Gate python-glanceclient on PyPy  https://review.openstack.org/7607202:08
clarkbkashyap: correct02:08
clarkbwhich is clunky because an auth failure usually means you don't want to retry02:09
kashyapclarkb, Yeah, I can see that. I'll talk to Fedora cloud folks on IRC (and probably raise it on mailing list) later this day and see if this can be amended02:09
fungi"While testing ssh access: [Errno 111] Connection refused"02:10
Alex_Gaynorflaper87|afk: Not sure if you're involved in glanceclient, but if yes, https://review.openstack.org/#/c/76072/ may be of interest :-)02:10
fungigood to know that still works02:10
*** Ryan_Lane1 has joined #openstack-infra02:11
kashyapclarkb, Just to be clear, you're referring to these, when you say "fedora images", right? -- http://fedoraproject.org/en/get-fedora#clouds02:12
clarkbkashyap: I believe those are the images used by tripleo02:12
* kashyap doesn't know what time-zone clarkb is in (probably EST/PST). If it's too late, just shoo me away02:12
clarkblifeless: can you confirm that the fedora images you are using are those ^02:13
clarkbkashyap: PST, and I am about to walk home :)02:13
* kashyap surprised "Valley" folks walk home at 18:00 :-)02:14
clarkbkashyap: oh I am not one of them :) I live in seattle02:14
clarkbdinner is ready in 15 minutes I need to hurry02:15
lifelessclarkb: I think so02:15
*** dkranz has joined #openstack-infra02:15
clarkbkashyap: so yes, those are the images we are using (lifeless thinks so)02:15
kashyapclarkb, Ah, okay. Yeah, I worked from there (Vallye) a few years ago, just saying what I see :-)  G'night!02:15
kashyaplifeless, Heya, who can confirm? Maybe Dan Prince?02:16
*** salv-orlando_ has joined #openstack-infra02:16
*** salv-orlando has quit IRC02:16
*** salv-orlando_ is now known as salv-orlando02:16
*** mgagne has joined #openstack-infra02:16
lifelesskashyap: hit the fedora webpage, download an image02:16
lifelesskashyap: if you're talking about nodepool stuff02:16
kashyaplifeless, Ok, then I know what you're referring to, these ones -- http://fedoraproject.org/en/get-fedora#clouds. Thanks for confirming. I'll just inspect with guestfish to confirm the behavior of fedora user02:17
kashyapThank you :-)02:18
lifelesskashyap: but if you're talking about dib built imagesfor tripleo02:18
lifelesssee the fedora element in dib02:18
kashyaplifeless, Yes, I remember the dib -- not talking about this.02:18
kashyaplifeless, Context is:02:18
kashyapTo have Fedora on Gate, trying to identify what are the issues/blockers. It's being coordinated here: https://etherpad.openstack.org/p/fedora-on-gate02:19
lifelessok, cool02:19
kashyapsoren, getting it into nodepool would be step-1 I see there02:19
kashyapErr, didn't mean to ping :-(02:19
*** gokrokve has joined #openstack-infra02:19
lifelessyeah, we want it for triple check jobs02:19
kashyap(ping him*)02:19
lifelesswhich is similar but different to gate02:20
lifelessfor d-g you'll need public cloud providers with images02:20
lifelesswhich may be different02:20
kashyaplifeless, Yeah, few steps at a time :-). First trying to get into nodepool /me has not much clue about infra, still learning02:21
*** moted has quit IRC02:24
*** kiall has joined #openstack-infra02:29
*** unicell has quit IRC02:30
*** pcrews has quit IRC02:30
*** unicell has joined #openstack-infra02:31
*** Ryan_Lane1 has quit IRC02:33
*** reed has quit IRC02:34
*** markmcclain has quit IRC02:36
pleia2kashyap: we got the paramiko issue sorted, and https://review.openstack.org/#/c/74825/ *should* fix the user+ssh issue (I haven't tested it, tripleo cloud was down earlier when I was going to)02:36
*** emagana has joined #openstack-infra02:37
pleia2kashyap: the rest of the etherpad should be up to date, I still have https://review.openstack.org/#/c/70007/ lingering, and I need to figure out why puppet3 on fedora20 doesn't like pip https://review.openstack.org/#/c/70031/02:37
kashyappleia2, Yeah, I'm _just_ looking through git commits, this seems to be it from dprince, where you use Paramiko? -- 1963731f7de0e6fc97e2771a043aba60d49a91e802:37
* kashyap clicks on the review02:37
pleia2https://review.openstack.org/#/c/69510/ is my WIP change for other things (mostly networking)02:39
kashyappleia2, Yes, 74825 is the what the commit I noted above corresponds to. Good to know.02:39
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Extend unittest jenkins builder  https://review.openstack.org/7051102:39
*** lnxnut has quit IRC02:39
*** lnxnut has joined #openstack-infra02:40
*** krotscheck has quit IRC02:41
kashyappleia2, Thanks for the work and your quick response here02:41
pleia2kashyap: no problem :) I was out a bit last week and then at a conference, so I could use the help catching up02:42
pleia2so jump in anywhere02:42
kashyappleia2, Sure. Yeah, it's been hectic here too, been pulled into several directions :-(.  And, I'll be away next week traveling. So, just thought I'd check here02:43
*** lnxnut has quit IRC02:44
fungiclarkb: so close! novaclient.exceptions.OverLimit: Quota exceeded for ram: Requested 61440, but already used 983040 of 1024000 ram (HTTP 413) (Request-ID: req-1a4c46f8-90d4-43bf-a556-06388cfacd0a)02:46
fungithat was on the sixth server02:46
clarkbsilly limits02:48
fungiclarkb: we have a few things we can probably clean out02:48
fungilike a couple extra zuuls for a start02:48
fungii'll see what else we haven't gotten around to cleaning up02:49
*** matsuhashi has quit IRC02:50
clarkbok thanks I am eating dinner now02:50
*** bhuvan has joined #openstack-infra02:50
openstackgerritA change was merged to openstack-infra/devstack-gate: start first partition at 1M not 0  https://review.openstack.org/7570302:51
*** oubiwann_ has quit IRC02:52
*** tjones has quit IRC02:53
*** pmathews has joined #openstack-infra02:55
*** matsuhashi has joined #openstack-infra02:55
*** sarob has quit IRC02:57
*** pcrews has joined #openstack-infra02:57
*** sarob has joined #openstack-infra02:57
*** thomasem has joined #openstack-infra02:59
*** thomasem has quit IRC02:59
*** e0ne has joined #openstack-infra02:59
*** bhuvan has quit IRC03:00
*** talluri has joined #openstack-infra03:00
zhiweifungi: hi03:01
*** talluri has quit IRC03:02
*** talluri has joined #openstack-infra03:02
*** sarob has quit IRC03:02
*** e0ne has quit IRC03:04
*** sarob has joined #openstack-infra03:04
openstackgerritA change was merged to openstack-infra/devstack-gate: annotate stages of script  https://review.openstack.org/7570503:05
openstackgerritA change was merged to openstack-infra/devstack-gate: calculate remaining timeout locally  https://review.openstack.org/7572603:05
*** talluri has quit IRC03:06
*** wchrisj has joined #openstack-infra03:07
*** jhesketh__ has quit IRC03:08
*** jhesketh__ has joined #openstack-infra03:09
*** weshay has quit IRC03:12
lifelessoh wow03:12
lifelessnodepool is now failing thusly:03:12
lifelessERROR: Quota exceeded for instances: Requested 1, but already used 100 of 100 instances (HTTP 413) (Request-ID: req-0c96b3bc-dc37-4685-8227-02398b3bea6b)03:12
lifelessnova list | wc -l03:12
lifelessso nova quota stats are -way- off03:12
lifelesswe haven't done any DB surgery at any point, so this is bugs03:13
*** jcooley_ has quit IRC03:16
*** wchrisj has quit IRC03:18
*** wenlock has joined #openstack-infra03:19
*** lnxnut has joined #openstack-infra03:19
*** vkozhukalov_ has joined #openstack-infra03:20
fungiclarkb: really turned out just to be the two old zuul servers. alone they wont be enough, but along with deleting the first elasticsearch server it should be possible03:23
fungizhiwei: hi there03:23
*** matsuhashi has quit IRC03:23
lifelessah, compute agents are down. stab.03:24
zhiweifungi: can you tell me the status of renaming the stackforge project?03:24
lifelessdstufft: around ?03:27
*** pmathews has quit IRC03:32
lifelessfungi: can you perhaps tickle nodepool? we had another outage, as listed above, everything is back but I'm not seeing guests03:33
*** nati_ueno has quit IRC03:36
*** lcheng has joined #openstack-infra03:38
*** morganfainberg is now known as morganfainberg_Z03:39
*** rcleere has joined #openstack-infra03:40
*** sdake__ has joined #openstack-infra03:40
*** sdake__ has quit IRC03:40
*** sdake__ has joined #openstack-infra03:40
*** sdake_ has quit IRC03:42
*** basha has joined #openstack-infra03:45
fungilifeless: i see a bunch in a building state for around half an hour. are those illegitimate? i can delete them if so03:45
openstackgerritLin Tan proposed a change to openstack-infra/zuul: Use six.moves.urllib.parse instead of urlparse  https://review.openstack.org/7282403:46
*** CaptTofu has quit IRC03:46
bashahey I have a question around failing builds in https://review.openstack.org/#/c/23424/1403:48
fungizhiwei: i had to refresh my memory from last week's meeting minutes. we're waiting for official or incubating openstack projects to need renames, and will batch stackforge project renames up with those. it seems like savanna may be ready to rename soon--they've sent their name options off to the foundation for a trademark search already03:49
clarkbbasha: go for it03:50
clarkbfungi: :/ about needing more room. We can defnitely start with teh first 5 then spin up the 6th after we free up some of the old es servers03:50
bashaclarkb: if you look at the python 26, 27 logs, theres just one test running and failing. SUrprisingly its passing in the local boxes here03:51
bashaalso clarkb fromn the logs it looks like its failing somewhere inside the library03:51
clarkbmock is failing03:51
clarkbthere should be a pip freeze at the end of the lgs showing you the version of mock used03:52
clarkbyou will want to make sure that the same version is used in local testing03:52
clarkbalso the test orders may be different locally and in jenkins03:52
fungiclarkb: tomorrow i'll put the fake dns rr back for 06 so we can merge that puppet change03:52
clarkbfungi: sounds good03:52
clarkbbasha: I would try running just that test locally and see if it fails in isolation03:53
bashaclarkb: it didnt03:54
clarkbbasha: and the mock versions match?03:54
bashaclarkb: you mean the requirements? yyup03:55
*** lnxnut has quit IRC03:55
clarkbnot the requirements, the actual installed versions03:55
*** lnxnut has joined #openstack-infra03:56
bashayes thats what I meant03:56
fungilifeless: on the assumption that new node builds should take less than half an hour, i'm deleting any in the state for that long03:56
*** pcrews has quit IRC03:56
clarkbbasha: ok, so the other possibility as far as test order goes is that if some other test runs first it breaks that test03:57
clarkbbasha: what you can do is grab the subunit log, load that into testr locally `testr load subunutlogfile`, then do `testr run --analyze-isolation` and it will do test bisection and try to determine why that test failed03:57
bashaclarkb: sure will try that03:58
bashaThanks :)03:58
*** e0ne has joined #openstack-infra03:59
*** markmcclain has joined #openstack-infra03:59
*** lnxnut has quit IRC04:00
* fungi offlines. it's getting late04:00
clarkbbasha: the other thing to note is that zuul will merge that change atop master before testing it04:00
*** rcleere has quit IRC04:00
clarkbbasha: if you haven't done that locally it may explain the failure04:00
clarkbfungi: good night04:00
*** emagana has quit IRC04:01
*** talluri has joined #openstack-infra04:03
lifelessfungi: ok, bunch errors, but we do have nodes now  Ithink04:03
*** jp_at_hp1 has quit IRC04:03
*** rcleere has joined #openstack-infra04:04
*** markmcclain has quit IRC04:04
*** e0ne has quit IRC04:04
*** talluri has quit IRC04:07
*** jcooley_ has joined #openstack-infra04:08
*** sarob has quit IRC04:08
*** sarob has joined #openstack-infra04:09
bashaclarkb: hmmmm, i thought I'd rebased it with master. but I'll check once more04:09
*** pmathews has joined #openstack-infra04:11
*** lnxnut has joined #openstack-infra04:11
dstufftlifeless: I am now04:11
lifelessdstufft: hey cool, wondering if you can fix the bad loop issue I reported on distutils-sig04:11
lifelessdstufft: since it really wigs bandersnatch out :)04:12
*** harlowja is now known as harlowja_away04:12
*** sarob has quit IRC04:13
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip  https://review.openstack.org/7594504:13
*** jeckersb is now known as jeckersb_gone04:14
*** pcrews has joined #openstack-infra04:15
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142504:16
*** lnxnut has quit IRC04:16
dstufftlifeless: probably someone renamed something04:16
dstufftwhich means a redirect got cached04:17
lifelessdstufft: for how long would such a cache persist?04:18
lifelessdstufft: note that it redirects *to itself*04:18
*** wchrisj has joined #openstack-infra04:21
*** tjones has joined #openstack-infra04:21
dstufftlifeless: normally 24 hours04:21
dstufftshould be fixed now04:21
lifelessdstufft: thanks!04:22
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: fix some small syntax issues in puppet manifest.  https://review.openstack.org/7249604:23
lifelessnow to stab this a few times until it does a run without timeouts04:23
lifelessand I should stop putting nuts load on the server04:23
*** david_lyle_ has joined #openstack-infra04:23
*** pcrews has quit IRC04:24
lifelesssigh there's always one package that times out04:26
*** lnxnut has joined #openstack-infra04:26
*** ryanpetrello_ has quit IRC04:28
*** moted has joined #openstack-infra04:28
*** moted has quit IRC04:28
*** relaxdiego has joined #openstack-infra04:30
*** matsuhashi has joined #openstack-infra04:31
*** rcleere has quit IRC04:32
*** harlowja_away is now known as harlowja04:32
*** jcooley_ has quit IRC04:32
*** jcooley_ has joined #openstack-infra04:33
dstufftlifeless: you know you can icnrease the timeout?04:33
*** tjones has quit IRC04:36
*** sarob has joined #openstack-infra04:38
*** sdake__ has quit IRC04:39
lifelessdstufft: are the defaults wrong?04:40
*** lnxnut has quit IRC04:40
dstufftlifeless: they might be for you :)04:40
dstufftI hardly ever get timeouts in bandersnatch04:41
*** mayu has quit IRC04:41
*** jcooley_ has quit IRC04:41
zhiweifungi: thanks.04:41
lifelessdstufft: heh04:43
lifelessdstufft: so we're on a 10Gbps link04:43
lifelessdstufft: its not clear whats causing the timeouts :)04:43
*** dkehn__ has joined #openstack-infra04:44
dstufftlifeless: are you down under? or do I have you confused with someone else?04:45
*** dkehn__ has quit IRC04:46
*** jcooley_ has joined #openstack-infra04:46
lifelessI live in NZ, the server in question is in las vegas :)04:46
*** dkehn__ has joined #openstack-infra04:46
*** dkehn has quit IRC04:47
*** dkehn_ has quit IRC04:47
dstufftlifeless: ah04:48
*** dkehn has joined #openstack-infra04:49
lifelessI don't have 10gbps to the net in my house :)04:49
*** talluri has joined #openstack-infra04:49
*** dkehn__ is now known as dkehn_04:49
*** lcheng has quit IRC04:53
*** lcheng has joined #openstack-infra04:56
lifelessdstufft: anyhow it finally got a complete run, and now has a proper serial, yay.04:57
*** jcooley_ has quit IRC04:58
*** pmathews has quit IRC04:59
*** jcooley_ has joined #openstack-infra04:59
*** e0ne has joined #openstack-infra04:59
*** jcooley_ has quit IRC05:03
*** e0ne has quit IRC05:04
*** reed has joined #openstack-infra05:04
*** talluri has quit IRC05:05
*** jcooley_ has joined #openstack-infra05:11
*** amotoki has joined #openstack-infra05:18
*** sarob has quit IRC05:19
*** sarob has joined #openstack-infra05:20
*** talluri has joined #openstack-infra05:22
*** talluri has quit IRC05:22
*** talluri has joined #openstack-infra05:22
*** sarob has quit IRC05:24
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers  https://review.openstack.org/6829705:32
*** nicedice has quit IRC05:34
*** lnxnut has joined #openstack-infra05:36
*** lcheng has quit IRC05:38
*** chenxu has quit IRC05:39
*** lnxnut has quit IRC05:41
*** CaptTofu has joined #openstack-infra05:47
*** lcheng has joined #openstack-infra05:50
openstackgerritA change was merged to openstack-infra/config: Upgrade manila's devstack-job to tempest-job  https://review.openstack.org/6684605:51
*** CaptTofu has quit IRC05:52
*** coolsvap has joined #openstack-infra05:58
*** e0ne has joined #openstack-infra05:59
*** tian has joined #openstack-infra06:01
*** jcooley_ has quit IRC06:01
*** harlowja is now known as harlowja_away06:01
*** yaguang has joined #openstack-infra06:02
*** e0ne has quit IRC06:04
*** chandan_kumar has joined #openstack-infra06:07
*** nati_ueno has joined #openstack-infra06:10
*** wchrisj has quit IRC06:14
*** DinaBelova_ is now known as DinaBelova06:17
*** sarob has joined #openstack-infra06:20
*** saju_m has joined #openstack-infra06:21
openstackgerritA change was merged to openstack/requirements: Remove vim header  https://review.openstack.org/7582106:25
*** sarob has quit IRC06:26
*** lnxnut has joined #openstack-infra06:26
*** reed has quit IRC06:26
*** yolanda_ has joined #openstack-infra06:27
*** saju_m has quit IRC06:29
*** ildikov_ has quit IRC06:29
*** lnxnut has quit IRC06:31
*** gyee has quit IRC06:42
*** saju_m has joined #openstack-infra06:42
*** vkozhukalov_ has quit IRC06:48
*** MarkAtwood has joined #openstack-infra06:48
openstackgerritA change was merged to openstack-infra/config: Add Zuul Job Queue graph  https://review.openstack.org/7570606:49
*** mrda is now known as mrda_away06:54
*** mkoderer has joined #openstack-infra06:54
*** e0ne has joined #openstack-infra06:59
*** yolanda_ has quit IRC06:59
*** CaptTofu has joined #openstack-infra07:00
*** denis_makogon has joined #openstack-infra07:01
*** jlibosva has joined #openstack-infra07:02
*** e0ne has quit IRC07:04
*** mwagner_wfh has quit IRC07:04
*** CaptTofu has quit IRC07:06
*** gokrokve has quit IRC07:06
*** skolekonov has joined #openstack-infra07:07
*** DinaBelova is now known as DinaBelova_07:09
*** gokrokve has joined #openstack-infra07:09
*** marun has quit IRC07:11
*** gokrokve_ has joined #openstack-infra07:11
*** saju_m has quit IRC07:13
*** saju_m has joined #openstack-infra07:13
*** gokrokve has quit IRC07:14
*** afazekas has quit IRC07:17
*** sarob has joined #openstack-infra07:23
*** sabari has joined #openstack-infra07:25
*** lnxnut has joined #openstack-infra07:26
*** sarob has quit IRC07:30
*** lnxnut has quit IRC07:31
*** lcheng has quit IRC07:32
*** vkozhukalov_ has joined #openstack-infra07:37
*** jcooley_ has joined #openstack-infra07:44
*** zhiyan is now known as zhiyan_07:48
*** rcarrillocruz1 has joined #openstack-infra07:53
*** pblaho has joined #openstack-infra07:53
*** flaper87|afk is now known as flaper8707:54
*** rcarrillocruz has quit IRC07:55
*** salv-orlando_ has joined #openstack-infra07:56
*** salv-orlando has quit IRC07:59
*** salv-orlando_ is now known as salv-orlando07:59
*** e0ne has joined #openstack-infra07:59
*** sabari has quit IRC08:00
*** e0ne has quit IRC08:04
openstackgerritValeriy Ponomaryov proposed a change to openstack-infra/config: Fix manila-tempest-job to use exported env var  https://review.openstack.org/7611508:06
*** denis_makogon has quit IRC08:15
*** NikitaKonovalov_ is now known as NikitaKonovalov08:17
*** mrmartin has joined #openstack-infra08:17
*** rlandy has joined #openstack-infra08:19
*** malini_afk is now known as malini08:19
*** dizquierdo has joined #openstack-infra08:21
*** jcoufal has joined #openstack-infra08:21
*** malini has left #openstack-infra08:23
*** dstanek has quit IRC08:25
*** sarob has joined #openstack-infra08:26
*** MarkAtwood has quit IRC08:27
*** DinaBelova_ is now known as DinaBelova08:27
*** lnxnut has joined #openstack-infra08:27
*** gokrokve_ has quit IRC08:28
*** mwagner_lap has joined #openstack-infra08:28
*** lnxnut has quit IRC08:31
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip  https://review.openstack.org/7594508:31
*** sarob has quit IRC08:32
openstackgerritFengqian.gao proposed a change to openstack-infra/jeepyb: Keep py3.X compatibility for urllib  https://review.openstack.org/7612208:34
*** jcooley_ has quit IRC08:34
*** luqas has joined #openstack-infra08:35
*** masayukig has joined #openstack-infra08:36
*** luqas has quit IRC08:36
*** hashar has joined #openstack-infra08:37
*** ildikov_ has joined #openstack-infra08:38
*** afazekas has joined #openstack-infra08:38
*** jgallard has joined #openstack-infra08:39
openstackgerritFengqian.gao proposed a change to openstack-infra/nodepool: Keep py3.X compatibility for urllib/urllib2  https://review.openstack.org/7612408:40
*** luqas has joined #openstack-infra08:41
*** DinaBelova is now known as DinaBelova_08:44
*** vkozhukalov_ has quit IRC08:53
*** DinaBelova_ is now known as DinaBelova08:57
*** wenlock has quit IRC08:57
*** rossella-s has joined #openstack-infra08:58
*** jpich has joined #openstack-infra08:59
*** salv-orlando has quit IRC08:59
*** CaptTofu has joined #openstack-infra09:02
*** CaptTofu has quit IRC09:06
*** protux has joined #openstack-infra09:09
*** jcooley_ has joined #openstack-infra09:10
*** e0ne has joined #openstack-infra09:11
*** yassine has joined #openstack-infra09:11
*** gokrokve has joined #openstack-infra09:12
*** gokrokve has quit IRC09:17
*** derekh has joined #openstack-infra09:17
*** zhiwei has quit IRC09:18
*** fbo_away is now known as fbo09:22
*** nati_ueno has quit IRC09:23
*** luqas has left #openstack-infra09:23
*** luqas has joined #openstack-infra09:25
*** lnxnut has joined #openstack-infra09:26
*** jcooley_ has quit IRC09:27
*** ociuhandu has quit IRC09:29
*** sarob has joined #openstack-infra09:30
*** ildikov_ has quit IRC09:30
*** lnxnut has quit IRC09:31
*** matsuhashi has quit IRC09:31
*** matsuhashi has joined #openstack-infra09:34
*** sarob has quit IRC09:36
openstackgerritJulien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency  https://review.openstack.org/6272609:37
*** zhiwei has joined #openstack-infra09:41
*** luqas has quit IRC09:47
*** jcoufal has quit IRC09:49
*** rossella-s has quit IRC09:50
*** salv-orlando has joined #openstack-infra09:50
*** yassine has quit IRC09:50
*** jp_at_hp has joined #openstack-infra09:51
*** salv-orlando_ has joined #openstack-infra09:54
*** davidhadas__ has quit IRC09:57
*** davidhadas_ has quit IRC09:57
*** zhiwei has quit IRC09:59
*** salv-orlando has quit IRC09:59
*** salv-orlando_ is now known as salv-orlando09:59
*** matsuhashi has quit IRC09:59
*** matsuhashi has joined #openstack-infra09:59
*** che-arne has joined #openstack-infra10:02
*** coolsvap has quit IRC10:06
*** gokrokve has joined #openstack-infra10:12
*** nosnos has quit IRC10:14
*** rcarrillocruz has joined #openstack-infra10:14
*** rossella has joined #openstack-infra10:16
*** rcarrillocruz1 has quit IRC10:17
*** matsuhashi has quit IRC10:17
*** gokrokve has quit IRC10:17
*** luqas has joined #openstack-infra10:19
*** protux has quit IRC10:20
*** david_lyle_ has quit IRC10:21
*** CaptTofu has joined #openstack-infra10:23
*** lnxnut has joined #openstack-infra10:26
*** salv-orlando has quit IRC10:29
*** vkozhukalov_ has joined #openstack-infra10:31
*** lnxnut has quit IRC10:31
*** sarob has joined #openstack-infra10:33
*** rlandy_ has joined #openstack-infra10:34
*** coolsvap has joined #openstack-infra10:36
*** rcarrillocruz1 has joined #openstack-infra10:36
*** boris-42_ has quit IRC10:37
*** rlandy has quit IRC10:38
*** rcarrillocruz has quit IRC10:38
*** sarob has quit IRC10:38
*** davidhadas has joined #openstack-infra10:39
*** NikitaKonovalov is now known as NikitaKonovalov_10:50
*** rlandy_ has quit IRC10:51
*** yassine has joined #openstack-infra10:57
*** relaxdiego has quit IRC11:01
*** psedlak has quit IRC11:02
*** denis_makogon has joined #openstack-infra11:04
*** rlandy_ has joined #openstack-infra11:04
denis_makogonhello, guys, could someone help me with gate failure explanation ?11:05
denis_makogoni'm seeing weired issue on jenkins gate http://logs.openstack.org/40/76140/1/check/gate-trove-python26/e449799/console.html (patchset https://review.openstack.org/#/c/76132/)11:05
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add script trigger.  https://review.openstack.org/7097111:06
*** jp_at_hp has quit IRC11:08
*** dkranz has quit IRC11:09
denis_makogonplease, i really need help11:09
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: added logstash plugin support and fixed test typo  https://review.openstack.org/7091711:10
*** dkranz has joined #openstack-infra11:12
*** jp_at_hp has joined #openstack-infra11:12
*** gokrokve has joined #openstack-infra11:12
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Add attachment pattern expression to email-ext.  https://review.openstack.org/7014811:12
*** mrmartin has quit IRC11:14
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Content-Type can now be set for email-ext publisher  https://review.openstack.org/7591911:14
*** gokrokve has quit IRC11:17
*** psedlak has joined #openstack-infra11:18
*** jhesketh has quit IRC11:21
*** jhesketh__ has quit IRC11:21
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin  https://review.openstack.org/7360111:22
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Test for email-ext publisher  https://review.openstack.org/7587211:25
*** lnxnut has joined #openstack-infra11:26
*** jerryz has quit IRC11:26
*** lnxnut has quit IRC11:31
*** salv-orlando has joined #openstack-infra11:36
*** fifieldt has quit IRC11:36
*** sarob has joined #openstack-infra11:36
*** jgallard has quit IRC11:38
*** boris-42_ has joined #openstack-infra11:38
*** jcooley_ has joined #openstack-infra11:41
*** DinaBelova is now known as DinaBelova_11:41
*** sarob has quit IRC11:42
*** hashar has quit IRC11:45
*** jcoufal has joined #openstack-infra11:46
*** jcooley_ has quit IRC11:48
zigomordred: Around?11:50
zigop1otr uploaded SQLA 0.9 to Sid last Sunday, this broke python-migrate at least, and probably some more.11:50
zigoI've tried fixing this, but I couldn't.11:50
zigoIt'd be awesome if you can find out what the error is.11:51
zigoWhat's Sean Dague nickname?11:51
sdaguezigo: what's up?11:54
zigosdague: just read above! :)11:54
* zigo is happy to have found sdague!11:54
sdagueI don't know the actual fix, I have a partial solution here - https://review.openstack.org/#/c/66156/11:55
sdaguebut it's not the whole fix11:55
zigosdague: I already include this fix in my code base! :)11:56
sdagueyeh, well I haven't dug into the rest of it11:56
zigosdague: https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=127 <--- This one suggests to use return ret instead of return self.preparer.quote(ret, cons.quote)11:56
zigoIt seems it works! :)11:56
*** yamahata has quit IRC11:57
*** NikitaKonovalov_ is now known as NikitaKonovalov11:57
zigoI don't understand why but it does.11:58
*** DinaBelova_ is now known as DinaBelova12:00
openstackgerritThomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient  https://review.openstack.org/7617012:01
sdagueso I'd be a little concerned about doing that blindly, because that assumes the value is already safely quoted12:02
*** luqas has quit IRC12:03
*** salv-orlando has quit IRC12:03
zigosdague: If you have a better patch, then great!12:04
sdaguethough let me play with that idea12:04
sdaguezigo: yeh, I'd just be concerned that you opened up a security hole by blindly dropping quoting12:04
zigoI got it.12:04
*** markmc has joined #openstack-infra12:06
*** ArxCruz has joined #openstack-infra12:09
*** alexpilotti has joined #openstack-infra12:10
*** luqas has joined #openstack-infra12:11
*** luqas has quit IRC12:11
*** gokrokve has joined #openstack-infra12:12
*** che-arne has quit IRC12:14
*** alexpilotti has quit IRC12:15
*** rlandy__ has joined #openstack-infra12:17
openstackgerritFengqian.gao proposed a change to openstack-infra/nodepool: Keep py3.X compatibility for urllib/urllib2  https://review.openstack.org/7612412:17
*** gokrokve has quit IRC12:17
*** rlandy_ has quit IRC12:18
*** basha has quit IRC12:19
*** rlandy__ is now known as rlandy12:20
*** che-arne has joined #openstack-infra12:22
*** lnxnut has joined #openstack-infra12:26
*** lnxnut has quit IRC12:31
*** hashar has joined #openstack-infra12:35
*** jang has quit IRC12:35
*** hashar_ has joined #openstack-infra12:38
*** sarob has joined #openstack-infra12:39
*** hashar has quit IRC12:41
*** hashar_ has quit IRC12:45
*** hashar has joined #openstack-infra12:45
*** sarob has quit IRC12:46
*** sandywalsh has quit IRC12:48
*** alexpilotti has joined #openstack-infra12:50
*** yaguang has quit IRC12:56
*** ilyashakhat has quit IRC12:57
*** talluri has quit IRC12:59
*** lcostantino has joined #openstack-infra12:59
*** mgagne1 has joined #openstack-infra13:00
*** ilyashakhat has joined #openstack-infra13:00
*** sandywalsh has joined #openstack-infra13:01
*** mgagne has quit IRC13:03
*** coolsvap has quit IRC13:04
*** weshay has joined #openstack-infra13:05
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Campfire had a malformed doc syntax  https://review.openstack.org/7618513:05
*** afazekas has quit IRC13:06
*** markmcclain has joined #openstack-infra13:06
*** luqas has joined #openstack-infra13:07
zigosdague: Here's my unit test logs now: http://paste.debian.net/83979/13:11
zigoAny idea where the "Failed for mysql://openstack_citest:openstack_citest@localhost/openstack_citest" is comming from?13:11
zigoAnd about these other FAIL?13:11
sdagueso you aren't testing against mysql there13:12
sdaguetest_db.cfg will give you the ability to set connect strings for mysql engines13:12
sdagueand realize that if you don't test against mysql, a lot of it isn't tested, because sqlite doesn't implement foreign keys, which is where most of the issues are13:13
zigosdague: I *do* test against MySQL.13:13
zigosdague: That's my shell script to do the unit tests: http://paste.debian.net/83980/13:14
*** gokrokve has joined #openstack-infra13:14
sdagueyeh, so realitistically this probably a bigger thing than I can sift through for a while. Got to get back to some other things.13:15
zigoOk, no worries.13:16
*** smarcet has joined #openstack-infra13:17
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Use venv to build documentation  https://review.openstack.org/7619013:17
*** andreaf has joined #openstack-infra13:18
*** gokrokve has quit IRC13:18
*** dstanek has joined #openstack-infra13:19
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Move requirement files to root of repository  https://review.openstack.org/7619213:22
*** dizquierdo is now known as dizquierdo_afk13:23
*** afazekas has joined #openstack-infra13:23
*** rcarrillocruz1 is now known as rcarrillocruz13:23
*** CaptTofu has quit IRC13:24
*** lnxnut has joined #openstack-infra13:26
sdaguefungi / SergeyLukjanov: so we need this to go through soon - https://review.openstack.org/#/c/75719/ as we no longer put devstack logs in the console.html13:29
SergeyLukjanovsdague, hey13:29
*** lnxnut has quit IRC13:31
*** jcooley_ has joined #openstack-infra13:31
*** exz has quit IRC13:31
*** exz has joined #openstack-infra13:31
*** dolphm_503 is now known as dolphm13:34
*** salv-orlando has joined #openstack-infra13:34
*** jgallard has joined #openstack-infra13:34
*** mbacchi has joined #openstack-infra13:36
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: remove old queries  https://review.openstack.org/7619613:37
*** wenlock has joined #openstack-infra13:38
*** jcooley_ has quit IRC13:38
*** dkliban has quit IRC13:38
*** lcostantino has quit IRC13:38
openstackgerritBjörn Michaelsen proposed a change to openstack-infra/git-review: add query option  https://review.openstack.org/7619713:39
*** wenlock has quit IRC13:42
*** valentinbud has joined #openstack-infra13:42
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr  https://review.openstack.org/7620013:43
valentinbudHello all. I have setup a Logstash, Elasticsearch, Kibana logging infrastructure using the openstack-infra/config repository.13:44
fungivalentinbud: neat! how's it working out for you?13:45
valentinbudThe multiline Logstash filters for TRACE messages don't work.13:45
valentinbudI have Logstash 1.3.3 and Elasticsearch 1.0.0 on Ubuntu 14.04.13:45
valentinbudThe system is processing logs from HyperV instances.13:46
*** yassine has quit IRC13:47
openstackgerritKhai Do proposed a change to openstack-infra/config: update comment filter for gerrit 2.8  https://review.openstack.org/7086413:47
valentinbudfungi it works quite, I mean the system is stable and the logs are gathered and indexed by elasticsearch, but the filters aren't working though.13:49
*** saju_m has quit IRC13:50
*** dizquierdo_afk is now known as dizquierdo13:51
*** dims has quit IRC13:51
openstackgerritKhai Do proposed a change to openstack-infra/config: update comment filter for gerrit 2.8  https://review.openstack.org/7086413:51
fungivalentinbud: interesting... i haven't used it to look for tracebacks so haven't noticed whether we get combined multiline records for those13:53
*** rcleere has joined #openstack-infra13:54
fungivalentinbud: have you looked for similar latterns on http://logstash.openstack.org/ to see whether it's working on ours?13:54
*** afazekas has quit IRC13:54
*** dims has joined #openstack-infra13:54
*** dkranz has quit IRC13:54
*** pdmars has joined #openstack-infra13:55
*** pdmars has quit IRC13:55
*** pdmars has joined #openstack-infra13:55
*** dprince has joined #openstack-infra13:56
openstackgerritMartin Mágr proposed a change to openstack-infra/config: Watch also havana branch fro packstack  https://review.openstack.org/7620613:57
*** mburned_` is now known as mburned14:00
valentinbudfungi have searched for TRACE at logstash.openstack.org now. http://logstash.openstack.org/#eyJzZWFyY2giOiJUUkFDRSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzMzNjgyNTEwMn0=.14:02
*** lnxnut has joined #openstack-infra14:02
*** afazekas has joined #openstack-infra14:02
valentinbudI don't know if the link takes you there, but if you search for TRACE you have events consisting of multiline TRACEs.14:03
*** sandywalsh has quit IRC14:03
tristanCHello folks, I seek guidance to fix #1281540 (stable/havana gate job fails because of partial swift client upgrade)14:03
fungivalentinbud: ahh, yep. it looks like we are able to glob those together. not sure why it's not working in yours, but clarkb might have some ideas when he's awake in a few hours14:04
fungitristanC: yeah, sorry, i did say yesterday i'd look into that, didn't i14:04
valentinbudOk, I'll wait for him and pose him a question regarding this problem. Thank you.14:05
tristanCfungi: no worry :) Someone posted a patch for that but he abandoned it: https://review.openstack.org/7562414:05
fungitristanC: i believe we just need to make grenade not run the swift exercises in grizzly when testing havana changes14:05
tristanCIt basically does the same thing that have been done for grizzly (see https://review.openstack.org/#/c/74451/ )14:05
tristanCfungi: thanks :)14:06
fungitristanC: yeah, it's close. we only want to apply that in grenade though, i think, so i'll throw it in an else after the if [ "$DEVSTACK_GATE_GRENADE" -ne "1" ]14:06
fungipatch coming in moments14:06
*** yassine has joined #openstack-infra14:09
*** boris-42_ has quit IRC14:10
*** mgagne1 is now known as mgagne14:10
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr  https://review.openstack.org/7620014:10
*** salv-orlando has quit IRC14:10
*** pmathews has joined #openstack-infra14:10
*** salv-orlando has joined #openstack-infra14:10
*** wchrisj has joined #openstack-infra14:11
*** nijaba has joined #openstack-infra14:11
*** nijaba has joined #openstack-infra14:11
*** gokrokve has joined #openstack-infra14:12
*** lcostantino has joined #openstack-infra14:12
*** jeckersb_gone is now known as jeckersb14:13
*** ociuhandu has joined #openstack-infra14:13
*** salv-orlando_ has joined #openstack-infra14:14
*** pmathews has quit IRC14:15
*** markmc has quit IRC14:16
*** salv-orlando has quit IRC14:16
*** changbl has quit IRC14:17
*** gokrokve has quit IRC14:17
*** sandywalsh has joined #openstack-infra14:18
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade  https://review.openstack.org/7562414:19
fungitristanC: ^14:19
*** markmc has joined #openstack-infra14:19
fungioh, my indenting is full of fail. fixing14:19
*** salv-orlando_ is now known as salv-orlando14:20
tristanCfungi: you rock sir! :)14:20
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade  https://review.openstack.org/7562414:21
*** jraim has quit IRC14:22
*** jraim_ has joined #openstack-infra14:22
*** beagles is now known as beagles_brb14:24
*** pblaho has quit IRC14:24
*** yamahata has joined #openstack-infra14:24
*** thomasem has joined #openstack-infra14:25
*** prad_ has joined #openstack-infra14:27
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr  https://review.openstack.org/7620014:28
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Move requirement files to root of repository  https://review.openstack.org/7619214:28
*** jcooley_ has joined #openstack-infra14:28
hasharrebasing is a good spam system14:28
*** sandywalsh has quit IRC14:29
fungihashar: especially if you stack 30 changes as dependencies and then rebase the bottom14:29
*** sarob has joined #openstack-infra14:29
fungithough i tend to prefer the mergespam we get when a big stack of changes all actually make it through the gate. much more satisfying14:30
*** persia has quit IRC14:30
hasharyeah those one are lovely14:30
openstackgerritZhiQiang Fan proposed a change to openstack/requirements: Upgrade six to 1.5.2  https://review.openstack.org/6842414:31
*** dkliban has joined #openstack-infra14:31
*** persia has joined #openstack-infra14:31
*** lnxnut has quit IRC14:31
*** miqui has joined #openstack-infra14:31
*** persia is now known as Guest6339314:31
fungisdague: i'm noticing that of the 14 backward-compat jobs we run against d-g changes, we don't have one which makes sure stable grenade still works... do you think it's warranted?14:31
sdagueyeh, though we're about to drop grizzly, right?14:32
fungisdague: we are, in 9 days, but we'll have a new stable grenade soon, right? ;)14:32
sdagueyeh, stable grenade should be in there14:33
fungisdague: the only reason i notice it is that i'm looking at the jobs for 75624 which should fix grenade on grizzly for the next week, but we don't actually have a job to confirm it works14:33
fungisomething to keep in mind for the future, where we all have time to make things work correctly ;)14:34
sdaguewhen is that going to be again? :)14:35
fungii never can remember if we measure that in geological time scales or cosmic time scales14:35
*** julim has joined #openstack-infra14:36
*** jcooley_ has quit IRC14:36
*** salv-orlando has quit IRC14:38
*** flaper87 is now known as flaper87|afk14:39
*** yamahata has quit IRC14:39
*** sarob has quit IRC14:41
*** sandywalsh has joined #openstack-infra14:42
*** ok_delta has joined #openstack-infra14:42
*** yamahata has joined #openstack-infra14:43
*** zehicle_at_dell has quit IRC14:43
*** lnxnut has joined #openstack-infra14:48
*** lnxnut has quit IRC14:48
*** katyafervent_awa has quit IRC14:49
*** miqui_ has joined #openstack-infra14:50
*** eharney has joined #openstack-infra14:52
*** miqui has quit IRC14:52
*** dkranz has joined #openstack-infra14:52
*** lnxnut has joined #openstack-infra14:53
*** yolanda_ has joined #openstack-infra14:53
*** valentinbud has quit IRC14:54
*** valentinbud has joined #openstack-infra14:54
*** markmcclain has quit IRC14:56
openstackgerritJeremy Stanley proposed a change to openstack-infra/jeepyb: Make welcome-message --verbose work like -v  https://review.openstack.org/7622214:56
*** katyafervent_awa has joined #openstack-infra14:58
*** beagles_brb is now known as beagles14:59
*** spredzy_ is now known as spredzy14:59
*** ryanpetrello has joined #openstack-infra15:00
*** Guest50062 is now known as sbadia15:01
*** tjones has joined #openstack-infra15:01
*** johnthetubaguy has joined #openstack-infra15:01
*** gokrokve has joined #openstack-infra15:02
*** bknudson1 has joined #openstack-infra15:02
*** bknudson has quit IRC15:05
*** bknudson1 has quit IRC15:07
*** skolekonov has quit IRC15:07
*** bknudson has joined #openstack-infra15:10
openstackgerritA change was merged to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config)  https://review.openstack.org/6963615:16
*** igor_ has joined #openstack-infra15:17
*** pdmars has quit IRC15:17
fungismarcet: ^ my --noop tests of that looked okay, but that didn't include executing any of the scripts so you'll want to make sure it's deploying as you and mrmartin intend15:18
smarcetfungi: ok cool, i will chat with mrmartin about it, tnx15:19
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Auth support  https://review.openstack.org/7321915:22
fungisdague: are you cool with approving 75624 to un-wedge stable/grizzly security fixes?15:24
fungier, i mean stable/havana security fixes15:24
openstackgerritA change was merged to openstack-infra/jeepyb: Make welcome-message --verbose work like -v  https://review.openstack.org/7622215:24
sdaguelet me look15:24
*** CaptTofu has joined #openstack-infra15:25
fungivery easy review15:25
openstackgerritSahdev Zala proposed a change to openstack-infra/config: New StackForge project heat-translator  https://review.openstack.org/7598815:25
sdaguethough... you sure that actually fixes things15:26
sdagueI thought that just made the fail happen later15:26
*** rcleere has quit IRC15:26
*** boris-42_ has joined #openstack-infra15:26
valentinbudclarkb: are you around?15:26
*** david_lyle_ has joined #openstack-infra15:26
fungisdague: oh, maybe? yeah without a backward-compat job to actually run it, this is probably the fastest way to find out15:27
fungisdague: i guess we may end up needing to skip tempest tests for swift in havana on grenade jobs too?15:27
*** david_lyle_ is now known as david_lyle15:27
*** boris-42_ has quit IRC15:27
sdaguewell, actually we'd have to disable swift15:28
fungiat least, any that end up interacting with swiftclient15:28
sdaguebecause I think that glance backs to swift in those tests15:28
fungioh, runs that deep huh?15:28
*** afazekas has quit IRC15:28
sdagueit's integration tests15:28
fungiglance uses swiftclient to interact with swift i guess15:28
sdagueanyway, we'll see15:28
*** basha has joined #openstack-infra15:28
*** markmcclain has joined #openstack-infra15:28
fungisure, this is a cheap way to find out what else we're missing15:28
*** markmc has quit IRC15:29
*** CaptTofu has quit IRC15:30
*** sandywalsh has quit IRC15:30
*** coolsvap has joined #openstack-infra15:30
*** david_lyle_ has joined #openstack-infra15:31
*** mrodden has joined #openstack-infra15:31
*** sarob has joined #openstack-infra15:34
*** salv-orlando has joined #openstack-infra15:34
*** salv-orlando has quit IRC15:34
*** david_lyle has quit IRC15:35
*** david-lyle is now known as dklyle15:35
*** david_lyle_ is now known as david_lyle15:35
*** miqui_ is now known as miqui15:36
*** david_lyle_ has joined #openstack-infra15:36
*** VINOD has joined #openstack-infra15:37
*** wenlock has joined #openstack-infra15:38
*** david_lyle has quit IRC15:39
*** david-lyle has joined #openstack-infra15:41
openstackgerritA change was merged to openstack-dev/pbr: Updated from global requirements  https://review.openstack.org/7311915:41
*** davidhadas_ has joined #openstack-infra15:42
*** sarob has quit IRC15:42
*** relaxdiego has joined #openstack-infra15:43
*** NikitaKonovalov is now known as NikitaKonovalov_15:43
*** sarob has joined #openstack-infra15:43
*** cody-somerville has quit IRC15:44
*** davidhadas has quit IRC15:45
*** david_lyle_ has quit IRC15:45
*** sandywalsh has joined #openstack-infra15:46
*** CaptTofu has joined #openstack-infra15:46
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Make sure lvm2 tools are installed  https://review.openstack.org/7624415:47
fungisomething else ^ missing from rackspace's ubuntu 12.10 pvhvm images15:48
fungier, 12.0415:48
*** basha has quit IRC15:49
*** amotoki_ has joined #openstack-infra15:51
*** atiwari has joined #openstack-infra15:52
*** mfer has joined #openstack-infra15:52
valentinbudfungi: what OS is OpenStack Logs sitting on?15:52
valentinbudUbuntu 12.04 or 12.10?15:52
fungivalentinbud: if you're still talking about logstash, kibana and elasticsearch we're running it all on ubuntu 12.04 lts (precise pangolin)15:53
fungithough actually the server where we're hosting our static log content also runs on the same ubuntu version15:53
*** rcleere has joined #openstack-infra15:54
valentinbudfungi: yes, about that. I am trying to identify potential issues with our deployment. In our case the system is on Ubuntu 14.04.15:54
fungivalentinbud: ahh, 14.04 lts hasn't been released yet, so we haven't tried it15:55
fungivalentinbud: also, all our servers run in public cloud providers who don't have snapshot images for 14.04 (as far as i've seen) and don't provide a way for us to upload our own base images15:56
fungiso to test it out that way we'd probably have to do an in-place upgrade from a released ubuntu version anyway15:56
*** reed has joined #openstack-infra15:57
*** jgrimm has joined #openstack-infra15:59
*** reed has quit IRC15:59
*** sarob has quit IRC15:59
*** DinaBelova is now known as DinaBelova_15:59
*** jcoufal has quit IRC15:59
valentinbudThanks fungi. I am going to test it on 12.04.4. On Debian Wheezy I get the same behaviour.16:00
*** jcoufal has joined #openstack-infra16:02
*** skraynev is now known as skraynev_afk16:02
*** luqas has quit IRC16:02
*** pmathews has joined #openstack-infra16:03
*** sandywalsh has quit IRC16:04
*** BobBall is now known as BobBallAway16:04
openstackgerritsahid proposed a change to openstack-infra/config: Adds new project Warm  https://review.openstack.org/7624716:04
openstackgerritAlexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement  https://review.openstack.org/6586316:05
*** lcheng has joined #openstack-infra16:07
valentinbudfungi: I don't know why Ubuntu 14.04 was chosen. I am working on the HyperV CI Team and have received the OS pre installed and the task to get the logs up and running.16:08
*** mgagne has quit IRC16:08
*** changbl has joined #openstack-infra16:10
*** sergmelikyan has joined #openstack-infra16:13
sergmelikyanGuys, can someone help me to debug issue with gate-job?16:13
sergmelikyanTests running perfectly on local machine but hang on gate16:14
sergmelikyanNo idea why it may happen :(\16:14
dhellmanngood morning16:14
sergmelikyanMorning, Doug16:15
sergmelikyanLooks like hangs code related to running RPC server16:17
sergmelikyandhellmann, did you encountered something like this?16:17
dhellmannsergmelikyan: looking16:18
* sergmelikyan has no idea why test is looking like he is looking, but actual running code is a simple RPC server16:18
*** jcooley_ has joined #openstack-infra16:18
*** pcrews has joined #openstack-infra16:19
dhellmannsergmelikyan: so the tests hang but work for you locally?16:20
*** thedodd has joined #openstack-infra16:20
*** thedodd has quit IRC16:20
sergmelikyanI am running tests locally by 'tox -e py27' command16:20
*** sandywalsh has joined #openstack-infra16:20
dhellmanndo the unit tests start a copy of the server?16:22
dhellmannjeblair, fungi, clarkb: if you have a few minutes, I'd appreciate some feedback on the gating approach for oslotest - https://review.openstack.org/#/c/74408/16:23
*** sahumada has joined #openstack-infra16:24
*** jlibosva has quit IRC16:24
*** dhellmann is now known as dhellmann_16:24
sergmelikyandhellmann, yes16:24
*** rcarrillocruz1 has joined #openstack-infra16:25
*** rcarrillocruz has quit IRC16:26
jeblairdhellmann_: so actually i think we should use the devstack-gate approach for this.  most of what d-g does is actually setting up the git repos correctly for cross-project gating16:27
*** mrodden has quit IRC16:27
*** sabari has joined #openstack-infra16:27
*** markmcclain has quit IRC16:28
*** dizquierdo has quit IRC16:28
*** dizquierdo has joined #openstack-infra16:28
*** DinaBelova_ is now known as DinaBelova16:28
jeblairdhellmann_: probably the easiest way to do this at the moment is to do something like the swift functional test job16:29
jeblairdhellmann_: where we use the d-g scripts to set everything up and then use the hooks in that job to actually run the tests16:29
*** jergerber has joined #openstack-infra16:30
jeblairdhellmann_: so, at a really high level, remove the git repo setup stuff from run_cross_tests, and just leave the bits where you install the projects and run tests16:30
jeblairdhellmann_: then you call run_cross_tests from the devstack-gate gate_hook16:30
*** branen has quit IRC16:30
jeblairer, actually post_test_hook16:30
*** pdmars has joined #openstack-infra16:31
jeblairdhellmann_: does that make sense (at a high level)?16:31
*** pdmars has quit IRC16:31
*** mgagne has joined #openstack-infra16:31
*** thuc has joined #openstack-infra16:32
*** pdmars has joined #openstack-infra16:32
*** dizquierdo has quit IRC16:36
*** gyee has joined #openstack-infra16:37
*** UtahDave has joined #openstack-infra16:37
jeblairdhellmann_: left comment in review; i can work through the details with you if you want.16:38
*** luqas has joined #openstack-infra16:39
*** mrodden has joined #openstack-infra16:41
*** markwash has joined #openstack-infra16:42
fungijeblair: types faster than i do. now to see if i left the same comments on it16:42
*** mwagner_lap has quit IRC16:42
fungiyep... *sigh*16:43
anteayawe just had an az-2 node go offline: https://jenkins01.openstack.org/job/gate-neutron-python27/7770/console16:43
*** wchrisj has quit IRC16:43
anteayano slave log and it looks like the same java error jog0 was talking about last night16:44
*** tjones has quit IRC16:44
anteayathe java error is that there is no slave log16:44
*** wchrisj has joined #openstack-infra16:45
anteayaneutron patch https://review.openstack.org/#/c/67556/ currently in the gate16:45
jeblairjenkins is constantly finding new ways to annoy us16:45
anteayait seems16:45
*** rcarrillocruz has joined #openstack-infra16:46
*** rcarrillocruz1 has quit IRC16:46
fungijeblair: is this another incidence of the possible gearman job deregistering issue you were suggesting yesterday?16:47
jeblairfungi: yeah.  my initial thought is that either the deregistration is having an adverse impact.  or possibly it's just coincidence and just otherwise somehow related to job start (lots of things are going on then).16:48
fungihave we seen it anywhere besides hpcloud-az2 i wonder...16:48
fungiwe already know we have some stability issues in there16:48
*** VINOD has quit IRC16:49
*** wchrisj has quit IRC16:49
openstackgerritA change was merged to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade  https://review.openstack.org/7562416:51
jeblairoh good question16:51
*** wchrisj has joined #openstack-infra16:52
jeblairfungi: mostly az2 and a little az116:53
fungitristanC: now that's ^ merged, you probably want to try rechecking one of your havana backports and see if/where it fails again16:53
*** chandankumar_ has joined #openstack-infra16:53
jeblair2 out of 17 hits in last 24 hours are az1, 15 are az216:53
*** VINOD has joined #openstack-infra16:53
*** cody-somerville has joined #openstack-infra16:54
*** chandan_kumar has quit IRC16:54
fungii smell a wumpus16:54
jeblairif there's one thing i don't miss about nc it's smelling wumpuses16:54
anteayaha ha ha16:54
tristanCfungi: yep thanks, was watching the gate already ;)16:55
VINODmy name is vinod and i am currently working at CERN16:55
VINODi have uploaded code called Domain Quota API driver to Gerrit16:55
VINODi can see that some of the tests are failing16:56
VINODthe gerrit page is16:56
VINODsome of the tests failing are16:56
VINODi have no idea why these are failing as i have added the code to nova to manage the quota16:57
*** oubiwann_ has joined #openstack-infra16:57
*** tjones has joined #openstack-infra16:57
VINODcan anybody help me in this16:57
anteayahi VINOD16:57
VINODeven a test "gate-nova-pep8" failing saying that the nova.conf.sample is out of date16:58
anteayathat patch is still running it's check tests16:58
anteayaonce the check tests are finished the logs for the tests will be posted to the patch16:58
VINODyou mean jenkins will do a re-check16:58
*** valentinbud has left #openstack-infra16:58
VINODyes, i have got logs16:58
anteayajenkins will post the urls for the test logs16:58
anteayaso after the test logs are posted to the patch16:58
VINODbut i could not understand the log error is by any means related to my code16:58
VINODfor example even a test "gate-nova-pep8" failing saying that the nova.conf.sample is out of date16:59
anteayalet's wait until the logs are posted to the patch16:59
anteayait is easier then for me to go through them with you16:59
*** valentinbud has joined #openstack-infra16:59
*** boris-42_ has joined #openstack-infra17:00
*** valentinbud has quit IRC17:02
*** mriedem1 has joined #openstack-infra17:02
VINODi have not touched anything in nova.conf.sample file and i downloaded the latest from the upstream17:03
VINODthe logs are available17:03
VINODplease go to https://review.openstack.org/#/c/75967/17:03
VINODand scroll down to see jenkins tests17:03
*** mriedem has quit IRC17:04
*** davidhadas has joined #openstack-infra17:04
*** davidhadas_ has quit IRC17:04
VINOD6 tests have failed out of 1617:04
VINODalso vmware mine sweeper build also failed giving the error17:04
VINODFAIL: setUpClass (tempest.api.compute.images.test_image_metadata.ImagesMetadataTestJSON)17:04
VINODDetails: {"itemNotFound": {"message": "Image not found.", "code": 404}}17:04
VINODagain, i have done nothing to the images classes17:04
fungijeblair: do you need anything off two old zuul servers? if not, i was thinking of (carefully) deleting them to regain some of our openstackci quota in dfw17:05
*** oubiwann_ has quit IRC17:05
*** oubiwann_ has joined #openstack-infra17:06
anteayaVINOD: hi as soon as I finish eating I will have a look17:06
VINODthank you so much17:06
jeblairfungi: go for it.  also, while you're there if you want to delete any of the non-production pbx servers17:06
VINODi will be grateful of you...because i have just submitted one more set of code for novaclient at https://review.openstack.org/#/c/76256/ and i fear that the same problems may happen to it as well17:07
VINODagain thank you so much17:07
fungijeblair: can do. i assume you still need your jeblair-{centos,precise,raring} vms17:07
sdaguejeblair: you want to send this one in - https://review.openstack.org/#/c/75719/17:08
jeblairfungi: oh, nope those can go too, sorry.17:08
VINODso i am leaving currently...any comments you can post to me at  vinod.kumar.boppanna@cern.ch17:08
jeblairsdague: devstack-gate stack landed?17:08
sdaguejeblair: yes17:08
sdaguelook at the console logs this morning, they are very short17:08
fungijeblair: oh, cool! i might actually have room to spin up the last elasticsearch replacement in that case without having to do part of the replacement incrementally17:09
*** VINOD has quit IRC17:09
jeblairsdague: done17:09
sdaguethanks much17:09
*** chandankumar_ has quit IRC17:10
*** pmathews has quit IRC17:10
*** pmathews has joined #openstack-infra17:10
*** hashar has quit IRC17:10
*** chandan_kumar has joined #openstack-infra17:10
*** lcheng has quit IRC17:11
*** relaxdiego has quit IRC17:11
openstackgerritA change was merged to openstack-infra/config: index sublog files  https://review.openstack.org/7571917:11
*** jlibosva1 has joined #openstack-infra17:12
*** jlibosva1 has quit IRC17:12
*** jlibosva has joined #openstack-infra17:13
tristanCfungi: the grenade havana check failed on swift there: https://jenkins01.openstack.org/job/check-grenade-dsvm/7960/console17:13
fungitristanC: okay, sort of expected that, based on discussion with sdague. i can iterate on it a bit more to see what else it needs17:14
*** marun has joined #openstack-infra17:14
*** jlibosva has quit IRC17:14
*** jlibosva has joined #openstack-infra17:14
fungioh! javelin next17:15
*** cadenzajon has joined #openstack-infra17:15
sdagueyeh, this is really a much deeper issue17:15
fungitime to see what i can configure in javelin17:15
ekarlsojaveln ?17:16
ekarlsowhat's that17:16
fungiekarlso: the "production-like" mock cloud built as part of grenade tests17:16
sdaguefungi: so realistically you need to stop running with git swiftclient17:16
sdagueor disable swift for havana17:16
sdaguethere really aren't other options17:16
openstackgerritA change was merged to openstack-dev/pbr: Remove unused _parse_mailmap()  https://review.openstack.org/7427917:17
fungisdague: just disable swift for havana grenade jobs though, right?17:17
fungisdague: seems it runs okay for other havana jobs17:17
sdaguewell, I guess we don't run the swift cli then?17:18
SergeyLukjanovmorning folks17:18
sdagueI guess that's what's different on this job17:18
SergeyLukjanovis it known issue?17:18
*** jlibosva has joined #openstack-infra17:18
*** hashar has joined #openstack-infra17:19
sdagueSergeyLukjanov: yeh it's in the er list somewhere17:19
SergeyLukjanovsdague, ok, thx, will search again17:19
SergeyLukjanovsdague, heh, the easy way - reload the page :( https://bugs.launchpad.net/nova/+bug/125562717:20
fungisdague: third (possibly even worse) option, we cease grenade testing of havana changes a week early17:21
*** jergerber has quit IRC17:21
sdagueyeh, the unwind seems kind of time consuming for 9 more days17:21
fungiit feels like giving up, but then again the other options sort of do too17:22
sdagueso what really happened is a hole in our approach for handling client major version changes17:22
fungisdague: well, that and we didn't test swiftclient master at all17:23
*** e0ne has quit IRC17:23
sdagueyeh, and that17:23
fungifor integration tests i mean17:23
*** amcrn has joined #openstack-infra17:23
fungiotherwise they probably would have seen and worked on solving the backward-compat issues there17:23
* fungi wanders off to propose the patch for the distastefully simple solution17:24
*** lcheng has joined #openstack-infra17:25
openstackgerritSergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627  https://review.openstack.org/7626517:25
openstackgerritsahid proposed a change to openstack-infra/config: Adds new project Warm  https://review.openstack.org/7624717:25
*** moted has joined #openstack-infra17:26
*** kfarr has joined #openstack-infra17:27
openstackgerritSergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627  https://review.openstack.org/7626517:28
*** hogepodge has joined #openstack-infra17:28
*** yassine has quit IRC17:28
*** markwash has quit IRC17:29
*** markmcclain has joined #openstack-infra17:29
*** relaxdiego has joined #openstack-infra17:30
*** mwagner_lap has joined #openstack-infra17:30
*** jcoufal has quit IRC17:30
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add queue-name parameter to projects  https://review.openstack.org/7626817:30
jeblairsdague, jog0: ^ that's step 1/2 to get us overall stats for the 'integrated' queue.17:31
*** jcoufal has joined #openstack-infra17:31
*** dhellmann_ is now known as dhellmann17:31
openstackgerritLogan Paschke proposed a change to openstack-infra/config: Create gerrit-powered-agenda Stackforge Repo  https://review.openstack.org/7627017:31
*** valentinbud has joined #openstack-infra17:31
dhellmannjeblair: thanks, I was pulled away right as you started responding -- is there documentation on any of that?17:31
* dhellmann reads comments on review17:31
anteayarussellb: do you accept 3000+ line patches in nova? https://review.openstack.org/#/c/75967/17:32
fungiwowie zowie17:32
anteayarussellb: I ask because the contributor asked for some guidance on how to understand the log output17:32
anteayarussellb: read scrollback for my interaction with VINOD17:32
anteayahe seemed... in a hurry17:33
russellb-2 though ... not approved for icehouse17:33
russellbwill have to wait17:33
anteayathat saves some time17:33
*** markmcclain1 has joined #openstack-infra17:33
fungiperhaps reaching out to tim bell or others at cern to give a heads up on better new contributor integration would help17:33
anteayarussellb: that sounds more like you than me17:33
anteayawouldn't you say?17:34
*** markmcclain1 has quit IRC17:34
*** davidhadas has quit IRC17:34
jeblairdhellmann: not so much documentation as examples (mentioned in review) plus some moderately documented code in d-g itself.17:35
* anteaya assumes russellb will chat with tim bell and moves on17:35
*** oubiwan__ has joined #openstack-infra17:35
*** oubiwann_ has quit IRC17:35
dhellmannjeblair: ok, I'll go study and come back with questions17:35
*** markmcclain1 has joined #openstack-infra17:36
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add queue-name parameter to projects  https://review.openstack.org/7626817:36
*** markmcclain1 has quit IRC17:36
jeblairdhellmann: okay, i'm happy to help with whatever you need17:36
*** dkliban is now known as dkliban_afk17:36
*** beagles is now known as beagles_brb17:36
*** sarob has joined #openstack-infra17:36
*** markmcclain has quit IRC17:37
*** dkehn_ has quit IRC17:37
*** chuck__ has joined #openstack-infra17:37
*** dkehn has quit IRC17:39
*** dkehn has joined #openstack-infra17:40
*** branen has joined #openstack-infra17:41
*** oubiwan__ has quit IRC17:41
*** dkehn_ has joined #openstack-infra17:42
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Stop running Grenade on Havana changes early  https://review.openstack.org/7628017:44
*** derekh has quit IRC17:44
fungitristanC: sdague: ^17:44
*** markwash has joined #openstack-infra17:45
*** DuncanT- has quit IRC17:45
*** markmcclain has joined #openstack-infra17:46
wenlockhi jeblair, clarkb: saw there was some activity yesterday on https://review.openstack.org/#/c/51425/ that needed another re-base, i took care of it last night, hopefully it gets a review again.   Thanks!17:46
anteayajaypipes: you around?17:47
anteayajaypipes: I was hoping to discuss http://lists.openstack.org/pipermail/openstack-dev/2014-February/028110.html17:47
anteayajaypipes: I have some concerns17:47
*** krotscheck has joined #openstack-infra17:50
*** pdmars has quit IRC17:50
kfarrHello all! I would appreciate feedback on a patch I'm working on.17:50
kfarrThis patch that adds scenario tests in Tempest for cinder volume encryption ( https://review.openstack.org/#/c/58018/ ) depends on having a value for fixed_key for the nova keymgr set.  The patch to add a default value for fixed_key in devstack ( https://review.openstack.org/#/c/62481/ ) was merged back in January, so all the Jenkins tests for the Tempest patch *should* pass.17:51
*** NikitaKonovalov_ is now known as NikitaKonovalov17:51
kfarrHowever, the check-grenade-dsvm Jenkins test fails because it first checks out and stacks the stable/havana branch of devstack, which was released before the fixed_key patch merged.  Even though it upgrades the repos, the settings stay the same because devstack is not restacked nor the settings updated.17:51
kfarrThe solution so far is to modify the grenade project and add a value for fixed_key when grenade runs through the upgrades, perhaps in this file: https://github.com/openstack-dev/grenade/blob/master/from-havana/upgrade-nova.  This idea doesn't seem ideal, so if anybody had alternative suggestions, I would appreciate hearing them.  Waiting for the next stable release of devstack is an option, but I would like to get the patch17:51
jeblairkfarr: #openstack-qa is probably the best place to ask about that17:51
jeblairanteaya, jaypipes: i do too.17:53
kfarrjeblair: ok, thanks!17:53
anteayathought you might17:53
*** NikitaKonovalov is now known as NikitaKonovalov_17:53
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Stop running Grenade on Havana changes early  https://review.openstack.org/7628017:54
fungigoogle hangouts?17:54
*** harlowja_away is now known as harlowja17:55
*** dkehn__ has joined #openstack-infra17:55
*** valentinbud has quit IRC17:55
fungii mean, they're free to use whatever they want, but that choice pretty much guarantees i won't be participating, and probably neither will many (any?) other infra team people17:56
jeblairfungi: exactly.  i would think that being able to ping infra folks would be very useful for this17:57
*** dkehn has quit IRC17:57
anteayamy concern is that the target audience is already isolated from the communication network17:57
*** dkehn_ has quit IRC17:57
anteayathis re-enforces their isolation17:57
fungianteaya: that is an extremely valid concern17:57
jeblairjaypipes: after all, i believe we provided considerable help to you as you were establishing this.  i think we might be able to continue to provide it if you don't exclude us from it.17:57
fungiand we have seen quite a lot of evidence to this effect17:57
jeblairanteaya: agreed17:57
*** mriedem has joined #openstack-infra17:58
*** dkehn has joined #openstack-infra17:58
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev  https://review.openstack.org/6132117:59
*** chuck__ has quit IRC17:59
*** dkehn__ is now known as dkehn_17:59
*** morganfainberg_Z is now known as morganfainberg17:59
*** yolanda_ has quit IRC18:00
anteayablah https://review.openstack.org/#/c/67556/ is failing in the gate again, on a test it passed last testing set18:00
anteayaso far I can't see in the log file where the problem is18:01
*** beagles_brb is now known as beagles18:01
*** mriedem1 has quit IRC18:01
pleia2morning clarkb18:01
anteayathis might be it: BadRequest: No nw_info cache associated with instance18:01
anteayamorning clarkb pleia218:02
*** gokrokve has quit IRC18:03
*** david_lyle_ has joined #openstack-infra18:03
*** david_lyle_ is now known as david_lyle18:04
*** dkehn__ has joined #openstack-infra18:04
*** mwagner_lap has quit IRC18:04
*** CaptTofu_ has joined #openstack-infra18:04
*** lcostantino has quit IRC18:04
*** CaptTofu has quit IRC18:04
anteayafound it: https://bugs.launchpad.net/nova/+bug/124906518:04
anteayaand the gate reset18:05
*** thomasem_ has joined #openstack-infra18:06
*** nati_ueno has joined #openstack-infra18:06
*** jgriffit2 has joined #openstack-infra18:07
*** david-lyle has quit IRC18:07
*** jgriffith has quit IRC18:07
*** thomasem has quit IRC18:07
*** dkehn_ has quit IRC18:07
*** luqas has quit IRC18:07
*** oubiwann_ has joined #openstack-infra18:08
mferclarkb fungi or anyone else... if you could take a look at https://review.openstack.org/#/c/71956/18:08
mferwe'd love to be able to start coding in there18:09
*** hashar has quit IRC18:09
*** dprince_ has joined #openstack-infra18:09
*** dkehn_ has joined #openstack-infra18:10
anteayato save time, currently 71956 will need to be monitored if it merges since it doesn't have a current repo to draw from18:11
*** dkliban_afk is now known as dkliban18:13
*** SumitNaiksatam has joined #openstack-infra18:14
*** reed has joined #openstack-infra18:16
*** che-arne has quit IRC18:16
*** rfolco has quit IRC18:17
clarkbfungi: so ES stuff. It just occurred to me that we have a couple things that don't talk to the whole cluster but directly to elasticsearch.o.o so we will want to do that node after we have at least on of the new nodes up and can point kibana and apache proxy at a new node18:17
clarkbfungi: also I find it super useful to run a local copy of elasticsearch bigdesk and elasticsearch head in my browser pointed at a port forward to the cluster18:17
clarkbfungi: these tools need privs we don't want to expose publicly hence the local copies and port forward, I would recommend setting those up on your end before we start adding and removing nodes18:18
*** kmartin has quit IRC18:19
*** pmathews1 has joined #openstack-infra18:19
*** kmartin has joined #openstack-infra18:19
*** rlandy has quit IRC18:19
*** rlandy has joined #openstack-infra18:20
*** changbl has quit IRC18:20
*** relaxdiego has quit IRC18:20
*** jgallard has quit IRC18:20
*** dkehn has quit IRC18:20
*** hogepodge has quit IRC18:20
*** jgriffit2 is now known as jgriffith18:21
*** jcoufal has quit IRC18:21
*** dprince has quit IRC18:22
*** chandan_kumar has quit IRC18:22
fungiclarkb: awesome. well, i'm setting up the cinder volumes on the last couple now (was able to delete enough cruft to free up sufficient quota for the last one)18:22
SergeyLukjanovfolks, could we use templates in templates in zuul?18:22
SergeyLukjanovlayout.yaml I mean18:22
*** chandankumar_ has joined #openstack-infra18:22
clarkbfungi: that is great news18:22
clarkbSergeyLukjanov: good question18:23
fungiclarkb: one issue which sprang to mind though... the /var/lib/elasticsearch mount point may not have the right ownership depending on how the package sets that stuff up, so the first one could come up unable to write there... i'll keep an eye out for it when we add the module18:23
*** che-arne has joined #openstack-infra18:23
*** mfer has quit IRC18:23
clarkbfungi: hrm maybe we want to manage the elasticsearch user in puppet?18:24
clarkbprobably safe to see what happens in the first go though18:24
*** jpich has quit IRC18:24
clarkbiirc the data actually ends up in /var/lib/elasticsearch/elasticsearch which may be a workaround for issues like that18:24
fungiclarkb: also discovered that the pvhvm images lack the lvm2 package by default (i've proposed a change to add that in template.pp from now on)18:24
SergeyLukjanovclarkb, yup, will take a look on how it works18:25
*** pmathews has quit IRC18:25
*** kfarr has quit IRC18:25
SergeyLukjanovheh, one more crazy question - is it quite crazy if we'd like to publish master / release image built by dib but there are about 12 images for 10-20 GB in total?18:26
anteayain the coverage logs url: logs.o.o/2 digits from sha/sha/post/<project>-coverage/7 random digits/cover/name of file18:27
*** valentinbud has joined #openstack-infra18:27
fungiSergeyLukjanov: /dev/mapper/main-tarballs     197G   30G  168G  16% /srv/static/tarballs18:27
anteayawhere do the 7 random digits come from?18:27
fungiSergeyLukjanov: looks like we probably have room18:27
SergeyLukjanovfungi, yay!18:27
clarkbanteaya: from the python random uuid generator18:27
anteayaclarkb: no wonder they looked random18:28
fungianteaya: click the link in the sidebar for the job parameters if you're trying to figure out what a running job will use18:28
anteayaso how can one find the coverage logs without having fore knowledge of the 7 random digits?18:28
SergeyLukjanovfungi, my dream is to publish images for all savanna plugins - versions matrix ;)18:28
anteayafungi: yes, how about a job that has finished running18:28
*** lcostantino has joined #openstack-infra18:28
anteayaI can find the ones for the running ironic job18:28
fungianteaya: is the coverage job run in post?18:28
anteayatrying to track down logs for neutron patches18:29
*** dstufft has quit IRC18:29
anteayagoing by markmcclain's question of me, I would go with yes18:29
fungianteaya: if it's in the post pipeline, look at the gitsha... give me an example project and commit18:29
anteayaproject is neutron18:29
anteayaI'll find you a commit18:29
fungikeep in mind that it will be what jenkins merged to the project, so it could be a merge commit rather than the parent18:30
*** thedodd has joined #openstack-infra18:30
fungiyeah, that wasn't the actual merge commit, that was just the proposed commit18:30
fungiyou'd need the merge commit sha instead18:30
*** chandankumar_ has quit IRC18:30
fungisince that's what actually gets the coverage run18:31
*** chandan_kumar has joined #openstack-infra18:31
*** julim has quit IRC18:31
*** julim has joined #openstack-infra18:31
fungibasically http://logs.openstack.org/<first two chars of sha>/<full sha>18:32
fungithen browse from there18:32
*** cody-somerville has quit IRC18:32
*** rohitk has joined #openstack-infra18:32
anteayafungi: thank you18:32
*** andreaf has quit IRC18:32
fungiso the coverage report was http://logs.openstack.org/af/af1aacff158d0004c081b8d8ee77f93baa6244a4/post/neutron-coverage/610a9d8/cover/18:32
markmcclainfungi: thanks!18:32
*** gokrokve has joined #openstack-infra18:33
*** rfolco has joined #openstack-infra18:33
*** vkozhukalov_ has quit IRC18:34
markmcclaindhellmann just told me he wrote a tool to locate these18:34
*** changbl has joined #openstack-infra18:35
fungii probably already knew that existed, but have a mind like a sieve18:35
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract check/gate req jobs to check-requirements  https://review.openstack.org/7631218:35
dhellmannfungi: me, too, that's why I write stuff like that18:36
rohitkWould the Jenkins issue on stable/havana be fixed anytime soon?18:36
*** mfer has joined #openstack-infra18:36
*** pmathews has joined #openstack-infra18:37
fungirohitk: probably not directly. instead i've proposed a change to just stop running that job since it becomes obsolete in a week when we drop grizzly support18:37
jaypipesjeblair, anteaya: I'm surprised to hear you think I am excluding the infra team from my proposed Q&A/workshop thing :(18:37
rohitkfungi: Ok, thanks18:37
anteayajaypipes: I don't think that is your motivation18:38
anteayabut I think it might be one of the consequences18:38
fungijaypipes: only indirectly by your choice of non-free (as in speech) software18:38
*** oubiwann_ has quit IRC18:38
*** khyati has joined #openstack-infra18:38
anteayadue to the fact that the target audience is coming from a place that doesn't understand opensource development18:38
anteayaand how we communicate18:38
*** ianw has quit IRC18:38
*** ianw has joined #openstack-infra18:39
*** oubiwann_ has joined #openstack-infra18:39
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support for TestNG publisher.  https://review.openstack.org/7015718:39
anteayaso you may inadvertently be sending a message to them by your choice of medium18:39
fungiyes, so any examples we set by using non-free software to interact with them will potentially send an inaccurate picture of what the project is about18:39
*** jamielen- has joined #openstack-infra18:39
*** igor_ has quit IRC18:39
anteayaand I just wanted to ensure it was a concious choice18:39
*** dstufft has joined #openstack-infra18:39
jaypipesanteaya, fungi, jeblair: i am more than happy to change the choice of medium. like I mentioned, I wanted something that I could share screens on.18:39
anteayaI wonder if perhaps a short tutorial on the benefits of using paste might help18:40
anteayaincluding the trigger command18:40
* jaypipes really not sure how he's gotten everyone so upset :(18:40
anteayaand full prompt18:40
fungijaypipes: i'm not upset at all18:40
fungii don't *think* anyone else is either18:40
anteayanot upset18:40
clarkband don't even know what is going on yet :) catching up on email now18:40
*** pmathews1 has quit IRC18:41
anteayajust aware of the messaging of using non-free software18:41
anteayawith a group struggling to understand open source18:41
fungianteaya: paste.o.o doesn't cover issues with showing people a demo of configuring the jenkins webui, i think was his point18:41
jaypipesanteaya: what do you suggest as an alternative?18:41
jeblairjaypipes: i know the lack of screenshare is :( but i think it will be offset by being able to interact with us, and as anteaya says, pulling the new folks into the community more fully18:41
anteayathat is a fair point18:41
jeblairjaypipes: i'd like to use irc18:42
jaypipesjeblair: totally. so would you want me to propose changing to an interactive IRC meeting instead?18:42
jaypipesk, will do.18:42
jaypipesnot a problem.18:42
jeblairjaypipes: cool, thanks18:42
anteayathen it will be logged18:42
jeblairjaypipes: we have 3 channels now, so there are some good slots open18:42
anteayaand links to images can be logged to18:42
fungiwe even have 3 meeting channels now, and you can make use of the meeting minutes that way for a record to point people back at if they can't attend in their timezone18:42
fungier, what jeblair said18:42
*** nicedice has joined #openstack-infra18:42
*** jamielennox has quit IRC18:42
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Fix unused variable in ci-skip wrapper.  https://review.openstack.org/7587918:43
jeblairjaypipes: the time you proposed works, or if you want to get me after breakfast, 1 or 2 hours later, on either monday or tuesday.  :)18:43
*** johnthetubaguy has quit IRC18:43
*** kmartin has quit IRC18:43
jaypipesjeblair: ok, thx. and guys, apologies for causing heartburn. was not intended at all. and I hope you all know how much I appreciate all of the help you have given me on this front.18:44
jeblairjaypipes: we _insist_ that we must continue to give you help! :)18:44
fungisolving problems/confusion for these people early means less work for us later18:45
anteayathanks for understanding jaypipes18:45
fungiand probably also gets them integrated into the ecosystem more quickly too18:45
* mordred hugs jaypipes18:45
anteayaknew it was just a matter of talking about it18:45
clarkbout of curiousity why so early PST time? strong interest from EU?18:46
*** kmartin has joined #openstack-infra18:46
clarkbmaybe its the time I am normally awake creating bias but there seems to be a lot of people west of PST interested in third party testing that may be exlcluded at the proposed time slot18:47
jaypipesclarkb: well, frankly, there's lots of interest from all over the place...18:47
jaypipesclarkb: I can certainly move it back...18:47
fungiwe've seen a ton of third-party ci influx from eu and apac18:47
jaypipesnever really a good time for everyone. I'll propose a staggered one week this time, next week diff time18:47
clarkbyou probably have a better idea of what would work well than I do, hence my question18:47
openstackgerritZiad Sawalha proposed a change to openstack-infra/config: Add satori as a new project  https://review.openstack.org/7366718:48
* anteaya thinks of poor mayu in china18:48
* jaypipes scratches head trying to figure out how to delete a Google event....18:48
anteayahe got his system up btw18:48
anteayahe was very grateful18:48
jeblairjaypipes: i'm not sure you can delete anything from google :)18:48
jaypipesyes, anteaya, I was working with mayu on #openstack-neutron yesterday.18:49
jaypipesjeblair: heh. or the NSA.18:49
jeblairfungi, clarkb, mordred, SergeyLukjanov: i'm going to be afk thursday and friday this week18:49
*** oubiwann_ has quit IRC18:49
fungijeblair: thanks for the heads up!18:49
jaypipesperhaps the NSA has advice on setting up third party CI? ;)18:49
clarkbjeblair: noted18:50
SergeyLukjanovjeblair, noted18:50
*** markmcclain has quit IRC18:50
*** mkoderer has quit IRC18:51
*** thomasem_ has quit IRC18:52
sdagueso I'm not sure why a hangouts based bootstrapping approach isn't seen as complementary to irc based ones. Honestly, I was just excited to see someone else taking effort off the plate for the infra team :)18:52
anteayanova test failing in the gate looks like a tempest test failure18:52
jaypipesaha! figured out how to delet the event. :)18:52
anteayabut the fails on the requirements patch might be infra related18:52
clarkbsdague: in particular it creates a divide in participation. Third party testers should be here but the majority are not18:53
*** valentinbud has quit IRC18:53
clarkbbut it also doesn't come with recorded text that is searchable18:53
sdagueclarkb: well, I think there are stages of getting them here18:53
clarkbfor the folks on the other side of the planet that may not be able to participate due to scheduling18:54
sdaguesure, but I think there already exists a ton of useful text on the subject, so more archived text is not something I expect will change things18:54
sdagueso if training wheels included another medium that they felt more comfortable with to then start engaging more regularly in infra, I see that as a good thing18:55
clarkbfungi: I am reviewing stuff now, please do ping when you are ready to do more ES stuff18:56
fungiclarkb: okay, i've got all 6 of the new 60g performance ubuntu 12.04 pvhvm elasticsearch servers in dns with 1tb cinder mounted on each. if you want to look one over (they should all be identical) go for it. puppet's not set to run on any of them, so next it should be safe to merge https://review.openstack.org/76048 right?18:56
clarkblol timing18:56
*** pdmars has joined #openstack-infra18:56
*** jcoufal has joined #openstack-infra18:56
*** mwagner_lap has joined #openstack-infra18:56
fungialso, almost meeting times18:57
*** pdmars has quit IRC18:57
clarkboh right18:57
*** markmcclain has joined #openstack-infra18:57
*** pdmars has joined #openstack-infra18:58
clarkbfungi: yes, I think it should be safe to merge that change then we can toggle puppet on nodes one at a time as we iterate through adding node then deleting node18:58
clarkbsdague: I might just be stubborn, but I think getting folks comfortable in the existing mediums is the greatest set of training wheels18:59
*** SumitNaiksatam has quit IRC18:59
fungiokay, let's allow it to merge and update while the meeting is underway and then i can get started with migration once the emeting ends18:59
clarkbteach a person to fish and all that19:00
clarkbfungi: sounds good19:00
anteayaso in http://logs.openstack.org/39/75539/9/gate/gate-devstack-dsvm-cells/ae77fc5/logs/devstacklog.txt.gz19:00
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract tarballs and pypi upload to pypi-release  https://review.openstack.org/7632219:00
anteayaI see: There are problems and -y was used without --force-yes19:00
jeblairit's meeting time over in #openstack-meeting19:00
*** thomasem has joined #openstack-infra19:00
anteayalooks like during the apt-get stage19:01
*** jp_at_hp has quit IRC19:01
sdagueclarkb: sure, it's all about activation energy. Much like summits and mid cycle meetups, google hangouts provides a way to bring more folks into the fold.19:01
anteayatwo more failing tests on a neutron patch, looking the same: https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-pg/664/console19:02
*** jamielen- is now known as jamielennox19:02
*** dkehn_ is now known as dkehn19:02
*** dkehn__ is now known as dkehn_19:03
anteaya Failed to fetch http://mirror.rackspace.com/ubuntu/dists/precise-security/multiverse/binary-amd64/Packages19:03
anteayais mirror.rackspace.com/ubuntu ours?19:03
anteayafrom what I am seeing in the logs, devstack is unable to fetch ubuntu dependencies from it19:04
*** markmcclain has quit IRC19:04
*** rohitk has quit IRC19:04
fungianteaya: rackspace manages that (or canonical may on behalf of rackspace--i'm somewhat unclear on the relationship there)19:04
*** cadenzajon_ has joined #openstack-infra19:04
anteayaokay, well from what I think I am seeing there is a problem with that mirror19:05
anteayaand patches are starting to fail as a result19:05
*** hogepodge has joined #openstack-infra19:05
anteayaanyone in channel whom we can ping?19:05
*** markmcclain has joined #openstack-infra19:05
fungipvo: do you happen to know anything about rackspace ubuntu apt mirrors having problems?19:05
sdagueanteaya: is this persistent? or just a one off?19:06
*** pblaho has joined #openstack-infra19:06
anteayait started with the requirements patch19:06
anteayaand has crept into the neutron patch19:06
anteayaI haven't looked at teh failing nova patch yet19:06
anteayait looks like it is expanding to me19:07
sdagueanteaya: right, I was more curious if yuo have a logstash query for it19:07
anteayaI haven't taken a full look19:07
*** cadenzajon has quit IRC19:07
anteayano no logstash yet19:07
sdaguethat's typically a good starting point, to figure out if it's localized, or systematic19:07
*** esker has joined #openstack-infra19:07
anteayak, trying to do meeting to19:08
anteayaThere are problems and -y was used without --force-yes19:08
anteayalooks like a key line19:08
*** DinaBelova has quit IRC19:08
*** NikitaKonovalov_ has quit IRC19:08
*** ruhe has quit IRC19:08
*** SergeyLukjanov has quit IRC19:08
anteayain devstacklog.txt.gz19:08
*** DinaBelova_ has joined #openstack-infra19:08
*** DinaBelova_ is now known as DinaBelova19:08
*** tteggel has quit IRC19:09
*** NikitaKonovalov_ has joined #openstack-infra19:09
*** ruhe has joined #openstack-infra19:09
anteayasdague: no events matched19:09
*** SumitNaiksatam has joined #openstack-infra19:09
*** NikitaKonovalov_ is now known as NikitaKonovalov19:09
anteayalogstash hasn't got the logs yet, since it is just in running tests19:09
sdagueso then it's new19:09
*** SergeyLukjanov has joined #openstack-infra19:10
*** oubiwann_ has joined #openstack-infra19:10
*** valentinbud has joined #openstack-infra19:10
*** SergeyLukjanov2 has joined #openstack-infra19:10
anteayaseems to be, yes19:10
SpamapSsoren: I wonder.. could we maybe move uvirtbot under openstack-infra's control?19:11
SpamapSsoren: it seems to go missing quite often...19:11
SpamapSand it is a really invaluable tool for easing conversations19:11
*** oubiwann_ has quit IRC19:11
*** SergeyLukjanov has quit IRC19:11
*** SergeyLukjanov2 is now known as SergeyLukjanov19:11
anteayait is affecting the nova patch: http://logs.openstack.org/18/75918/1/gate/gate-tempest-dsvm-postgres-full/42a02b8/logs/devstacklog.txt.gz19:12
anteayaand ceilometer: http://logs.openstack.org/69/76069/1/gate/gate-tempest-dsvm-neutron-large-ops/b9a7a7a/logs/devstacklog.txt.gz19:13
*** igor_ has joined #openstack-infra19:13
*** ryanpetrello has quit IRC19:13
anteayaand this ceilometer: http://logs.openstack.org/87/66687/13/gate/gate-tempest-dsvm-full/495c46f/logs/devstacklog.txt.gz19:13
*** tteggel has joined #openstack-infra19:14
anteayaso yes this mirror.rackspace.com ubuntu issue is affecting us19:14
anteayafungi: whom can I ping about this?19:14
openstackgerritA change was merged to openstack-infra/zuul: Merger: log non Git exceptions on merge  https://review.openstack.org/7097719:16
*** rossella has quit IRC19:17
*** igor_ has quit IRC19:18
fungianteaya: it looks like they may have changed how they're mirroring that, or have updated from an upstream mirror in a way such that only the compressed sources and packages files are present for a couple of suites. i can open a trouble ticket with rackspace to give them a heads up about that19:19
*** ryanpetrello has joined #openstack-infra19:19
anteayashall I open a bug on our end for those who will need to recheck/reverify?19:19
jaypipesanteaya, jeblair: lemme know if OK: http://lists.openstack.org/pipermail/openstack-dev/2014-February/028124.html19:22
anteayajaypipes: looks good to me, thank you19:24
*** mwagner_lap has quit IRC19:24
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract common dsvm to integrated-gate  https://review.openstack.org/7633319:25
kashyapclarkb, pleia2 Heya, just a note --  "Defaults requiretty" from sudo will be removed from Fedora's sudo. The maintainer just confirmed it on the bug19:27
kashyapI updated the etherpad here -- https://etherpad.openstack.org/p/fedora-on-gate19:27
clarkbkashyap: great news19:27
* kashyap hits the hay, too late here. Just wanted to note the above.19:27
pleia2kashyap: great :)19:28
*** sdake_ has joined #openstack-infra19:28
*** cody-somerville has joined #openstack-infra19:29
anteayathat is the mirror.rackspace.com bug report19:30
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273908  https://review.openstack.org/7633519:30
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract tarballs and pypi upload to pypi-release  https://review.openstack.org/7632219:32
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract common dsvm to integrated-gate  https://review.openstack.org/7633319:32
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Extract check/gate req jobs to check-requirements  https://review.openstack.org/7631219:32
anteayaso far just 5 patches that I see taken out by the mirror issue19:32
*** valentinbud has quit IRC19:33
anteaya75539,9 73086,5 75918,1 76969,1 66687,1319:34
openstackgerritA change was merged to openstack-infra/config: Install and run elasticsearch on new workers  https://review.openstack.org/7604819:34
*** che-arne has quit IRC19:35
reedmoto-g or nexus 5?19:36
anteayareed hi what are you asking about?19:36
reedanteaya, random stuff :) I lost ... my phone was stolen in a cab19:37
clarkbreed: nexus 519:37
*** fbo is now known as fbo_away19:37
clarkbmoto g is decent if you want cheap but nexus 5 is much better19:37
anteayareed: :(19:37
reedclarkb, I figured... is that worth +$200 more and +7days wait?19:38
fungianteaya: i think that apt mirror issue must have been short-lived. by the time i got enough of a break to log into a system and confirm it by hand, it seems to not be happening19:38
reedah, the 1st world problems19:38
reedthanks anteaya19:38
anteayafungi: I am thinking the same19:38
anteayasome sort of transference hiccup maybe19:39
clarkbfungi: I see the logs in the bug, were those the only logs available? gerrit lib logs at debug level a lot of additional info19:39
clarkbfungi: including return code and stderr19:39
clarkband stdout19:39
clarkbwhich would probably help us debug19:39
anteayafungi: but at least those 5 patches have a bug number19:39
*** esker has quit IRC19:40
fungiclarkb: yes, that's all puppet dropped into the syslog for us19:40
pleia2jaypipes: not sure if anyone else has asked, but would it be possible for you to convert your blog posts into creating a better http://ci.openstack.org/third_party.html ?19:40
sorenSpamapS: Yeah, it's a bit of a POS.19:41
sorenSpamapS: There's not much to it, really. It's a standard Supybot with a plugin I stole from ubottu half a decade ago.19:41
jaypipespleia2: most certainly -- all content on my blog is creative commons share-alike licensed. I was thinking of letting the blog posts bake on joinfu.com for at least another week or two (as I'm still updating the posts as folks send comments and corrections). But after that, I'd be happy to push up content patches to ci.openstack.org with some or all of that content.19:43
pleia2jaypipes: awesome, thanks :) I was reading through the posts last night and they're really great19:44
*** igor_ has joined #openstack-infra19:44
fungisoren: SpamapS: if someone wanted to whip up a similar bot which did link mapping for the services we use, shortening, all those sorts of things, i expect infra would be happy to run it19:44
jaypipespleia2: very much eager for criticism, comments, corrections, bugs,e tc :)19:44
*** atiwari has quit IRC19:45
pleia2jaypipes: I don't like maintaining the os-ext-testing separate from our infrastructure, but I don't have a better suggestion at the moment19:46
*** atiwari has joined #openstack-infra19:46
*** esker has joined #openstack-infra19:46
jaypipespleia2: me neither :) happy to push it to stackforge or somewhere in openstack-infra/ eventually...19:46
* pleia2 nods19:47
mesteryHi folks: I'm seeing some bizzare failures for some patches similar to the one here: https://review.openstack.org/#/c/73372/419:47
pleia2probably openstack-infra at some point, or maybe include it as another part of config (since there is overlap)19:47
mesteryLooks like the devstack node never even made it up. Have others seen this? I don't see a recheck bug for this, thus asking here.19:47
jaypipespleia2: of course, the idea behind os-ext-testing was to reuse absolutely as much as possible the stock puppet manifests from openstack-infra/config and only create new manifests where needed19:47
jaypipespleia2: the os-ext-testing-data repository was an attempt to introduce folks setting up these CI systems to the notion of keeping all config data in source control. I know that I could have used Hiera for the data store, too.19:49
*** mwagner_lap has joined #openstack-infra19:49
anteayamestery: we just got hit with this: https://bugs.launchpad.net/openstack-ci/+bug/128478619:49
mesteryanteaya: Thank you, that's what I was looking for!19:50
anteayaaffected 5 gate patches before it went away and I don't know how many in check19:50
anteayaI checked your devstacklog file I didn't see this error19:50
anteayamestery: is this the correct bug?19:50
mesteryanteaya: I saw it yesterday.19:50
*** shashank_ has joined #openstack-infra19:51
anteayamestery: did you19:51
*** bhuvan has joined #openstack-infra19:51
anteayado you have the logs?19:51
anteayawe just spotted it within the last hour19:51
sdaguequestion, a while ago someone had this need javascript library that let you do markup language for things like sequence diagrams19:52
sdagueand I've since lost it19:52
mesteryanteaya: http://logs.openstack.org/72/73372/15/check/check-tempest-dsvm-neutron/10663b0/19:52
mesteryanteaya: Not sure it's the same as the one you pointed me at.19:52
fungimestery: looks like it died while or immediately after setting up the metadata agent? http://logs.openstack.org/72/73372/15/check/check-tempest-dsvm-neutron/10663b0/logs/devstacklog.txt.gz19:52
mesteryanteaya: But I saw this same problem as the logs I just gave you yesterday with another Neutron patch.19:52
*** hogepodge has quit IRC19:52
sdaguemestery: yeh, we split out logs19:53
mesteryfungi: I think you're right, and then it hit a timeout and was reaped, is that it?19:53
fungimestery: yeah, i don't see any screen-n logs for it...19:54
anteayamestery: it is not looking like the same one to me, in 1284786 devstack dies because it can't apt-get from mirror.rackspace.com19:56
anteayait appears your logs get past that step19:56
*** pblaho has quit IRC19:57
mordredjaypipes: AaronGr is working on some refactor ideas that should help with the amount of needed divergence19:57
jaypipesmordred: yup, I saw that email. ++!19:57
mordredjaypipes: cause we're tracking an exact copy right now and there are ... more than one thing that _must_ be forked :)19:57
morganfainbergjeblair, oooooo19:57
morganfainbergi mean19:57
bhuvansdague: are you looking for something like http://plantuml.sourceforge.net/index.html?19:57
morganfainbergjaypipes, mordred, really?19:57
mordredmorganfainberg: yah19:57
morganfainbergjaypipes, mordred, because i was about to toss up my hands and use chef (or the like) to setup something very similar19:58
clarkbsdague: I can dig a couple up shortly, I looked at them for my LCA talk before giving up on js19:58
morganfainbergmordred, the current puppet manifests were/are very hard-coded o.o infra specific19:58
mesterythanks anteaya, I agree with that assessment19:58
shashank_Hi All... I see that my devstack is failing because it is unable to clone nova repository19:58
clarkbsdague: http://bramp.github.io/js-sequence-diagrams/19:58
*** boris-42_ is now known as boris-4219:58
sdagueclarkb: thanks19:58
shashank_2014-02-25 11:25:30.076 | + git clone git://git.openstack.org/openstack/nova.git /opt/stack/nova19:59
shashank_2014-02-25 11:25:36.643 | Cloning into '/opt/stack/nova'...19:59
shashank_2014-02-25 11:26:36.645 | fatal: The remote end hung up unexpectedly19:59
shashank_2014-02-25 11:26:36.648 | fatal: early EOF19:59
mordredmorganfainberg: they still are - but it's totally possible to do - and we're actively working on making it way less/not hardcoded infra specific19:59
shashank_2014-02-25 11:26:36.651 | fatal: recursion detected in die handler19:59
shashank_Any idea what could be wrong?19:59
morganfainbergmordred, seriously awesome to hear19:59
jeblairmorganfainberg: there's a lot of work being staged to fix that.  most of it's waiting on puppetboard to be up and running, but we're getting pretty close to that19:59
anteayashashank_: instead of pasting error codes into the channel can you instead post the url to the patch in question?19:59
jeblairmorganfainberg: then i expect we'll start merging puppet refactoring changes quickly19:59
morganfainbergjeblair, ++ nice.  i might need to stand stuff up on my own then still (i need to have an infra-like env up by next weeks end)20:00
* fungi is reminded to see if the tweak to the jetty config is up and can be merged20:00
morganfainbergwhich limits what i can do, but very very happy to hear this20:00
shashank_anteaya: sorry, which patch?20:00
morganfainbergi'll def. contribute where i can once things are moving.20:00
sdaguemestery: actually something failed in devstack setup, but something is weird about our cleanup, so it doesn't seem to have gotten logged20:00
shashank_I am using devstack and haven't specified any nova branch or patch in the localrc20:00
anteayashashank_: yes, which patch has the error you are posting about?20:01
jeblairmorganfainberg: yeah, if you can stand it, maybe muddle through as best you can for now, but don't invest heavily in alternate infrastructure and instead, come back and contribute when you aren't rushed and we're hading in the right direction20:01
*** NikitaKonovalov is now known as NikitaKonovalov_20:01
*** rlandy has quit IRC20:01
morganfainbergjeblair, yep20:02
clarkbshashank_: are you just running localhost locally?20:02
shashank_clarkb: yes20:02
clarkbif so you might be having trouble with git protocol through a proxy/firewall20:02
morganfainbergjeblair, i already have the "just do it outside of CMS due to timing" so i'll use the puppet stuff you guys are doing as the template in either case so i'm not massively diverged.20:02
morganfainbergjeblair, and i'll look at making things much more in line/leverage the o.o stuff as it becomes viable to do so / i have time to contribute to that20:03
mesterysdague: So it will be impossible to track down the exact reason for the failure here?20:03
sdagueunfortunately, yes, we've been trying to sort out these flush issues which are creeping up from time to time20:03
shashank_clarkb: My localrc is as follows: http://paste.openstack.org/show/69528/20:04
jeblairmorganfainberg: sounds like a plan!20:04
clarkbshashank_: are you behind a firewall or proxy?20:04
shashank_yes, I am20:05
shashank_but other repositories are cloned successfully20:05
*** davidhadas has joined #openstack-infra20:05
clarkbshashank_: try changing the repo root to use https20:05
anteayasdague: do we have a bug number for mestery?20:06
shashank_2014-02-25 11:24:53.331 | + git clone git://git.openstack.org/openstack/neutron.git /opt/stack/neutron20:06
shashank_2014-02-25 11:24:54.695 | Cloning into '/opt/stack/neutron'...20:06
shashank_2014-02-25 11:25:17.522 | + cd /opt/stack/neutron20:06
shashank_2014-02-25 11:25:17.525 | + git fetch git://git.openstack.org/openstack/neutron.git refs/changes/81/67281/1020:06
shashank_2014-02-25 11:25:19.645 | From git://git.openstack.org/openstack/neutron20:06
shashank_2014-02-25 11:25:19.649 |  * branch            refs/changes/81/67281/10 -> FETCH_HEAD20:06
shashank_2014-02-25 11:25:19.652 | + git checkout FETCH_HEAD20:06
anteayashashank_: please stop doing that20:06
shashank_That works!20:06
anteayaplease use paste20:06
*** denis_makogon_ has joined #openstack-infra20:06
*** mrmartin has joined #openstack-infra20:06
clarkbI forget the localrc setting for that but there is one you should see in stackrc that can be overridden in localrc20:06
shashank_sure... sorry20:06
anteayathank you20:06
*** vishy has joined #openstack-infra20:07
*** igor_ has quit IRC20:07
shashank_hmmm... but I am curious why cloning nova repository fails consistently20:07
shashank_while others succeed20:07
fungishashank_: nova is a lot larger than some other repositories. you may be hitting proxy limits on your originating network20:08
*** denis_makogon has quit IRC20:08
*** denis_makogon_ is now known as denis_makogon20:08
*** dmakogon_ has joined #openstack-infra20:09
mrmartinfungi: hi, can I see somehow a server's puppet log without logging in through ssh?20:09
fungimrmartin: i can get it for you... just a few minutes20:09
*** dcramer_ has joined #openstack-infra20:09
mrmartinfungi: allright, thanks, I need the puppet logs of openstackid-dev.openstack.org20:09
*** boris-42 has quit IRC20:10
*** boris-42 has joined #openstack-infra20:10
*** dcramer_ has quit IRC20:10
fungimrmartin: i assumed that was what you wanted. i tested the change with --noop before i approved it, but that's not entirely comprehensive of course and i hadn't gone back to see how it ended up20:10
*** dcramer_ has joined #openstack-infra20:11
* anteaya goes for a walk20:11
openstackgerritA change was merged to openstack-infra/config: Set puppetdb server to listen on all interfaces  https://review.openstack.org/7595020:11
*** vkozhukalov_ has joined #openstack-infra20:12
mrmartinfungi: I guess we have a failed deployment due the missing distro tarball, at tarballs.openstack.org/openstackid, but need to check the exact reason.20:12
*** oubiwann_ has joined #openstack-infra20:13
*** mbacchi has quit IRC20:13
*** fbo_away is now known as fbo20:15
*** ianw has quit IRC20:15
*** ianw has joined #openstack-infra20:16
*** coolsvap has quit IRC20:16
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: add some helpfulness to our log output  https://review.openstack.org/7634920:17
*** dcramer_ has quit IRC20:17
*** cadenzajon has joined #openstack-infra20:17
*** valentinbud has joined #openstack-infra20:17
*** boris-42 has quit IRC20:19
fungimrmartin: i think you're right... http://paste.openstack.org/show/69537/20:19
valentinbudHello clarkb, I have a problem with a logging infrastructure built from the openstack-infra/config bits. Multiline filters don't work as expected.20:19
*** boris-42 has joined #openstack-infra20:19
valentinbudI have been pointed in your direction.20:19
*** bhuvan has quit IRC20:19
clarkbvalentinbud: hey20:19
*** cadenzajon_ has quit IRC20:20
mrmartinfungi: may I ask you to do an rm -rf /srv/openstackid ?20:20
clarkbvalentinbud: what version of logstash are you suing with the ruleset?20:20
valentinbudHey clarkb. I am working on the HyperV CI and using the same log processing infra.20:20
fungimrmartin: sure, doing that now20:20
mrmartinfungi: how much time do we need to wait for another puppet run?20:20
clarkbvalentinbud: ok so logtash version matches20:21
*** ociuhandu has quit IRC20:21
clarkbvalentinbud: how are you testing/noticing the problem? and does it manifest itself as every line in the mulitline event being a distinct event?20:21
*** rcarrillocruz1 has joined #openstack-infra20:21
fungimrmartin: it refreshes every 10 minutes20:21
*** oubiwann_ has quit IRC20:22
valentinbudclarkb: I am feeding logstash via stdin and/or file inputs and tag the events with oslofmt.20:22
*** rcarrillocruz has quit IRC20:22
valentinbudThe logs come from nova-compute.20:22
fungimrmartin: the last update completed just before i removed that directory, at 20:21:11, so we've got about 9 minutes before it fires again20:22
*** tjones has quit IRC20:23
*** bhuvan has joined #openstack-infra20:23
clarkbvalentinbud: and you are using the json_lines codec on stdin?20:23
*** boris-42 has quit IRC20:23
*** boris-42_ has joined #openstack-infra20:23
valentinbudclarkb: no, I am not, that wouldn't work because the lines are raw text, or is my assumption wrong?20:24
clarkbvalentinbud: give me a second to reread what goes on there20:24
*** oubiwann_ has joined #openstack-infra20:25
*** shashank_ has quit IRC20:25
valentinbudThank you clarkb.20:25
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Require openstack-doc-tools 0.8.2  https://review.openstack.org/7635120:25
clarkbvalentinbud: yeah json lines is what we use because the daemon we have talking to the logstash input spits out json20:25
*** tomhe has joined #openstack-infra20:25
openstackgerritSergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627  https://review.openstack.org/7626520:26
*** mrmartin has quit IRC20:26
*** bhuvan_ has joined #openstack-infra20:27
jog0jeblair: nice20:28
*** boris-42_ is now known as boris-4220:29
valentinbudThe log-gearman-worker which also outputs to CRM114.20:29
clarkbvalentinbud: correct20:29
*** tjones has joined #openstack-infra20:29
*** bhuvan has quit IRC20:30
valentinbudI like how you've done the logging infrastructure, robust and easily scalable.20:30
*** dcramer_ has joined #openstack-infra20:30
valentinbudI have noticed something strange with logstash.20:30
*** gyee has quit IRC20:30
tomheOur Zuul setup has recently had a problem with the embedded Gearman server not registering a Jenkins job. I had to delete and recreate the job a couple of times until it listed as registered. Any ideas to what could be causing this? (On Zuul from master as of Fri Feb 21 08:28:58)20:30
clarkbvalentinbud: thanks, making it scalable was definitely one of the goals20:30
valentinbudIf I feed him only TRACE lines it filters the message as multiline.20:31
valentinbudBut if I feed him one INFO and one DEBUG line, before or after the TRACE, it doesn't do it.20:31
tomheOn a side note: The new Zuul merger architecture really improved our performance. Great work you guys!20:31
clarkbvalentinbud: interesting, so the way multiline works in logstash is a bit of a hack20:31
*** dmsimard has joined #openstack-infra20:32
clarkbvalentinbud: logtash keeps a buffer with the previous event in it, if multiline matches it updates that buffer by appending the new data until something doesn't match the multiline condition20:32
clarkbvalentinbud: when that happens the item in the buffer is flushed and the item that didn't match the condition goes into the buffer20:32
dmsimardAny issues with paste.openstack.org ? Unresponsive here.20:32
clarkbits possible that something around matching conditions is causing that buffer to flush when we don't expect it to20:32
*** bhuvan has joined #openstack-infra20:33
*** tjones has quit IRC20:33
dmsimardLooks like paste is back, nevermind about that.20:33
valentinbudclarkb: it might be that. How can I debug further. I have downloaded a nova-compute log from a Jenkins build failure and feed it via input file with start beginning.20:34
valentinbudclarkb: I output everything on stdout with codec rubydebug.20:34
valentinbudclarkb: For what you say I must go deeper. How can I do that?20:34
*** bhuvan_ has quit IRC20:34
clarkbvalentinbud: I do testing locally but spinning up a geard server and a logstash-gearman-worker pointed at localhost 9999 and have a logstash instance with tcp input listening there20:35
*** hogepodge has joined #openstack-infra20:35
clarkbthen I have a simple script that adds a job to the geard server causing logs to be fetched and go through the whole process20:35
clarkbvalentinbud: give me a few minutes and I can share that small script with you20:35
clarkbbut mocking up the whole pipeline like that would probably help20:36
*** mrmartin has joined #openstack-infra20:36
*** bhuvan_ has joined #openstack-infra20:38
*** bhuvan_ has joined #openstack-infra20:38
*** dhellman_ has joined #openstack-infra20:40
*** dhellman_ has quit IRC20:40
clarkbfungi: have you got elasticsearch head and bigdesk running locally?20:40
clarkbfungi: they run better in chromium fwiw20:40
fungiclarkb: took a detour to catch openstackid and puppetdb work up so i'm not blocking people. getting started on bigdesk installation now20:40
clarkbno rush, I think once we start iterating on nodes will have lots of time in between steps20:40
clarkbbecause data needs to be copied around in order to not lose it20:40
fungiis there any need to sandbox this stuff? just wondering if i should be sticking it into a vm of some sort rather than running straight on my workstation20:40
fungii assume it works in debian20:40
clarkbit is all js so sandboxed by browser20:40
fungioh, wfm20:40
*** oubiwann_ has quit IRC20:42
*** bhuvan has quit IRC20:42
*** dhellman_ has joined #openstack-infra20:42
*** bhuvan has joined #openstack-infra20:42
*** bhuvan has joined #openstack-infra20:42
openstackgerritA change was merged to openstack-infra/config: Enable puppetdb from puppetmaster  https://review.openstack.org/7461220:42
clarkbfungi: I just clone their github repos and open index.html in chromium20:42
*** jlibosva has quit IRC20:42
*** bhuvan__ has joined #openstack-infra20:43
fungioh, no installing needed. great20:43
*** dcramer_ has quit IRC20:43
*** ekarlso has quit IRC20:43
*** bhuvan has quit IRC20:43
*** bhuvan__ has quit IRC20:43
fungihttps://github.com/lukas-vlcek/bigdesk.git ?20:43
*** bhuvan has joined #openstack-infra20:44
*** DinaBelova is now known as DinaBelova_20:44
*** ekarlso has joined #openstack-infra20:44
*** bhuvan_ has quit IRC20:45
valentinbudclarkb: I got to this after checking that all the other components of the system work. The gearman client subscribes to Jenkins ZMQ, receives the message, pass it to the workers that pull the logs via HTTP and feed it to CRM114 and to Logstash which indexes in Elasticsearch.20:45
*** mrda_away is now known as mrda20:46
valentinbudclarkb: I didn't want to modify substantially the indexer.conf file to be able to test it locally so I tag input with oslofmt and type jenkins. That means they are processed in the else if oslofmt in [tags] filter.20:48
clarkbmy current hunch is that the input codecs are doing smoething subtly different in ways that break multiline20:49
valentinbudBoth the multiline filters make sense to me. Match lines that have TRACE and the ones that don't begin with TIMESTAMP_ISO8601.20:50
*** ildikov_ has joined #openstack-infra20:50
valentinbudI even tried to use multiline codec - http://logstash.net/docs/1.3.3/codecs/multiline with the same pattern. Had the same results.20:51
*** weshay has quit IRC20:51
*** lcostantino has quit IRC20:51
clarkbyeah, maybe they are all getting a timestamp in some weird way?20:51
fungiBigdesk may not work correctly! Found ES node version: 0.90.9; Requires ES node version: >= 1.0.0.RC120:52
fungii guess i should checkout an older tag20:52
valentinbudclarkb: do this timestamps look weird?20:53
clarkbfungi: oh yeah, their readme should tell you which versions support other versions20:53
clarkbfungi: I can also give you my git sha if it helps20:53
*** dangers_away is now known as dangers20:53
*** jergerber has joined #openstack-infra20:54
*** Ajaeger has joined #openstack-infra20:54
fungiclarkb: v2.2.2 works, later tags give that warning20:54
fungi(for future reference)20:54
*** dcramer_ has joined #openstack-infra20:55
AjaegerInfra-team, any ideas why Zuul is not picking up this review https://review.openstack.org/#/c/67223/ ?20:55
clarkbvalentinbud: they look ok to me, but I would compare them to upstream jenkins logs20:55
fungiclarkb: so what specifically should i be looking at now if anything? (i see what you mean now when you say "the cluster is green")20:55
*** jraim_ is now known as jraim20:56
valentinbudI would do that first then. Thank you.20:56
clarkbfungi: under nodes, you can click individual nodes and see a bunch of metrics20:56
clarkbfungi: we will be itnerested in indexing requests per second, ram and cpu use20:56
clarkbfungi: elasticsearch head is the other tool you want to grab, same process as bigdesk20:56
fungiAjaeger: was it wip when it got approved?20:56
openstackgerritAndrew Lazarev proposed a change to openstack/requirements: Bump python-savannaclient to 0.5.0  https://review.openstack.org/7635720:56
clarkbelasticsearch head is helpful because it shows you shard state in an easy to view way20:56
*** mbacchi has joined #openstack-infra20:57
Ajaegerfungi: yes, it was. And then I approved it again20:57
fungiAjaeger: we may have a hole in our layout.yaml gate enqueuing logic there, if the change already has +2 vrfy and it not merged20:58
fungisince that happens rarely20:58
Ajaegerso, what can I do?20:59
fungiAjaeger: i just rechecked it... it will re-gate after the current check run finishes. please open a bug about this against openstack-ci so we remember to address it20:59
Ajaegerfungi: will do!20:59
clarkbfungi: let me know when you have elasticsearch head up to and have had a chance to poke around with the tools21:00
*** dcramer_ has quit IRC21:00
fungiclarkb: what's the plan for upgrading elasticsearch? (presumably after we get the cluster migrated to the new nodes)21:00
clarkbfungi: note head in particular gives you the ability to delete stuff so be a little careful :)21:00
*** dhellman_ has quit IRC21:00
clarkbfungi: that is a tough one since there are breaking changes between 0.90.921:01
fungiclarkb: ohhh... you're suggesting i run a copy of elasticsearch too, not just elasticsearch bigdesk connected to the existing cluster? i totally misunderstood what you meant there21:01
clarkbfungi: no no21:01
clarkb'elasticsearch head'21:01
clarkbit is another tool like bigdesk runs in your browser shows you stuff21:01
*** hashar has joined #openstack-infra21:01
fungiis not the same thing as the head of elasticsearch's master branch21:01
dhellmannsdague: about https://review.openstack.org/#/c/74408/ -- I need to read up, but I wanted to make sure you saw the comments before I start re-implementing anything21:01
*** david_lyle is now known as david-lyle21:01
clarkbfungi: in particular it shows you shard state21:02
clarkbwhich is really handy when we start adding and removing nodes21:02
*** thedodd has quit IRC21:02
fungiclarkb: so https://github.com/mobz/elasticsearch-head21:02
sdaguedhellmann: yeh, ok, so the road is really a functional test suite?21:03
Ajaegerfungi: filed - https://bugs.launchpad.net/openstack-ci/+bug/128484221:03
fungiclarkb: neat, i've got it up in another tab too21:04
fungiAjaeger: thanks!21:04
Ajaegerfungi: I have to say thanks!21:04
*** rcarrillocruz has joined #openstack-infra21:04
clarkbfungi: feel free to poke around in both tools and get a feel for what they are showing you21:04
*** Ajaeger has quit IRC21:05
clarkbfungi: as I think having that mental model is a good thing21:05
nibalizerfungi: so looks like puppetdb fix + connector landed21:05
clarkbparticularly for the shard layout21:05
funginibalizer: yes, i'm about to go back and see what the logs have to say now21:05
*** rcarrillocruz1 has quit IRC21:05
dhellmannsdague: I want to make sure we don't commit anything to oslo.test that breaks unit tests in the projects using it21:05
dhellmannsdague: I don't know if that counts as "functional" or not, but it seems like the way to test it is to run the unit tests of those projects21:06
nibalizertail -f /var/log/puppetdb/puppetdb.log should say things like 'executed "replaced facts" for node foo.o.o"21:06
*** dprince_ has quit IRC21:06
sdagueright, so you'll have a job which jumps into the various unit test directories and runs the tests21:07
funginibalizer: the syslog says...21:07
funginibalizer: Feb 25 20:49:20 ci-puppetmaster puppet-agent[32308]: (/Stage[main]/Puppetdb::Master::Config/Service[puppetmaster]) Failed to call refresh: Could not find init script or upstart conf file for 'puppetmaster'21:07
sdagueso... have you thought about the duration of said job?21:07
jeblairdhellmann: so this is really (test + incubator), (test + config), (test + messaging)?21:08
sdaguebecause neutron 2.7 unit tests come in at 40 minutes21:08
jeblairdhellmann: and not (test + incubator + config + messaging)21:08
dhellmannjeblair: right, the former21:08
*** eharney has quit IRC21:08
dhellmannsdague: I'd be happy if we could do it as multiple parallel jobs21:08
funginibalizer: but regardless, looking at the puppetdb log we seem to be getting events, so should hopefully be good to go for slapping puppetboard atop of this?21:08
sdaguedhellmann: well that's already parallel using all the codes21:08
dhellmannsdague: this was a first stab, so we could start using the lib in oslo without breaking ourselves :-)21:08
dhellmannsdague: I mean neutron in one job and nova in another, on different hosts21:09
dhellmannlike we do with py27 and py26 tests21:09
jeblairdhellmann: you're also running neutron and nova tests?21:09
nibalizerfungi: I assume we run puppet under apache/mod_passsenger/rack ?21:09
dhellmannjeblair: we will, when they start consuming the library21:09
tomheOur Zuul setup has recently had a problem with the embedded Gearman server not registering a Jenkins job. I had to delete and recreate the job a couple of times until it listed as registered. Any ideas to what could be causing this? (On Zuul from master as of Fri Feb 21 08:28:58)21:09
nibalizeri'll send a review your way21:09
dhellmannjeblair: *all* integrated projects21:09
sdaguejeblair: yeh, this is about forward testing21:09
sdagueso this library can't break those21:10
dhellmannI want to add the job to prevent us breaking before we land the change to start using the new lib21:10
fungiclarkb: in bigdesk, what feats has elasticsearch4 performed to earn a medal of honor?21:10
jeblairyeah, i just didn't realize the scope extended outside oslo.  got it.21:10
fungiit looks plenty important21:10
clarkbfungi: it was elected the master21:10
dhellmannjeblair: sure, that wasn't clear from the review21:10
jeblairdhellmann: so in the future, you might have a test that's testing (oslo.test + oslo.messaging + nova) together?21:11
jeblairdhellmann: or would you still be doing (oslo.test + nova) ?21:11
dhellmannjeblair: I don't think so, no21:11
dhellmannjeblair: just oslo.test + foo21:11
clarkbfungi: other points of interest, on the GC graph, if the pink line goes above the blue line it usually means big trouble21:12
*** rcarrillocruz1 has joined #openstack-infra21:12
clarkbfungi: that implies a few GC runs are taking massive amounts of CPU time21:12
dhellmannclarkb: don't cross the streams21:12
clarkbwhich usually leads to the cluster falling over beacuse intercluster pings stop getting processed21:12
jeblairdhellmann: nice! :)21:12
*** smarcet has quit IRC21:13
*** rcarrillocruz has quit IRC21:13
fungiclarkb: noted21:13
clarkbfungi: other points of interest, in elasticsearc-head if yo uclick on a green shard box you get a pop up thingy with lots of info21:13
*** mrmartin has quit IRC21:14
clarkbthis is handy when you watch shards get restored as you can see progress in the size of the recovering shard21:14
clarkbfungi: when you are comfortable I think we can go ahead and turn on puppet on the first node and see what happens21:15
fungiokay, i think 67223 must not be an enqueuing logic issue in zuul... probably need to go digging in the log21:15
sdaguedhellmann: so look at solum, barbican, they have gate_hook scripts to do this21:15
sdagueand the swift functional test21:15
clarkbfungi: just ping me and I will pay attention wen you are ready for that21:16
sdaguewhich gives you the d-g environment, but you can do anythign you want from there21:16
dhellmannsdague: ok, sounds good -- I went this direction based on our previous conversation so I wanted to make sure you knew about the change21:16
fungiclarkb: will do--thanks@21:16
clarkbfungi: agree re 6722321:16
jeblairdhellmann: ok.  i think that still the combination of two projects is still worth using the d-g code to set up the repos.  also, i think we can template that so you can do this as 3 separate jobs21:16
sdagueyeh, I just didn't fully grok it21:16
openstackgerritSergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627  https://review.openstack.org/7626521:16
dhellmannjeblair: that sounds great21:16
sdagueappologies for sending you down a bad path21:16
jeblairdhellmann: that way it's just a matter of substituting in 'foo' to add a new one21:16
dhellmannsdague: not a big deal, just trying to keep everyone on the same page21:16
dhellmannjeblair: I'll see if I can take a stab at implementing it that way, unless you think you'll have time21:17
jeblairsomeday we'll genericize this a bit more so you don't have to use something called "devstack-gate" to get your repos set up... hashar wants that too.  :)21:17
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Set apache as the puppet service name  https://review.openstack.org/7636621:17
*** dolphm is now known as dolphm_50321:18
nibalizerfungi: that ^^ should deal with the 'no service puppetmaster' problem'21:18
*** dmsimard has left #openstack-infra21:18
*** miqui has quit IRC21:18
jeblairdhellmann: i'll take a stab at setting up the jjb template to call your script21:18
jeblair(via devstack-gate)21:18
*** jhesketh has joined #openstack-infra21:18
dhellmannjeblair: ok, let me know what args I'll need to handle (I need to look at how those hooks work still, today has been all meetings)21:19
*** jhesketh_ has joined #openstack-infra21:19
*** tjones has joined #openstack-infra21:19
jeblairjhesketh: good morning!21:21
hasharjeblair: dhellmann I even have a patch pending somewhere that reimplements devstach shell script in pure python.  Nicknamed that zuul-cloner.21:21
jheskeththat stuff was cool hashar :-)21:21
jeblairhashar: yeah, i can see a future where we install that on all our slaves, and then devstack-gate calls it, and any other job can too21:21
*** thedodd has joined #openstack-infra21:21
hashardhellmann: the idea is to pass a list of repositories clone them then for each repository attempt to apply the Zuul ref it exists, fallback to the branch if it exists, else master.21:21
mordredhashar: ++21:22
jeblairhashar: also, since it's in python, we can have it produce sensible debug/error messages which we don't get in bash21:22
*** julim has quit IRC21:23
*** oubiwann_ has joined #openstack-infra21:23
hasharI resurrected the patchset which is https://review.openstack.org/#/c/70373/21:23
fungijeblair: https://launchpad.net/bugs/1284842 may be of interest to you21:24
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Openstackid deployment symlink override fix  https://review.openstack.org/7636821:24
hasharI am busy integrating Selenium tests to be triggered with Gerrit. So I am unlikely to work on that zuul cloner anytime soon :-/21:24
jeblairhashar: cool, thanks21:24
*** mrmartin has joined #openstack-infra21:24
anteayatomhe: the ones who can answer your question are jeblair fungi and clarkb and all are currently in the middle of something21:25
jeblairfungi: ack21:25
anteayatomhe: I suggest patience and try again once one of them appears free21:25
mrmartinfungi: a very trivial fix for the openstackid deployment problem: https://review.openstack.org/#/c/76368/1/modules/openstackid/files/functions if you have some time plz. approve21:26
mrmartinthen we also need an rm -rf /srv/openstackid to cleanup the application directory.21:26
fungimrmartin: okay, can do21:26
hasharmordred: jeblair: Turbo-hipster author might be interested as well though I forgot to follow up.21:26
jeblairhashar: that's jhesketh21:26
jeblairhashar: he just said it's cool.  ;)21:27
jog0hoping to get an infra review on this e-r fingerprint https://review.openstack.org/#/c/76043/1/queries/1284371.yaml21:27
fungimrmartin: i see only some of your symlinks use force. do the others not need it?21:27
mrmartinI guess no21:27
fungithat answer works for me then ;)21:27
jog0being its an ifra bug21:27
*** chandan_kumar has quit IRC21:27
mrmartinwe can enhance those scripts later, but lets deliver something minimal that works21:28
*** zul has quit IRC21:28
funginibalizer: thanks!21:28
hasharjeblair: jhesketh hipster thing might well let me replace Jenkins one day :-] If manage to add some plugins to turbo-hipster hehe.21:28
funginibalizer: is puppet_service_name (apache) going to be different from the initscript service name (apache2) or is that a typo?21:29
jog0jeblair: thanks!21:29
jeblairhashar: that's the idea and what we're working toward.  the swift log upload is the next step in that (replaces jenkins scp artifact archiving)21:29
hasharjeblair: swift is a good idea. I guess it is more scalable than a 2TB disk21:30
jeblairtomhe: i don't have an immediate suggestion for a fix, but to help debug, you can look in the jenkins logs (gearman-plugin is fairly verbose) and you can enable DEBUG level logging for the zuul gear server21:30
nibalizerfungi: typo for sure21:30
nibalizeri've been in 'httpd' (ungh) land for tooo long21:30
jeblairtomhe: with that you should be able to at least find out if the problem is gearman-plugin or gear server21:31
funginibalizer: thanks--no worries21:31
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Set apache as the puppet service name  https://review.openstack.org/7636621:31
nibalizerfungi: updated21:31
funginibalizer: lgtm. thanks again21:33
*** alexpilotti has quit IRC21:33
*** gyee has joined #openstack-infra21:34
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1284371  https://review.openstack.org/7604321:34
nibalizerhappy to do it21:34
nibalizeri'll start working on the puppetboard change21:34
nibalizerprobably wont give it to you today, perhaps tomorrow21:34
funginibalizer: awesome... i can't wait to look it over21:34
*** dizquierdo has joined #openstack-infra21:34
nibalizerwe were thinking puppetboard on the puppetdb host at first for simplicity right?21:35
funginibalizer: that was my understanding21:35
nibalizerokay cool21:35
fungiclarkb: okay, i'm starting (and configuring to start on boot) the puppet agent on elasticsearch01 unless you have objections21:35
*** cody-somerville has quit IRC21:36
clarkbno objections 01 is a fine place to start21:37
*** flaper87|afk is now known as flaper8721:37
* fungi always likes to start at the beginning21:37
openstackgerritA change was merged to openstack-infra/config: Openstackid deployment symlink override fix  https://review.openstack.org/7636821:38
fungimrmartin: in another 7 minutes that ^ will be on our puppetmaster, at which point i'll clear away /srv/openstackid again in preparation for the next update cycle21:39
mrmartinfungi, great, with this patch it was working in my vargant environment21:39
fungiclarkb: so, we've got an elasticsearch01 already appearing in bigdesk. that was quick21:39
*** amcrn has quit IRC21:40
fungiclarkb: also in es-head21:40
clarkbfungi: cool, the heap mem numbers bother me though21:40
clarkbfungi: it should have mlockall'd 30g of memory21:40
clarkbbut doesn't appear to have done that21:40
fungiclarkb: i'll check its config21:41
clarkbfungi: you should be able to safely stop the service if you need to since that node doesn't have any data on it yet21:41
jog0heh jenkins will rerun if a 3rd party CI server comments21:43
jog0hopefully 3rd party wont rerun if jenkins comments otherwise we may get a loop21:44
fungiclarkb: the es-head jvm.mem is where you were looking, right?21:44
fungi"heap_max: 989.8mb"21:44
clarkbfungi: I was looking at bigdsek but similar numbers21:45
anteayajog0: where are you seeing that?21:45
*** thomasem_ has joined #openstack-infra21:45
clarkbfungi: committed heap mem is just over 200MB21:45
clarkbthat should be like 29.9GB21:45
*** dhellman_ has joined #openstack-infra21:46
*** luis_ has joined #openstack-infra21:47
fungiclarkb: as seen in /etc/default/elasticsearch...21:47
fungi# Heap Size (defaults to 256m min, 1g max)21:47
clarkbok so that went in properly, I am going to blame mlockall then21:47
fungimaybe the file was puppeted in after the package was installed/service was started?21:47
clarkbfungi: ohhhh21:47
*** oubiwann_ has quit IRC21:47
*** thomasem has quit IRC21:47
*** lnxnut_ has joined #openstack-infra21:48
clarkbfungi: lets double check the mlockall settings too then restart the service21:48
* fungi always suspects puppet order of operations logic first21:48
*** eharney has joined #openstack-infra21:48
clarkbfungi: in /etc/elasticsearch/configsomething.yml there should be an mlockall setting set to true21:48
*** fbo is now known as fbo_away21:48
clarkbbut it will only work if ES_HEAP_SIZE is properly set21:48
*** ok_delta has quit IRC21:48
fungibootstrap.mlockall: true21:49
fungiif so, then yes it's there21:49
*** luisg has quit IRC21:50
fungi"Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set to the same value...make sure that the ElasticSearch process is allowed to lock the memory, eg. by using `ulimit -l unlimited`."21:50
fungido we need to check any of that?21:50
clarkbno, ES_HEAP_SIZE sets it for us21:50
clarkbso I think your theory is sound21:50
*** lnxnut has quit IRC21:51
clarkbjust do an service stop then start21:51
fungiokay, i'll restart the service and see whether it changes21:51
clarkband let us see how that goes21:51
*** kynes has quit IRC21:51
*** cody-somerville has joined #openstack-infra21:51
*** cody-somerville has quit IRC21:51
*** cody-somerville has joined #openstack-infra21:51
fungi"heap_max: 29.8gb"21:52
clarkbmuch better21:53
*** cadenzajon has quit IRC21:53
fungiso i'll see about fixing that in the puppetingness21:53
*** yolanda_ has joined #openstack-infra21:53
*** dkliban has quit IRC21:53
fungimrmartin: btw your updated deploy has happened, looks like21:53
clarkbthough I think the cluster is crankier about the flap than I anticipated... oh well it should eventually become appy again21:53
mrmartinoh great, we just need to setup the membership db with some fake test data21:54
*** jcoufal has quit IRC21:54
mrmartinfungi: thanks for the help21:54
fungimrmartin: sure thing. if you need me to source a mysqldump into it or something, let me know21:54
*** apevec has joined #openstack-infra21:54
*** gokrokve_ has joined #openstack-infra21:55
clarkbfungi: ok it is happier now, over in elasticsearch-head you can see two purple shards, they are relocating to the new node21:55
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: add some helpfulness to our log output  https://review.openstack.org/7634921:56
clarkbit will relocate shards to balance things out, I think we should let it do that until it is appy then we can turn off es621:56
clarkbit will then move more data around, then we can add 0221:56
mrmartinfungi: I think we need some real x509 SSL certs, because for dev we are currently using self-signed certificates, but I'll open a blueprint about that21:56
clarkbrince and repeat21:56
*** jcoufal has joined #openstack-infra21:56
*** Sukhdev has joined #openstack-infra21:56
fungiclarkb: sounds great. i'm going to pop out for dinner shortly and then resume with the fun21:57
fungimrmartin: okay, we can get those for you when needed for production--no need for a blueprint21:57
*** gokrokve has quit IRC21:57
fungimrmartin: jeblair usually takes care of them21:57
*** dcramer_ has joined #openstack-infra21:57
*** esker has quit IRC21:58
clarkbfungi: sounds good, thank you for the help21:58
mrmartinfungi: ok, I did some testing yesterday with a local gerrit installation, and I had problems with self-signed certs, needed to tweak the java cert store21:58
fungimrmartin: very good point. it may be easier in this case for us to add one for dev as well, given the nature of that particular project21:59
fungirather than trying to hunt down all the ways we need to set other consuming dev instances of applications to not validate or manually update their trusts21:59
*** dhellman_ has quit IRC21:59
openstackgerritA change was merged to openstack-infra/elastic-recheck: remove old queries  https://review.openstack.org/7619622:00
*** thomasem_ has quit IRC22:02
*** bhuvan_ has joined #openstack-infra22:03
*** lnxnut_ has quit IRC22:03
*** lnxnut has joined #openstack-infra22:04
mrmartinjeblair: could you make a valid x509 certs for openstackid-dev.openstack.org domain?22:04
jeblairmrmartin: yes, i'll do that this week; i have another cert to get too.22:05
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Correct elasticsearch config dependency ordering  https://review.openstack.org/7637622:05
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers  https://review.openstack.org/7605122:05
*** jeckersb is now known as jeckersb_gone22:06
fungiclarkb: merge or fiddle with 76376 as needed while i'm out at dinner, and then we can test it when we bring up the next cluster member22:06
mrmartinjeblair, please let me know when the cert is ready, and I'll prepare the config for that. What is the usual way, will we inject the private key / cert through hiera variables?22:06
*** bhuvan has quit IRC22:07
fungimrmartin: that is precisely how we do it, yes. you can see examples of that throughout the openstack-infra/config repository22:07
mrmartingreat, thank you guys for the help.22:07
*** bhuvan_ has quit IRC22:08
*** bhuvan has joined #openstack-infra22:08
fungimrmartin: i believe i already configured the openstackid module that way22:08
fungiso we just update what's in hiera if so22:08
*** lnxnut has quit IRC22:08
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware  https://review.openstack.org/7553922:08
mrmartinfungi: we need to enlist the ssl params at the head of openstack_project::openstackid_dev class, but openstackid supports the ssl path variables.22:10
*** rossella has joined #openstack-infra22:11
fungiokay, great22:12
* fungi heads to dinner... back in a while22:12
tomhejeblair: Ok, thanks22:14
jheskethjeblair: your queue-name patch... would it be more sensible for the queue name to be on the job description in the layout.yaml? Specifically it is the job that joins queues together, not projects22:14
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1273908  https://review.openstack.org/7633522:15
*** oubiwann_ has joined #openstack-infra22:16
*** hashar has quit IRC22:16
mrmartinfungi: could you send me the latest puppet run of openstackid?22:16
*** dcramer__ has joined #openstack-infra22:17
*** jcoufal has quit IRC22:17
apevecsdague, I need one more devstack core review for https://review.openstack.org/76189 (issue blocks stable/grizzly and havana)22:17
*** dcramer___ has joined #openstack-infra22:19
sdagueapevec: done22:19
jeblairjhesketh: that's true.  that may be more correct and not significantly less convenient to configure.  presumable we would set that for 'gate-mumble-tempest-mumble'...22:20
*** jcoufal has joined #openstack-infra22:20
jheskethyeah, that's my thinking22:20
*** dcramer___ has quit IRC22:20
*** dcramer___ has joined #openstack-infra22:21
*** flaper87 has quit IRC22:22
*** flaper87 has joined #openstack-infra22:22
*** dcramer_ has quit IRC22:22
*** dcramer__ has quit IRC22:23
*** shashank_ has joined #openstack-infra22:24
jheskethjeblair: I've added a few more thoughts in my review22:25
jeblairjhesketh: thx22:25
*** lnxnut has joined #openstack-infra22:26
*** bhuvan has quit IRC22:26
*** bhuvan has joined #openstack-infra22:27
*** dcramer___ has quit IRC22:27
*** thomasem has joined #openstack-infra22:28
*** salv-orlando has joined #openstack-infra22:29
*** dolphm_503 is now known as dolphm22:29
*** denis_makogon has quit IRC22:30
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add oslo.test integration test  https://review.openstack.org/7638122:30
*** mfer has quit IRC22:30
jeblairdhellmann, sdague: ^ there's a rough cut of what that looks like.  i realize i have been operating under the assumption that we want a symmetric gate between oslo.test and foo....22:31
*** dcramer___ has joined #openstack-infra22:32
jeblairdhellmann, sdague: however, if that's not the case, and we only want to test proposed oslo.test changes with foo master in an asymmetric gate, then the original approach (with some changes to use cached git repos and the pypi mirror) would probably be ok.22:32
*** ociuhandu has joined #openstack-infra22:33
jeblairdhellmann, sdague: (this would work for that too, but is slightly overkill)22:33
*** valentinbud has quit IRC22:33
*** miqui has joined #openstack-infra22:34
*** yolanda_ has quit IRC22:35
*** jergerber has quit IRC22:36
*** CaptTofu_ has quit IRC22:38
*** mwagner_lap has quit IRC22:39
openstackgerritA change was merged to openstack-infra/config: Add logstash job queue graph  https://review.openstack.org/7605622:40
*** thomasem has quit IRC22:42
*** jergerber has joined #openstack-infra22:43
*** mriedem has quit IRC22:44
*** thedodd has quit IRC22:44
*** jcoufal has quit IRC22:46
*** weshay has joined #openstack-infra22:46
*** cody-somerville has quit IRC22:49
*** jerryz has joined #openstack-infra22:49
*** mbacchi has quit IRC22:50
*** shashank_ has quit IRC22:51
*** rcleere has quit IRC22:53
*** vkozhukalov_ has quit IRC22:57
pleia2can we get this nudged along? https://review.openstack.org/#/c/70007/ (would help me out to carry fewer local patches right now)22:58
pleia2clarkb: you mentioned you wanted to test it in case of weird firewall things22:58
openstackgerritA change was merged to openstack-infra/zuul: Update trigger script for new zuul url parameter  https://review.openstack.org/7339222:59
clarkbpleia2: ah yes, I can do a quick application of that somewhere22:59
*** shashank_ has joined #openstack-infra22:59
clarkbjeblair: jhesketh: mordred: https://review.openstack.org/#/c/76057/1 trying to decide if I actually like that23:00
openstackgerritA change was merged to openstack-infra/meetbot: Fix misspellings in meetbot  https://review.openstack.org/7204823:01
clarkbjeblair: jeblair mordred part of me thinks that the zuul merger should be trusted then it can push to untrusted git replicas which the jobs consume instead of the zuul merger directly23:01
clarkbjhesketh: ^ tab complete fail sorry23:01
openstackgerritA change was merged to openstack-infra/os-loganalyze: Fix misspellings in os-loganalyze  https://review.openstack.org/7205223:01
mordredI like that you talk to jeblair twice23:01
jheskethhis opinion matters twice as much ;-)23:01
lifelessor he listens half as much :P23:02
clarkbwhat do you guys think? should I continue to follow the train of thought in that WIP'd change or make mordred finish the zuul merger replication work23:02
*** bhuvan_ has joined #openstack-infra23:02
*** dcramer___ has quit IRC23:02
openstackgerritA change was merged to openstack-infra/zuul: Correct the example of zuul enqueue command  https://review.openstack.org/7230823:02
clarkbjhesketh: your comment makes me think this ins't a terrible idea, but at the same time it seems clunky. Liek a more generic thing would be to have zuul mergers hand off a url to an arbitrary place then they can be trusted23:02
*** dcramer___ has joined #openstack-infra23:03
*** ryanpetrello has quit IRC23:05
jheskethclarkb: not sure what you mean by the arbitrary place that they can be trusted?23:05
*** openstack has joined #openstack-infra23:06
clarkbat a network level they do23:06
clarkbwhich is what we are working around, if there is a btter term I can use let me know23:07
jeblairclarkb: zuul already supports a completely private access model -- don't run a public http server, and require your workers to ssh into the zuul mergers to fetch.23:07
*** bhuvan_ has quit IRC23:07
*** bhuvan has joined #openstack-infra23:07
jeblairclarkb: these changes are instead both just to work around arbitrary firewall rules.  "don't allow incoming connections" is not actually a security model based on trust.23:07
jheskethso I kind of like the thought process that a merger should be trusted. Separating out the mergers was only ever a way of scaling horizontally and therefore should sit next to zuul... with mordred's change to push refs elsewhere that could be enough for workers to grab their refs from?23:08
jeblairthat's kind of beside the point other than i would suggest that talking about "trust" isn't the right way to think about this.23:08
clarkbthats fair23:08
*** lcostantino has joined #openstack-infra23:09
*** changbl has quit IRC23:09
*** dcramer__ has joined #openstack-infra23:09
*** ryanpetrello has joined #openstack-infra23:10
*** dcramer___ has quit IRC23:10
mordredyeah. I can agree withthat23:10
clarkbso to reframe, slaves can only talk to a similarly hosted git mirror (either zuul merger serving out its refs or a replica) in "external" space and they cannot talk to an internal gerrit23:11
clarkbat least not from external -> internal. We can have internal push out to external all we want23:12
clarkbfrom zuul's perspective does making the zuul merger push to an external replica make more sense than running an external zuul merger that can serve refs without talking to gerrit?23:12
jeblairclarkb: in either case you are doubling the number of times you transfer the refs.  the overall performance is probably similar.23:13
jeblairclarkb: from an application engineering perspective, you've got the mergers on either the inside or outside of the evil corporate firewall...23:14
*** dcramer__ has quit IRC23:14
*** markmcclain has quit IRC23:14
*** ryanpetrello has quit IRC23:14
jeblairclarkb: so one thing to consider is: where do you really want them?  inside the firewall for stability?  outside the firewall for scalability?23:14
jeblairclarkb: because basically the use case here is that you're running most of the app internally but using external cloud resources for some of the computation.23:15
jeblairclarkb: of course, if you put the mergers inside, you still need git replicas outside.23:15
*** dstanek has quit IRC23:15
*** dkliban has joined #openstack-infra23:15
clarkbmordred: ^ you are in a btter position to answer these questions than I am23:16
*** jgrimm has quit IRC23:16
mordredwell, although I agree with jeblair on the general idiocy of the points of view of evil corporate overlords23:17
*** dkranz has quit IRC23:17
mordredthe current situation is that outbound ssh is "trusted" - so replicating out to a git replicate is "more better safer ESTUPID"23:17
*** pdmars has quit IRC23:17
jeblairclarkb: anyway, i'm okay with both patches in principle.  as for which approach would be better in this case: i'm available for implementation consulting at extremely unreasonable rates.  ;)23:17
mordredbut ... I think I'd rather that we don't just put ina  crutch for this - but rather if one of the approaches is actually a better fit in some way for zuul23:18
mordredjeblair: :)23:18
clarkbright which is why I ask, I don't think we need both patches23:18
clarkbwe here == zuul upstream23:18
jeblairclarkb: oh yay welcome back. :)23:19
clarkbzuul shouln't need to have N firewall workarounds. Just one sufficiently useable one23:20
jheskethmy vote for the two options is to expect mergers to sit next to zuul and therefore use zuul/gerrit's repos and push references arbitrarily. I also think this gives zuul more flexibility. It allows zuul to place refs wherever is required as opposed to requiring a zuul-merger to be installed at the destination23:20
jeblairokay, i'm trying to strain to think of a use for each of these patches other than as a firewall workaround23:20
jeblairclarkb: yours is a general 'put something in front of gerrit' approach which might help people in seriously large gerrit situations (like where there is no direct gerrit access)23:21
jeblairmordred: jhesketh makes a good point about yours, though it's hard for me to see a concrete application of it other than as a firewall workaround.23:23
*** ociuhandu has quit IRC23:23
jeblair(like _why_ would you want to push refs to samething other than the merger?  maybe you want really big permanent git servers somewhere and ephemeral mergers?  not sure why, but your patch would support that)23:23
clarkbyeah I like mordreds too23:23
jeblairclarkb: oddly enough, i think yours is the most likely to be useful by openstack-infra.  because it's possible we may someday decide we want to have zuul hit the apache mirror.23:24
clarkboh good point23:24
fungii was leaning the same direction23:25
clarkbespecially if the number of zuul mergers becomes quite large23:25
* fungi also caught up on scrollback now23:25
jeblairclarkb: indeed.23:25
mordredyeah. I think I agree with jeblair23:26
jheskethas in instead of gerrit, fetch from git.o.o?23:26
mordredwe've hit the scaling wall of needing to move off of a single location for cloning a few times now :)23:26
jeblairclarkb, mordred: if you want to twist my arm and force me to care about this, i think we should seriously consider both.23:27
jeblairjhesketh: that, but also we run an apache mirror at git.o.o that overlays gerrit23:28
jeblairjhesketh: that, but also we run an apache mirror at review.o.o that overlays gerrit23:28
clarkbjeblair: I am open to considering both too, my initial reaction is only one should be necessary but we seem to be coming up with decent argumetns for both that don't overlap23:28
jeblairjhesketh: so if you fetch from https://review/p/foo23:28
mordredjeblair: as in, finishing/applying both? or consider their relative merits further?23:28
jeblairjhesketh: that's actually served by apache from a location gerrit replicates to23:29
jheskethright, yes23:29
fungiit just *happens* to reside on the same machine23:29
jeblair(to reduce load on gerrit itself which as it turns out isn't the worlds best static content server)23:29
jeblairmordred: apply both.23:30
fungiyaknow... "java"23:30
mordredjeblair: interesting23:31
jeblairmordred: cause if the goal is implementation flexibility to deal with evil firewalls, i could see two different sites deciding they wanted to solve that two different ways.23:31
jeblairmordred: but hey, you're the only one with a use case for this so far.23:31
jeblairmordred: so if you and clarkb only want to push one of them that's fine.23:32
*** weshay has quit IRC23:32
mordredjeblair: that we know about - apparently jaypipes it making an army of infra cloners out there ...23:32
fungisome behind a very great firewall indeed23:32
fungi(great in the sense of its size, though from what i've seen not so great in terms of comprehensivity)23:33
jeblairclarkb's change has a very small but non-zero chance of being applied independently of your requirement.  i don't foresee infra's need for mordred's patch, atm, but i certainly did when i wrote it the first time so who knows.  :)23:33
*** bhuvan has quit IRC23:33
fungiclarkb: the cluster view in bigdesk, while amusing, does suggest that es01 is now fairly shardy (though not terrifically). does this mean downing es6 is safe at this point?23:33
fungiand i agree es-head provides a better functional view of that23:34
*** zul has joined #openstack-infra23:34
clarkbfungi: its technically safe to down es6 at any point beacuse we can lose one node with our redundancy23:34
*** dims has quit IRC23:34
clarkbfungi: but by waiting longer and allowing as much data to copy to es01 now we reduce the time period we are vulnerable to data loss due to losing a node23:35
*** bhuvan has joined #openstack-infra23:35
*** oubiwann_ has quit IRC23:35
fungiclarkb: how do i tell when it's copied all it wants to? when it has a shard from each logstash worker?23:35
clarkbfungi: no, when there are no remaining purple shards23:35
clarkbpurple means copying23:35
funginice. like in python where whitespace is significant, in elasticsearch colors seem to have significance23:36
* fungi will wait for the purple problem to subside23:36
jeblairman i wish i was on whatever you guys are on23:36
clarkbjeblair: its good stuff, made with 100% real java23:37
dstufftjava, not even once23:37
fungijeblair: it's like computers, but with pictures23:37
jeblairspeaking of colors, reload http://status.openstack.org/zuul/23:37
jeblairthat sort of brown line is waiting and total drawing on top of each other for the logstash queue23:38
fungijob queue!23:38
*** david-lyle has quit IRC23:38
clarkbjeblair: in that case I will clean up my existing change23:38
fungiwith my java goggles on, i think that sort-of-brown is called "brick"23:38
fungior maybe "rust"23:38
* fungi might need to get out the pantone catalog23:39
jeblairsdague, jog0: http://status.openstack.org/zuul/ shows the logstash queue now so you can see how backed up it is23:39
*** dizquierdo has quit IRC23:39
*** bhuvan has quit IRC23:39
*** bhuvan has joined #openstack-infra23:40
fungiclarkb: do we need to try https://review.openstack.org/76376 from a dev env when we're ready to bring up es02, or are you cool with it as-is?23:41
*** sandywalsh has quit IRC23:41
fungialso, i note only two "purple" shards in es-head... are we just waiting for those to clear, or does the cluster start new copies at a set rate so as to not swamp intercommunication channels between cluster members?23:43
clarkbit is rate limited23:43
clarkbthat change lgtm I don't think you need to test it on es0223:43
fungiokay, so when one of these finishes another shard may go from green to purple?23:44
clarkbso this can take a while :)23:44
*** bhuvan has quit IRC23:44
fungidoes the number of workers or cluster members allow us to predict the number of shards we should expect es01 to accumulate?23:44
clarkbfungi: so we could just add all of the nodes to the cluster first if you want to get over that hurdle, then remove old ones as things settle23:45
clarkbfungi: roughly. we have an index per day, 10 shards per index and 2 weeks of data. that is 140 shards. divide that by te number of cluster members to get a rough estimate23:45
clarkbin this case 2023:45
fungiit looks to me like each elasticsearch cluster member has at least one shard for each logstash worker, in come cases more than one23:46
*** prad_ has quit IRC23:46
fungier, some cases23:46
clarkbfungi: the shards don't map to logstash workers23:46
clarkbeach worker can index on any shard23:46
fungioh, wait, they map to dates23:46
clarkbsince we create a new index each day23:46
fungi"logstash-2014.02.15" wasn'y just a truncated worker name and date. it's per-day shards23:47
fungimore lightbulbs23:47
*** masayukig has quit IRC23:47
*** dims has joined #openstack-infra23:47
*** bhuvan has joined #openstack-infra23:47
clarkbfungi: jeblair pleia2 https://review.openstack.org/#/c/70007/1 was noop applied to zuul-dev and the files updated by the change were not modified on zuul-dev23:47
pmathewsI don't think versionlock is working on the live controllers as expected23:47
fungiso, given that all the new servers are built, and replication is slowish, i'm in favor of adding more new cluster members and then scaling off the old ones later. should i go particularly slowly with that?23:48
clarkbfungi: jeblair pleia2 happy with my approving that change?23:48
fungiclarkb: you bet23:48
clarkbfungi: that sounds fine with me, I would start each one and let it join the cluster before doing the next23:48
fungiclarkb: okay. in that case i'll approve 76376 and give it a chance to get in place first23:49
clarkbkeep in mind we can only delete one server at a time regardless of how many total servers we have23:49
fungii don't intend to delete any of the old ones until you're good with it23:50
clarkband wait for the cluster to recover before deleting the next23:50
fungithe old ones, and the outer gods23:50
openstackgerritA change was merged to openstack-infra/config: Correct elasticsearch config dependency ordering  https://review.openstack.org/7637623:51
*** mspreitz has joined #openstack-infra23:52
clarkbfungi: you ready for me to approve 70007? making sure I understand your response above. Do you think we shuold have jeblair weigh in first? I am always paranoid about firewall rule changes23:52
mspreitzHas anybody succeeded in doing a `get fetch` from an Ubuntu 13.10 (saucy) machine?23:53
fungiclarkb: yes, if you're satisfied with your testing on 70007, i'm cool with it23:53
clarkbpmathews: I don't have enough context to understand thsoe comments. wrong window or do I need to catch up on mail23:53
*** bhuvan_ has joined #openstack-infra23:53
anteayamspreitz: hi, can you share some context?23:53
clarkbfungi: approved23:53
*** sandywalsh has joined #openstack-infra23:53
*** bhuvan has quit IRC23:54
mspreitzanteaya: I am trying to cherry pick a patch, and the `git fetch` fails with a connection problem23:55
anteayamspreitz: would you be able to paste the output of the error and post the url of the paste in here?23:55
clarkbmspreitz: where are you fetching from and are you behind a firewall/proxy?23:55
openstackgerritA change was merged to openstack-infra/config: Update some deprecated Puppet variable references  https://review.openstack.org/7000723:55
fungimspreitz: i strongly suspect at least several people in this channel run ubuntu saucy (13.10?) and use the git fetch subcommand very often, but if you have a specific error message you wish to share (better pasted into http://paste.openstack.org/ with some additional contaxt and the url provided here), we're pretty good at interpreting git errors23:56
mspreitzSee http://paste.openstack.org/show/69602/23:56
*** Sukhdev has quit IRC23:56
fungisometimes running git is like throwing bones23:57
fungiand that error does indeed look like a broken transparent https proxy23:57
anteayamspreitz: do you have an account on review.openstack.org?23:57
*** ociuhandu has joined #openstack-infra23:57
*** lcostantino has quit IRC23:58
anteayassh key on it and you can upload to it?23:58
*** bhuvan has joined #openstack-infra23:58
mspreitzI copied my command line from Gerrit23:58
fungimspreitz: also, try http:// or git:// to see if it helps23:58
*** zehicle_at_dell has joined #openstack-infra23:58
*** ociuhandu has quit IRC23:58
*** flaper87 is now known as flaper87|afk23:58
mspreitzWow!  Same complaint when I try http23:59
*** bhuvan_ has quit IRC23:59
cyeohmspreitz: don't know if this might help or not: http://stackoverflow.com/questions/13524242/error-gnutls-handshake-failed-git-repository (the rebuild git answer looks like a lot of work)23:59
fungignutls should definitely not be complaining about malformed tls packets on http23:59
mspreitzBTW, you all know that there are some middle-boxes that abort SSH connections if either Key Exchange Init message is long, right?23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!