Tuesday, 2013-10-29

*** julim has joined #tripleo00:02
*** julim has quit IRC00:06
*** matsuhashi has joined #tripleo00:18
*** tzumainn has quit IRC00:26
*** CaptTofu has quit IRC00:33
*** CaptTofu has joined #tripleo00:34
*** nosnos has joined #tripleo00:34
*** CaptTofu has quit IRC00:38
*** UtahDave has joined #tripleo00:41
*** UtahDave has quit IRC00:56
*** edmund has joined #tripleo01:08
*** krotscheck has quit IRC01:17
*** CaptTofu has joined #tripleo01:23
*** CaptTofu has quit IRC01:23
*** CaptTofu has joined #tripleo01:24
*** CaptTofu has quit IRC01:24
*** CaptTofu has joined #tripleo01:25
*** anteaya has quit IRC01:43
*** noslzzp has quit IRC01:48
*** cwolferh has quit IRC02:30
*** cwolferh has joined #tripleo02:46
*** MarkAtwood has quit IRC03:07
*** matsuhashi has quit IRC03:20
*** matsuhashi has joined #tripleo03:21
*** matsuhashi has quit IRC03:25
*** edmund has quit IRC03:37
*** cwolferh has quit IRC03:46
*** spzala has quit IRC03:51
*** jcoufal has joined #tripleo04:06
*** ccrouch is now known as ccrouch-afk04:07
*** akuznetsov has joined #tripleo04:24
*** openstackgerrit has quit IRC04:36
*** openstackgerrit has joined #tripleo04:36
*** matsuhashi has joined #tripleo04:42
*** MarkAtwood has joined #tripleo04:43
*** MarkAtwood has left #tripleo04:49
*** rpodolyaka1 has joined #tripleo04:49
*** rpodolyaka1 has quit IRC04:52
*** rpodolyaka1 has joined #tripleo04:53
tseronglifeless, FYI, i'll be in HK for the openstack summit, along with at least half a dozen other SUSE crew05:00
lifelesstserong: sweet05:02
lifelesstserong: hey, there are folk in HP asking after suse support in dib05:02
lifelesstserong: I said you had been doing some stuff but there wasn't a 'just works' element yet.05:02
*** matsuhashi has quit IRC05:02
tserongyeah, i'05:02
lifelesstserong: and that I would enquire ;)05:02
tserongi've got a partial implementation of an opensuse element05:02
tserongwhich i was hacking on week before last05:02
*** matsuhashi has joined #tripleo05:03
tserongit did actually build something, but that thing didn't boot :-/05:03
tserongi didn't commit it anywhere public yet though (it's a bit more WIPy than I'd like)05:04
tserongi was hoping to squeeze in a bit of cleanup on it this week before going to HK05:04
tserongbut i've also got a pile of other stuff to get through, so i'll have to see how i go :-/05:05
devanandalifeless: hi! #'s question for one o fmy slides -- how often per day is tripleo deploying a new cloud?05:06
*** matsuhashi has quit IRC05:08
rpodolyaka1morning guys05:08
*** coolsvap has joined #tripleo05:09
rpodolyaka1I believe, it takes about an hour to build and deploy overcloud, so 24 times per day05:10
rpodolyaka1maybe, a few clouds more :)05:11
lifelessdevananda: yeah05:23
lifelesstserong: ok05:23
lifelesstserong: let me know if you need input05:23
*** rushiagr has joined #tripleo05:26
lifelesstripleo-cd status is suspiciously quiet05:27
lifeless# service tripleo-cd status05:27
lifelesstripleo-cd stop/waiting05:27
tseronglifeless, thanks, will do05:27
lifelesserror: Couldn't resolve host 'git.openstack.org' while accessing https://git.openstack.org/cgit/openstack/diskimage-builder/info/refs05:28
lifelessfatal: HTTP request failed05:28
lifelessDNS again05:28
lifelesswho wants to be on mellanox firmware EPIC FAIL again ?05:28
lifelessseems reliable in a while-true loop05:28
lifelessso, starting it up again05:29
lifelesstserong: I don't know if Hari will be in HK, if he is I'll introduce you05:30
lifelesstserong: since his team is responsible for getting a bunch of vendor images together05:31
tserongok, cool05:31
tserongthe openSUSE image I've been basing off, I generated on build.opensuse.org in my home project05:32
tserongbut (without having spoken to anyone else about this), I don't see any reason we couldn't publish more official-ish05:32
tserongimages in the Cloud:OpenStack project on OBS (somewhere in http://download.opensuse.org/repositories/Cloud:/OpenStack:/)05:33
tserongthen DIB would grab one of those, much the same as it does for ubuntu, and do its magic on top05:33
lifelessthat would be idea :)05:36
rushiagrhi folks05:44
rushiagrI am trying the dev_test setup for tripleo05:44
rushiagrI thought while it is downloading packages, I'll ask a question05:45
rushiagrQ: Is my laptop (8gb ram, i7 processor) enough for the devtest environment? I see that it mentions that it will create 5 VMs!05:45
rpodolyaka1rushiagr: I think, you are a bit short of RAM and might possibly end up using swap05:46
rushiagrrpodolyaka1: can I make it just 3 VMs by changing some config? one for seed, one undercloud and the other overcloud?05:47
rpodolyaka1rushiagr: though you might try to lower the mem requirements for created VMs (NODE_MEM env variable)05:47
lifelessrushiagr: it's four VM's05:47
lifelessrushiagr: the 5th is a qemu instance within the overcloud compute node05:48
rushiagr4 + 1 seed, isnt it?05:48
lifelessrushiagr: so it has no direct footprint on the host.05:48
lifelessseed, undercloud, overcloud control plane, overcloud compute.05:48
rushiagrlifeless: ah, okay05:48
lifelessalso, once the undercloud is running, you can 'save' the seed via virt-manager05:48
lifelessif you do some routing stuff, I shouldn't have mentioned that right now; it's in the advanced fiddling category ;)05:49
rushiagrlifeless: yeah :)05:49
rushiagrlifeless: that made me feel I know so little :/05:50
* rushiagr hopes NODE_MEM=512 would be enough05:52
lifelessopenstack won't work with tht05:53
lifelessopenstack services are fat memory hogs05:53
rushiagrowh, 1G a good compromise?05:53
rpodolyaka1rushiagr: 1.5-1.7 could work, I think05:53
lifelessI have a running undercloud here05:53
lifelessKiB Mem:  98997760 total, 41625076 used, 57372684 free,  2194164 buffers05:53
rpodolyaka1but 2GB is known to work great05:54
rushiagrI have only 8 G ram, so I'm trying to optimise it as much as I can05:54
lifelessrushiagr: yeah :(05:54
lifelessrushiagr: we've had deploy failures due to the node being slow due to swapping with 1G nodes05:54
lifelessrushiagr: we then tried 1.5G - same05:54
lifelessrushiagr: be sure to use a 32-bit build, 64-bit is a waste for this scenario05:55
lifelessrushiagr: 1.5G may work w/32bit05:55
rushiagrcant I configure the undercloud nodes with a different memory than the overcloud, seed nodes05:55
rushiagrlifeless: good advice, will follow that05:55
lifelessrushiagr: in theory yes, but we haven't put that sort of complexity into the dev harness05:56
rushiagrlifeless: okay, that's exactly the ans I was looking for05:56
lifelessso yeah this machine05:57
lifeless400M for nova compute, 245M resident05:57
lifeless400M for conductor, 230M resident05:57
lifeless2.5G for rabbitmq, 210M resident05:57
lifeless4.3G for mysqld, 200M resident05:57
lifeless200M for nova-api, 140M resident05:58
lifelessmeh, let me copy-paste05:58
*** tzumainn has joined #tripleo05:58
rushiagrwell, I'll do with 1.7G, and let you people know05:58
rushiagrwhat do you mean by resident?05:59
lifelessactual RAM used05:59
lifelessthe virt column includes memory mapped libraries and files05:59
lifelessand swapped out pages05:59
lifelessand ram05:59
lifelessso virt-res = stuff on disk that this program may access, res = currently in ram05:59
rushiagrahh, okay06:00
lifelessit's common to have lots of shared pages amongst programs using the same libraries06:00
lifelessso you can't add the virt column up to get a sane total06:00
lifelessbut you can add the res column up06:00
rushiagrI misunderstood that resident = disk-resident. Completely opposite06:00
rushiagrlifeless: yeah, good to know06:01
lifelessanyhow, point is that I can visually see 1.9G of memory resident on this 64 bit node06:01
rushiagrhmmm.. I'll start with 1.7G RAM and 32bit machines..06:02
lifelessnote that deploying bare metal images involves about 2.5G of IO too06:03
lifelessreading the qcow out of glance06:03
lifelesswriting it to a temp file06:03
lifelessconverting it to a raw file06:03
lifelessthen it gets copied to the target via iscsi06:03
rushiagrthat should be okay i guess, I have 1TB HDD06:05
*** akuznetsov has quit IRC06:05
lifelessrushiagr: right, point was that that goes into the pagecache overhead of the relevant VM06:06
lifelesse.g. the seed06:06
rushiagrah okay06:07
*** jcoufal has quit IRC06:28
*** jeckersb_gone has quit IRC06:28
*** GheRivero has quit IRC06:28
*** akuznetsov has joined #tripleo06:30
*** GheRivero has joined #tripleo06:30
*** jeckersb_gone has joined #tripleo06:31
*** rpodolyaka1 has quit IRC06:37
*** jcoufal has joined #tripleo06:37
*** akuznetsov has quit IRC06:40
*** cd-undercloud has joined #tripleo06:40
cd-undercloud************** overcloud complete status=1 ************06:40
*** cd-undercloud has quit IRC06:40
lifelesstimed out waiting for the overcloud stack to be ready06:41
lifelessI bet one or more of the new machines I registered is fail.06:42
lifelesshmmm, both active06:42
lifelessI can ping both06:42
lifelessI can ssh into the compute node06:43
lifelessI can ssh into the not compute node06:43
lifelessit's vlan hasn't come up06:43
lifeless[2013-10-29 02:20:44,975] (os-refresh-config) [ERROR] Aborting...06:44
lifeless2013-10-29 02:20:44.979 1683 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 106:44
lifeless2013-10-29 02:20:44.979 1683 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.06:44
lifelesslast thing logged at 20 paste06:44
lifeless20 past06:44
lifelessbut it's running:06:44
lifelessservice os-collect-config status06:44
lifelessos-collect-config start/running, process 168306:44
lifelesscat /var/lib/heat-cfntools/cfn-metadata-server06:44
lifelessso that looks ok06:45
lifelessand it can ping it06:45
lifelessheat-local data is there06:45
lifelessI'm confused why this didn't recover06:45
lifelessthe clock is out by 20m06:46
lifelessI wonder if that breaks heat auth06:46
SpamapSlifeless: some orc script being silent?06:46
SpamapSlifeless: heat auth is not time based06:46
lifeless2013-10-29 02:23:05.698 1683 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 106:46
lifeless2013-10-29 02:23:05.698 1683 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.06:46
lifelessSpamapS: keystone tokens06:46
lifelessSpamapS: ?06:46
SpamapSlifeless: no it uses ec2 keypairs06:46
lifelesslog into and have a poke06:46
lifelessSpamapS: ^06:47
lifelessSpamapS: it's written a seemingly valid /etc/os-collect-config.conf06:47
lifelesspath = notcomputeConfig.Metadata06:47
SpamapSlifeless: key I added isn't working06:47
lifelessSpamapS: thats too opaque for me06:47
lifelessSpamapS: other than 'blame SpamapS'06:48
SpamapSspamaps-clint-HP is not working to login to cd-undercloud.tripleo.org06:48
lifelessSpamapS: oh, cause thats manually synced and I had not06:48
SpamapSI was hoping you'd say that :)06:48
*** tzumainn has quit IRC06:49
lifelessok try cd-undercloud again ?06:50
lifelessI'll do the bastion in a sec06:51
*** rdopieralski has joined #tripleo06:52
lifelessnova updated06:53
lifelessnow the bastion06:53
SpamapSlifeless: looks like something was definitely wonky.. cfn data source is working now06:53
lifelessSpamapS: what fixed it?06:53
*** jtomasek has joined #tripleo06:54
SpamapSas in, something went slow06:54
lifelessI didn't reset the clock06:54
lifelessSpamapS: try the bastion host ?06:54
lifelessSpamapS: hang on06:54
lifelessSpamapS: you sshed into the undercloud06:54
SpamapSlifeless: aye06:54
lifelessSpamapS: how can you tell the cfn data source is working? You're on the wrong machien...06:54
SpamapSlifeless: I'm on
SpamapSI bounced06:55
lifeless2013-10-29 02:31:30.601 1683 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 106:55
lifeless2013-10-29 02:31:30.602 1683 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.06:55
lifelessstill looks broken to me06:55
SpamapSdo we have two's?06:55
lifelessSpamapS: how did you bounce? Your key isn't on
lifelessSpamapS: whats the hostname you see ?06:56
lifelessthats the undercloud06:56
lifelessnote the letters u-n-d-e-r06:56
SpamapSoh did not catch the $? == 106:56
SpamapSI didn't note any letters06:57
lifelessstack name in the hostname is super useful06:57
SpamapSyes it is06:57
lifelessI've now added your key to the 174 host06:57
lifelessso you should be able to log in there06:57
lifelessand to the bastion which has ilo access setup etc06:57
lifelessdone dan's too while there06:57
lifelessexcept bastion, we dont' have non-HP on there yet.06:57
lifelessleaving it with you for a little (insomniac!) while I sooth Cynthia06:59
SpamapS| notcompute          | 37443 | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-85b74461-de49-4fed-8601-62ae42d4b5a4) | CREATE_FAILED      | 2013-10-29T05:58:36Z |07:00
SpamapSlifeless: looks like novaclient exploded and heat stopped stack creation.07:01
*** coolsvap has quit IRC07:04
SpamapSlifeless: no visible logs of that request ID anywhere on undercloud..07:05
SpamapSlifeless: I recommend starting tripleo-cd again. I'm off to bed.07:06
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381507:20
rdopieralskilifeless: do you have a moment to talk about this patch ^^ ?07:20
*** jtomasek has quit IRC07:25
lifelessSpamapS: ok, thanks07:28
lifelessSpamapS: since it's late, I'm going to stab the firmware thing then restart07:28
lifelessrdopieralski: sure07:30
*** marios has quit IRC07:31
*** marios has joined #tripleo07:31
rdopieralskilifeless: do you think I should add werkzeug to tuskar-ui's and horizon's dependencies just for that function, or should I rather rewrite it on my own using the same principles as this one?07:33
rdopieralskilifeless: note, that werkzeug is a http library, unrelated to caching07:34
rdopieralskilifeless: I can write my own code that does that -- actually, if you look at the previous patches, I did -- but this trick is quite smart, so I figured it would be nice to give credit07:35
lifelessrdopieralski: there are many such things out there07:39
lifelessrdopieralski: I don't think we want to add yet another07:39
lifelessrdopieralski: reuse is good.07:39
*** bauzas has joined #tripleo07:39
rdopieralskilifeless: you didn't answer07:41
lifelessrdopieralski: I did in the review07:42
* rdopieralski looks07:42
lifelessrdopieralski: I don't know if adding werkzeug is the right thing07:43
lifelessrdopieralski: but copying the code is the wrong thing07:43
lifelessrdopieralski: writing a new caching decorator is the wrong thing07:43
rdopieralski"but it's work it" <-- what does that mean?07:43
lifelessthats me failing to type. Sorry!07:43
lifelessbut, lets work through it07:44
lifelessis what I should have written.07:44
rdopieralskiok, thanks, I will do something more productive then and maybe someday when we have more time for politics we will talk about including werkzeug in the dependencies07:44
lifelessI don't expect it to be very political07:45
lifelessbut it does require changing global-requirements07:45
lifelessand consensus on the new dep07:45
lifelessIf you don't have time to do that now, I can understand that, but please don't be put off by the fact we're part of a larger ecosystem ;)07:45
rdopieralskican't help it07:46
rdopieralskiI will just find some other part of code that I can somehow improve07:46
rdopieralskiand hopefully it won't attract too much attention07:46
lifelessrdopieralski: I'm lost, I don't understand the objection to putting a one line patch into global-requirements and pinging a couple of folk about it07:47
lifelessrdopieralski: receiving attention in this project is generally a good thing; 1500 devs...07:47
rdopieralskilifeless: no objection07:47
rdopieralskiwell, I don't like attention, I like fixing stuff07:48
lifelessok, so lets fix this?07:48
rdopieralskiI can't think of a commit message that would justify pulling in werkzeug just for one of its internal utility function.07:50
lifelessrdopieralski: how big is werkzeug ?07:51
*** jprovazn has joined #tripleo07:51
lifelessrdopieralski: AIUI it's a library, it's *meant* to be reused.07:51
rdopieralskiseveral thousand lines, afair07:51
lifelessrpodolyaka: so the commit message should just be 'add werkzeug'07:51
lifelessseveral thousand lines - tiny.07:51
lifelessnot 100 lines tiny, but still not big.07:51
rdopieralskiyeah, it's a small http header parsing and web utilities library07:51
lifelessso lets reuse it.07:52
rdopieralskipretty much all that is in django.core07:52
rdopieralskiok, let me try it07:52
lifelessit'll take 5 minutes to prep and put forward07:52
*** martyntaylor has joined #tripleo07:52
*** martyntaylor has left #tripleo07:52
lifelessworst that can happen is soemone has a good reason not to use it07:52
lifelessif that happens, we've lost 5 minutes, oh well, we can move on.07:52
lifelesslet me just check the license is sympatico for OpenStack07:53
lifelessif folk complain about it's size, I'll poke Armin about splitting it apart07:54
lifelessit is a decently sized tarball, I'm not sure why07:54
lifelessI want a round tuit to run https://pypi.python.org/pypi/radon against nova ;)07:55
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381507:55
rdopieralskilifeless: the packaging people will kill me :P07:56
rdopieralskilifeless: so what's the process now?07:57
rdopieralskilifeless: same as with any change?07:57
*** coolsvap has joined #tripleo07:57
lifelessrdopieralski: make a change to the openstack-infra/global-requirements repo adding it07:57
lifelessrdopieralski: add a reviewer (me, so I know the url to help discuss it) and add a review (the horizon PTL)07:58
lifelessrdopieralski: wait for reviews; if it gets in then the tuskar-ui change is a no-brainer07:58
*** rushiagr has quit IRC08:02
rdopieralskiI can't see global-requirements at https://github.com/openstack-infra08:04
*** jcoufal_ has joined #tripleo08:05
rpodolyakait's here git://git.openstack.org/openstack/requirements08:06
rpodolyakaor here https://github.com/openstack/requirements08:07
*** lsmola has joined #tripleo08:07
*** jcoufal has quit IRC08:08
rpodolyakardopieralski: sorry for pointing you to the 'long way' instead of short one :(08:08
rdopieralskirpodolyaka: what do you mean?08:09
rpodolyakardopieralski: reusing of Werkzeug decorator instead yours one08:09
rdopieralskinah, I should have done it from the beginning08:09
rdopieralskiI'm just too lazy when it comes to interacting with people08:09
rpodolyakabut it's community, it's open source, you have to talk to people here :)08:10
rdopieralskithank you for the help with it guys, by the way, I really appreciate it08:10
rpodolyakanp, we are glad to help08:10
rpodolyakathe more people we had, the more task we can finish ;)08:11
* rpodolyaka is typing too bad today08:11
*** ifarkas has joined #tripleo08:11
*** marun has joined #tripleo08:13
lifelessrpodolyaka: +108:15
rdopieralskithis is a little embarassing, but who is horizon's ptl?08:16
rdopieralskiGabriel Hurley?08:17
*** akuznetsov has joined #tripleo08:17
rdopieralskiit's hard to find on the wiki :)08:17
rpodolyakaI found the letter in the mailbox - " September 2013 PTL elections - Final results" :)08:18
rdopieralskinow, I suppose the best way to have my patches to horizon reviewed is to come to the horzion meeting and mention them?08:20
lifelessin theory no08:20
lifelessin practice, perhaps :)08:20
rdopieralskior at least mention the blueprints08:20
ogelbukhrdopieralski: you could use https://review.openstack.org/#/admin/groups/08:20
ogelbukhjust look for -ptl groups08:21
rdopieralskioh, nice, thanks!08:21
ogelbukhrdopieralski: np )08:21
*** marun has quit IRC08:27
*** nosnos_ has joined #tripleo08:29
*** nosnos has quit IRC08:30
*** marun has joined #tripleo08:33
*** marun has quit IRC08:33
*** cd-undercloud has joined #tripleo08:38
cd-undercloud************** overcloud complete status=1 ************08:38
*** cd-undercloud has quit IRC08:38
*** jistr has joined #tripleo08:57
*** nosnos has joined #tripleo09:02
*** nosnos_ has quit IRC09:03
*** jtomasek has joined #tripleo09:03
*** marun has joined #tripleo09:03
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381509:05
*** derekh has joined #tripleo09:09
*** markmc has joined #tripleo09:09
ogelbukhNg: good morning09:16
*** rushiagr2 has joined #tripleo09:17
Nghey hey09:21
*** akuznetsov has quit IRC09:21
*** lucasagomes has joined #tripleo09:24
*** akuznetsov has joined #tripleo09:25
*** rushiagr2 has quit IRC09:27
*** akuznetsov has quit IRC09:30
*** cd-undercloud has joined #tripleo09:33
cd-undercloud************** overcloud complete status=0 ************09:33
*** cd-undercloud has quit IRC09:33
*** rushiagr3 has joined #tripleo09:34
*** jcoufal_ is now known as jcoufal09:40
*** epim has joined #tripleo09:52
*** epim has quit IRC09:56
derekhlifeless: should there be a triple section on https://wiki.openstack.org/wiki/Summit/Icehouse/Etherpads for us to create etherpads under ?10:00
lifelessderekh: yes!10:00
derekhlifeless: k, I can create it if you want10:01
lifelessderekh: fantastic :)10:03
*** akuznetsov has joined #tripleo10:06
derekhlifeless: done https://wiki.openstack.org/wiki/Summit/Icehouse/Etherpads#Tuesday_610:09
lifelessthanks ;)10:09
* lifeless is a big believer in delegation10:10
*** nosnos has quit IRC10:31
*** nosnos has joined #tripleo10:32
*** nosnos has quit IRC10:36
*** cd-undercloud has joined #tripleo10:42
cd-undercloud************** overcloud complete status=1 ************10:42
*** cd-undercloud has quit IRC10:42
lifelessderekh: now mightbe the time to exercise your sysadmin access :) - see why it's failing10:43
derekhlifeless: k, will take a look around10:43
lifelessI reset the firmware a couple hours back, and thats usually stable for days10:43
*** ehelms-afk is now known as ehelms10:43
lifelesslast three runs were fail, pass, fail10:43
derekh[2013/10/29 06:42:19 AM] [ERROR] key db-password does not exist in ['/var/run/os-collect-config/heat_local.json', '/var/run/os-collect-config/ec2.json']10:50
derekh+ db_pass=10:50
derekhstill digging into whats caused that10:51
lifelessderekh: ok so that should happen once on boot10:51
lifelessthen it gets the real config from cfn10:51
lifelessderekh: one reason that can keep occuring is if heat thinks the instance deploy failed10:52
lifelessderekh: e.g. heat event-list overcloud10:52
lifeless| notcompute          | 37620 | CREATE aborted                                                                                                                                                        | CREATE_FAILED      | 2013-10-29T09:58:46Z10:52
lifeless| NovaCompute0        | 37619 | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f4603762-9da9-4f79-b7d8-8f53fc1c5875) | CREATE_FAILED      | 2013-10-29T09:58:46Z10:52
lifelessderekh: so, question is why did nova barf10:53
*** rushiagr3 has quit IRC10:55
derekhlifeless: 2013-10-29 09:58:45,790.790 19625 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: Maximum attempts reached10:56
lifelessderekh: ahha10:56
lifelessderekh: stabbity stab stab stab10:56
derekhlifeless: hmm, looks like this is relevant in neutron-server.log11:00
derekh2013-10-29 09:57:06.448 19414 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'18b96b8d-c989-4820-b330-0c0f7c19c8d5'], 'name': u'ctlplane', 'provider:physical_network': u'ctlplane', 'admin_state_up': True, 'tenant_id': u'4956c533154c476799c688eda7ed65ab', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': u'e85b44c7-1136-4217-954e-cdf0acdddfe1', 'provider:segmentation_id':11:00
derekh None}11:00
derekhhappened at the same time as the heat fail event11:01
derekh2013-10-29 09:57:50.587 18834 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']. Exception:11:04
rpodolyakais dhcp agent running?11:05
lifelessderekh: what exception ?11:05
*** che-arne has joined #tripleo11:05
derekhOct 29 09:57:41 undercloud-notcompute-jws3awlsb2kh kernel: [1826679.931936] Buffer I/O error on device loop0p1, logical block 226496011:05
lifelessderekh: I think thats the stuff-stuck-on-unmount thing showing up lazily11:06
lifelesswe have tonnnnes of that in the logs11:06
lifelessnot great, but not correlated with failures over the last couple weeks11:06
lifeless00:04 < derekh> 2013-10-29 09:57:50.587 18834 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute11:07
lifeless                ['ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']. Exception:11:07
lifelesswhat exception followed it ?11:07
derekhjust the exit code from ovs-vsctl http://paste.openstack.org/show/50117/11:08
rpodolyakahmm, does br-int exist at all?|11:08
lifelessrpodolyaka: should do, this is the undercloud node which has been doing all the deploys :)11:09
derekhrpodolyaka: yup and that command is working for me now11:10
lifelesslets capture this in a bug at minimum11:10
*** jtomasek has quit IRC11:10
lifelessI forsee someone climbing through ovs-vsctl code to see what causes this :)11:10
derekhok, will gather up the details11:11
rpodolyakalifeless: ah, I forgot for a minute that we didn't deploy undercloud :)11:11
*** jtomasek has joined #tripleo11:14
lifelessok, past-midnight gnight; see you all at the meeting ;)11:24
derekhlifeless: night ^ bug filed, will see if I can find anything more out11:25
rpodolyakainteresting, my undercloud fails to get CFN metadata11:34
rpodolyakaheat-admin@undercloud-notcompute-gvqnorbsopna:/var/run/os-collect-config$ sudo os-collect-config11:34
rpodolyaka2013-10-29 11:33:42.425 31715 WARNING os_collect_config.cfn [-] 400 Client Error: InvalidParameterValue11:34
rpodolyaka2013-10-29 11:33:42.425 31715 WARNING os-collect-config [-] Source [cfn] Unavailable.11:34
rpodolyakaheh, | notcomputeConfig | 9  | TypeError: Index to "Fn::Select" should be a string: "Fn::Select": ["4", ["str1", "str2"]] | CREATE_FAILED      | 2013-10-29T11:10:52Z11:37
rpodolyakathe funny part is that corresponding commit to heat has been merged recently (https://review.openstack.org/#/c/54063/3/heat/engine/template.py)11:41
* rpodolyaka investigating11:41
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Check of existence of directory 'lost+found'  https://review.openstack.org/5431111:44
rpodolyakaaha, and we are using int values in Fn::Select calls in our templates11:44
*** jtomasek has quit IRC11:44
rpodolyakaso this must be the reason11:44
derekhrpodolyaka: I just deployed a fresh devtest this morning and didn't get this error11:45
rpodolyakaderekh: could you please check if you have commit 9df15c134261f6b0675d5039562b3a893e879e44 in heat git repo?11:45
derekhrpodolyaka: nope don't have it, looks like DIB_OFFLINE=1 so it mustn't have gotten pulled down11:48
rpodolyakaderekh: aha, btw does "heat event-show overcloud" in tripleo-cd cloud show something similar to this error?11:49
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381511:49
*** akuznetsov has quit IRC11:49
rpodolyakaderekh: but we don't upgrade heat in undercloud, so it should be ok11:49
rpodolyakaso it's a bug, but it should not affect tripleo-cd11:50
*** cd-undercloud has joined #tripleo11:51
cd-undercloud************** overcloud complete status=1 ************11:51
*** cd-undercloud has quit IRC11:51
*** markmc has quit IRC11:51
derekhrpodolyaka: agreed, cd error looks different http://paste.openstack.org/show/50119/11:51
rpodolyakaok, so Heat docs say that selector can be an integer value (http://docs.openstack.org/developer/heat/template_guide/functions.html#fn-select), going to file/fix a bug in Heat then11:58
*** akuznetsov has joined #tripleo12:05
*** rushiagr has joined #tripleo12:05
*** ccrouch-afk is now known as ccrouch12:09
*** akuznetsov has quit IRC12:11
*** akuznetsov has joined #tripleo12:21
*** jdob has joined #tripleo12:23
*** john-n-seattle2 has quit IRC12:26
*** lucasagomes is now known as lucas-hungry12:27
rushiagris the devtest setup need to be done as root?12:40
slaglerushiagr: no, it uses sudo where it needs to12:41
*** dprince has joined #tripleo12:50
rushiagrslagle: I remember that the command at point number 14 of this page http://docs.openstack.org/developer/tripleo-incubator/devtest.html gave me some trouble12:51
rushiagrslagle: the output gave me a 'permission denied' somewhere in between12:52
slaglerushiagr: did it still build an image successfully?12:52
rushiagrslagle: so I ran it via sudo (and probably some other commands too), which now is giving a 'permission denied' on the execution of 'devtest-seed-vm' line on devtest_seed page12:53
rushiagrslagle: yes, it built it successfully12:54
slaglerushiagr: you shouldn't have to add a 'sudo' anywhere. the called scripts themsevles use sudo where it's needed12:55
rushiagrhmm.. I'll do everything all over again12:56
rushiagrslagle: thanks12:56
slaglerushiagr: it's possible there could be one missing.  might be worth filing a bug. or if you have a log you could paste, that could help too12:56
rushiagrI think my usage of 'sudo' while building images might have given a 'permission denied' at later stages. I'll try again and keep this channel posted if I find issues still12:57
*** jdob has quit IRC12:57
*** jdob has joined #tripleo12:57
*** akuznetsov has quit IRC12:58
*** cd-undercloud has joined #tripleo12:59
cd-undercloud************** overcloud complete status=1 ************12:59
*** cd-undercloud has quit IRC12:59
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-heat-templates: Use string selector values in Fn::Select calls  https://review.openstack.org/5432212:59
rushiagrI am hoping running 'cleanup-env' wont clean up my downloaded images. Am I correct? I havent set up proxies yet, and dont want that 200 something megs image to be downloaded13:00
rpodolyakarushiagr: it won't, it just deletes created VMs13:01
rushiagrrpodolyaka: thanks13:01
rpodolyakarushiagr: but you'd better set up an http proxy anyway :)13:02
rushiagrrpodolyaka: looks like I should. I wanted to sprint through the devtest setup, but I am realising going slowly but steadily will give me lesser trouble13:03
rpodolyakarushiagr: btw, one of the last commit to Heat breaks our devtest story13:05
rpodolyakaI proposed two different fixes, waiting on input from Heat guys13:06
rushiagrrpodolyaka: oh :(13:07
rpodolyakathough it's easy to overcome13:07
rushiagrrpodolyaka: nice to know. I think changing the template files temporarily would fix this problem (whenever it will occur for me)13:07
rushiagrrpodolyaka: yeah13:07
rushiagrany help on this? except wiping off these files and downloading the images again?13:09
*** julim has joined #tripleo13:09
*** anteaya has joined #tripleo13:10
rpodolyakarushiagr: try "sudo chown your_user_name:your_user_name ~/.cache/image-create/" first13:11
rpodolyakaoh, I forgot -R13:11
rpodolyakato make it recursive13:11
rushiagrokay, trying13:11
*** jdob has quit IRC13:11
*** jdob has joined #tripleo13:12
*** jeckersb_gone is now known as jeckersb13:12
rushiagrrpodolyaka: that worked! it is now downloading packages13:13
*** morazi has joined #tripleo13:14
*** akuznetsov has joined #tripleo13:14
*** tzumainn has joined #tripleo13:17
* rushiagr just asked his colleague and got his groups and permissions myths busted13:17
rushiagrI knew only two things till today -- use a 'sudo' or do a recursive 'chmod 777' :D13:18
*** john-n-seattle has joined #tripleo13:27
*** jayg|g0n3 is now known as jayg13:27
dkehnmorning all13:31
*** akuznetsov has quit IRC13:33
*** akuznetsov has joined #tripleo13:35
rushiagrdkehn: good morning :)13:42
rushiagrrpodolyaka: is it okay if I set up squid proxy after I create the seed VM? the devtest_seed thingy is still downloading packages13:47
rpodolyakarushiagr: yep, packages will also be downloaded when building images for undercloud/overcloud/user VMs13:48
rushiagrrpodolyaka: okay. Will the packages downloaded now (for seed VM) be cached for these VMs? or only after setting up the squid proxy will they be cached?13:50
rushiagrI doubt this will happen. But if you can confirm, I can stop the package downloads for seed VM right now13:52
*** coolsvap has quit IRC13:54
rpodolyakarushiagr: nope, it will take effect only when you set http_proxy env variable and build a new image13:58
rpodolyakarushiagr: though I can see fedora packages being cached in .cache/image-create/yum/. Not sure, where we set up this manual caching13:59
*** lucas-hungry is now known as lucasagomes13:59
rushiagrrpodolyaka: okay. Thanks14:00
rpodolyakarushiagr: but I can't see any *.deb files in image-create and subdir14:00
rpodolyakaso http proxy is a universal solution14:01
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Check existence of directory 'lost+found' before attempting to remove it  https://review.openstack.org/5431114:05
*** noslzzp has joined #tripleo14:08
*** cd-undercloud has joined #tripleo14:09
cd-undercloud************** overcloud complete status=1 ************14:09
*** cd-undercloud has quit IRC14:09
rushiagrrpodolyaka: yeah.. I am doing just that14:12
*** julim has quit IRC14:12
*** spzala has joined #tripleo14:15
*** akuznetsov has quit IRC14:17
dkehn/window/show 514:19
NobodyCamgood morning TripleO14:29
NobodyCammorning dkehn14:29
*** jergerber has joined #tripleo14:29
*** blamar has quit IRC14:30
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381514:31
*** akuznetsov has joined #tripleo14:34
*** blamar has joined #tripleo14:34
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Check existence of directory 'lost+found' before attempting to remove it  https://review.openstack.org/5431114:39
*** che-arne has quit IRC14:47
*** edmund has joined #tripleo14:53
*** edmund has quit IRC14:53
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Check existence of directory 'lost+found'  https://review.openstack.org/5431114:54
*** spzala has quit IRC14:55
*** coolsvap has joined #tripleo14:58
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Rename heat_watch_server_url to watch_server_url  https://review.openstack.org/5434714:59
openstackgerritDerek Higgins proposed a change to openstack/tripleo-heat-templates: Rename heat_watch_server_url to watch_server_url  https://review.openstack.org/5434814:59
*** openstackgerrit has quit IRC15:00
*** openstackgerrit has joined #tripleo15:01
openstackgerritA change was merged to openstack/diskimage-builder: Check existence of directory 'lost+found'  https://review.openstack.org/5431115:04
*** julim has joined #tripleo15:05
SpamapSrdopieralski: so is the end of the story with https://review.openstack.org/#/c/53859/ that django already has this cached_property decorator, we don't need to even import werkzeug?15:07
rdopieralskiSpamapS: yes, but the django's one is slightly broken, so it doesn't work well with mox, so I need to work around it in tuskar-ui tests15:10
SpamapSrdopieralski: _ahhh_15:10
SpamapSrdopieralski: and have we reported that bug to django? :)15:10
rdopieralskiSpamapS: but I'm getting there :)15:10
rdopieralskiSpamapS: no, because I just found it15:10
rdopieralskiI will report it, and propose a patch15:11
rdopieralskibut we still need the workaround for earlier versions15:11
SpamapSrdopieralski: yeah, totally.15:11
* rdopieralski goes back to recursive yak shaving15:11
*** john-n-seattle has quit IRC15:12
SpamapSrdopieralski: well thanks for staying the course. I thought copying the code was o-k but I figured you had a better reason than "don't want to change requirements" ;)15:12
rpodolyakaSpamapS: you might want to check this one https://bugs.launchpad.net/heat/+bug/1245863 :)15:13
SpamapSUgh yeah I totally missed the regression introduced there.15:15
SpamapSglaringly obvious when you know it is there. DOH.15:15
SpamapSwhy would anyone use "4" and not 4. :-P15:15
rpodolyakaphp guys? :)15:16
SpamapShah probably. :)15:17
SpamapSrpodolyaka: fix approved, now it just has to wind its way through the gate15:17
rpodolyakathanks :)15:18
SpamapSno thank **YOU**15:18
rpodolyakanp :)15:18
*** cd-undercloud has joined #tripleo15:21
cd-undercloud************** overcloud complete status=1 ************15:21
*** cd-undercloud has quit IRC15:21
SpamapSstill barfing on neutron rate limits or something15:22
derekhSpamapS: https://bugs.launchpad.net/tripleo/+bug/124585215:23
*** SpamapS changes topic to "CRITICAL BUG: 1245852 | Using OpenStack to deploy OpenStack; meetings Tuesday 1900 UTC in #openstack-meeting-alt"15:24
*** che-arne has joined #tripleo15:25
*** ruhe has joined #tripleo15:26
rpodolyakahmm, interesting. why do we have two interfaces for dhcp if there is only one neutron network?15:27
*** john-n-seattle has joined #tripleo15:27
rpodolyakaneutron doesn't clean up for himself?15:28
SpamapSrpodolyaka: that may be the problem15:29
rpodolyakaon the other hand, we don't configure ctlplane network in devtest_overcloud.sh15:30
rpodolyakaso it's not recreated on each deploy of overcloud15:30
*** akuznetsov has quit IRC15:30
* rpodolyaka is confused15:30
SpamapSrpodolyaka: networking makes my head explode. :-P15:31
SpamapSI almost always have to draw it.15:32
*** akuznetsov has joined #tripleo15:32
*** marun has quit IRC15:34
*** jprovazn has quit IRC15:40
*** dprince has quit IRC15:49
openstackgerritTomas Sedovic proposed a change to openstack/tripleo-image-elements: Add Horizon element  https://review.openstack.org/5091815:49
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381515:50
SpamapSderekh: so I imagine you're near EOD... are you still trying to work that bug or should I pick it up?15:53
*** cwolferh has joined #tripleo15:54
SpamapS08:54 < openstackgerrit> A change was merged to openstack/heat: Make Fn::Select accept an integer selector again   https://review.openstack.org/5432115:55
SpamapSrdopieralski: ^^15:55
rdopieralskiSpamapS: ?15:55
derekhSpamapS: I'll be here for another bit but havn't gone back since my last update to the bug, I'll poke around on it a bit more now for a bit but feel free to pick it up also15:55
*** akuznetsov has quit IRC15:56
SpamapSrdopieralski: doh sorry, too many r's15:57
SpamapSrpodolyaka: ^^15:57
SpamapSderekh: ok I will likely pick it up after the meeting.15:57
*** cwolferh has quit IRC16:00
derekhSpamapS: ok cool16:00
*** ruhe has quit IRC16:01
derekhIf I discover anythin else before that I'll add it to the bug16:01
*** julim has quit IRC16:02
*** spzala has joined #tripleo16:09
rpodolyakaSpamapS: cool ;)16:10
*** ruhe has joined #tripleo16:12
*** cwolferh has joined #tripleo16:12
*** ifarkas has quit IRC16:17
*** ruhe has quit IRC16:21
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the cached api calls into @cached_property  https://review.openstack.org/5381516:25
rushiagrseed vm up and running \\o//16:29
SpamapSrushiagr: ^516:30
rushiagrSpamapS: :)16:30
*** rdopieralski has quit IRC16:31
*** MarkAtwood has joined #tripleo16:31
*** edmund has joined #tripleo16:31
*** cd-undercloud has joined #tripleo16:32
cd-undercloud************** overcloud complete status=1 ************16:32
*** cd-undercloud has quit IRC16:32
*** akuznetsov has joined #tripleo16:36
*** jcoufal has quit IRC16:39
*** viktors has left #tripleo16:39
openstackgerritClint Byrum proposed a change to openstack/os-apply-config: Preserve symlinks when writing output  https://review.openstack.org/5437716:42
*** dprince has joined #tripleo16:48
*** ruhe has joined #tripleo16:51
*** rushiagr has quit IRC16:51
*** marun has joined #tripleo16:54
*** julim has joined #tripleo17:01
*** krotscheck has joined #tripleo17:05
*** ruhe has quit IRC17:13
*** UtahDave has joined #tripleo17:15
*** jcoufal has joined #tripleo17:15
Ngone week today it shall be TripleO Tuesday :D17:19
*** blamar has quit IRC17:19
*** blamar has joined #tripleo17:20
SpamapSNg: woooot17:25
slaglei wonder if tripleo's will deliver burgers to the conference17:26
openstackgerritJiri Stransky proposed a change to openstack/tripleo-incubator: Respect storage pool path in create-nodes script  https://review.openstack.org/5438517:27
Ngslagle: I am going to go to a TripleO's at some point, irregardless of their delivery abilities :)17:27
Ngand I would like to take many of us with me :)17:27
slagleoh i'm going for sure :)17:27
*** rpodolyaka1 has joined #tripleo17:35
*** derekh has quit IRC17:38
*** cd-undercloud has joined #tripleo17:42
cd-undercloud************** overcloud complete status=1 ************17:42
*** cd-undercloud has quit IRC17:42
*** jistr has quit IRC17:46
hewbroccaNg: what is TripleO's????17:50
Nghewbrocca: there's a Canadian burger chain that has some outlets in Hong Kong, called Triple-O's17:50
hewbroccawhen are we going???17:50
hewbroccaI wonder if they have t-shirts17:51
Ngthat would be pretty cool17:51
hewbroccaOK this is also funny:17:52
hewbroccathey appear to have t-shirts and hats, but only for employees?17:54
*** ifarkas has joined #tripleo17:55
*** julim_ has joined #tripleo18:00
*** krotscheck has quit IRC18:01
*** julim has quit IRC18:01
pleia2lifeless: I left some notes under iteration 2, could use some feedback particularly on the first grey comment18:08
* Ng dinners18:08
lifelesspleia2: k18:09
SpamapSlifeless: 'morning18:16
lifelessSpamapS: blergh :P. C woke at 053018:16
SpamapSlifeless: she got you a whole extra hour of daytime!18:16
*** krotscheck has joined #tripleo18:19
*** ehelms is now known as ehelms-afk18:19
lifelessmeeting time!19:01
*** anteaya has quit IRC19:03
*** jprovazn has joined #tripleo19:05
*** cd-undercloud has joined #tripleo19:05
cd-undercloud************** overcloud complete status=1 ************19:05
*** cd-undercloud has quit IRC19:05
*** anteaya has joined #tripleo19:08
*** lucasagomes has quit IRC19:10
*** che-arne has quit IRC19:13
lifelessNg: are you here for the meeting ?19:15
rpodolyaka1looks like somebody is going to wear a UTC watch :)19:15
*** coolsvap has quit IRC19:17
*** athomas has joined #tripleo19:19
*** martyntaylor has joined #tripleo19:19
openstackgerritA change was merged to openstack/tripleo-incubator: Add James Slagle to tripleo-cd-admins.  https://review.openstack.org/5358719:19
*** marun has quit IRC19:30
Nglifeless: I am now. we've come off daylight savings, so the meeting is now exactly co-incidental with baby-bedtime :/19:37
lifelessNg: ugh19:40
Ngyeah :/19:40
lifelessNg: perhaps you can paste notes from you for the meeting here in advance next meeting?19:40
Nglifeless: good plan19:40
dkehnNg: after running devtest.sh a few time, can one jsut go to the devtest_seed.sh and othere scripts easily19:45
dkehnNg: assuming executing all the exports in there19:46
Ngdkehn: yeah if your environment variables are all set up19:46
dkehnNg: trying to nail down the ml2 piece19:47
*** vipul is now known as vipul-away19:49
*** marios has quit IRC19:51
*** marios has joined #tripleo19:51
*** vipul-away is now known as vipul20:05
*** athomas has quit IRC20:05
*** rpodolyaka1 has quit IRC20:11
*** cd-undercloud has joined #tripleo20:28
cd-undercloud************** overcloud complete status=1 ************20:28
*** cd-undercloud has quit IRC20:28
*** jdob has quit IRC20:31
*** boris-42_ has joined #tripleo20:32
*** spzala has quit IRC20:35
*** boris-42 has quit IRC20:35
*** markmc has joined #tripleo20:35
*** rdopieralski has joined #tripleo20:38
*** julim_ has quit IRC20:47
*** jprovazn has quit IRC21:01
*** lsmola_ has joined #tripleo21:02
SpamapSlifeless: sorry was offline accidentally due to my phone not wanting to tether in saucy.21:03
*** ifarkas has quit IRC21:07
*** akuznetsov has quit IRC21:07
*** jergerber has quit IRC21:07
*** lsmola has quit IRC21:07
*** kiall has quit IRC21:07
*** Ng has quit IRC21:07
lifelessSpamapS: doh21:08
*** edmund has quit IRC21:12
*** akuznetsov has joined #tripleo21:14
*** jergerber has joined #tripleo21:14
*** kiall has joined #tripleo21:14
*** Ng has joined #tripleo21:14
*** hobana.freenode.net sets mode: +v Ng21:14
SpamapSlifeless: have you looked at the cd cloud again yet? Just about to pick that back up.21:17
lifelessSpamapS: no, still in meetings21:18
lifelessSpamapS: you saw the bug / ticket in trello?21:18
*** kiall has quit IRC21:24
*** kiall has joined #tripleo21:26
SpamapSlifeless: yeah, on it21:28
*** ehelms-afk is now known as ehelms21:28
openstackgerritJenkins proposed a change to openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/5442621:32
lifelessSpamapS: any thoughts?21:50
SpamapSlifeless: I was pondering rolling back to the last commit that worked.21:50
SpamapSlifeless: as in roll back all git trees21:50
lifelessSpamapS: huh21:51
lifelessSpamapS: why? we know its neutron shi**ing itself21:51
SpamapSlifeless: it may be due to changes elsewhere.21:51
SpamapShave not really dug into the logs though so :-P21:51
lifelessSpamapS: but we're not deploying to the undercloud21:51
lifelessSpamapS: abstractions are useful, they let us totally ignore things contained within ;)21:52
*** cd-undercloud has joined #tripleo21:52
cd-undercloud************** overcloud complete status=1 ************21:52
*** cd-undercloud has quit IRC21:52
SpamapSlifeless: I'm sorry I thought it was overcloud neutron having problems... which explains why I was getting nowhere finding the errors.21:52
lifelessSpamapS: heat reported nova fail; nova logs report neutron fail.21:52
lifelessSpamapS: take 5, have coffee :>21:53
* SpamapS has been trying to kill other distractions while digging in21:53
*** julim has joined #tripleo21:53
*** martyntaylor has left #tripleo22:08
*** morazi has quit IRC22:12
*** jayg is now known as jayg|g0n322:14
SpamapSlifeless: I'm wondering if we just have a stuck ovs port on br-int22:14
SpamapSOct 29 22:12:22|408190|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap8e1c39ef-b8 device failed: No such device22:14
SpamapSOct 29 22:12:33|408191|netdev|WARN|Dropped 45 log messages in last 12 seconds (most recently, 1 seconds ago) due to excessive rate22:14
lifelessip link show dev tap8e1c39ef-b822:14
lifelessDevice "tap8e1c39ef-b8" does not exist.22:14
lifelessno such port22:14
SpamapSshows in list-ports for br-int22:15
lifelessovs-vsctl show | grep tap22:15
lifeless        Port "tap8e1c39ef-b8"22:15
lifeless            Interface "tap8e1c39ef-b8"22:15
lifelessso how did it get there is a question22:15
SpamapSlifeless: neutron bug? ovs bug?22:15
lifelessI suspect the firmware stuff22:15
lifelesspulling eth2 out from under it and then readding it22:15
lifelesscould well have tickled something22:15
SpamapSyeah we may need to be more careful and quiesce the ovs daemons then22:15
lifelessI think we'll want to forward this to neutron, but I agree lets remove that port from ovs and see what happens22:16
SpamapSlifeless: ok ovs thinks it is gone.22:17
SpamapSjust in time to delete the old overcloud22:18
SpamapS| c68d20dc-2d81-42d8-bff3-600d2312a508 | overcloud-notcompute-nnbzko2heo7k | BUILD  | deleting   | NOSTATE     | ctlplane= |22:24
*** tzumainn has quit IRC22:29
SpamapSoh m I guess that still works22:30
lifelesswe've 37 or so machines available22:33
lifelessa wedged one isn't the ned of the world22:33
*** rdopieralski has quit IRC22:46
*** lsmola_ has quit IRC22:47
*** tzumainn has joined #tripleo22:54
*** spzala has joined #tripleo23:01
*** cd-undercloud has joined #tripleo23:13
cd-undercloud************** overcloud complete status=1 ************23:13
*** cd-undercloud has quit IRC23:13
lifelessat least heat-list shows now errors this time23:21
lifelessSpamapS: ^23:21
*** UtahDave has left #tripleo23:22
*** sdake has quit IRC23:25
*** jergerber has quit IRC23:31
*** jcoufal has quit IRC23:43
*** tzumainn has quit IRC23:47
*** devanand1 has joined #tripleo23:53
*** marios_ has joined #tripleo23:53
*** MarkAtwood has quit IRC23:54
*** markmc has quit IRC23:56
*** jcooley_ has joined #tripleo23:57
*** marios has quit IRC23:58
*** devananda has quit IRC23:58
*** lynxman has quit IRC23:58
*** jesusaurus has quit IRC23:58
*** jcooley has quit IRC23:58
*** jcooley_ is now known as jcooley23:58
*** lynxman has joined #tripleo23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!