| *** achanda has quit IRC | 00:01 | |
| *** achanda_ has quit IRC | 00:02 | |
| *** dims has quit IRC | 00:03 | |
| *** sdake_ has joined #kolla | 00:08 | |
| *** britthou_ has joined #kolla | 00:09 | |
| *** jasonsb has joined #kolla | 00:09 | |
| *** sdake has quit IRC | 00:11 | |
| *** britthouser has quit IRC | 00:12 | |
| bmace | you tagging l3 today sdake? or just pushing up / tagging some images to dockerhub? | 00:15 |
|---|---|---|
| *** diogogmt has quit IRC | 00:18 | |
| *** sdake has joined #kolla | 00:18 | |
| *** jruano has joined #kolla | 00:19 | |
| *** sdake_ has quit IRC | 00:21 | |
| *** weiyu has quit IRC | 00:22 | |
| sdake | bmace i will tag l3 when its ready | 00:32 |
| sdake | atm it isnt' working multinode for me | 00:32 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1490780 | 00:32 |
| openstack | Launchpad bug 1490780 in kolla "rabbitmq fails during startup" [Critical,Triaged] | 00:32 |
| sdake | bmace more importantly: | 00:33 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1490778 | 00:33 |
| openstack | Launchpad bug 1490778 in kolla "multinode deploy results in intermittent authentication failures" [Critical,Triaged] - Assigned to Steven Dake (sdake) | 00:33 |
| sdake | however the mariadb permission problem is fixed | 00:33 |
| sdake | once i can reliably get a multinode deploy l3 will be tagged | 00:34 |
| sdake | bmace do those bugs ring any bells? | 00:34 |
| *** blahRus has quit IRC | 00:41 | |
| *** zhiwei has joined #kolla | 00:44 | |
| bmace | i have seen some failures in various places, but since we are using ndb rather than mariadb.. mileage probably may vary :) | 00:45 |
| *** sdake_ has joined #kolla | 00:47 | |
| *** sdake__ has joined #kolla | 00:49 | |
| *** sdake has quit IRC | 00:50 | |
| *** sdake_ has quit IRC | 00:53 | |
| sdake__ | bmace what is nbd? | 00:54 |
| *** sdake__ is now known as sdake | 00:54 | |
| sdake | open source database? | 00:55 |
| *** banix has joined #kolla | 00:55 | |
| *** weiyu has joined #kolla | 01:07 | |
| *** banix has quit IRC | 01:12 | |
| *** Haomeng has joined #kolla | 01:14 | |
| *** Haomeng|2 has quit IRC | 01:17 | |
| *** dims has joined #kolla | 01:20 | |
| *** banix has joined #kolla | 01:20 | |
| *** erkules_ has joined #kolla | 01:22 | |
| *** erkules has quit IRC | 01:25 | |
| *** zhiwei has quit IRC | 01:37 | |
| *** SourabhP has quit IRC | 01:38 | |
| sdake | ubuntu gate just merged yay ;) | 01:57 |
| *** jruano has quit IRC | 01:58 | |
| sdake | samyaple you alive yet | 02:01 |
| *** rhallisey has quit IRC | 02:11 | |
| *** banix has quit IRC | 02:13 | |
| *** jruano has joined #kolla | 02:27 | |
| *** jruano has quit IRC | 02:33 | |
| *** jruano has joined #kolla | 02:39 | |
| *** klint has joined #kolla | 02:51 | |
| *** britthou_ has quit IRC | 02:53 | |
| *** britthouser has joined #kolla | 02:53 | |
| *** alisonh has quit IRC | 02:56 | |
| *** dayou has joined #kolla | 03:01 | |
| *** achanda has joined #kolla | 03:07 | |
| *** alisonh has joined #kolla | 03:13 | |
| *** jruano has quit IRC | 03:19 | |
| *** alisonh has quit IRC | 03:22 | |
| SamYaple | sdake: yea | 03:23 |
| sdake | multinode is completely busted: https://bugs.launchpad.net/kolla/+bug/1490778 | 03:25 |
| openstack | Launchpad bug 1490778 in kolla "multinode deploy results in intermittent authentication failures" [Critical,Triaged] - Assigned to Steven Dake (sdake) | 03:25 |
| sdake | but your mariadb bug fix did the trick there | 03:25 |
| sdake | eating bbiaf - can you read that bug log | 03:25 |
| *** alisonh has joined #kolla | 03:26 | |
| sdake | samyaple here is the diff I'm working with - still busted | 03:27 |
| sdake | http://ur1.ca/nmldb -> http://paste.fedoraproject.org/261788/41078016 | 03:27 |
| SamYaple | hey before you go digging into that let me submit my keepalive config changes that we need | 03:29 |
| SamYaple | might solve your issue | 03:29 |
| SamYaple | (again... no issues on ubuntu. centos seems bad) | 03:29 |
| *** unicell has quit IRC | 03:31 | |
| sdake | cool i'll be happy to test | 03:32 |
| sdake | please git review it ;) | 03:33 |
| sdake | i think the issue is haproxy tho not keepalived | 03:33 |
| sdake | probably ubuntu has some different defaults then centos | 03:33 |
| SamYaple | probably but the patch affects both | 03:33 |
| SamYaple | no we lay down the haproxy config | 03:33 |
| sdake | i understand that | 03:33 |
| SamYaple | and keepalived for that matter | 03:33 |
| SamYaple | there are no other options coming into play though | 03:34 |
| sdake | i have been digging itn othe code for the last 8 hurs i understand how it works now :) | 03:34 |
| sdake | + stick on dst | 03:34 |
| sdake | + stick-table type ip size 2 | 03:34 |
| sdake | going to give that a go | 03:34 |
| sdake | i htink this is the last thing pending for liberty-3 | 03:34 |
| sdake | and i can tag it and we can get on our way to rc1 | 03:34 |
| SamYaple | we might as well tag it since we arent fixing that tonight... | 03:35 |
| SamYaple | were definetely not using those sticky options | 03:35 |
| sdake | we aren't tagging until its fixed | 03:36 |
| sdake | its totally busted atm | 03:36 |
| SamYaple | no its not | 03:36 |
| sdake | multinode is completely unusable | 03:36 |
| SamYaple | its busted in an inactive environment | 03:36 |
| SamYaple | if you actually used the environment it would be fine | 03:37 |
| sdake | if i sit ther eand spam glance image-list | 03:37 |
| SamYaple | we are not solving for an idle case | 03:37 |
| sdake | the first 2 go through | 03:37 |
| sdake | the 3rd fails | 03:37 |
| sdake | that is not inactive | 03:37 |
| sdake | then it never works after that | 03:37 |
| sdake | again completely unusable | 03:37 |
| SamYaple | alright well we are not thorwing options at it to fix it | 03:37 |
| SamYaple | we will need to understand the issue and then fix it and the sticky options are not the fix | 03:37 |
| sdake | i dont even know what sticy options do, just in the red hat bugzilla tracker | 03:38 |
| SamYaple | exactly | 03:38 |
| sdake | but ya we have until the 3rd to fix it | 03:38 |
| SamYaple | thats my point | 03:38 |
| sdake | so lets get cracking | 03:38 |
| sdake | if you need access to my enviornment i am happy to provide it | 03:38 |
| sdake | people will actually try out liberty-3 | 03:38 |
| SamYaple | I may have to since centos is the only one affected | 03:38 |
| sdake | i dont want centos completely busted | 03:38 |
| sdake | email me your ssh key | 03:39 |
| sdake | public key that is | 03:39 |
| SamYaple | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD02SUqeQSk4sapStfsh8HbBfSxNoipYAeKJHzhhldHHXXuW+csByqWfyFQBXAKGdDa1RBk7JNAz9WpyUpmqtYwGvlDofzY52RYdm3jzw9tZCS5ERuMrYB14xokFA5nMyOBluiEkkZvlkJpTywh9Be9kaMJutx46Li+grcSXCtUo3l8mcbMHF0uzg+JFm8Q6KpW0hmmyiPyvwMd4m1+jXw4rVJiHW8GD5Pkhmfwo6srT0mDaOGnrfdXoucoP0Is0U4uwnBxpPfyvG/MKuekyEeoM+zc7kBhHwXyPVYSnOkqgzO4j2eUIYt8ca+ya/8cVdSC/GsGAxBcx4ICMpvxhGXV null@null | 03:39 |
| sdake | what login id do you want | 03:40 |
| SamYaple | doesnt matter | 03:41 |
| sdake | tell me what you want plz | 03:41 |
| sdake | s8m? | 03:41 |
| SamYaple | yea i really dont care | 03:42 |
| SamYaple | thats fine | 03:42 |
| sdake | sticky doesn't seem to fix it | 03:44 |
| sdake | bummer | 03:44 |
| sdake | sec making creds | 03:44 |
| *** britthou_ has joined #kolla | 03:46 | |
| *** thetrav has joined #kolla | 03:48 | |
| *** britthouser has quit IRC | 03:48 | |
| thetrav | do you guys still publish docker containers to docker_hub? | 03:49 |
| thetrav | the one sitting there under fedora-keystone is out of date and broken | 03:49 |
| SamYaple | thetrav: we do not | 03:50 |
| SamYaple | thetrav: we will for stable branches | 03:50 |
| SamYaple | we cant keep up with publishing for every commit due to how slow docker hub is | 03:50 |
| thetrav | are there any stable branches yet? | 03:50 |
| SamYaple | thetrav: now for about 6 weeks | 03:51 |
| SamYaple | not* | 03:51 |
| thetrav | meaning you are planning on releasing something stable in about 6 weeks? | 03:51 |
| SamYaple | I may push images once we snapshot L3 thetrav, ive had a few requests | 03:51 |
| SamYaple | thetrav: officially, yes | 03:51 |
| thetrav | oooooh | 03:51 |
| *** achanda has quit IRC | 03:51 | |
| thetrav | I've used working containers published in the past, it's useful | 03:52 |
| SamYaple | yea weve gone through alot of changes recently so the containers need to be built fro mthe same commit id that the playbooks are run from | 03:52 |
| SamYaple | thats part of the issue | 03:52 |
| SamYaple | we are solving this through a few areas, including having packaged Kolla so it can be pip installed | 03:53 |
| thetrav | ? | 03:53 |
| SamYaple | right now the ansible playbooks and docker images are very closely tied together | 03:53 |
| SamYaple | any images we push owuld quickly go out of date | 03:53 |
| SamYaple | so we do not push any | 03:53 |
| sdake | thetrav its pretty painless to build your own images | 03:54 |
| thetrav | so you're moving into a model where the containers are service centric, ideally allowing anyone to use them for various orchestrators? | 03:54 |
| sdake | and that way they match the dev commit your using | 03:54 |
| SamYaple | thetrav: yea that was always the plan, but we are very close to it | 03:54 |
| thetrav | sdake: i've built my own in the past, both from scratch and by forking the kolla github | 03:54 |
| sdake | thetrav i'd like to see the api to the containers stabilize to such a point that tripleo could use them equally | 03:54 |
| SamYaple | thetrav: Triple-O uses puppet right now i believe, so we are halfway there | 03:55 |
| sdake | however we aren't quite there yet | 03:55 |
| SamYaple | yea | 03:55 |
| *** britthou_ has quit IRC | 03:55 | |
| thetrav | so should I build from master or stable/kilo? | 03:56 |
| SamYaple | thetrav: master | 03:57 |
| SamYaple | stable/kilo is not really a valid branch | 03:57 |
| sdake | sam try ssh into 101,102,103 | 03:57 |
| sdake | make sure your ssh keys are working there | 03:58 |
| sdake | its a branch just not maintained | 03:58 |
| sdake | i expect in liberty the branches will be better maintained thetrav | 03:58 |
| sdake | the liberty branch that is | 03:58 |
| SamYaple | sdake: im not going to copy my private key to your server if thats what you are asking | 03:59 |
| sdake | no | 03:59 |
| sdake | i already have you setup with a private key on my machines | 03:59 |
| SamYaple | ah yes 101 seems borked | 03:59 |
| SamYaple | the other two are fine | 03:59 |
| sdake | define borked? | 03:59 |
| sdake | oh you can't ssh you mean | 03:59 |
| SamYaple | yea | 03:59 |
| SamYaple | 101 is asking for pass | 04:00 |
| *** diga has joined #kolla | 04:00 | |
| SamYaple | rejecting key | 04:00 |
| sdake | try 101 again | 04:00 |
| SamYaple | gtg | 04:00 |
| SamYaple | ah wrong window | 04:00 |
| SamYaple | yea its working | 04:00 |
| sdake | you should be able to sudo | 04:01 |
| thetrav | do you guys have a docker container for building kolla containers? | 04:01 |
| sdake | holy batman summer storm time | 04:01 |
| sdake | thetrav no | 04:01 |
| sdake | SamYaple 101-103 are the deployment targets | 04:02 |
| sdake | 100:4000 is the registry server | 04:02 |
| sdake | i rebuilt the images from master this morning and no new commits have hit the repo | 04:02 |
| sdake | this is after your change to fix mariadb | 04:03 |
| sdake | use this openrc on YOUR hardware | 04:03 |
| sdake | http://ur1.ca/nmlof -> http://paste.fedoraproject.org/261804/41080233 | 04:04 |
| SamYaple | are you saying I would be able to sudo? passwordless? | 04:04 |
| sdake | yes | 04:05 |
| SamYaple | asking for passwords | 04:05 |
| sdake | do you know how to turn the password request off | 04:05 |
| sdake | and i'll make that change | 04:06 |
| sdake | your in the wheel group | 04:06 |
| SamYaple | s8m ALL=(ALL) NOPASSWD: ALL | 04:06 |
| SamYaple | the NOPASSWD: is key there | 04:06 |
| sdake | you will have to relog for tht to take efect | 04:08 |
| sdake | should be set on all 4 machines | 04:08 |
| sdake | i deploy from 100 to 101-103 | 04:08 |
| SamYaple | nah you dont have to relog for sudo | 04:09 |
| SamYaple | it rereads each time | 04:09 |
| SamYaple | all good | 04:09 |
| sdake | cool | 04:09 |
| SamYaple | you have to relog for group changes | 04:09 |
| sdake | dont damage my 100 box plz ;) | 04:09 |
| SamYaple | i wont | 04:09 |
| sdake | the others can be reloaded | 04:10 |
| sdake | all 4 boxes can be rebooted | 04:10 |
| sdake | feel free to delete the images or containers on any box | 04:10 |
| sdake | what is wierd about this problem is openstack endpoint list in a loop works fine | 04:11 |
| sdake | but glance image list does not | 04:11 |
| sdake | openrc to be used on your laptop http://ur1.ca/nmlqz -> http://paste.fedoraproject.org/261809/10807541 | 04:12 |
| sdake | oh i pasted that already i see above | 04:13 |
| sdake | let me know when your planning to take it over | 04:13 |
| sdake | btw AIO works fantasitcally well on 100 | 04:14 |
| SamYaple | can I install tmux on 100? | 04:15 |
| sdake | thetrav i'd highly encourage you to try out master and follow the docs as best as possible | 04:15 |
| sdake | feel free | 04:15 |
| SamYaple | thanks | 04:15 |
| sdake | just don't yum upgrade plz | 04:15 |
| sdake | i don tthink yo ucan use that openrc from my hardware, but i am not certain | 04:15 |
| sdake | thetrav and provide feedback - only 3 weeks left until we wrap up dev for this cycle | 04:16 |
| sdake | 101-103 are centos 100 is fedora 21 | 04:17 |
| sdake | notice i am a bit tight on disk space | 04:17 |
| thetrav | sdake: ok I'll have a look, I assume you're still focussing on single node though? I see mention of multi-node, but only mentions | 04:17 |
| sdake | thetrav lbirty-3 will be full on multinode | 04:17 |
| sdake | 3 control + ~100 compute + storage nodes | 04:18 |
| thetrav | sorry, what is lbirty-3? | 04:19 |
| sdake | liberty-3 is the deadline approaching in the next 3 days | 04:19 |
| sdake | after that we have liberty-rc1 on 25th - hard deadline | 04:19 |
| sdake | after the 25th no new feateures can hit the code base, only bug fixes | 04:19 |
| sdake | until liberty is releaed | 04:19 |
| thetrav | ahh, ok, openstack liberty | 04:19 |
| thetrav | getting it ready for the summit? | 04:20 |
| sdake | right | 04:20 |
| sdake | getting it ready yes ;) | 04:20 |
| thetrav | ok, so this might be time for me to try it out then... I've got two environments, one with 6 servers one with 14, if kolla can get me 3 control nodes and 3 compute nodes in the smaller env that's a win for me | 04:21 |
| thetrav | is it going to run with the kilo codebase or the indev work for openstack liberty? | 04:22 |
| sdake | theoretically ha shoudl be working well to but atm its busted | 04:22 |
| sdake | it runs with master of liberty atm | 04:22 |
| thetrav | ok | 04:22 |
| sdake | at end of liberty it will run with liberty stable branches | 04:22 |
| sdake | and then we will run with master of mitaka | 04:22 |
| thetrav | so I'm trying to get Telstra to send me to the summit. I'll try to find the Kolla guys there and chat about it | 04:23 |
| sdake | we will have design session space | 04:23 |
| thetrav | hmm, I'd be flying out on the Friday if it happens | 04:23 |
| thetrav | My plan is to use Kolla or something similar to manage the on prem cloud for: http://www.crn.com.au/News/407398,telstra-opens-software-skunkworks-in-melbourne.aspx | 04:24 |
| sdake | we have design summit space tue-thur - time is undecided at this point | 04:24 |
| thetrav | ok, I'll keep an eye on it | 04:24 |
| sdake | SamYaple have you taken usage of my machine or shall I keep at it | 04:25 |
| SamYaple | oh yea im messing with this now | 04:28 |
| sdake | cool its almost time for me to be human and sleep :) | 04:28 |
| sdake | i have a bad cold or flu or something | 04:29 |
| sdake | i'd recommend sticking to debugging justglane + keystone | 04:29 |
| sdake | not quite sure why centos behaves differently then ubuntu | 04:29 |
| sdake | that said it may be an environmental thing | 04:30 |
| sdake | wehre ubuntu also behaves the same way in my environment | 04:30 |
| *** dims has quit IRC | 04:30 | |
| sdake | but since you and i are about the only two people who have done multinode deploys.. | 04:30 |
| sdake | maybe harmw | 04:30 |
| SamYaple | so one thing ive noticed is that your internal url is pointed to your external address | 04:31 |
| sdake | let me look - that would explain it | 04:32 |
| sdake | kolla_internal_address: "192.168.1.148" | 04:32 |
| sdake | 148 = free network ip on my network | 04:32 |
| sdake | broked.selfip.net is my public address | 04:32 |
| sdake | config file looks ok? | 04:32 |
| SamYaple | [s8m@bigiron ~]$ openstack endpoint show f90304592bd04ca88910eea1c3b4e1b5 | 04:33 |
| SamYaple | +--------------+----------------------------------+ | 04:33 |
| SamYaple | | Field | Value | | 04:33 |
| SamYaple | +--------------+----------------------------------+ | 04:33 |
| SamYaple | | adminurl | http://192.168.1.148:9292 | | 04:33 |
| SamYaple | | enabled | True | | 04:33 |
| SamYaple | | id | f90304592bd04ca88910eea1c3b4e1b5 | | 04:33 |
| SamYaple | | internalurl | http://broked.selfip.net:9292 | | 04:33 |
| SamYaple | | publicurl | http://broked.selfip.net:9292 | | 04:33 |
| SamYaple | | region | RegionOne | | 04:33 |
| SamYaple | | service_id | 24935aaca8e24f03858d69a353447e10 | | 04:33 |
| SamYaple | | service_name | glance | | 04:33 |
| SamYaple | | service_type | image | | 04:33 |
| SamYaple | +--------------+----------------------------------+ | 04:33 |
| sdake | admin_url='http://{{ kolla_internal_address }}:{{ glance_api_port }}' | 04:35 |
| sdake | internal_url='http://{{ kolla_internal_address }}:{{ glance_api_port }}' | 04:35 |
| sdake | public_url='http://{{ kolla_external_address }}:{{ glance_api_port }}' | 04:35 |
| sdake | bug in shade? | 04:36 |
| SamYaple | we dont really use shade | 04:36 |
| sdake | where does kolla_keystone_service come from? | 04:36 |
| SamYaple | this is keystone v2 nonsense | 04:36 |
| SamYaple | i wrote that | 04:37 |
| sdake | internalurl=public_url, | 04:38 |
| sdake | publicurl=public_url, | 04:38 |
| sdake | there is a bug there | 04:38 |
| SamYaple | yup | 04:38 |
| SamYaple | thats not the issue though | 04:39 |
| *** weiyu has quit IRC | 04:39 | |
| sdake | nice diagnoitics hope thats it | 04:39 |
| SamYaple | its not | 04:39 |
| SamYaple | or i take that back | 04:39 |
| SamYaple | it _could_ be | 04:39 |
| sdake | can you fix and build that container | 04:39 |
| SamYaple | but that would be your routers fault | 04:39 |
| sdake | if it is - win :) | 04:39 |
| thetrav | so right away it appears to want to publish to docker hub | 04:40 |
| *** diga has quit IRC | 04:40 | |
| thetrav | can I tell it with .buildconf to push to a private repo? | 04:40 |
| sdake | like i said you can't use openrc from my hardware ;) | 04:40 |
| sdake | dot fiesl are gone thetrav | 04:40 |
| sdake | use -n to set the repo name | 04:41 |
| thetrav | they still exist in the docs | 04:41 |
| thetrav | https://github.com/stackforge/kolla/blob/master/docs/image-building.rst | 04:41 |
| sdake | e.g -n 192.168.1.100:4000/kollaglue will publish to my local registry | 04:41 |
| sdake | yes docs are out of date | 04:41 |
| sdake | that one in particular | 04:41 |
| *** diga has joined #kolla | 04:41 | |
| sdake | wtb peole to improve doccs ;0 | 04:41 |
| thetrav | sure, it's just that you told me to follow the docs and give feedback :P | 04:41 |
| sdake | oh right | 04:42 |
| thetrav | I will use -n | 04:42 |
| sdake | i dont know if that iwll let you push to a privat registry | 04:42 |
| sdake | but it will let you push to your own registry server | 04:42 |
| thetrav | yeah that's what I meant | 04:42 |
| thetrav | so tools/build-all-docker-images --release --push -n docker.ipc.telstra.net:5000/ should work? or am I setting a namespace, and require /kollaglue | 04:43 |
| sdake | tools/build.py is what your after | 04:44 |
| sdake | our imge building docs are completely out of date | 04:44 |
| sdake | you ned a namespace as well - such as telestra | 04:44 |
| sdake | or kolla | 04:44 |
| sdake | tools/build.py -n docker.ipc.telstra.net:5000/kolla | 04:45 |
| SamYaple | idk sdake i updated the endpoints in the db same behaviour | 04:45 |
| thetrav | so /tools/build.py --release --push -n docker.ipc.telstra.net:5000/kolla | 04:45 |
| *** achanda has joined #kolla | 04:45 | |
| thetrav | does --release and --push happen anymore? | 04:45 |
| sdake | tools/build.py --push -n docker.ipc.telstra.net5000/kolla | 04:45 |
| sdake | there is no --release anymore | 04:45 |
| SamYaple | my only suggestion right now is to limit the active haporxy servers to one like galera | 04:45 |
| sdake | run build.py -h to see docs on build | 04:45 |
| thetrav | ahh, I need to pip install some stuff first | 04:45 |
| sdake | SamYaple that doesnt really get at the source of the problem though | 04:47 |
| SamYaple | no | 04:47 |
| SamYaple | i think there is something up with delorean | 04:47 |
| SamYaple | there are no errors | 04:47 |
| SamYaple | and glance appropriately fetchs credentials | 04:47 |
| sdake | if you read that ask.openstack article they imply that it has to do with haproxy killing connections prematurely | 04:48 |
| SamYaple | which just isn't correct | 04:48 |
| SamYaple | when shutting down the other 2 glance servers it works 100% of the itme | 04:48 |
| sdake | not sure what one has to do with the other | 04:49 |
| SamYaple | glance is saying bad token when it clearly fetched a good one? | 04:50 |
| sdake | keystone says it can't find the token in the registry | 04:50 |
| sdake | rather in the db | 04:50 |
| SamYaple | havent seen that error anywhere | 04:52 |
| sdake | if it always is syncornized throug hone glance server there is never a condition where glance wont know what the toekn is | 04:52 |
| sdake | its in my logs | 04:52 |
| sdake | in the bug tracker | 04:52 |
| sdake | "RBAC token not found" | 04:52 |
| SamYaple | it may be in the bug tracker, but its not in your logs | 04:52 |
| sdake | docker exec keystone cat /var/log/keysotne/keysotne.log | 04:52 |
| SamYaple | im watching it right now dude | 04:53 |
| SamYaple | 2015-08-31 21:54:40.310 11 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/site-packages/keystone/common/controller.py:161 | 04:53 |
| sdake | well it was earlier - i saw it with my own eyes ;) | 04:53 |
| SamYaple | and yet it said authorization failed | 04:53 |
| sdake | which node is that on, i want to see the log output | 04:53 |
| sdake | did yu turn on debug mode or something | 04:53 |
| SamYaple | yup | 04:55 |
| SamYaple | restarted the container and now its saying "Auth token not in the request header" | 04:57 |
| sdake | keystone is only running on 101 atm | 04:57 |
| SamYaple | sometimes | 04:57 |
| SamYaple | yes | 04:57 |
| sdake | glane image-list still fails | 04:57 |
| SamYaple | i know | 04:58 |
| sdake | se if glance -d image-list is helpful to you | 04:59 |
| SamYaple | i always run with debug | 05:00 |
| sdake | i got an error 500 | 05:00 |
| sdake | http://ur1.ca/nmm5k -> http://paste.fedoraproject.org/261839/83726144 | 05:02 |
| *** unicell has joined #kolla | 05:02 | |
| sdake | seems to imply mariadb is boned in some way | 05:02 |
| sdake | since that i the only dependency in the call path | 05:02 |
| SamYaple | why on earth would oyu think that a 500 client error had anything to do with the database? | 05:03 |
| SamYaple | you know the client never talks to the database right? | 05:03 |
| sdake | yes | 05:03 |
| SamYaple | if ther eare no database complaints in the service logs, its not the database | 05:03 |
| SamYaple | and since there arent... its not hte database | 05:04 |
| sdake | if you run 1 glance and 1 keystone does it work? | 05:04 |
| sdake | in other words killign off the 2 ha keystone and glance services? | 05:04 |
| SamYaple | it works if i run just 1 glance server | 05:05 |
| sdake | does the haproxy roundrobin to the other glance servers? | 05:05 |
| sdake | or does it pick one and stick to it | 05:07 |
| sdake | try 3 glance 1 keystone - that eliminates glance as the source of the problem | 05:10 |
| SamYaple | that doesnt work | 05:10 |
| sdake | doesn't work at all or produces authentication failure? | 05:11 |
| sdake | basically with 1 glance 3 keystone, you still are not serializing all db access thorugh keystone | 05:12 |
| sdake | anything that uses glance would be serialized | 05:12 |
| sdake | but with 1 keystone, db would be serialized | 05:12 |
| *** inc0 has joined #kolla | 05:15 | |
| sdake | yo inc0 | 05:15 |
| inc0 | good morning | 05:15 |
| sdake | inc0 you have done multinode deployment haven't you? | 05:16 |
| inc0 | I did, but with just one controller and lately we had lots of problems | 05:16 |
| inc0 | but shoot | 05:16 |
| sdake | oh doing multicontroller - no bueno | 05:16 |
| sdake | did you get authentication failures with services like glance? | 05:17 |
| inc0 | yup | 05:17 |
| sdake | inc0 https://bugs.launchpad.net/kolla/+bug/1490778 | 05:17 |
| openstack | Launchpad bug 1490778 in kolla "multinode deploy results in intermittent authentication failures" [Critical,Triaged] - Assigned to Steven Dake (sdake) | 05:17 |
| inc0 | keystone is busted | 05:17 |
| sdake | pretty sure its db not keystone | 05:17 |
| inc0 | well, yeah | 05:17 |
| inc0 | keystone failed to add user | 05:18 |
| inc0 | I'd guess because we didn't set up keepalived+haproxy properly | 05:18 |
| sdake | were you going to wiat for the apocoplyse to report this? :) | 05:18 |
| inc0 | no, we were trying to fix it;) it's from Friday | 05:19 |
| sdake | inc0 your using ubuntu or centos | 05:19 |
| inc0 | centos | 05:19 |
| sdake | what have you tried | 05:19 |
| sdake | read that bug log plz | 05:19 |
| inc0 | yeah, I didn't go that far, I've just started to suspect ks | 05:21 |
| inc0 | keepalived* | 05:21 |
| inc0 | I'll work on it today | 05:22 |
| sdake | this bug is holding up the release | 05:23 |
| sdake | what would keepalived have to do with it inc0 | 05:24 |
| inc0 | well, problem is keystone failed to add it's own user | 05:25 |
| inc0 | so I | 05:25 |
| sdake | i can run keystone endpoint list in a loop and it never fails to authenticate | 05:25 |
| sdake | rther openstack endpoint list | 05:25 |
| inc0 | so you're futher than I was | 05:25 |
| sdake | this is just using master with a 3 node ha setup | 05:26 |
| inc0 | ok | 05:26 |
| sdake | but glance image-list in a loop fails with authentication problems | 05:27 |
| *** weiyu has joined #kolla | 05:27 | |
| sdake | how do the two differ from a haproxy + keepalive perspective? | 05:28 |
| inc0 | well they shouldn't | 05:28 |
| inc0 | in your openrc you point out to keepalived right? | 05:28 |
| sdake | does keystone cache tokens? | 05:28 |
| inc0 | what do you mean? | 05:29 |
| inc0 | by default ks keeps tokens for 2hrs afair | 05:29 |
| sdake | i pointed to my external endpoint which NATs to my VIP address | 05:29 |
| sdake | does it re-read the database if it gets a rquest for a token it doesn't have? | 05:29 |
| inc0 | do you have glance user in database? | 05:29 |
| sdake | i.e. is there a cachin problem | 05:29 |
| inc0 | ah not like that | 05:30 |
| sdake | no idea SamYaple has taken over my dev box ;) | 05:30 |
| inc0 | it checks db on every request I think | 05:30 |
| sdake | does haproxy roundrobin ortarget one specific server? | 05:30 |
| inc0 | its rr by default | 05:31 |
| inc0 | however, we didn't get haproxy working right?:) | 05:31 |
| inc0 | SamYaple, around? | 05:31 |
| sdake | huh? | 05:31 |
| inc0 | this problem with "no key ansible_eth1" is solved? | 05:31 |
| SamYaple | sorry went food | 05:34 |
| inc0 | one you ran into last week | 05:34 |
| SamYaple | not fixed yet inc0 | 05:35 |
| SamYaple | fixable, just have to implement it | 05:35 |
| SamYaple | wont be a problem | 05:35 |
| inc0 | how do you want to fix it? | 05:35 |
| SamYaple | ive got a couple of ways, im going to try them out and see which one works best | 05:35 |
| inc0 | do share plz;) | 05:36 |
| sdake | inc0 there is a workaround | 05:37 |
| sdake | set all nodes to the same for control network etc | 05:37 |
| sdake | for the moment use that | 05:37 |
| sdake | we need to stay focused on the fact that multinode deploy is busted atleast on centos | 05:37 |
| inc0 | yeah, let me get to office and I'll try to deploy it | 05:38 |
| inc0 | I should have fresh images | 05:38 |
| sdake | you need to rebuild with master | 05:38 |
| sdake | some bug fixes hit the repo which fix the mariadb problem | 05:39 |
| inc0 | ok, thanks for heads up | 05:39 |
| inc0 | I'll focus on this problem today | 05:39 |
| inc0 | afk for 1hr | 05:40 |
| sdake | i'd like to tag and release asap | 05:40 |
| sdake | so lets get this one solved | 05:40 |
| sdake | the keytone docs say to setup keystone with a memcache server | 05:41 |
| sdake | and that keystone on ubuntu ships with sqlite as the database of choice | 05:42 |
| SamYaple | ignore those | 05:42 |
| SamYaple | they are tainted with OSAD | 05:42 |
| SamYaple | you absolutely do not need a memcache server | 05:42 |
| sdake | i dont particuarly care at this point how its done as long as it works ;) | 05:43 |
| sdake | my point is maybe that is why ubuntu behaves differently then centos | 05:43 |
| SamYaple | .... | 05:43 |
| SamYaple | it installs from source | 05:43 |
| SamYaple | why do you think openstack installed from source would have a default per idstro | 05:43 |
| SamYaple | that entire guide is talking about cloud-archive packaging | 05:44 |
| SamYaple | besides that entire thing is controlled by configuration options | 05:44 |
| sdake | ya | 05:44 |
| *** inc0 has quit IRC | 05:44 | |
| sdake | install from centos uses two configuration files | 05:45 |
| *** SourabhP has joined #kolla | 05:46 | |
| sdake | 2015-08-31 20:41:50.317 1 DEBUG glance.common.config [-] config files: ['/usr/share/glance/glance-api-dist.conf', '/etc/glance/glance-api.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2219 | 05:46 |
| sdake | glance-api-dist.conf -> http://ur1.ca/nmmia -> http://paste.fedoraproject.org/261850/44108647 | 05:48 |
| sdake | could be trigger of the problem | 05:50 |
| sdake | does ubuntu read in a defualt api-dist.conf file? | 05:50 |
| sdake | samyaple try running glance without that glance-api-dist.conf garbage | 05:51 |
| sdake | that is a huge delta | 05:51 |
| SamYaple | yea im doing that now | 05:51 |
| sdake | probably want to do a full clean and deploy on the deployment nodes ;) | 05:52 |
| SamYaple | i rekicked the glance containers | 05:52 |
| SamYaple | this is a test of upgrades | 05:54 |
| SamYaple | what is that | 05:54 |
| SamYaple | 'this is a test of upgrades | 05:54 |
| SamYaple | ' | 05:54 |
| sdake | i just added an echo | 05:54 |
| SamYaple | thats cool | 05:55 |
| sdake | during a demo which failed | 05:55 |
| SamYaple | you said this was master though | 05:55 |
| sdake | it was earlier today | 05:55 |
| SamYaple | ok | 05:55 |
| sdake | it was, but i did reuibld glance | 05:55 |
| sdake | forgot about that | 05:55 |
| sdake | everything else is master | 05:55 |
| sdake | this problem could explain why heat is busted on centos too | 05:56 |
| SamYaple | i said it was delorean packaging a long time ago :) | 05:56 |
| sdake | you did | 05:57 |
| SamYaple | im testing that though before laying final blame | 05:57 |
| sdake | and you would be correct if this fixes it :) | 05:57 |
| SamYaple | i think you should trust me more | 05:58 |
| *** weiyu has quit IRC | 05:58 | |
| sdake | you hae root on my box dude | 05:58 |
| sdake | i'm not sure what else you want ;) | 05:58 |
| SamYaple | valid point | 05:58 |
| SamYaple | carry on | 05:58 |
| SamYaple | youve convinced me | 05:58 |
| *** dims has joined #kolla | 05:59 | |
| sdake | whatever the case is we need to get rid of that dist crap in the delorean packaging | 06:00 |
| sdake | kolla is responsible for config ;) | 06:00 |
| SamYaple | yea there is some delorean crust in the ubuntu-source ontainer too | 06:01 |
| SamYaple | havent had a chance to remove | 06:01 |
| *** dims has quit IRC | 06:04 | |
| *** sdake_ has joined #kolla | 06:05 | |
| SamYaple | still nothing but also no errors | 06:06 |
| SamYaple | none of the bugs you posted share erros wtih what youre seeing on the server side | 06:06 |
| SamYaple | server side i see " Auth token not in the request header. Will not build auth context. " | 06:06 |
| SamYaple | but that makes me blame the glance client | 06:06 |
| SamYaple | idk man i gots other stuff to do at the moment | 06:07 |
| SamYaple | let me check ubuntu and verify its not hte same behaviour | 06:07 |
| SamYaple | also sdake you need to update teh kolla namespace move patch | 06:08 |
| sdake_ | yup waiting for the rest of the reviews to be approved for project-config | 06:08 |
| *** sdake has quit IRC | 06:08 | |
| SamYaple | cool | 06:08 |
| sdake_ | the move is the 11th | 06:09 |
| sdake_ | so I have time | 06:09 |
| sdake_ | i've got one more for removal of the f21 gate | 06:09 |
| SamYaple | when will the ubuntu gate be active? | 06:11 |
| *** Haomeng|2 has joined #kolla | 06:12 | |
| *** sdake has joined #kolla | 06:13 | |
| sdake | samyaple if your done debugging for the moment let me take over for a bit | 06:13 |
| sdake | i have a few things i want to try | 06:13 |
| SamYaple | yea go for it | 06:13 |
| SamYaple | you may wany to rekick at least glance | 06:13 |
| sdake | i am gong to destory it all | 06:15 |
| *** unicell has quit IRC | 06:15 | |
| *** sdake_ has quit IRC | 06:15 | |
| sdake | keystone also has a default dist file | 06:15 |
| *** unicell has joined #kolla | 06:15 | |
| SamYaple | yea i saw | 06:15 |
| SamYaple | not sure it thats whats going on but thats why im going to check ubuntu when i get a chance | 06:15 |
| SamYaple | to ensure that from source pure works | 06:15 |
| *** Haomeng has quit IRC | 06:16 | |
| SamYaple | hey sdake in regards to ansible modules, i dont want to keep having the same arguemnets over and over again | 06:16 |
| SamYaple | ansible will always import * | 06:16 |
| SamYaple | it is required | 06:16 |
| SamYaple | lets drop this | 06:16 |
| sdake | huh? | 06:17 |
| sdake | I am pretty sure i agreed with you on that point | 06:17 |
| SamYaple | sdake: im saying formally, its coming up again | 06:17 |
| SamYaple | https://review.openstack.org/#/c/218570/6/ansible/library/bslurp.py | 06:17 |
| SamYaple | im not pointing the finger at you | 06:17 |
| *** weiyu has joined #kolla | 06:22 | |
| sdake | can't we just use the openstack client instead of shade for keystone v3 stuff? | 06:27 |
| SamYaple | again, we dont use shade irght now | 06:28 |
| SamYaple | just because i import shade doesnt mean im using it | 06:28 |
| SamYaple | if you notice im calling the keystone client _within_ shade | 06:29 |
| SamYaple | the only thing shade is doing is a bit of authentication abstraction | 06:29 |
| SamYaple | and a few other nice things | 06:29 |
| SamYaple | but in the end im using the keystone client directly | 06:29 |
| SamYaple | i can do v3 right now, i just hae to do it | 06:30 |
| sdake | the less dependencies a system has the better | 06:30 |
| sdake | if we don't need shade no reason to import it ;0 | 06:30 |
| SamYaple | i just said we do need it, i said we dont _use_ it (as in the keystone part of shade) | 06:30 |
| SamYaple | it still sets stuff up | 06:31 |
| SamYaple | this reduces the lines of code i eed from 300 to like 50 | 06:31 |
| SamYaple | but im still calling keystone directly | 06:31 |
| *** SourabhP has quit IRC | 06:35 | |
| *** inc0 has joined #kolla | 06:35 | |
| inc0 | back | 06:36 |
| openstackgerrit | Hengqing Hu proposed stackforge/kolla: Expose exception objects via the as keyword https://review.openstack.org/218757 | 06:37 |
| sdake | samyaple can you tell me if ubuntu has defualt config options it runs | 06:37 |
| SamYaple | how do you mean? | 06:38 |
| SamYaple | openstack itself has default options | 06:38 |
| sdake | i mean default config files it loads up from packaging | 06:41 |
| sdake | like delorean does | 06:41 |
| sdake | and do you know how they do that in the packaging? | 06:41 |
| SamYaple | i dont know. probably. i havent done a cloud-archive install in about a year | 06:41 |
| *** tfukushima has joined #kolla | 06:47 | |
| coolsvap | SamYaple, the ceph review will fail with pep8 untill you change the license imho | 06:49 |
| sdake | SamYaple can you paste your diff for fixing glance pleae | 06:50 |
| sdake | i want to see how you fixed the paste deploy problem on centos | 06:50 |
| SamYaple | sdake: i never fixed it | 06:51 |
| SamYaple | sdake: i gave up and said i would check ubuntu to ensure that isnt happening | 06:52 |
| SamYaple | then we can revisit | 06:52 |
| SamYaple | coolsvap: yea ill change the licensing | 06:52 |
| SamYaple | that stuff is copypasta from my yaodu modules | 06:52 |
| sdake | ok well i am fixing now, can you look at what defualt file is loaded on ubuntu | 06:52 |
| SamYaple | again, I dont know what the ubuntu packaged defaults are | 06:53 |
| sdake | from source | 06:53 |
| sdake | look at the logs | 06:53 |
| sdake | see the glance-api.conf line | 06:53 |
| sdake | it tells you which stuff is loaded by default | 06:53 |
| SamYaple | i se what you mean | 06:53 |
| SamYaple | ill look at that when i rekick | 06:53 |
| sdake | clearly it is loading somehting extra | 06:53 |
| sdake | to pull in the psate deploy file | 06:54 |
| SamYaple | if it is doing any hackery, it owuld be in /usr/bin/glance-api | 06:54 |
| SamYaple | athat isn't provided by the source packaging | 06:54 |
| sdake | where does that come from? | 06:54 |
| SamYaple | the centos packaging has thier own script i mean | 06:54 |
| SamYaple | i believe | 06:54 |
| SamYaple | i know ubuntu did | 06:54 |
| sdake | ubuntu start.sh still calls glance-api | 06:55 |
| SamYaple | ubuntu packaging | 06:55 |
| SamYaple | cloud-archive | 06:55 |
| sdake | no, from source | 06:55 |
| sdake | our start.sh calls the same binary | 06:55 |
| SamYaple | that doesnt mean is the same binary | 06:55 |
| SamYaple | where is the disconnect | 06:55 |
| sdake | possibly not | 06:55 |
| sdake | it has the same name, i'd like to see what is in it | 06:56 |
| SamYaple | i mean the ubuntu cloud-archive (binary) had a special glance-api | 06:56 |
| SamYaple | centos might as well | 06:56 |
| sdake | and what it loads by default | 06:56 |
| sdake | ok, build from ubuntu source creates a glance-api | 06:56 |
| sdake | beecause start.sh calls it | 06:56 |
| SamYaple | yes | 06:56 |
| sdake | woul be helpful to see ap aste of that file | 06:57 |
| sdake | and the logs it generates | 06:57 |
| sdake | because it is definately loading in some defaults that are not in the configuration file | 06:57 |
| sdake | beause if i run glance with --config-file /etc/glance/glance-api.conf | 06:57 |
| sdake | it looks for a paste deploy section | 06:57 |
| sdake | i assume ubuntu would do precisely the same hting so my change would break ubuntu | 06:58 |
| SamYaple | it wouldn't | 06:58 |
| SamYaple | that paste file is in /usr/share/glane | 06:58 |
| SamYaple | it (by default) looks for it in /etc/glance/ | 06:58 |
| sdake | ok well it is oading the config for it | 06:58 |
| SamYaple | thats where it lives on ubuntu | 06:58 |
| SamYaple | anyway i gotta stay focused for a bit | 06:59 |
| sdake | just paste the log and glance-api file plz | 06:59 |
| sdake | next time you deploy | 06:59 |
| sdake | whih hopefully is soon | 06:59 |
| openstackgerrit | Mick Thomspon proposed stackforge/kolla: Add ansible support for Murano https://review.openstack.org/218894 | 07:01 |
| *** slagle has quit IRC | 07:06 | |
| *** achanda has quit IRC | 07:11 | |
| *** achanda has joined #kolla | 07:17 | |
| *** chandankumar has joined #kolla | 07:24 | |
| *** achanda has quit IRC | 07:25 | |
| *** achanda has joined #kolla | 07:26 | |
| *** achanda has quit IRC | 07:30 | |
| *** kjelly has quit IRC | 07:35 | |
| *** shardy has joined #kolla | 07:35 | |
| *** kjelly has joined #kolla | 07:36 | |
| *** thetrav has quit IRC | 07:40 | |
| kjelly | Hi all, I try to use ansible to deploy openstack. And it gives me the error, unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \"Host '172.22.2.81' is not allowed to connect to this MariaDB server | 07:40 |
| kjelly | What's wrong for me? | 07:40 |
| *** tfukushima has quit IRC | 07:42 | |
| *** tfukushima has joined #kolla | 07:42 | |
| *** tfukushima has quit IRC | 07:44 | |
| *** tfukushima has joined #kolla | 07:45 | |
| *** tfukushima has quit IRC | 07:47 | |
| *** tfukushima has joined #kolla | 07:48 | |
| *** tfukushima has quit IRC | 07:49 | |
| *** tfukushima has joined #kolla | 07:50 | |
| inc0 | kjelly, check if your mysql binded to correct ip | 07:52 |
| inc0 | however, also try to rebuild mariadb images because there were bugs fixed this night | 07:52 |
| kjelly | inc0: Ok. Thank you for suggestion. | 07:54 |
| sdake | SamYaple so built ubuntu keystone and ubuntu glance api and glance registry | 08:03 |
| sdake | glance image-list still producing identiy errors | 08:03 |
| sdake | so pretty sure its not the openstack packaged services that are at fault - although the configs are indeed different | 08:04 |
| openstackgerrit | Steven Dake proposed stackforge/kolla: Register internal_url properly https://review.openstack.org/219171 | 08:10 |
| sdake | kjelly the reason that is happeing if yur doing multinode deployment on centos + binary is because mariadb was busted before t-12 hours ago | 08:13 |
| sdake | if your not doing multinode deployment, I have no idea why that is happening - docker logs mariadb should help you understand whats happening | 08:13 |
| SamYaple | sdake: I am doing multinode | 08:16 |
| SamYaple | but not right now | 08:16 |
| SamYaple | real work calls | 08:16 |
| sdake | ok | 08:16 |
| sdake | if you could confirm glance image list in a loop works that would be great | 08:17 |
| sdake | just type it in irc | 08:17 |
| SamYaple | no problem | 08:17 |
| sdake | or if it doesn't work type that in irc too | 08:17 |
| SamYaple | if oyu say its broken it probably is | 08:17 |
| SamYaple | well figure it out | 08:17 |
| sdake | i am preparing to deploy full ubuntu on centos kernel | 08:17 |
| sdake | hpeofully that works | 08:17 |
| sdake | building now | 08:17 |
| sdake | inc0 have you thought of using a data container for logging | 08:19 |
| *** jmccarthy has quit IRC | 08:19 | |
| sdake | in other words everything goes to /var/log and the data container exports /var/log to all containers in the system | 08:19 |
| kjelly | sdake: I use all-in-one node. Now I try to rebuild images. | 08:19 |
| sdake | all in one didn't have this problem kjelly | 08:19 |
| sdake | kjelly make sure to use mater | 08:19 |
| sdake | master | 08:19 |
| SamYaple | sdake: there isn't a single data container | 08:19 |
| inc0 | also, I don't want to multiply containers | 08:20 |
| SamYaple | but my idea was a loging data container so all hosts can see all logs | 08:20 |
| inc0 | later rsyslog will forward logs instead of writing to file | 08:20 |
| *** jmccarthy has joined #kolla | 08:20 | |
| SamYaple | inc0: actually it will do both | 08:20 |
| SamYaple | it will write to disk and forard for logstash | 08:20 |
| SamYaple | local copy never goes away | 08:20 |
| inc0 | well, as long as we add log rotation there | 08:21 |
| SamYaple | yea | 08:21 |
| sdake | so my idea is one data contianer per host that goes to one data container per deplooyment | 08:21 |
| sdake | that way you don't have to hunt down a million different containers to find the log files | 08:21 |
| openstackgerrit | Merged stackforge/kolla: Expose exception objects via the as keyword https://review.openstack.org/218757 | 08:21 |
| inc0 | sdake, my question is...do we really need yet another container? | 08:21 |
| kjelly | sdake: I see. Maybe my envirement have another problem. I will use new envirement to try it. | 08:21 |
| SamYaple | the perdeployment thing sdake is central logging, and thats mitaka at earliest | 08:21 |
| inc0 | it lands on volume, so it's visible by host | 08:22 |
| sdake | the container contains all /var/log for the entire node | 08:22 |
| sdake | so yes, we do need another contianer for that | 08:22 |
| sdake | hunting down these log files is a real chore | 08:22 |
| inc0 | sdake, new patchset will create /var/log/kolla | 08:22 |
| sdake | and that contains what? | 08:23 |
| inc0 | and when we'll add logging to services, we'll add logging per service as well | 08:23 |
| inc0 | so it will be nova-all, neutron-all | 08:23 |
| sdake | and that goes in what, a data continaer mounted by all the different containers on the host? | 08:24 |
| inc0 | each host will have /var/log/kolla/nova-all.log and there where logs from this hosts nova is | 08:24 |
| sdake | we need logs seprated by service | 08:24 |
| SamYaple | why are you combining logs inc0 | 08:24 |
| SamYaple | yea | 08:24 |
| sdake | not jamming all nova logs in one file | 08:24 |
| inc0 | SamYaple, well, that's just templating we're going to do | 08:24 |
| sdake | combinging logs is a different services job | 08:24 |
| SamYaple | yea all the logs need to spit out per service | 08:25 |
| SamYaple | any compining parsing is done later | 08:25 |
| inc0 | ok | 08:25 |
| SamYaple | thats teh central logging blueprint | 08:25 |
| sdake | put in a data container | 08:25 |
| sdake | it will make operators life much easier | 08:25 |
| SamYaple | lies | 08:25 |
| SamYaple | bound to host makes operators livfe easier | 08:25 |
| inc0 | I agree with Sam | 08:25 |
| sdake | yes one data container for the host | 08:26 |
| inc0 | it's almost automatic to look at /var/log | 08:26 |
| sdake | not one per service | 08:26 |
| inc0 | not to enter some container | 08:26 |
| SamYaple | no one is doing one logging container per services sdake | 08:26 |
| sdake | we are not doing a bindmount for logging | 08:26 |
| SamYaple | sdake yes we are | 08:26 |
| sdake | -2 inc | 08:27 |
| inc0 | oO | 08:27 |
| SamYaple | just do it inc0 ill show you how to make it optional | 08:27 |
| sdake | containers do not write to the host filesystem in kolla | 08:27 |
| SamYaple | sdake: its happening this was talked about in the midcycle | 08:27 |
| sdake | and we aer not going to start | 08:27 |
| SamYaple | worst project ever if we dont | 08:27 |
| sdake | you can do the work, i can -2 it, big waste of time | 08:27 |
| sdake | put in a data container | 08:27 |
| SamYaple | get ready to kill the project then sdake | 08:27 |
| inc0 | wow that escalated quickly | 08:28 |
| sdake | ya sam likes to escalate :) | 08:28 |
| sdake | SamYaple i'm pretty sure your overreacitng | 08:28 |
| inc0 | sdake, but really I'm kinds in favor of Sam's approach | 08:28 |
| sdake | but we are not writing data from the container to the host os | 08:28 |
| inc0 | ops will look for logs there | 08:28 |
| SamYaple | sdake: we are, as discussed in the midcycle | 08:28 |
| SamYaple | but as i said it will be optional | 08:28 |
| sdake | which part of we are not writing data from the contianer to the host os are you not hearing | 08:29 |
| inc0 | if logs won't be there, they'll panic | 08:29 |
| SamYaple | sdake: its happening | 08:29 |
| sdake | like i said, -2 inc | 08:29 |
| inc0 | and only after they stop panicking they'll start thinking and look at containers | 08:29 |
| sdake | we are not writing data from the container to the host os | 08:29 |
| sdake | its a huge security problem to do so | 08:29 |
| sdake | not going to happen | 08:29 |
| SamYaple | sdake: stop. we talked about this. it will be optional. but it _will_ be an option | 08:29 |
| inc0 | soo...I'm off to a meeting | 08:30 |
| sdake | if you want o make bindmounting optional that is fine, but it needs to come from a data container so when the option is turned off by default | 08:30 |
| inc0 | in the meantime, please sort it out | 08:30 |
| inc0 | and let me know | 08:30 |
| sdake | there is one place to look for logs | 08:30 |
| sdake | i just did sort it out inc0 | 08:30 |
| inc0 | bb in 1.5h | 08:30 |
| SamYaple | how many times did i say optional sdake? | 08:30 |
| SamYaple | seriously. read | 08:30 |
| sdake | i want it in one place | 08:30 |
| sdake | when its optinoally turned off | 08:30 |
| SamYaple | thats what was discussed | 08:30 |
| SamYaple | like i said inc0 do it | 08:30 |
| sdake | i can live with an optional bindmount if its in one place | 08:31 |
| SamYaple | inc0: ill show you how to make it optional | 08:31 |
| sdake | as long as the data is in a bindmounted data container | 08:31 |
| SamYaple | thats not how bindmounts work, but dont worry because its _optional_ | 08:31 |
| sdake | so people can look in one place for the logs assuming the option is turned off by default | 08:31 |
| *** inc0 has quit IRC | 08:35 | |
| openstackgerrit | James McCarthy proposed stackforge/kolla: Swift WIP - add some missing services, also ansible configuration https://review.openstack.org/218795 | 08:40 |
| *** dims has joined #kolla | 08:42 | |
| *** dims has quit IRC | 08:48 | |
| kjelly | https://github.com/stackforge/kolla/blob/master/docs/image-building.rst The document seems to be out of date. Could I update the document ? | 08:56 |
| sdake | plesae do update kjelly | 08:59 |
| sdake | are you familiar with the openstack workflow? | 08:59 |
| kjelly | sdake: I know. I contributed code to horizon ever. | 09:01 |
| sdake | nce ;) | 09:01 |
| *** vbel has joined #kolla | 09:02 | |
| vbel | good morning and evening :) | 09:03 |
| sdake | hey vbel | 09:03 |
| sdake | hey quick | 09:03 |
| sdake | q | 09:03 |
| sdake | i noticed the footer was after CMD | 09:03 |
| sdake | does that still work for the proxy building use case? | 09:04 |
| vbel | yes | 09:04 |
| sdake | good news eveyrone our docs job was merged | 09:04 |
| sdake | so soon our docs will be published to docs.openstack.org | 09:04 |
| vbel | it also was easy to add to all containers :) | 09:04 |
| sdake | the bad news is our docs are in bad shape ;( | 09:05 |
| *** dayou has quit IRC | 09:05 | |
| vbel | i am going to fix Dockerfile.j2 for novnc - debian, ubuntu AND oraclelinux don't have it as binary and should install from git. I wonder if the correct approach would be to get it from git for all distros for source build type | 09:06 |
| sdake | samyaple heads up the ubuntu source job has merged but has gate failures: https://review.openstack.org/#/c/218757/ | 09:06 |
| kjelly | sdake: Should I post a bug in launchpad for update document ? | 09:07 |
| vbel | currently rpm based distros get novnc from rdo for source builds | 09:07 |
| sdake | vbel we don't implement binary ubuntu or debian | 09:07 |
| sdake | kjelly docs do not require bug ids | 09:07 |
| sdake | to lower the bar to encourage people to submit inremental updates ;) | 09:07 |
| vbel | I mean the section is for 'source' but rpm dists get it from rdo | 09:08 |
| kjelly | sdake: Ok. I see. | 09:08 |
| vbel | http://paste.openstack.org/show/437223/ | 09:09 |
| sdake | vbel looks ok to me? | 09:10 |
| vbel | one problem is that we don't use rdo for source based oraclelinux as much as we can, another is that we could generalize things and fetch novnc from url/git | 09:10 |
| vbel | the same way we do for nova/cinder/neutron .. | 09:10 |
| sdake | oraclelinux binary = rdo + oraclelinux / oraclelinux + source = build from source | 09:11 |
| vbel | yes, I am ok with binary, but source could be really 'source' :) | 09:12 |
| sdake | oh right line 13 | 09:13 |
| sdake | ya that should be a binary conditional | 09:13 |
| vbel | we have binary above with the same stuff, but here we could just leave cross-distro ADD | 09:14 |
| *** zhiwei has joined #kolla | 09:14 | |
| sdake | file a bug submit a patch cr's will review it | 09:15 |
| vbel | yep, will do | 09:15 |
| sdake | SamYaple full source deploy from master yields same problem with glance image-list repeated in a loop | 09:17 |
| sdake | full ubutnu source deploy | 09:17 |
| sdake | centos kernel | 09:17 |
| sdake | vbel have you done any multinode deploys? | 09:18 |
| *** inc0 has joined #kolla | 09:18 | |
| vbel | sdake: so far without haproxy | 09:18 |
| sdake | vbel using what in its place? | 09:18 |
| vbel | sdake: but I did check haproxy and keepalived working | 09:18 |
| inc0 | sdake, SamYaple soo...I'm making logging container with data container, and if we want bindmount, let's do it in another patch ok? | 09:19 |
| vbel | just no proxy :) | 09:19 |
| sdake | inc0 sounds good ;) | 09:19 |
| sdake | i'm +2 on a bindmount patch if it comes from a data container | 09:19 |
| sdake | and its optional | 09:19 |
| SamYaple | sdake: thats not how bindmoutns work | 09:19 |
| sdake | each service uses volume_from to mount data container | 09:19 |
| sdake | tht data container bindmounts /var/log | 09:20 |
| sdake | then we can expand the dat container to log via logstash or whatever forwarding we deicde on later | 09:21 |
| inc0 | sdake, my question....is there anything against rsyslog container being data container as well? | 09:21 |
| sdake | inc0 nope | 09:21 |
| inc0 | so we can just make it single container? | 09:21 |
| sdake | yes | 09:22 |
| inc0 | so if I'll make bindmounting /var/log optional | 09:22 |
| inc0 | it works? | 09:22 |
| inc0 | for you? | 09:22 |
| sdake | from singel container where all the individual volume_froms on a node access yup | 09:22 |
| inc0 | no other container even needs volume from | 09:22 |
| inc0 | I mean, nova doesn't even need to have access to /var/log | 09:23 |
| sdake | i'd like the files stored on disk | 09:23 |
| inc0 | it will be in rsyslog | 09:23 |
| inc0 | container | 09:23 |
| sdake | i want rsyslog to read from files on disk not sent by openstack services via syslog | 09:23 |
| inc0 | nova will have volume on /dev/log, but /var/log doesn't have to be one | 09:23 |
| inc0 | hmmm...why? | 09:24 |
| sdake | syslog blocks | 09:24 |
| sdake | syslog in nonblocking mode loses data | 09:24 |
| sdake | i dont want openstack services to block on logging | 09:24 |
| sdake | been doing diags for 20 years | 09:24 |
| inc0 | rsyslog too? | 09:24 |
| sdake | you will just have to trust me this is the best way to proceed | 09:24 |
| inc0 | allright | 09:24 |
| inc0 | well...hmm | 09:25 |
| sdake | i'm ok on logging blocking on write to disk not wirte to network | 09:25 |
| inc0 | currently it's only writting to disk | 09:25 |
| sdake | what i mean is dont setup the service to log directly via syslog | 09:26 |
| inc0 | yeah, I understand | 09:26 |
| sdake | service/services | 09:26 |
| inc0 | unless service doens't know how to log to fiel | 09:26 |
| inc0 | file | 09:26 |
| sdake | that is a special circumstance | 09:26 |
| sdake | nothing we can do about that | 09:27 |
| sdake | lets not optimize around the one or two services tht cant log to file | 09:27 |
| * inc0 thinking if every service will write to log in syslog-readable way | 09:28 | |
| sdake | if not, lagstash ftw ;-) | 09:28 |
| inc0 | well, anyway, this will be approached case by case | 09:28 |
| sdake | lets take baby steps get em logged to files first ;) | 09:28 |
| inc0 | sdake, it has to be understood by syslog because it's syslog what will forward it to logstash | 09:28 |
| sdake | logstash-forwarder? | 09:29 |
| SamYaple | sdake why exactly are the services going to be logging to a file again? | 09:29 |
| SamYaple | and if so, why do we need syslog at this time? | 09:29 |
| SamYaple | do you know how this is being setup? | 09:30 |
| sdake | i am not sure we need syslog | 09:30 |
| SamYaple | jeez if you just want the logs to hit the disk why are we doign all of this thne | 09:30 |
| sdake | i'm suffering deja vu because i've had this same conversation in like 10 other programs | 09:30 |
| SamYaple | this would have been good information to have during all of those logging conversations you particpated in | 09:30 |
| sdake | took me awhile to process | 09:31 |
| SamYaple | 09:27 < sdake> lets not optimize around the one or two services tht cant log to file | 09:31 |
| SamYaple | so the opposite of what you said? | 09:31 |
| SamYaple | everything now logs to files | 09:31 |
| sdake | i thought inc0 said some services only log to syslog | 09:32 |
| SamYaple | correct | 09:32 |
| SamYaple | and some only to files | 09:32 |
| sdake | those are the one or two services iw as referencing | 09:32 |
| SamYaple | but you want just files | 09:32 |
| SamYaple | well you get no keepalived or haproxy logs at all now | 09:33 |
| SamYaple | which is bad enough | 09:33 |
| sdake | syslog blocks - to operate syslog in noblocking mode it has to lose data | 09:33 |
| openstackgerrit | Vladislav Belogrudov proposed stackforge/kolla: Source based builds should get novnc from tarball (git/url) https://review.openstack.org/219204 | 09:33 |
| SamYaple | but inc0 migh as well scrap that rsyslog container that would be needed | 09:33 |
| SamYaple | sdake: writing to a file blocks as well :/ | 09:33 |
| sdake | inc0 my apologies on any time you have wasted thus far | 09:33 |
| sdake | yes writing to files blocks for short periods | 09:34 |
| sdake | writing to network blocks for long periods | 09:34 |
| SamYaple | dude. syslog isnlt talking over the netowkr yet | 09:34 |
| sdake | there in't much way to get around blocking except to used memory mapping | 09:34 |
| SamYaple | thats central logging | 09:34 |
| SamYaple | i dont think you know whats going on | 09:34 |
| sdake | i really want to tag a release, meaning i need to focus on fixing the fact that multinode deployment just plain doesn't work | 09:36 |
| sdake | so no, I don't have time to go dig into the work already completed at this time | 09:36 |
| *** dims has joined #kolla | 09:36 | |
| SamYaple | so maybe dont comment on it? | 09:37 |
| sdake | i dont get to talk to inc0 very often | 09:37 |
| sdake | i wanted to sync up with him | 09:37 |
| SamYaple | so confuse the hell out of him when you do | 09:37 |
| sdake | hence the comments | 09:37 |
| SamYaple | but you dont know whats happening and are trying to redirect him | 09:37 |
| sdake | inc0 are you confused? | 09:37 |
| SamYaple | youre kidding right sdake? What do you think the current direction is? do we have a syslog container or not? does each service write out its own file? or write it to the syslog container? | 09:39 |
| sdake | each service writes to its own file | 09:40 |
| vbel | gentlemen, we are near release and want to add syslog right now? | 09:40 |
| SamYaple | sdake: thats not what we are doing right now. and thats not what were going to do | 09:40 |
| SamYaple | syslog isnt talking over the network, its over the /dev/log device. | 09:40 |
| sdake | vbel i agree, I am trying ti simplif y things | 09:40 |
| sdake | this is one area of simplification | 09:40 |
| *** dims has quit IRC | 09:41 | |
| SamYaple | lies | 09:41 |
| sdake | i would prefer not to have a syslog service running at all tbh | 09:41 |
| SamYaple | syslog was already identified as a requirement | 09:41 |
| sdake | i would prefer a data container | 09:41 |
| SamYaple | we need it before central logging | 09:41 |
| SamYaple | this was teh path to central logging which you said you wanted not 15 minutes | 09:41 |
| sdake | but we dont need it before rc1 | 09:41 |
| SamYaple | this is how all the logging was going to be written to a file | 09:42 |
| SamYaple | this was teh design | 09:42 |
| sdake | how does the data get into syslog? | 09:42 |
| SamYaple | as stated by myself and inc0 several times, over /dev/log | 09:42 |
| SamYaple | aka normal syslog way | 09:42 |
| vbel | and you want to consolidate those from all the hosts via rsyslog? | 09:43 |
| SamYaple | no | 09:44 |
| SamYaple | syslog per host | 09:44 |
| SamYaple | central logging is mitaka | 09:44 |
| SamYaple | syslog is needed for central logging | 09:44 |
| SamYaple | so we do syslog now, no remote stuff | 09:44 |
| SamYaple | then we add remote stuff in mitaka | 09:44 |
| sdake | agenda item for wednesday's meeting | 09:45 |
| vbel | I see. Wonder if docker itself does not have something similar already | 09:45 |
| sdake | problem solved | 09:45 |
| SamYaple | the worst part is sdake, you were in the conversatoins were we were discussing this and now you are comming in at the eleventh hour this stuff | 09:45 |
| sdake | volumes from, make each service log to /var/log, take 4-8 hours to implement | 09:46 |
| SamYaple | no joke. super easy | 09:47 |
| SamYaple | any yet whats in the queue is almost done as is | 09:47 |
| SamYaple | because we have been working on it for a while | 09:47 |
| SamYaple | but you come in and dont understand the situation and start making decisions about things | 09:47 |
| vbel | why not to do this - https://docs.docker.com/reference/logging/overview/ | 09:48 |
| sdake | multinode deploy doesn't work, syslog log perfection is not really high on the priority list for the project | 09:48 |
| SamYaple | agreed. and yet you tried to control it for some reason | 09:48 |
| vbel | --log-driver=syslog | 09:48 |
| vbel | implemented :) | 09:49 |
| SamYaple | vbel: docker logging is... not so good. But it was discussed. loogging in 1.8 got alot better so maybe we can come back to it | 09:49 |
| SamYaple | the real issue is some thing will not log to stdout/stderr | 09:49 |
| SamYaple | and in those cases we cant use that | 09:49 |
| sdake | looking for a fast simple solution to any of our blueprints or bugs for rc1 | 09:49 |
| vbel | yes, like keystone | 09:49 |
| sdake | we have 25 days to seal rc1 | 09:50 |
| sdake | nothing like diags to raise poeople's blood pressure ;) | 09:53 |
| *** coolsvap has quit IRC | 10:01 | |
| *** coolsvap has joined #kolla | 10:02 | |
| sdake | vbel you aid you got multinode deploy to work without haproy | 10:04 |
| sdake | what did you use instead of haproxy | 10:04 |
| SamYaple | i setup an F5 with Kolla recently | 10:05 |
| SamYaple | not to bad, but i dont have a script to generate the config for the F5 yet | 10:05 |
| sdake | samyaple havey ou been running without haproxy in your multinode deployments? | 10:05 |
| SamYaple | some of them | 10:06 |
| SamYaple | i test about 75% haproxy+keepalived | 10:06 |
| SamYaple | eerrr 25% | 10:06 |
| *** dwalsh has joined #kolla | 10:10 | |
| sdake | ubuntu source: | 10:11 |
| sdake | 2015-09-01 03:10:14.191648 2015-09-01 03:10:14.191 20 WARNING keystone.middleware.core [-] RBAC: Invalid token | 10:11 |
| sdake | 2015-09-01 03:10:14.191773 2015-09-01 03:10:14.191 20 WARNING keystone.common.wsgi [-] The request you have made requires authentication. | 10:11 |
| inc0 | sorry, I had ad-hoc meeting- going through logs | 10:14 |
| inc0 | ok...guys | 10:18 |
| inc0 | let's talk this through tomorrow | 10:19 |
| inc0 | I'm going back to multinode fixes, logging is targeted for rc1 anyway | 10:19 |
| sdake | this is an interesting workaround | 10:22 |
| sdake | If i used mariadb in haproxy conf i would run in the exact same problem. Instead i configured a VIP on all galera nodes and pointed all [database] sections in openstack configs to that VIP. | 10:22 |
| sdake | So i would try to remove galera cluster from haproxy config and try the VIP route for mysql. | 10:22 |
| sdake | with this galera doesn't round robin | 10:23 |
| sdake | i am pretty sure the problem is galera writes and reads are racing | 10:23 |
| SamYaple | galera doesnt round robin now | 10:24 |
| SamYaple | there are no deadlocks, how oculd there be | 10:24 |
| sdake | inc0 said earlir haproxy round robins | 10:24 |
| SamYaple | only one databsae is ever read/write, the others are backup | 10:24 |
| *** akwasnie has joined #kolla | 10:24 | |
| SamYaple | haproxy does | 10:24 |
| SamYaple | galera is set to backup | 10:24 |
| SamYaple | as weve been over | 10:24 |
| sdake | ya haproxy roundrobin reads to galera | 10:25 |
| sdake | based upon what inc0 stated | 10:25 |
| SamYaple | it does not | 10:25 |
| inc0 | haproxy round robins by default | 10:25 |
| inc0 | to APIs | 10:25 |
| SamYaple | someone look at a config please | 10:25 |
| inc0 | let me look at config for galera | 10:25 |
| inc0 | ah my mistake | 10:27 |
| inc0 | for mysql its for backup | 10:27 |
| SamYaple | yep | 10:27 |
| inc0 | but just for mysql | 10:27 |
| inc0 | I mean maria | 10:27 |
| SamYaple | exactly as it should be | 10:27 |
| inc0 | agree | 10:27 |
| sdake | ok making progrss | 10:28 |
| sdake | inc0 which one is the roundrobin flg? | 10:28 |
| inc0 | https://github.com/stackforge/kolla/blob/master/ansible/roles/haproxy/templates/haproxy.cfg.j2#L23 | 10:29 |
| inc0 | there is if statement | 10:30 |
| SamYaple | yup | 10:30 |
| inc0 | and it's marked as backup unless it's first in the loop | 10:30 |
| SamYaple | correct | 10:30 |
| SamYaple | just like i wrote it 6 months ago | 10:30 |
| *** dims has joined #kolla | 10:30 | |
| *** dayou has joined #kolla | 10:31 | |
| *** tfukushima has quit IRC | 10:33 | |
| sdake | yes i see | 10:34 |
| sdake | going to try makign keystone acive backup backup | 10:35 |
| *** dims has quit IRC | 10:36 | |
| vbel | sdake: I used one of hosts' IPs instead of VIP, will be checking haproxy soon | 10:36 |
| vbel | mysql should not have roundrobin haproxy setting. | 10:37 |
| *** diga has quit IRC | 10:37 | |
| sdake | it doesn't | 10:37 |
| vbel | source based routing is recommended | 10:41 |
| vbel | we don't have it. It is important for db and dashboard | 10:41 |
| vbel | http://docs.openstack.org/high-availability-guide/content/ha-aa-haproxy.html | 10:41 |
| vbel | basically anything that has 'session' should have "balance source" | 10:42 |
| SamYaple | vbel: agreed on dashboard | 10:57 |
| SamYaple | doesnt matter for database at this time | 10:57 |
| *** dwalsh has quit IRC | 10:57 | |
| SamYaple | without source based routing the dashboard wont carry session info and we would have to use memcache | 10:57 |
| vbel | yes, db is smart enough. If you use client you will see "reconnecting" but it does not fail | 10:57 |
| SamYaple | i have a keepalived patch with some haproxy tweaks for this already | 10:57 |
| SamYaple | vbel: well no the db in this case only has one 'active' connections that can server requests | 10:58 |
| inc0 | sdake, about multinode...does your rabbit run? | 11:00 |
| sdake | inc0 yes | 11:00 |
| inc0 | freshly built container - INFO - This container has already been configured; Refusing to copy new configs | 11:00 |
| SamYaple | inc0: its probably restarting | 11:01 |
| inc0 | yeah it is | 11:01 |
| SamYaple | inc0: change the container restart policy | 11:01 |
| SamYaple | i set it to 'no' | 11:01 |
| inc0 | kk let me check what's actual error | 11:03 |
| SamYaple | mutlinode rabbitmq has a race condition for initial cluster startup | 11:03 |
| inc0 | yeah that might be it, change rabbit to just one node in inventory? | 11:04 |
| inc0 | that's informative... docker logs rabbitmq -> su: System error | 11:05 |
| SamYaple | oh yea vbel reported a su error with docker its a kernel thing | 11:06 |
| SamYaple | iwas unable to reproduce on ubuntu | 11:06 |
| SamYaple | whats your system info | 11:06 |
| vbel | inc0 - you have bad kernel :) | 11:06 |
| SamYaple | vbel ^ | 11:06 |
| vbel | we do too :) | 11:06 |
| inc0 | it's 14.04 cloud img | 11:06 |
| vbel | I use sudo instead of su for db init commands and run rabbit via sudo -u rabbitmq, otherwise rabbit will try to use su | 11:07 |
| inc0 | vbel, you do it in container? | 11:08 |
| *** weiyu has quit IRC | 11:08 | |
| SamYaple | inc0: kernel? | 11:08 |
| vbel | I have CMD="sudo -u rabbitmq /usr/sbin/rabbitmq-server" in my Dockerfile.j2 | 11:08 |
| inc0 | vbel, mind upstreaming it as a patch? | 11:09 |
| SamYaple | inc0: no thats not the way to do it | 11:09 |
| *** britthouser has joined #kolla | 11:09 | |
| SamYaple | we need ot be dropping permsiisions in the containers | 11:09 |
| inc0 | SamYaple, what is way to do it then? | 11:09 |
| vbel | USER rabbitmq | 11:10 |
| SamYaple | that would be the correct way | 11:10 |
| SamYaple | then a special sudoers file that allows us to change the rabbitmq folder perms since the data contaienr will screw those perms up | 11:10 |
| SamYaple | thats the model from yoadu and i have full permissions dropping there | 11:10 |
| SamYaple | it works well | 11:11 |
| inc0 | why aren't we doing that then? | 11:11 |
| SamYaple | dont look at me | 11:11 |
| vbel | time and people | 11:11 |
| SamYaple | its a major change that sdake targed for mitaka | 11:11 |
| SamYaple | i wanted t oget it done in RC1 but im not fighting him on everything | 11:11 |
| SamYaple | ;) | 11:11 |
| inc0 | ok, can we add this workaround till we do? | 11:12 |
| inc0 | rabbitmq isn't working by default now | 11:12 |
| *** slagle has joined #kolla | 11:12 | |
| SamYaple | i suggest we do it correctly now | 11:12 |
| vbel | inc: you will get this problem in all other containers where we have 'su' | 11:12 |
| SamYaple | i can submit a patch for this right now | 11:12 |
| inc0 | SamYaple, please do | 11:13 |
| SamYaple | inc0: just keep in mind what vbel said | 11:13 |
| sdake | things that iprove security i wont generally push back on | 11:13 |
| SamYaple | i know sdake but you did say no to it for RC1 | 11:13 |
| inc0 | maybe as full change | 11:13 |
| SamYaple | anyway looks like we have bugs and reasons to address the issue now | 11:13 |
| inc0 | but simple container by container | 11:14 |
| sdake | i dont think rc1 is settled yet | 11:14 |
| SamYaple | i am fully familar with how to do this thanks to yoadu | 11:14 |
| SamYaple | i can lead the conversion | 11:14 |
| SamYaple | ill submit something for rabbitmq | 11:14 |
| SamYaple | we can go form there | 11:14 |
| inc0 | yeah, keystone is affected too | 11:15 |
| SamYaple | inc0: all containers will be broke for you | 11:16 |
| SamYaple | we use su in just about every one | 11:16 |
| inc0 | that's encouraging | 11:16 |
| SamYaple | su is what is broked iwth your kernel | 11:16 |
| inc0 | sdake, that's not problem you're facing? | 11:17 |
| SamYaple | i just wrote this udev rule to fix openstack in juno | 11:19 |
| SamYaple | KERNEL=="brq*", ACTION=="add", RUN+="/bin/bash -c 'export dummy=taprpc-$(awk -Fbrq\|- {print\ \$2} <<< %k);(sleep 15; ip l a $dummy type dummy && brctl addif %k $dummy)'" | 11:19 |
| SamYaple | awesome i know | 11:19 |
| inc0 | I hate operators... | 11:19 |
| inc0 | :P | 11:19 |
| inc0 | or rather I hate bash, and ops just happen to use it | 11:20 |
| SamYaple | because we can actually make things work :D | 11:20 |
| SamYaple | oh comone that is perfectly readable | 11:20 |
| SamYaple | and i only use bash because the default shell is sh | 11:21 |
| inc0 | http://devopsreactions.tumblr.com/post/84407988311/scripts-written-by-developers-versus-scripts | 11:21 |
| SamYaple | so useful everyday versus easily broken when you throw a square into the mix ;) | 11:22 |
| *** banix has joined #kolla | 11:23 | |
| sdake | vbel > 9000 | 11:24 |
| sdake | http://ur1.ca/nmp5j -> http://paste.fedoraproject.org/261924/11066851 | 11:25 |
| *** dims has joined #kolla | 11:25 | |
| sdake | glance image-list in a loop works like a champ with above change | 11:25 |
| *** jruano has joined #kolla | 11:26 | |
| SamYaple | that is not the root cause of the issue though... | 11:26 |
| SamYaple | and you have limited your environment severly | 11:27 |
| SamYaple | keystone does a whole bunch of requests | 11:27 |
| *** dwalsh has joined #kolla | 11:27 | |
| sdake | in the oifficla ha docs | 11:27 |
| SamYaple | yep | 11:28 |
| SamYaple | my point still stands | 11:28 |
| vbel | true, official ha docs put 'balance source' everywhere :) | 11:28 |
| sdake | proably because openstack is bust without it | 11:28 |
| inc0 | I just wonder why it's even problem | 11:29 |
| sdake | wht does that balane shit do | 11:29 |
| SamYaple | its not | 11:29 |
| SamYaple | never used it in yaodu | 11:29 |
| *** dims has quit IRC | 11:29 | |
| inc0 | it shouldn't be, I mean...meh http is stateless | 11:29 |
| SamYaple | indeed | 11:29 |
| SamYaple | he is maksing the issue here | 11:29 |
| SamYaple | horizon is only because of that csrf token and stuff | 11:30 |
| SamYaple | and even then it can balance _if_ you use memcache backend | 11:30 |
| vbel | if you use keystone with localhost memcache you need it | 11:30 |
| inc0 | it might mean that onlu one of 3 ks is working and you happen to reach correct one | 11:30 |
| inc0 | for example | 11:30 |
| vbel | and localhost memcache is the only secure option for memcache :) | 11:30 |
| sdake | perhap yaodu didn't really work ;) | 11:30 |
| *** pbourke has quit IRC | 11:30 | |
| inc0 | vbel, yeah, we don't do local memcache | 11:30 |
| sdake | another option is there is a keystoon option missing from our ansible code | 11:30 |
| SamYaple | lets not insult the project that was working 4 months before kolla | 11:31 |
| sdake | but i a mrunning ubuntu userspace | 11:31 |
| inc0 | sdake, try to call keystone by actual ip, not floating | 11:31 |
| sdake | i was making no insult | 11:31 |
| sdake | you said kolla worked, it did not atleast on my hardware or inc0s or vbels | 11:31 |
| sdake | maybey ou never tried same test case as I ran | 11:32 |
| SamYaple | what is your test case? | 11:32 |
| sdake | glance image-list in a loop | 11:32 |
| SamYaple | i mean i used it until i replaced it with kolla | 11:32 |
| SamYaple | yea no i actually _used_ it multinode until about 2 months ago | 11:32 |
| vbel | glance image-list in loop is a good test case! | 11:32 |
| SamYaple | i do all my dev in virtualmachines running on Kolla in ha right now | 11:33 |
| sdake | simple and obvious ;) | 11:33 |
| inc0 | https://blueprints.launchpad.net/kolla/+spec/sanity-check-container | 11:34 |
| sdake | what exactly does that option do | 11:34 |
| inc0 | that was reason I posted this | 11:34 |
| SamYaple | inc0: that probably wont help whiles its running | 11:35 |
| SamYaple | we do retry loops on all interactoins with teh database | 11:35 |
| SamYaple | api* | 11:35 |
| inc0 | vbel, so you replaced su in every container? | 11:35 |
| inc0 | SamYaple, hence monitoring discussion | 11:35 |
| vbel | inc0: yes | 11:35 |
| *** pbourke has joined #kolla | 11:36 | |
| sdake | samyaple could you explain how that option limits how openstack behaves | 11:36 |
| vbel | inc0: if you can upgrade your kernel you won't need it | 11:36 |
| inc0 | vbel, which version is ok? | 11:37 |
| sdake | hey pbourke | 11:37 |
| SamYaple | sdake: if you have three controllers with source balancing potentially all or a large majority of your servers can end up _only_ going to one backend | 11:37 |
| inc0 | 3.13.0-54-generic <- I have this one | 11:37 |
| SamYaple | inc0: which version of docker? | 11:37 |
| inc0 | 1.6.2 | 11:38 |
| sdake | does it look at the source ip to balance to the target? | 11:38 |
| inc0 | pretty ancient... | 11:38 |
| SamYaple | inc0: try a newer version | 11:38 |
| SamYaple | i tried 1.71 and up with the 3.13 kernels and it was fine | 11:38 |
| vbel | inc: if you use magic backported RH - 3.10 | 11:38 |
| SamYaple | ubuntu for me | 11:38 |
| SamYaple | oh yea my ubuntu gate is running! | 11:39 |
| sdake | ya i told lyou that earlier | 11:39 |
| sdake | we should have docs pubhlsihing soon too | 11:39 |
| vbel | month ago I had ubuntu lts server (14) and it had this problem too | 11:39 |
| inc0 | yeah I'm running ubuntu 14.04 lts | 11:40 |
| SamYaple | inc0: what backend? | 11:40 |
| SamYaple | i think this may only affect AUFS... | 11:40 |
| SamYaple | i cant remember | 11:40 |
| inc0 | devicemapper | 11:40 |
| vbel | ubuntu desktop has different kernel and it works fine with su in containers | 11:40 |
| inc0 | our centos doesn't build on ubuntu+aufs | 11:40 |
| SamYaple | oh well that has all kinds of iisuse for ubuntu | 11:40 |
| vbel | devicemapper | 11:40 |
| *** rhallisey has joined #kolla | 11:40 | |
| SamYaple | inc0: centos doesnt work on ubuntu anymore | 11:41 |
| SamYaple | if youre on ubuntu you have to use ubuntu-source | 11:41 |
| inc0 | oh... | 11:41 |
| inc0 | ok.. | 11:41 |
| SamYaple | we removed the nopam hack sdake had | 11:41 |
| inc0 | that's sad | 11:41 |
| SamYaple | removing hacks is never sad :) | 11:41 |
| inc0 | removing "it works" is | 11:41 |
| SamYaple | it broke ceph | 11:41 |
| SamYaple | ceph > hack to make centos work for ubuntu | 11:41 |
| inc0 | ubuntu source has all bits ready? | 11:43 |
| inc0 | for ansiblish stuff | 11:43 |
| SamYaple | inc0: its the only one where they all build :) | 11:43 |
| SamYaple | yep | 11:43 |
| SamYaple | havent touched centos in almost 3 weeks for full builds | 11:44 |
| SamYaple | well | 11:44 |
| SamYaple | no i take that back | 11:44 |
| SamYaple | i did a full build ofr heat+horizon testing | 11:44 |
| vbel | https://review.openstack.org/#/c/219204/1 - for all bits ready ;) | 11:45 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 11:45 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 11:47 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 11:48 |
| sdake | inc0 i am running multinode now and rabbitmq appears to be suffering some kind of problem | 12:00 |
| SamYaple | which problem | 12:01 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 12:01 |
| sdake | running with this diff: | 12:01 |
| sdake | http://ur1.ca/nmpf2 -> http://paste.fedoraproject.org/261935/10887914 | 12:01 |
| sdake | all hypervisors appear down | 12:01 |
| sdake | but they did register | 12:01 |
| sdake | although the api services all got through an init-runonce | 12:01 |
| sdake | which was impossible prior | 12:01 |
| SamYaple | almost like you are masking the issue? | 12:02 |
| sdake | these are in the offical upstream docs... | 12:02 |
| sdake | probably a discussion t obe had on the mailing list | 12:02 |
| SamYaple | which were unmaintained until like 2 months ago | 12:02 |
| sdake | well what other solution would you propose | 12:03 |
| SamYaple | im working on it | 12:03 |
| SamYaple | havent let you down yet. have a little faith | 12:03 |
| inc0 | there is little faith after 20hrs being awake isn't it sdake ?;) | 12:04 |
| SamYaple | i find that lack of faith.... disturbing | 12:04 |
| sdake | pretty much inc0 i am beat ass tired of debugging this particular problem | 12:04 |
| inc0 | what kind of problems rabbitmq is showing? | 12:04 |
| sdake | inc0 not sure | 12:04 |
| sdake | going to do some mor emasking | 12:04 |
| sdake | see if that fixes it | 12:04 |
| sdake | the ha guide does not ecommend usin haproxy with rabbitmq | 12:06 |
| SamYaple | yea weve already been over this we arent going to be doing that | 12:06 |
| sdake | we are tagging in t-2 days, if you have a solution that works in that itmeline fine, if not my patch goes in | 12:07 |
| sdake | if you want to refactor later, feel free | 12:07 |
| SamYaple | well see | 12:07 |
| *** weiyu_ has joined #kolla | 12:07 | |
| * inc0 grabs popcorn | 12:07 | |
| * sdake grabs popcorn | 12:07 | |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 12:07 |
| sdake | the alternative is to tag something that totally doesnt work except aio | 12:08 |
| sdake | which seems like a nonstarter to me | 12:08 |
| sdake | so pick your poison | 12:08 |
| SamYaple | or you know, fix it correctly | 12:08 |
| sdake | i have tried everything i see to fix it | 12:08 |
| sdake | nothing | 12:08 |
| sdake | you gae up after 1 hour | 12:09 |
| sdake | i have a working solution after 20 | 12:09 |
| SamYaple | im at my job | 12:09 |
| SamYaple | i gave up nothing | 12:09 |
| SamYaple | ive fixed all kinds of shit today | 12:09 |
| sdake | t-2 days we are tagging | 12:09 |
| SamYaple | its just not kolla for once | 12:09 |
| sdake | i understand eveyrone has other shit to do | 12:09 |
| sdake | i'm not asking you to work miracles | 12:09 |
| inc0 | SamYaple, then tell us how it SHOULD be fixed | 12:09 |
| sdake | but i am at a loss as to a better solution | 12:09 |
| SamYaple | inc0: since no one has produced an error thats kind of hard | 12:10 |
| SamYaple | i need ot look at it | 12:10 |
| sdake | it got put in those docs for a reason :) | 12:10 |
| SamYaple | and no client errors dont count | 12:10 |
| *** banix has quit IRC | 12:10 | |
| SamYaple | sdake: its said source since those docs initially went up 3 years ago | 12:10 |
| SamYaple | it was not tought out i can assure oyu | 12:10 |
| SamYaple | its only been in the last few months that that doc has gotten any attention | 12:11 |
| sdake | ok well if you want to refactorit and fix it prperly thats fine | 12:11 |
| sdake | that is how normal software engineering works | 12:11 |
| sdake | but from what I can tell the proper fix is this integration solution | 12:11 |
| SamYaple | wait i thought it was the database that was the problem | 12:12 |
| SamYaple | you said that for the first 4 hours | 12:12 |
| inc0 | sdake, I'm about to setup brand new cluster of vms with ubuntu 14.04 | 12:12 |
| inc0 | is that your config? | 12:12 |
| sdake | yes it is a synrhoniation problem wiht the database in some way | 12:12 |
| sdake | i'm notentirely sure where in the stack | 12:12 |
| SamYaple | it is not | 12:12 |
| inc0 | sdake, ubuntu-source? | 12:12 |
| sdake | the sourc erouting forces lal data through one server | 12:13 |
| SamYaple | otherwise rabbitmq wouldnt be affected would it | 12:13 |
| sdake | inc0 ubuntu source fails, centos binary fails | 12:13 |
| SamYaple | thats not what source does | 12:13 |
| sdake | i've only asked 3 times how itworks | 12:13 |
| sdake | nobody will explain | 12:13 |
| inc0 | ubuntu-binary is working? | 12:13 |
| sdake | only that its evil | 12:13 |
| sdake | ubntu-binary doesn't build dude | 12:13 |
| SamYaple | inc0: ubuntu-binary doesnt exist | 12:13 |
| SamYaple | there is no delorean type repo for ubuntu | 12:13 |
| inc0 | ok...my question is | 12:13 |
| inc0 | if I build ubuntu-source, on ubuntu 10.04 | 12:14 |
| SamYaple | oh good god | 12:14 |
| inc0 | 14.04 | 12:14 |
| sdake | fwiw I don't think we need haproxy at all for rabbitmq | 12:14 |
| SamYaple | oh whew | 12:14 |
| inc0 | will it reproduce your failing env sdake? | 12:14 |
| SamYaple | sdake: as discussed many times, we dont | 12:14 |
| sdake | if you deploy to 3 nodes | 12:14 |
| sdake | samyaple there is haproxy config for rabbitmq in the haproxy file dude | 12:14 |
| inc0 | 3 controllers? | 12:14 |
| sdake | 3 ontrolls | 12:14 |
| sdake | you ccan see my diff of my inventory file | 12:15 |
| vbel | yes, haproxy is not needed for rabbitmq | 12:15 |
| SamYaple | sdake: im well aware | 12:15 |
| SamYaple | do you know why its there? | 12:15 |
| sdake | well its in there, not sure if its used | 12:15 |
| SamYaple | because we have no means to update the config at the moment | 12:15 |
| vbel | we should put rabbit hosts into openstack configs instead | 12:15 |
| SamYaple | so it has to stay like that until kolla-ansible can update configs | 12:15 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 12:24 |
| sdake | inc0 on the plus side of things today, I've become an expert in how our ha solution is implemented | 12:25 |
| sdake | looks pretty solid | 12:25 |
| inc0 | I'll treat it as a complement | 12:25 |
| sdake | nice simple tidy | 12:26 |
| SamYaple | inc0: i normally just say 'youre welcome' when i dont know if its an insult or compliment :) | 12:26 |
| sdake | if i dont have anything nice to say i dont say anything at all | 12:27 |
| sdake | write it down | 12:27 |
| SamYaple | IRC _is_ writing it down =D | 12:27 |
| *** weiyu_ has quit IRC | 12:28 | |
| *** dims has joined #kolla | 12:33 | |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 12:34 |
| sdake | someetimes I get a lockup here | 12:34 |
| sdake | TASK: [neutron | Ensuring OVS bridge is properly setup] *********************** | 12:34 |
| sdake | changed: [minime-one] | 12:34 |
| sdake | only solution is a reboot | 12:34 |
| SamYaple | it would be modprobing ovs right around that stage | 12:34 |
| SamYaple | actually, it wouldn't | 12:35 |
| SamYaple | it would have tried to do that earlier | 12:35 |
| SamYaple | it may not have successed | 12:35 |
| SamYaple | but at that stage it would try to start ocmmunicating with ovs-vsctld for the first time | 12:35 |
| sdake | only happens on multinode as well | 12:36 |
| *** dwalsh has quit IRC | 12:36 | |
| sdake | nova doesn't work multinode | 12:51 |
| sdake | http://paste.openstack.org/show/437638/ | 12:51 |
| *** klint has quit IRC | 12:57 | |
| *** dwalsh has joined #kolla | 12:58 | |
| openstackgerrit | Steven Dake proposed stackforge/kolla: Make API services work multinode https://review.openstack.org/219261 | 12:59 |
| sdake | this is our next blocking bug blocking the release of liberty-3 | 13:03 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1490973 | 13:03 |
| openstack | Launchpad bug 1490973 in kolla "compute nodes don't show hypervisors as active" [Critical,Triaged] | 13:03 |
| SamYaple | sdake: not seeing your issues on my multinode | 13:04 |
| SamYaple | at all | 13:04 |
| sdake | sam remove your -2 vote | 13:04 |
| SamYaple | multinode working for me with glane image-list loop | 13:04 |
| sdake | i am relaly not in any mood | 13:05 |
| SamYaple | im not either | 13:05 |
| sdake | let the other cores judge | 13:05 |
| SamYaple | i am -2 on the source balancing for anything other than horizon _unless_ it is needed to tag | 13:05 |
| sdake | it is needed to tag | 13:05 |
| sdake | if you want to rework it later feel free as i said | 13:05 |
| SamYaple | you said i have two days, AND i dont see the issue at all | 13:06 |
| sdake | inc0 confirmed it | 13:06 |
| SamYaple | inc0 had other issues... | 13:06 |
| SamYaple | he was running centos on ubuntu too | 13:06 |
| SamYaple | which would never work without nopam | 13:06 |
| sdake | it is not the packaging i ran full ubntu source build | 13:07 |
| SamYaple | which i am also running with no issues | 13:07 |
| inc0 | I'm buiilding sources now | 13:07 |
| inc0 | let me check it locally plz | 13:07 |
| SamYaple | so likely it is a problem with centos on the host level | 13:07 |
| rhallisey | what can I test that someone is not doing atm? | 13:07 |
| rhallisey | centos binary? | 13:08 |
| sdake | fix https://bugs.launchpad.net/kolla/+bug/1490973 | 13:08 |
| openstack | Launchpad bug 1490973 in kolla "compute nodes don't show hypervisors as active" [Critical,Triaged] | 13:08 |
| rhallisey | so multinode | 13:08 |
| rhallisey | k | 13:08 |
| SamYaple | also dont have that problem sdake | 13:08 |
| SamYaple | working fine | 13:08 |
| SamYaple | so lets take this time to _fix_ issues rather than kneejerking around | 13:08 |
| sdake | i am at a loss for further solutions | 13:09 |
| sdake | if you have a solution i am willing to test | 13:09 |
| sdake | you have access to my box | 13:09 |
| sdake | i want that patch approved so i can tag | 13:09 |
| SamYaple | if its alright with you know that my shift is over im am going to jump back into your bbox | 13:09 |
| sdake | yup its aok | 13:09 |
| SamYaple | fyi, i just rebuilt and rekicked on ubuntu _multienode_ not a problem at all | 13:10 |
| sdake | i understand why it works, it keeps all traffic to one keystone server | 13:10 |
| *** dwalsh has quit IRC | 13:10 | |
| SamYaple | right but why is that needed :) | 13:10 |
| SamYaple | because broken networking most likely | 13:10 |
| sdake | you said your deploying on virt? | 13:11 |
| inc0 | yeah, with RESTful APIs it shouldn't matter | 13:11 |
| sdake | it may if the performance of the boxes isvery high | 13:11 |
| sdake | we have already elimited the databae since its master/slave/slave | 13:12 |
| sdake | that means keystone in nwayactive is bust on bare metal | 13:12 |
| sdake | inc0 arey u on baremetal? | 13:12 |
| inc0 | both simultaneously | 13:12 |
| inc0 | akwasnie, is running baremetal aio | 13:13 |
| sdake | could you expand | 13:13 |
| inc0 | right now | 13:13 |
| inc0 | we're deploying 2 deployments independently | 13:13 |
| sdake | that may not work iirc haproxy's mcat addr is hard coded | 13:13 |
| sdake | or keepalive | 13:14 |
| sdake | samyaple are you running on virt deployment? | 13:17 |
| *** sdake has quit IRC | 13:17 | |
| *** sdake has joined #kolla | 13:21 | |
| sdake | laptop died, samyaple are you running on virt or baremetal? | 13:22 |
| SamYaple | sdake: ive just rekicked my two main boxes for baremetal | 13:24 |
| SamYaple | sdake: 5 minutes ill be on your box doing things | 13:25 |
| SamYaple | whats the current state? | 13:25 |
| SamYaple | are you in the middle of a test | 13:25 |
| sdake | centos binary and ubuntu source are built form master in the registry | 13:25 |
| vbel | should we add in inventory/multinode second host to [network] as default? To have 2 keepalived/haproxies .. | 13:25 |
| sdake | i do want to look at the compute node log | 13:26 |
| sdake | so let me save those | 13:26 |
| SamYaple | vbel: those inventory files are more like docs than anything ,so whatever makes it more clear ot the user in your opinion | 13:27 |
| vbel | I see | 13:27 |
| *** jtriley has joined #kolla | 13:28 | |
| sdake | ive got the logs | 13:28 |
| sdake | enjoy | 13:28 |
| SamYaple | for the record, still no issues on ubuntu | 13:29 |
| SamYaple | had an image-list going this whole time | 13:29 |
| SamYaple | while true; do glance image-list || break; done | 13:30 |
| SamYaple | still hasnt broken | 13:30 |
| sdake | on virt? | 13:32 |
| SamYaple | baremetal | 13:32 |
| SamYaple | didnt break on virt either | 13:33 |
| sdake | 3 node? | 13:33 |
| SamYaple | 2 node baremetal | 13:33 |
| SamYaple | 3 node virt | 13:33 |
| sdake | let me try 2 node baremetal | 13:33 |
| sdake | yo udont have a 3rd node i assume? | 13:33 |
| sdake | deploying, one moment | 13:35 |
| SamYaple | not a baremetal host no | 13:36 |
| sdake | looks like you chnged glance-api | 13:42 |
| sdake | and its failing to boot | 13:42 |
| sdake | so i can't test if two node reproduces | 13:42 |
| sdake | SamYaple your free to use the system now if you like | 13:45 |
| *** Haomeng has joined #kolla | 13:47 | |
| SamYaple | ok hey i saw 1 error! | 13:47 |
| SamYaple | it was a 500 | 13:47 |
| SamYaple | i tracked it back to too many db connectiosn ( a common issue) | 13:47 |
| SamYaple | but only 1 error | 13:47 |
| *** Haomeng|2 has quit IRC | 13:49 | |
| sdake | pbourke are you about | 13:50 |
| sdake | i could live with 1 in 500 | 13:50 |
| sdake | i am at 1 in 2 | 13:50 |
| *** ftcpops has joined #kolla | 13:52 | |
| SamYaple | sdake: ok sorry work cleanup | 13:53 |
| SamYaple | im in your box now | 13:53 |
| SamYaple | lets walk through this | 13:53 |
| SamYaple | 100 is a different machine from 101,102,103? | 13:54 |
| sdake | right | 13:54 |
| sdake | 100 is hp z820 xeon | 13:54 |
| SamYaple | any kolla stuff ruinning there? | 13:54 |
| sdake | 101/102/103 are dells | 13:54 |
| SamYaple | (not ansible) | 13:54 |
| sdake | nothing running on 100 | 13:54 |
| SamYaple | ok | 13:54 |
| sdake | if there ws, it joins the cluster | 13:54 |
| sdake | which causes chaos | 13:54 |
| SamYaple | so kolla not deployed atm | 13:54 |
| sdake | right | 13:55 |
| sdake | my ansible imae has that change for fixing the internal_url | 13:55 |
| sdake | the ansible-kolla | 13:55 |
| sdake | otherwise its mater | 13:55 |
| *** jruano has quit IRC | 13:55 | |
| SamYaple | im going to deploy ubuntu source since thats what i was using to rule that out | 13:55 |
| sdake | feel free | 13:56 |
| sdake | its already built | 13:56 |
| sdake | so no waiting 70 inutes or a build and push | 13:56 |
| SamYaple | i think 103 locked up | 13:57 |
| sdake | ya its busted | 13:57 |
| sdake | i'll go reboot | 13:57 |
| sdake | i rebooted all3 | 13:59 |
| SamYaple | ok | 13:59 |
| SamYaple | that explains that | 13:59 |
| SamYaple | are these vms? | 13:59 |
| sdake | 101/102/103? | 13:59 |
| SamYaple | yea | 13:59 |
| sdake | bare metal cat /proc/cpuinfo | 13:59 |
| sdake | 16gb ram | 13:59 |
| sdake | 256 gb ssd | 14:00 |
| sdake | maybe 32gb of ram actually | 14:00 |
| sdake | i don't recall | 14:00 |
| SamYaple | no problem | 14:00 |
| SamYaple | just checking | 14:00 |
| *** dayou has quit IRC | 14:00 | |
| sdake | model name: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz | 14:01 |
| vbel | cool i7 with ssd :) My dream laptop | 14:01 |
| sdake | ya 32gb ram | 14:02 |
| *** Kennan has joined #kolla | 14:03 | |
| SamYaple | sdake: you are welcome to join my tmux session while i work | 14:03 |
| SamYaple | also how do i check the version of an installed package in centos? | 14:03 |
| *** Kennan2 has quit IRC | 14:03 | |
| sdake | rpm -qi packagename | 14:04 |
| sdake | how do i join a tmux session | 14:05 |
| sdake | never eused tmux before | 14:05 |
| SamYaple | su to my user (s8m) | 14:05 |
| SamYaple | tmux attach | 14:05 |
| *** zhiwei has quit IRC | 14:05 | |
| *** dayou has joined #kolla | 14:06 | |
| SamYaple | you attached? | 14:07 |
| *** dwalsh has joined #kolla | 14:07 | |
| SamYaple | oh on the 100 server | 14:07 |
| SamYaple | not the dells | 14:07 |
| inc0 | mkdir: cannot create directory '/var/lib/rabbitmq/mnesia' that's a new - error from rabbitmq | 14:08 |
| sdake | http://docs.openstack.org/kilo/config-reference/content/section_keystone.conf.html | 14:08 |
| sdake | inc0 pbourke has a patch up for that | 14:08 |
| sdake | there are some keystone caching options | 14:08 |
| sdake | worth exploring turning those off | 14:08 |
| sdake | caching plus nwayactive = bad news | 14:08 |
| sdake | impossible to keep cache coherency | 14:09 |
| SamYaple | wow dude can you resize your window alow | 14:10 |
| SamYaple | alot* | 14:10 |
| SamYaple | i cannot work | 14:10 |
| sdake | let me star twith a fresh term | 14:10 |
| SamYaple | ok i detached you | 14:10 |
| SamYaple | micro window size there | 14:10 |
| sdake | so you want it max sized? | 14:11 |
| SamYaple | if possible | 14:11 |
| SamYaple | your window affects mine | 14:11 |
| SamYaple | thats fine | 14:11 |
| inc0 | must love tmux | 14:12 |
| SamYaple | i love it | 14:12 |
| SamYaple | so much | 14:12 |
| inc0 | I do too | 14:12 |
| SamYaple | :setw sync | 14:12 |
| inc0 | I have it on shortcut | 14:12 |
| SamYaple | yea i do too | 14:12 |
| inc0 | ctrl + s | 14:12 |
| SamYaple | on my boxes.... | 14:12 |
| SamYaple | sdake: no openstack client | 14:15 |
| SamYaple | i dont want to dirty these boxes alot | 14:15 |
| SamYaple | but i need build stuff or somethin | 14:15 |
| SamYaple | g | 14:15 |
| sdake | its fine | 14:15 |
| sdake | i can reload them if i need | 14:16 |
| SamYaple | alrighty | 14:16 |
| sdake | just tyr to to break them because it does take about 30 mintes to reload | 14:16 |
| vbel | question: we don't use keystone token caching at all currently, correct? | 14:23 |
| SamYaple | correct | 14:23 |
| vbel | SamYaple: thanks | 14:23 |
| SamYaple | tough the configs dont prevent it | 14:24 |
| *** banix has joined #kolla | 14:24 | |
| *** shardy_ has joined #kolla | 14:30 | |
| sdake | ya turn that token caching shit off | 14:31 |
| *** shardy has quit IRC | 14:32 | |
| *** shardy_ has quit IRC | 14:35 | |
| *** shardy has joined #kolla | 14:36 | |
| vbel | I am in glance image-list, 100 times ok. haproxy/keepalived/keystone/glance on 2 virtual nodes (virtualbox with host-only network + router vm) | 14:43 |
| SamYaple | vbel: i think i have a thread on the issue... | 14:43 |
| SamYaple | pulling it now | 14:43 |
| *** shardy_ has joined #kolla | 14:47 | |
| *** shardy has quit IRC | 14:48 | |
| openstackgerrit | Merged stackforge/kolla: Register internal_url properly https://review.openstack.org/219171 | 14:49 |
| *** shardy_ has quit IRC | 14:52 | |
| *** blahRus has joined #kolla | 14:53 | |
| *** shardy has joined #kolla | 14:53 | |
| *** sdake_ has joined #kolla | 14:53 | |
| openstackgerrit | Merged stackforge/kolla: rsyslog container https://review.openstack.org/214589 | 14:54 |
| *** sdake has quit IRC | 14:58 | |
| sdake_ | samyaple i think yo uwant something relate to immidate flushing | 14:58 |
| inc0 | I gtg | 14:58 |
| inc0 | will come back to this tomorrow | 14:59 |
| inc0 | cyas | 14:59 |
| sdake_ | you would think turning off caching woudl enable flushing but who nkows :)] | 14:59 |
| SamYaple | yea | 14:59 |
| SamYaple | i do _not_ want ot put memcache as a backing for keystone | 14:59 |
| *** inc0 has quit IRC | 14:59 | |
| SamYaple | but that would instantly make this work :( | 14:59 |
| sdake_ | ya let stry not to do that | 14:59 |
| SamYaple | major security hole | 14:59 |
| SamYaple | fernet tokens would also fix this | 15:00 |
| sdake_ | redis is a caching backend as well | 15:02 |
| *** dayou has quit IRC | 15:02 | |
| sdake_ | seems a bit like overkill | 15:02 |
| sdake_ | maybe some day we will want that | 15:02 |
| sdake_ | i just want the damn thing to work | 15:03 |
| SamYaple | fernet is the future | 15:04 |
| SamYaple | its pretty damn slick | 15:04 |
| SamYaple | i believe it is default for liberty | 15:04 |
| *** diogogmt has joined #kolla | 15:04 | |
| *** sdake_ is now known as sdake | 15:06 | |
| *** dayou has joined #kolla | 15:07 | |
| *** diogogmt has quit IRC | 15:07 | |
| vbel | I cannot recreate rabbitmq mkdir problem, if one can - please pastebin ansible output | 15:09 |
| *** diogogmt has joined #kolla | 15:09 | |
| sdake | samyaple asking around various people what the config options are for keystone | 15:09 |
| SamYaple | sdake: for master liberty ;) | 15:09 |
| SamYaple | sure there is lots of that | 15:09 |
| sdake | i cannot recreate rabbitmq mkdir problem either | 15:10 |
| sdake | the permissions are set properly in the contianers i debugged | 15:10 |
| *** banix has quit IRC | 15:10 | |
| *** dwalsh has quit IRC | 15:11 | |
| *** diogogmt has quit IRC | 15:11 | |
| * rhallisey is stuck in building limbo | 15:11 | |
| rhallisey | brb | 15:11 |
| *** dayou has quit IRC | 15:12 | |
| sdake | what types of tokens are we using samyaple | 15:14 |
| sdake | #openstack-keystone | 15:14 |
| sdake | pls join | 15:14 |
| *** banix has joined #kolla | 15:15 | |
| *** rhallisey has quit IRC | 15:15 | |
| *** dims has quit IRC | 15:17 | |
| *** diogogmt has joined #kolla | 15:21 | |
| vbel | uuid as default | 15:23 |
| *** jasonsb has quit IRC | 15:23 | |
| *** jasonsb has joined #kolla | 15:23 | |
| *** diogogmt has quit IRC | 15:24 | |
| *** jasonsb has quit IRC | 15:28 | |
| *** dayou has joined #kolla | 15:39 | |
| *** tfukushima has joined #kolla | 15:44 | |
| *** jtriley has quit IRC | 15:47 | |
| *** chandankumar has quit IRC | 15:49 | |
| *** banix has quit IRC | 15:50 | |
| *** dayou has quit IRC | 15:50 | |
| *** banix has joined #kolla | 15:51 | |
| *** juggler has joined #kolla | 15:57 | |
| *** rhallisey has joined #kolla | 15:59 | |
| *** diga has joined #kolla | 16:03 | |
| SamYaple | so to confirm, has _ANYONE_ been able to reproduce what sdake was seeing? | 16:05 |
| *** dims has joined #kolla | 16:05 | |
| *** sdake_ has joined #kolla | 16:05 | |
| SamYaple | I know i havent, inc0 never said he was able to | 16:05 |
| *** banix has quit IRC | 16:05 | |
| SamYaple | he was having the su kernel issue | 16:05 |
| *** ftcpops has quit IRC | 16:07 | |
| *** sdake has quit IRC | 16:09 | |
| *** jtriley has joined #kolla | 16:09 | |
| *** jasonsb has joined #kolla | 16:13 | |
| *** diga has quit IRC | 16:18 | |
| *** sdake_ is now known as sdake | 16:21 | |
| *** erkules_ is now known as erkules | 16:33 | |
| *** erkules has joined #kolla | 16:33 | |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 16:36 |
| *** jruano has joined #kolla | 16:39 | |
| *** jasonsb has quit IRC | 16:42 | |
| *** jasonsb has joined #kolla | 16:43 | |
| *** SourabhP has joined #kolla | 16:43 | |
| *** SourabhP_ has joined #kolla | 16:45 | |
| *** diogogmt has joined #kolla | 16:46 | |
| *** jasonsb has quit IRC | 16:47 | |
| *** SourabhP has quit IRC | 16:48 | |
| *** SourabhP_ is now known as SourabhP | 16:48 | |
| *** banix has joined #kolla | 17:02 | |
| *** unicell has quit IRC | 17:03 | |
| *** sdake_ has joined #kolla | 17:06 | |
| *** sdake has quit IRC | 17:06 | |
| openstackgerrit | Vladislav Belogrudov proposed stackforge/kolla: Source based builds should get novnc from tarball (git/url) https://review.openstack.org/219204 | 17:06 |
| *** sdake has joined #kolla | 17:07 | |
| *** jasonsb has joined #kolla | 17:08 | |
| *** SourabhP has quit IRC | 17:09 | |
| openstackgerrit | Mick Thomspon proposed stackforge/kolla: Add ansible support for Murano https://review.openstack.org/218894 | 17:10 |
| *** sdake_ has quit IRC | 17:10 | |
| *** jasonsb has quit IRC | 17:12 | |
| SamYaple | looks like keystone is borked in centos-binary bad pacakges | 17:17 |
| SamYaple | same with cinder | 17:17 |
| SamYaple | 2015-09-01 17:16:08.741 | DEBUG:kolla.cmd.build:cinder-api:[91mError: Package: 1:python-keystone-8.0.0.0b3-dev209.el7.centos.noarch (delorean) | 17:18 |
| SamYaple | 2015-09-01 17:16:08.741 | DEBUG:kolla.cmd.build:cinder-api: Requires: python-oslo-config >= 2:1.11.0 | 17:18 |
| SamYaple | 2015-09-01 17:16:08.741 | DEBUG:kolla.cmd.build:cinder-api: Installed: 2:python-oslo-config-1.9.3-1.el7.noarch (@openstack-kilo) | 17:18 |
| SamYaple | 2015-09-01 17:16:08.741 | DEBUG:kolla.cmd.build:cinder-api: python-oslo-config = 2:1.9.3-1.el7 | 17:18 |
| SamYaple | 2015-09-01 17:16:08.742 | DEBUG:kolla.cmd.build:cinder-api: Available: 2:python2-oslo-config-2.3.1-dev2.el7.centos.noarch (delorean) | 17:18 |
| SamYaple | 2015-09-01 17:16:08.742 | DEBUG:kolla.cmd.build:cinder-api: python-oslo-config = 2.3.1.dev2 | 17:18 |
| *** jasonsb has joined #kolla | 17:19 | |
| sdake | NS RHW PEOVLWM IA QHr? | 17:20 |
| sdake | and the problem is what? | 17:20 |
| SamYaple | ? | 17:21 |
| *** jasonsb has quit IRC | 17:21 | |
| rhallisey | still building | 17:21 |
| SamYaple | idk looks like the kilo repo is taking priority over delorean | 17:21 |
| SamYaple | i dont know how that stuff works | 17:21 |
| SamYaple | which package wins from which repo | 17:21 |
| sdake | that is correct that is wat should happen | 17:21 |
| SamYaple | well thats bad | 17:22 |
| SamYaple | the delorean repo has newer packages | 17:22 |
| SamYaple | in this case newer ones that are required | 17:22 |
| SamYaple | anyway just a heads up since its centos binary i just saw the gate fail | 17:22 |
| SamYaple | someone will need to patch | 17:22 |
| sdake | http://ur1.ca/nms9w -> http://paste.fedoraproject.org/262087/14411281 | 17:22 |
| sdake | sam what cuaes this problem | 17:23 |
| SamYaple | bad ram | 17:23 |
| sdake | inventory file http://ur1.ca/nmsa7 -> http://paste.fedoraproject.org/262088/14411281 | 17:23 |
| SamYaple | i legit dont know sdake but its that problem node again | 17:23 |
| sdake | i am tyring to deploy to single nodes | 17:23 |
| sdake | is that possibly caused by a borked install of centos? | 17:24 |
| SamYaple | python would have to be shot | 17:24 |
| SamYaple | but i guess maybe | 17:24 |
| SamYaple | i really cant tell you | 17:24 |
| SamYaple | if you changed nothing else it should be fine | 17:24 |
| sdake | old kernel on node-03 | 17:25 |
| sdake | i didn't yum update it after installing | 17:25 |
| sdake | but i yum updated the other nodes... | 17:25 |
| *** achanda has joined #kolla | 17:25 | |
| SamYaple | that would explain ovs bombing out | 17:25 |
| *** loth has joined #kolla | 17:27 | |
| sdake | so interesting fact | 17:27 |
| sdake | node 1 was yum updated | 17:27 |
| sdake | node 2 and node 3 were not | 17:27 |
| openstackgerrit | Sam Yaple proposed stackforge/kolla: Setup ubuntu build gate https://review.openstack.org/219243 | 17:28 |
| SamYaple | sdake: that makes sense | 17:28 |
| SamYaple | when i yum installed packages node 1 completed with way fewer instaleld packages than node 2 and 3 | 17:29 |
| SamYaple | i mean im not saying thats the issue but there are some inconstencies to work through | 17:29 |
| sdake | node 2 doesn't install | 17:29 |
| sdake | same erro about keystone dict object | 17:29 |
| *** unicell has joined #kolla | 17:30 | |
| sdake | did you install all the same software on all 3 nodes? | 17:31 |
| SamYaple | for openstackclient yea | 17:32 |
| sdake | is there anything you didn't install on node1? | 17:32 |
| SamYaple | no | 17:32 |
| sdake | that you installed on node 2 and node 3 | 17:32 |
| sdake | so my router somehow asigned minime-one to 192.168.1.148 | 17:44 |
| sdake | so who knows what tht would do | 17:45 |
| sdake | probably nothing good | 17:45 |
| sdake | after fixing that I get fatal: [minime-01] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'dict object' has no attribute 'keystone'", 'failed': True} | 17:48 |
| *** jasonsb has joined #kolla | 17:53 | |
| *** SourabhP has joined #kolla | 17:58 | |
| *** banix has quit IRC | 18:00 | |
| *** chandankumar has joined #kolla | 18:00 | |
| *** banix has joined #kolla | 18:02 | |
| *** SourabhP has quit IRC | 18:16 | |
| *** SourabhP has joined #kolla | 18:17 | |
| *** akwasnie has quit IRC | 18:19 | |
| *** dperaza has joined #kolla | 18:22 | |
| *** dperaza has left #kolla | 18:26 | |
| *** jruano has quit IRC | 18:30 | |
| *** dims has quit IRC | 18:40 | |
| *** dims has joined #kolla | 18:40 | |
| *** dims has quit IRC | 18:45 | |
| *** diogogmt has quit IRC | 18:48 | |
| *** diogogmt has joined #kolla | 18:54 | |
| *** SourabhP has quit IRC | 18:58 | |
| *** dims has joined #kolla | 19:09 | |
| *** sdake_ has joined #kolla | 19:12 | |
| *** sdake has quit IRC | 19:13 | |
| *** dims has quit IRC | 19:13 | |
| *** sdake has joined #kolla | 19:13 | |
| *** sdake_ has quit IRC | 19:18 | |
| *** dims has joined #kolla | 19:18 | |
| *** banix has quit IRC | 19:21 | |
| *** dims has quit IRC | 19:25 | |
| *** whydidyoustealmy has quit IRC | 19:34 | |
| *** whydidyoustealmy has joined #kolla | 19:34 | |
| *** whydidyoustealmy has quit IRC | 19:35 | |
| *** dims has joined #kolla | 19:35 | |
| *** whydidyoustealmy has joined #kolla | 19:36 | |
| *** unicell has quit IRC | 19:38 | |
| *** sdake_ has joined #kolla | 19:45 | |
| *** sdake has quit IRC | 19:49 | |
| *** sdake has joined #kolla | 19:49 | |
| *** sdake_ has quit IRC | 19:53 | |
| *** chandankumar has quit IRC | 19:57 | |
| *** unicell has joined #kolla | 20:05 | |
| *** SourabhP has joined #kolla | 20:31 | |
| *** absubram has joined #kolla | 20:43 | |
| *** sdake has quit IRC | 20:53 | |
| *** achanda has quit IRC | 20:54 | |
| *** tfukushima has quit IRC | 21:05 | |
| *** achanda has joined #kolla | 21:07 | |
| *** jtriley has quit IRC | 21:10 | |
| *** achanda has quit IRC | 21:30 | |
| *** SourabhP has quit IRC | 21:42 | |
| *** shardy has quit IRC | 22:01 | |
| *** tfukushima has joined #kolla | 22:05 | |
| *** tfukushima has quit IRC | 22:10 | |
| *** SourabhP has joined #kolla | 22:11 | |
| *** achanda has joined #kolla | 22:54 | |
| *** achanda has quit IRC | 22:59 | |
| *** dims has quit IRC | 23:03 | |
| *** blahRus has quit IRC | 23:05 | |
| *** tfukushima has joined #kolla | 23:06 | |
| *** tfukushima has quit IRC | 23:12 | |
| *** absubram has quit IRC | 23:24 | |
| *** dims__ has joined #kolla | 23:24 | |
| *** diogogmt has quit IRC | 23:40 | |
| *** diogogmt has joined #kolla | 23:43 | |
| *** diogogmt has quit IRC | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!